Web::Scraper for SF.pm LT

Practical Web Scraping

with Web::Scraper

Tatsuhiko Miyagawa [email protected]

Six Apart, Ltd. / Shibuya Perl MongersSF.pm Lightning Talk

Tatsuhiko MiyagawaTatsuhiko Miyagawa 2007/11/027 SF.pm Lightning Talks2007/11/027 SF.pm Lightning Talks

How many of youhave done

screen-scraping w/ Perl?


How many of youhave used

LWP::Simple and regexp?



<td>Current UTC (or GMT/Zulu)-time used: Monday, August 27, 2007 at 12:49:46 



> perl -MLWP::Simple -le '$c = get("http://timeanddate.com/worldclock/"); $c =~ m@(.*?)@ and print $1'Monday, August 27, 2007 at 12:49:46


It works!


WWW::MySpace 0.70


WWW::Search::Ebay 2.231


There are3 problems(at least)


(1)Fragile

Easy to break even with slight HTML changes(like newlines, order of attributes etc.)


(2)Hard to maintain

Regular expression based scrapers are good Only when they're used in write-only scripts


(3)Improper

HTML & encodinghandling


I &hearts; Shibuya

> perl –e '$c =~ m@(.*?)@ and print $1'I &hearts; Shibuya


Web::Scraperto the rescue


Web scraping toolkitinspired by scrapi.rb

DSL-ish


Example

#!/usr/bin/perl

use strict;

use warnings;

use Web::Scraper;

use URI;

my $s = scraper {

process "strong#ctu", time => 'TEXT';

result 'time';

};

my $uri = URI->new("http://timeanddate.com/worldclock/");

print $s->scrape($uri);


Basics

use Web::Scraper;

my $s = scraper {

# DSL goes here

};

my $res = $s->scrape($uri);


process

process $selector,

$key => $what,

…;


$selector:

CSS Selectoror

XPath (start with /)


CSS Selector: strong#ctuXPath: //strong[@id="ctu"]



$key:key for the result

hashappend "[]" for

looping


$what:'@attr''TEXT''RAW'

Web::Scrapersub { … }

Hash reference


<ul class="sites"><li><a href="http://vienna.openguides.org/">OpenGuides</a></li><li><a href="http://vienna.yapceurope.org/">YAPC::Europe</a></li></ul>


process "ul.sites > li > a",

'urls[]' => '@href';

# { urls => [ … ] }



process '//ul[@class="sites"]/li/a',

'names[]' => 'TEXT';

# { names => [ 'OpenGuides', … ] }



process "ul.sites > li > a",

'sites[]' => {

link => '@href', name => 'TEXT';

};

# { sites => [ { link => …, name => … },

# { link => …, name => … } ] };



Tools


> cpan Web::Scraper

comes with 'scraper' CLI


> scraper http://example.com/

scraper> process "a", "links[]" => '@href';

scraper> d

$VAR1 = {

links => [

'http://example.org/',

'http://example.net/',

],

};

scraper> y

---

links:

- http://example.org/

- http://example.net/


> scraper /path/to/foo.html

> GET http://example.com/ | scraper


Demo


Thank you

http://search.cpan.org/dist/Web-Scraperhttp://www.slideshare.net/miyagawa/

webscraper

Date post:	28-Nov-2014
Category:	Technology
Upload:	tatsuhiko-miyagawa
View:	15,882 times
Download:	2 times

Web::Scraper for SF.pm LT

Technology