Screen Scraping in Ruby
I’m not going to make a full post on this because over on Rubyrailways.com they have an excellent post covering the main libraries for screen scraping in ruby.
However it is a bit out of date being 7 months old, and since then 2 more excellent libraries have emerged.
Hpricot from the Ruby Superstar _why which has just been updated to version 0.5 today!
And Scrapi which is also excellent.
3 comments
Like this article? Subscribe to the SEO on Rails feed.
Comments
-
I guess you are interested in my newest creature, scRUBYt!: http://scrubyt.rubyforge.org/files/README.html It is based on Hpricot 0.5 and Mechanize, and it is fairly powerful... Cheers, Peter
-
Peter, thanks for pointing this out, it looks absolutely awesome. My main use case for this is extracting prices for a price comparision engine ( no I don't do black hat scraper stuff! ). When I checked out the options a few months ago, scrapi looked easiest.
Looking at your example, you have taken this a step further and I'm an truly impressed! -
wrt this post: I have just added Hpricot to the article you mentioned, and of course scRUBYt! :-) so it is not that outdated anymore I guess. wrt: scRUBYt!: Man, what's happening there is unbelievable. I got around 40 mails, 10+ feature reqs, 20+ bug fix reqs, lots of future ideas and other kind of feedback - I am totally psyched. OK, I know that scRUBYt! totally rocks :-)) but, still I did not know that everybody else will recognize this so fast! Price comparison... as soon as I will have a little time I would like to put together a little scenario which will present all Ruby/Rails books (this is already a challenge in itself) and their prices from say 10 webshops, updated daily. I am glad you like scRUBYt!, but this is just the beginning - I have lots of other things in store (the todo list has 200+ items ATM - no kidding) so there is much, much more to come...
