I wrote a Ruby library for crawling the Web. Use it to take down The Man, like so:
require ‘rubygems’
require ’spider’
include Spider
spider(['http://del.icio.us/mikeburns']) do |a_url, a_web_page|
puts "I am taking down The Man by knowing this URL: #{a_url}"
end
I used it to gets people’s addresses from around the Web. I plan to put them on a map. I [...]
Monthly Archives: March 2007
Spider the Web with Ruby
March 31, 2007 – 3:03 am
Ruby Can’t open(‘http://digg.com/’)
March 29, 2007 – 5:13 am
This is a bit of a Ruby bug, but it’s so much more of a Digg bug that it’s worth telling the public about:
This code, as of Ruby 1.8.5 and March 29th, 2007, will not work. In fact, it will crash Ruby (which I should report, yes).
require ‘open-uri’
open(‘http://digg.com/robots.txt’)
I suspect the crash happens when the timeout [...]