Monthly Archives: March 2007

Spider the Web with Ruby

I wrote a Ruby library for crawling the Web. Use it to take down The Man, like so:
require ‘rubygems’
require ’spider’
include Spider
spider(['http://del.icio.us/mikeburns']) do |a_url, a_web_page|
puts "I am taking down The Man by knowing this URL: #{a_url}"
end
I used it to gets people’s addresses from around the Web. I plan to put them on a map. I [...]

Ruby Can’t open(‘http://digg.com/’)

This is a bit of a Ruby bug, but it’s so much more of a Digg bug that it’s worth telling the public about:
This code, as of Ruby 1.8.5 and March 29th, 2007, will not work. In fact, it will crash Ruby (which I should report, yes).
require ‘open-uri’
open(‘http://digg.com/robots.txt’)
I suspect the crash happens when the timeout [...]