Proxied Spider

Aha: if you need to proxy your Spider calls, look no further than the HTTP Configuration gem. I didn’t write this, and have yet to use it, but I think it goes like this: So next up will be a tutorial with stuff like this and other cool stuff, plus a way to use memcached […]

An updated way to spider the Web with Ruby

I’ve released version 0.2.0 of Spider. Everything has changed: Use RSpec to ensure that it mostly works. Use WEBrick to create a small test server for additional testing. Completely re-do the API to prepare for future expansion. Add the ability to apply each URL to a series of custom allowed?-like matchers. BSD license. The new […]

Spider the Web with Ruby

I wrote a Ruby library for crawling the Web. Use it to take down The Man, like so: I used it to gets people’s addresses from around the Web. I plan to put them on a map. I like putting things on maps. It once took obscene amounts of memory, until I discovered that Ruby […]

Ruby Can’t open(‘’)

This is a bit of a Ruby bug, but it’s so much more of a Digg bug that it’s worth telling the public about: This code, as of Ruby 1.8.5 and March 29th, 2007, will not work. In fact, it will crash Ruby (which I should report, yes). I suspect the crash happens when the […]

response_from: the opposite of respond_to render :xml

Update: This can be replaced, mostly, with Rails 2.0′s ActiveRecord#from_xml. As I assured in a prior post, I spent today hacking on a plugin to make it easier to consume XML REST APIs. The result is the ResponseFrom plugin. It's the opposite of this:  respond_to { |w| w.xml { render :xml => User.find_first } } […]

REST Hacking

First off, many thank yous to Brian Del Vecchio for linking to MyNCRE from BostonWTF. It's an awesome idea and Brian's an awesome guy. Thanks!  In other news, I spent much of today hacking on a REST XML API. The good news: respond_to is awesome, as is render's :xml argument and especially ActiveRecord::Base's to_xml method. […]


Get every new post delivered to your Inbox.