The newest version of Spider, 0.3.0, is hitting your gem tree Real Soon Now. This release features:

Set the headers to a HTTP request.

This can be used to set the cookies, user agent, and many other fine things.

setup and teardown handlers.

Seems like a good place to set the headers if the headers are conditional on the URL.

Say :every, not :any.

Makes more sense this way, I claim.

All the handlers take the same three arguments.

The URL, the response, and—new—the calling URL.

Next on my list: proxies, a better way to store whether an URL has been seen, then a tutorial.

Get it the usual way: gem install spider