The newest version of Spider, 0.3.0, is hitting your gem tree Real Soon Now. This release features:
- Set the headers to a HTTP request.
- This can be used to set the cookies, user agent, and many other fine things.
- Seems like a good place to set the headers if the headers are conditional on the URL.
- Makes more sense this way, I claim.
- All the handlers take the same three arguments.
- The URL, the response, and—new—the calling URL.
Next on my list: proxies, a better way to store whether an URL has been seen, then a tutorial.
Get it the usual way:
gem install spider