John Nagro immediately reported errors with the Spider Ruby gem, so I’ve fixed them in 0.2.1. You should upgrade, especially if you want support for:
- URLs without any path component (e.g.
- HTTP redirects.
John also had some good ideas, so here is what is in the works:
- The ability to construct a complete graph of every node found.
- Defeat cycles using memcached instead of memory (closely related to the above bullet).
Net::HTTPabstraction that makes it easier to e.g. use a proxy or replace HTTP.