Comments for Mike Burns, Coder

Comment on An updated way to spider the Web with Ruby by Jesse Crouch

Jesse Crouch — Mon, 05 Oct 2009 16:54:00 +0000

Hey Mike -

Any way to specify levels of depth for the spider to go? Say it starts at mike-burns.com and I don’t want to limit it by domain name, but by levels of links to follow. First might take you to Digg and then a Digg link takes me to Flickr, etc. Say I only want to find the first level (Digg) and not go any further. Any easy way to do that currently?

Thanks again for the bot. It’s really helped me with my ruby project so far. I’m actually currently using it alongside a rails project that I’m working on.

Comment on Truncating HTML in Ruby by sandip

sandip — Tue, 23 Jun 2009 13:02:46 +0000

Hi,

Thanks for gr8 post!.
It saved my lot of time.
One quick question…I wanted to escape image and embed tag from my html text.

What changes do i need ????

Thanks,

Sandip R~

Comment on Newest Blog Post for Each User by Neo

Neo — Fri, 06 Mar 2009 03:24:40 +0000

ralph lauren paint golden doodle margarita recipes nsw tab full house cast artis indonesia memek seattle premium outlets rabbit review map of guam fleetwood campers

Comment on Spider the Web with Ruby by Carl

Carl — Thu, 15 Jan 2009 20:00:30 +0000

Sry but doesnt work for me.
I get no errors/warning with the installation but
if i try your ecample Code i get this:
—
TypeError: wrong argument type Class (expected Module)
Line 3
—
ruby 1.8.6 (2008-03-03 patchlevel 114) [universal-darwin9.0]
Mac OS X Leopard 10.5.6

thx for answers

Comment on Proxied Spider by Tom White

Tom White — Tue, 13 Jan 2009 21:01:15 +0000

It seems the spider should handle ports. I believe this is a little different then a proxy. Here is my construct_complete_url(), it allows you to spider sites at ports other than 80.

def construct_complete_url(base_url, additional_url, parsed_additional_url = nil) #:nodoc:
parsed_additional_url ||= URI.parse(additional_url)
case parsed_additional_url.scheme
when nil
u = base_url.is_a?(URI) ? base_url : URI.parse(base_url)
if additional_url[0].chr == ‘/’
“#{u.scheme}://#{u.host}:#{u.port}#{additional_url}”
elsif u.path.nil? || u.path == ”
“#{u.scheme}://#{u.host}:#{u.port}/#{additional_url}”
elsif u.path[0].chr == ‘/’
“#{u.scheme}://#{u.host}:#{u.port}#{u.path}/#{additional_url}”
else
“#{u.scheme}://#{u.host}:#{u.port}/#{u.path}/#{additional_url}”
end
else
additional_url
end
end

Comment on Uniquify an array of hashes in Ruby by Eric

Eric — Mon, 05 Jan 2009 11:48:57 +0000

Thanks a lot, I needed this for this problem:
http://projecteuler.net/index.php?section=problems&id=109 :D

Comment on Uniquify an array of hashes in Ruby by Uniq for Array or Hash with a Deeply Nested Structure | Chasing

Uniq for Array or Hash with a Deeply Nested Structure | Chasing — Sun, 04 Jan 2009 23:33:04 +0000

[...] this post, Mike Burns solve this problem by redefining #hash and #eql?. If you are doing this kind of [...]

Comment on Ruby Can’t open(‘http://digg.com/’) by Vamsee

Vamsee — Fri, 02 Jan 2009 17:55:44 +0000

Thank You. This bug, who ever responsible has been driving me crazy.

Comment on Truncating HTML in Ruby by dbr

dbr — Sun, 28 Dec 2008 04:20:26 +0000

Thanks for the code! I’ve made a few small changes:

- Tags without attributes were created like `` (so I made the `attrs_to_s` method return an space+attrs, and removed the space from the `start_element` case)

- I added the at_end parameter, so you can add a string before the closing tags, most likely “…”

For example:

Instead of..

>> puts “Something“.truncate_html(5) + “…”
=> Someth…

You can do..

>> puts “Something“.truncate_html(5, at_end = “…”)
=> Someth…

- I also changed `p_e[0].first(new_len)` to `p_e[0][0..new_len]` as p_e[0] didn’t seem to have a `.first` method?

Oh, the code:

http://pastie.org/347690

Comment on An updated way to spider the Web with Ruby by Michael Forrest

Michael Forrest — Thu, 20 Nov 2008 12:03:42 +0000

Hi Mike

You need to update this example: the handle 2xx should be:

s.on :success do |a_url, resp, prior_url|
puts “body: #{resp.body}”
end

Although, for the record, I disapprove of contracting ‘response’ to ‘resp’, but just put it that way for consistency with the rest of your example.

Thanks for the gem!