http://www.cheezhead.com/2005/10/21/more-on-oodle-craigslist-and-ebay/ Insight and opinion from the world of employment. Fri, 13 Nov 2009 04:01:22 -0700 http://wordpress.org/?v=2.8.6 hourly 1 http://www.cheezhead.com/2005/10/21/more-on-oodle-craigslist-and-ebay/comment-page-1/#comment-324 Joel Mon, 24 Oct 2005 18:20:34 +0000 http://www.cheezhead.com/2005/10/21/more-on-oodle-craigslist-and-ebay/#comment-324 Your robots.txt is a message to bots that come to your site. Technically, yes, I don't think you can block a bot, so it's more of a protocol of politeness than a brick wall. Beyond that, I assume the courts would decide scraping activity if it's done against the will of the site. Also, I think if a site really wanted to, they could show different content to a specific bot than the rest to defend itself against such "terrorism." Your robots.txt is a message to bots that come to your site. Technically, yes, I don’t think you can block a bot, so it’s more of a protocol of politeness than a brick wall. Beyond that, I assume the courts would decide scraping activity if it’s done against the will of the site. Also, I think if a site really wanted to, they could show different content to a specific bot than the rest to defend itself against such “terrorism.”

]]> http://www.cheezhead.com/2005/10/21/more-on-oodle-craigslist-and-ebay/comment-page-1/#comment-323 harmonyjones Sun, 23 Oct 2005 17:58:47 +0000 http://www.cheezhead.com/2005/10/21/more-on-oodle-craigslist-and-ebay/#comment-323 How does Craiglist actually "BAN" Oodle? I mean technically it's not possible. If an aggregator wants to scrape Craiglist listings they can. Simple as that. Aside from restrictions on robots.txt how do they stop Oodle? How does Craiglist actually “BAN” Oodle?
I mean technically it’s not possible.
If an aggregator wants to scrape Craiglist listings they can. Simple as that.
Aside from restrictions on robots.txt how do they stop Oodle?

]]>