« Craigslist Update | Main | Web 2.0 based on biz model shift »

November 30, 2005



You better ask before you scrape this time.

Craig Donato

we always check robots.txt...

i've posted on this before so i'll be brief in my response. it's not reasonable to expect a search engine to contact every site in its index. there would be no billion page indexes if this was the case. that's why the robots.txt protocol exists. it acts as a simple invitation (or gate) to search engines.


True about the robot.txt, however, scrapers are not search engines and they do not always listen to robot.txt files.

Craig Donato


Fair enough. I think there is a big difference between scrapers (that grab and display content) and search engines that point to it.

To be clear and more directly respond to Janne's comment, Oodle is a search engine not a scraper. Our goal is to deliver users to the right classified listing wherever it may reside.

The comments to this entry are closed.