We’re very excited about where things are going. Oodle provides a valuable service to consumers by enabling them to see all their choices. We’ve seen over 25% growth in users and traffic to partners every month since we launched. Last month alone we sent Craigslist well over 1 million referrals (i.e., users sent to their site). We also continue to have the largest classifieds index (Craigslist represented about 20%) and will continue to add millions of more listings to our index in the coming months. This week, we received over a dozen requests from classified sites wishing to be included.
This whole Craigslist thing has unfortunately mushroomed into quite a big deal. They certainly have every right not to be in our index and I’m genuinely sorry that we appear to have upset them. We continue to be great fans.
In a recent news article, Craigslist CEO Jim Buckmaster made a couple of comments about our actions. It’s never good to see these sorts of things play out in a public debate but we wanted to provide our point of view.
We have always been up front with Craigslist. Before we launched, we sent them a note asking them to check out our site. A month later we requested a meeting to discuss what we were doing and how we were accessing their site. Both times they politely declined to meet with us.
We do our best to act as a responsible search engine. We strictly follow the robots.txt protocol and check it every time we visit a site. For those sites that don’t want to be included, they simply need to add a single line of text to their robots.txt file (as of today, we are still not in Craigslist’s robot.txt file).
After Jim let us know that Craigslist wished to be removed from Oodle, we complied within two business days. During this time, we tried to establish a dialog to address their concerns and are still interested in doing so.
We always try to be very mindful in the way we index a site. We go to great lengths to minimize our impact on our partners’ servers (and visit most sites in the middle of the night when the cost to crawl is near zero). To put it in perspective, Google and Yahoo pull about three times the data from Oodle's servers each day than we pulled from Craiglist. It’s helpful to know the issue at hand is the load we place on their servers. We have a number of proposals on how to address their concerns. I hope we get a chance to share them.