When we first launched, I read a few blog post that contemplated whether or not what we were doing is legal. Not only do I firmly believe that it's legal, I think the debate is somewhat moot. We hold ourselves to a much higher standard.
As a search engine, indexing the content on other sites is a privelege not a right. Publishers should clearly see value in the relationship. If they don't, they can easily ask search engines (all or a selected few that are ill-behaved) to stay away. doing this is as simple as adding a single file to their web site -- robots.txt.
As I've stated in other posts, web publishers (including commerce sites, etc.) increasingly view search engines as playing an important role in their market ecology. To play this role, however, i think companies need to follow certain unwritten rules (or net-etiquette):
1. Summarize and point, don't aggregate
I've seen oodle referred to as a classifieds aggregator. We are not an aggregator in that we don't display listings. Our job is to point our users to listings published on other sites -- and in doing so, get users off our site as quickly as possible ( love-in not lock-in).
To this end, we've provide (attributed) summaries that contain enough information for users to qualify their interest but not enough for them to take action. To respond to a listing, users need to click-through to the full listings on the publishers site.
2. Be a search engine not a publisher (or said another way, don't compete with your content partners)
It's a fundamental conflict of interest when a company attempts to both play the role of search engine and play the role of a publisher. Why would publishers let their listings be indexed by a company building a competing business off their back? It will be interesting to see how this point plays out in the coming weeks. Rumors are ablaze with Google's entry into the classified listings business. It's also rumored to be adding classifieds listings into Froogle. Will classified publishers stand for this? I'm not sure why they would.
I was thinking to write a meta search engine to crawl all your listing, plus some others not indexed by yours.Is it ok? If not, why?
Posted by: Karl | August 11, 2005 at 03:12 AM
I'm anxious for an answer to the previous comment...oodle?
Posted by: Cody | August 12, 2005 at 06:41 AM
Sorry for the delayed response. Have been traveling.
Meta searching is fine as long as it's handled appropriately (within the bounds of net-etiquette). Such is the case with other search engines such as Google and Yahoo...
Posted by: Craig | August 16, 2005 at 10:35 AM
I noticed that the job section has monster, hotjobs, etc. Being a lawyer, I read their TOS and it specifically says that you cannot redisplay or manipulate their data w/o their consent. Do you have their written permission?
Posted by: Dennis | September 14, 2005 at 11:58 AM
Hey Dumb-Dumb,
"We are not an aggregator in that we don't display listings. Our job is to point our users to listings published on other sites."
An aggregator does not display content either. Therefore, you are an aggregator or a scraper.
Posted by: Dan | September 17, 2005 at 08:25 PM
Do you have their written permission? (
monster, hotjobs)
thanks.
Posted by: mod | September 28, 2005 at 01:07 PM
Dan,
The only point I was trying to make is that people don't think of Google as an aggregator. We act just like them (showing summaries to help people find the right listing).
That said, if Google is an aggregator to you then please consider us an aggregator.
Posted by: craig | September 29, 2005 at 09:36 AM
With respect to the question(s) above -- do we have written relationships with companies like Monster and do we check every terms of use agreement - the answer is no.
There would be no billion page indexes if search engines needed to check every terms of use agreement on every website. Instead, search engines rely on a well established etiquette using robots.txt. We check robots.txt every time we visit a site. We also conform to the net-etiquette around search results lists (display summaries, attribute, etc).
Posted by: Craig | September 29, 2005 at 05:03 PM
how, in what universe, can it occur to you that oodle does not compete with the content providers for viewers? why would a viewer go to a site that sells 10 widgets when it can select form a site that has 100? can you twist logic enough to answer that?
Posted by: Ray | October 17, 2005 at 06:53 PM
Dear Craig,
I recently subscribed to your blog, because of your great vertical website. I am also working on my own vertical search engine and I just wanted to say you and your team are all doing a great job at Oodle.com. Keep up the good work!
JD-
Posted by: JD | May 08, 2006 at 03:41 AM