« I took a ukulele lesson once... | Main | Wrap »

DMOZ had 9 lives. Used up yet?

RIP DMOZ: 1998-2006

aka Open Directory Project
aka Netscape Open Directory
aka directory.mozilla.org
aka NewHoo
aka GnuHoo

Peter Da Vanzo: Is DMOZ Dead?
Tom Lustina: Here Lies ODP
Sean Bolton: DMOZ, Please Die Already
Resource Zone: submit URL link not working
Trond Sorvoja: Will AOL allow a Open Directory Foundation?

Apparently the machine holding dmoz in AOL ops crashed. Standard backups had been discontinued for some reason; during unsuccessful attempts to restore some of the lost data, ops blew away the rest of the existing data on the system.

So for the past 6 weeks, a few folks have been trying to patch the system back together again (reverse engineering from the latest RDF dump, I suppose). But 6 weeks is a very long outage. Add in the massive AOL layoffs last week, and it's not clear if there's even any left over there who cares. Even if some form of the ODP editing system is brought back, the likelihood of continued existence within AOL seems extremely doubtful.

dmoz doesn't exactly operate on a model of transparency, to say the least, so they have been keeping the details of what happened private. Perhaps they're concerned about an exodus of the remaining editors, or gleeful proclamations of death from the SEM industry. The remaining ODP editors will probably be mad at me for discussing this, but they get mad at me whenever I talk about the ODP....ironic! :-) Hey guys, it's 2006, open up.


What do you do when you get an email like this?

To: "Rich Skrenta"
Date: Tue, 14 Jul 1998 18:47:15 -0700
Subject: Infoseek and NewHoo


I just got off the phone with Steve Kirsch, Infoseek's founder and Chairman of the Board. We are very much interested in purchasing the technology, content, and founders of NewHoo. This is our preferred option, but we would certainly consider discussing other partnering opportunities if this doesn't work out.

We think that the best way to continue the process would be for you to name a price range for a possible purchase, including the appropriate market and financial information justifying that price.

Next, we can continue our discussions if there is enough interest on this side.


We launched NewHoo in June, 1998. Within 4 months we had the CTO of LookSmart saying we wanted to quit and join us, an acquisition offer from Infoseek, a $5M funding offer from Lycos, an angel funding offer being brokered by the Venture Law Group, and an acquisition offer from Netscape. We took the Netscape offer; it was a great strategic fit, since they had a lot of traffic to pour on the directory, and were willing to give the data away for free.

Unfortunately, as with many (most?) acquisitions, the hopeful little product was eventually lost within the sprawling org.

In a 2003 talk, I predicted that the server would get lost in AOL ops, and, deprived of any staff who understood how it worked, it would just crash one day, and that would be it.

My (edited) reply to a dmoz meta editor who contacted me about the extended outage:

Not sure if you all have been following the drama going on within AOL, but I doubt they have any attention for dmoz at all at this point, less even than usual. In fact, my guess is that everyone involved in the management chain there over dmoz for the past 6 years is now gone.


So regardless of the tactics of whether specific front-line people in AOL ops can get the machine running again or not, I doubt that the environment there will be very good in the longer term. All of the folks there who had been championing product-lead growth are now gone. One possible outcome is that Time Warner is slimming AOL down for an eventual spin-out. A more cynical take is that they're going to deliberately torture the org first, as payback for the destruction in Time/Warner value following the AOL merger (this idea was put forward in an NY Times story a few months ago).

I do think it's a great time for a new directory to emerge, and human editing, if supported by sufficient technical automation to make them sufficiently productive, could be a powerful model. Bob Keating's ideas around building a faceted directory are spot-on IMO.

However, I maintain my belief that, without a monetary engine -- in other words, without making the directory a business at some level -- dependence on corporate patronage will eventually leave it weak and understaffed again. One option I might suggest is to look at something like Jimmy Wales' new Wikia service, and see if it could fit the bill, at least at some level. If so, the dmoz editors could move over there and start building again.

WikiPedia is another model to consider. It seems to have depended on patronage, and has probably been limited in the past by resource constraints. Modest advertising (e.g. adsense/adwords on search) on dmoz could easily have supported a staff of 10-20 full time employees, as well as hosting costs. Call it a nonprofit foundation, but you need the entity and some money coming in to pay for things like...proper ops (gosh you could have that from Rackspace for a monthly fee, including backups :-).

But unlikely to be possible within AOL, I'm afraid. I ran a scan of the forums to estimate active editorship...I count approx 4000 recent posters to the forums, given the old 50% measurement that suggests about 8-10k active editors -- plenty to build something fairly interesting again in a relatively short time.


In any case, if I can be helpful in any way, let me know.

-- Rich

I spoke to Bob Keating yesterday and apparently my post shook things up a bit inside of AOL with respect to the ODP. He credited it with getting them to finally assign a sysadmin back to dmoz, which hasn't had a dedicated SA for some time, part of the reason this outage was so long.

I was also contacted by Jimmy Wales of Wikipedia/Wikia, who very much would like to rescue dmoz and give it a good home.

So this post has directly lead to the server being fixed as well as a significant offer of help from a major industry figure.

AOL just had an impressive re-org. I actually briefly worked with Ron Grant while I was there. He's a scarily effective thinker and negotiator and frankly scares the living bejeezus out of me. You have something broken and rotting like AOL, you want some bold moves to try to fix it. Ron's the right guy for that.

Similarly I think the ODP is suffering from its closed, stultifying culture. There needs to be a re-org within the editor culture itself before the ODP will be able to truly move forward. Fire the handful of metas at the core of this rot and have a general housecleaning. Institute term limits for the senior ODP positions; that works great in politics to clean out the old corrupt guys and make way for fresh blood.


Listed below are links to weblogs that reference DMOZ had 9 lives. Used up yet?:

» Trackbacking your entry... from SearchGuild
Hehe, interesting....: http://www.skrenta.com/2006/12/dmoz_had_9_lives_used_up_yet.html [Read More]

» ODP Founder Skrenta On Dying Open Directory from Search Engine Land: News About Search Engines & Search Marketing
DMOZ had 9 lives. Used up yet? from one of the Open Directory's founders Rich Skrenta covers how some of the ODP's data has apparently been lost due to a main server crash, with backups not having been done. There's apparently some attempt at recovery,... [Read More]

» Panne en perspective from DMOZ blog
Sur son blog, Rich Skrenta, l'un des fondateurs de l'ODP, revient sur l'histoire du projet, sur la panne de serveurs qui est apparemment à l'origine d'une panne générale de l'annuaire depuis près de deux mois, et sur l'implication d'AOL,... [Read More]

» DMOZ founder says 'Time to Shape Up or Ship Out' from
Dmoz database crashes with no backup - and AOL with massive layoffs have more important ROI things to focus on. Dmoz founder reflects, and so does SearchEngineWatch's founder ============================================================http://search... [Read More]

» Rich Skrenta zum ODP from tentatives verlinken
Ob die Neun Leben des ODP vorbei seien fragts sich einer der Gründer vom ODP. Hinterher sind zwar alle immer schlauer und Unkenrufe wie In a 2003 talk, I predicted that the server would get lost in AOL ops, and, deprived of any staff who understood h... [Read More]

» DMOZ Dead? Details at 11 from SiteProNews Blog
Has the coffin closed on the Open Directory Project? After a six week outage in which editors and submitters have been greeted with variations of a DMOZ 404 page, one of the founders, Rich Skrenta, weighs in with an opinion that is less than optimistic in [Read More]

» Totgesagte leben länger from u1amo01
Das ODP - ein Anachronismus, ein Zombie, der noch nicht gemerkt hat, dass er tot ist? Selbst einer der Gründer bezweifelt in seinem Blog, ob von den neun Leben des ODPs noch eines übrig sei. Ach. Nach dem verheerenden Crash am 20.10.2006 daue... [Read More]

» Is DMOZ dead? from Platinax Business Forums
I posted before on Platinax News that there were technical issues with DMOZ. Now DMOZ founder Rich Skrenta has joined in to announce DMOZ is dead. According to his comments: Apparently the mach... [Read More]

» DMOZ back online from Platinax Small Business News
DMOZ - the Open Directory Project - has recovered it’s editor server. It comes after almost 2 months of DMOZ editors being unable to log in due to serious technical problems. Issues still remain - for example, site suggestion, update listing, ... [Read More]

Comments (13)

What's the value in the ODP anyway? Seems to me that the improvements in search technology over the past few years have rendered it obsolete anyway.

I am interested in starting a big directory like Dmoz (but not like Dmoz)

I have already created more than 1125categories

at www.annexagain.com

and you can see this here at www.annexagain.com/1125.html

also I am working on to make it as static html pages
I have also a started a thread


at forums.digitalpoint.com

and most of the replies I got there was negative

But I want to continue the project

Also I want some suitabel suggestions from you
also some support as you can tell ODP editors to
switch over to my site

I am creating static html pages via it since I do not have the resources to take a deciated server for it

Also I would be creating static html pages via php script so the php pages would be actually replaced by a static html pages but add url pages will be in php and it would run on a database

so I would be minimizing the load on server

Als editing will be easy and also if a new URL is added I would spider the site via my search engine to see if there are any new links added

If any was added I would type the page url on the browser some thing like www.annexagain.com/1168.html

and save it in the appropriate category name and upload it to the server

I want you to post in there and to spread the word among ODP editors so they could help me a lot

and also since the load on the server would be minimun there would b no chance of a crash like Dmoz experienced so I want you to spread the word among ODP editors

I would first first create 10,000 categories and that means 10,000 pages and then I would visit these 10,000 pages and save them as static html pages (.html)

and then I would upload these pages to different folders like www.annexagain.com/India/

and by then 10,000 pages would have more than 490,000 categories for 49 top countries using the web

and the editor login and editing would be depend on php

Now I want you to post in forums.digitalpoint.com as I could get support from there and also to spread the word among ODP editors so they can assist me in adding categories and other work

I will tell you again since I the directory apages would run on static html page there would be no problem as I can even host on shared hosting like servage.net

And I hope you would help me in this regard and reply to me vai Email

Best regards

AOL should shut DMOZ down because of the legal problems they face because of the adult listings, with NBC Dateline's focus on internet predators, DMOZ porn listings make visible AOL's conflict of interest in selling porn filtering software to parents.

To lose a database is sin number 1, but a database that included tons of editor affiliate links and links of those who bribed editors to get in is not worth much in the first place.

AOL search should simply sell DMOZ or convert it to a local search database and dump all porn listings if they know what is good for them!

I was the first to call for AOL to fire all the editors and replace them with automation, this may be the first step in the process that in the end will clean up the brand and make it relevant in the marketplace.

My thoughts on ODP have evolved a lot since our original dislike of the service and the way it evolved in practice.

No, I don't really go back on my dislike of that practical reality of it, and I actually think starting it again would be a bad idea - I think this is the kind of stuff that works best (and this is increasingly proven) in deep verticals, rather than trying to amass everything.

Then again, Wikipedia and Digg are having their heyday right now.

What ODP did was give us a starting point for key debates about inducements to voluntary rating, editing, and "ontologizing" online. That kind of activity has definitely not gone away, but as another poster suggests, today we have a more sophisticated view of how to do collaborative filtering. If ODP got us 2% along the path of a platform for collective wisdom and a "third voice" in rating particular websites, many of the other services and features out there today have us maybe 4-5% along that path.

Did someone say search is still in its infancy?


Jude Deguzman:

I personally agree with you Rich. I used to be a lowly editor at the ODP at one of its Regional Categories. Its time for the ODP to say goodbye. The big categories cannot be handled with a small amount of editors. Lies and deception are still rampant. Its time for a new one. I have presented a project to a couple of search engines and maybe it might be of interest to you too. We can build a new one with a new concept. I can be contacted at my e-mail address or at Digital Point Forums as popotalk.

DMOZ would be a more appropriate acquisition for ASK.com - with their 'EXPERT RANK' technology emphasis

Ask would lose all credibility with me if they bought DMOZ and incorporated it directly into their ranking technology. They are supposed to be a disinterested third party.


So it's about time to 'create derivative works from'


Basic License. Netscape grants you a non-exclusive, royalty-free license to use, reproduce, modify and create derivative works from, and distribute and publish the Open Directory and your derivative works thereof, subject to all of the terms and conditions of this Open Directory License. You may authorize others to exercise the foregoing rights; provided, however, that you must have an agreement with your sublicensees that passes on the requirements and obligations of Sections 2 and 4 below and which must include a limitation of liability provision no less protective of Netscape than Section 6 below.

Hi Rich,

Thanks for explaining to us what's been going on with the demise of DMOZ. I can't believe that AOL really blew it again so badly. It seems as though AOL's been not the neglected stepchild, but more so that irritating kid on the playground that always gets in the way when the big kids want to play ball.

Your call for a new sort of directory is both insightful & important. Can a wiki-style directory really pull it off though? What sort of entity can really negotiate the altruism that made DMOZ popular at its inception and the business-savvy required to make it really successful (as you put it well). I don't think a non-profit can fit that bill. Perhaps a W3/Wiki/Digg combobulation?

What wonderful food for thought...

Richard Gagnon:

All ODP really needs is a different model for how the directory is populated. Offering a link maintenance service, free to websites, with a licensing agreement that allows ODP to incorporate some of those link lists into their directory, will increase the number of "volunteers" to edit the directory.

Right now, ODP has too small an editorial staff to effectively catalog the web. At the same time, there are thousands of lists maintained by personal and commercial websites, that can fill in the gaps that ODP has in their directories. All ODP has to do is tap into those existing links.

There is a value in human indexing of the web. When I picked up the Corel Painter category, it had been dead for years with a no longer working link to Corel for a version that was two behind the latest one. Other sites on the net didn't have significantly better links. The best I found didn't have more than 20 tutorial links. I searched through thousands of Google listings to come up with a hundred useful tutorials. Without having a human being highlight the useful links, many of the ones I found would have remained lost on the net and not have been available to Painter users.

The issues I have with ODP are:
- An annoying delinking editorial review process (site listings, that don't 100% follow their style guidelines are temporarily removed from the directory until they are corrected).
- Moving links to what another editor considers a more appropriate category. I don't mind a Painter link I found being copied to another location, but I do resent having it deleted from Painter simply because it contains information about other programs or subjects.
- Not allowing more freedom to create categories so that the listings aren't a huge alphabetical listing.
- Not allow ranking of sites by value (other than one or two "cooled" sites.


Excepting Nullsoft[makers of winamp], i doubt if any other company survived after being acquired by AOL. Historically AOL is known to acquire and just ignore them.
Remember the old news/rumour that AOL was to acquire Redhat... and how the tech community reacted to it on slashdot?

It's definitely worth working on a new directory or even revive the existing DMOZ as suggested by a few above. But relying on DMOZ would eb scary going forward. Hmmm

Rick Morrow:

DMOZ had its time and influence in the past and it's time to say bye. What is the point of a human review that simply summarizes a website with a description such as: "Sells X and Y"?
I have always felt the editors, as a whole, do not have credentials as experts nor do they list useable descriptions as noted above. Google et al have sufficient expertise to catalog the web so "human reviews with ineffective descriptions" are efforts in futility.
The smart money would attempt to consolidate as many smaller SE's as possible - do we really need more then Google, Yahoo and MSN?

Tom Munroe:

With all the insightful and well informed comments being made I hate to just rank on DMOZ but I sincerely hate it. If you are in a catagory without an editor you are dead! If you need to update a URL forget it. DMOZ seems to crow that it may never do anything about listing or updating your site and if you don't like it take a flying leap.

I have never liked the Yahoo "we'll edit your site like we want to" so shut up and live with it. It's not really that bad, of course, but mistakes happen and good luck trying to fix them. For some reason I show up in the Canada directoty for Yahoo. Why? Beats me.

Give me bots and automation anyday. Kill the humans!


This page contains a single entry from the blog posted on December 16, 2006 12:09 PM.

The previous post in this blog was I took a ukulele lesson once....

The next post in this blog is Wrap.

Many more can be found on the main index page or by looking through the archives.

Powered by
Movable Type 3.33