Skrentablog

New blog for me... I've been blogging for the past several years over on the Topix Weblog, but it's gone all corporate on me and I wanted room to talk about stuff that wasn't necessarily related to Topix. Sadly, no more posts like this anymore. :-)

Google sends more traffic to WikiPedia than any other site, according to Hitwise. I've noticed that WikiPedia has top rankings now for more and more searches. Since they've doubled their article count over the past year, to 1.5M english articles, their search footprint has expanded. This trend will only continue.

This is also exactly the kind of destination content that Google loves. The articles are excellent quality, and WikiPedia is entirely non-commercial, so Google can feel good about having them rank at the top of the organic listings. Any algorithmic changes Google makes in the future are likely to preserve and reinforce this effect.

Google accounts for nearly 50% of their inbound traffic. But "wikipedia" is their top inbound search term, so people are seeking the brand out directly, as Steve Rubel notes a year ago.

The full list of top-10 Google downstream sites is:

1. Wikipedia
2. eBay
3. Amazon.com
4. Yahoo
5. IMDB
6. Walmart
7. Answers.com
8. Target
9. MapQuest
10. BizRate

This list doesn't distinguish between paid and non-paid listings, but clearly WikiPedia isn't buying adwords so 100% of their juice is organic. Given how often I see About.com come up in the results I'm surprised to not see them in this list.

It will be interesting to see whether Jimmy Wales can replicate this success with Wikia, the VC-backed commercial twin to WikiPedia. 100% revshare, wow -- I don't fully get the model yet but he's clearly got a bold plan up his sleeve.

More Hitwise coolness from SEOmoz.

He pointed out that I actually implemented the thing in java (this was for the IP security layer in Sun's Javastation kernel), and my comments in the code actually appear at the end of the patent. Truel remarked

Just as exciting is the discovery that Rich used to write code that had comments...in html. I'm not even sure how to process it.

Mr. Skrenta,

    As I understand you are one of the founders of the ODP. My question is simple and I'm hoping that you can help me. I've submitted our site about 15 times over the last 4 years (waiting months and months between submissions) and have not yet been listed. I've followed all of the rules each and every time. Is it possible that there is something not entirely kosher with the way editors handle things? Is it possible that the buzz on the internet that says there is some corruption going on in the ODP with editors is true? Something is really wrong when someone submits a site for 4 years and is ignored.

aka Open Directory Project
aka Netscape Open Directory
aka directory.mozilla.org
aka NewHoo
aka GnuHoo

Peter Da Vanzo: Is DMOZ Dead?
Tom Lustina: Here Lies ODP
Sean Bolton: DMOZ, Please Die Already
Resource Zone: submit URL link not working
Trond Sorvoja: Will AOL allow a Open Directory Foundation?

Apparently the machine holding dmoz in AOL ops crashed. Standard backups had been discontinued for some reason; during unsuccessful attempts to restore some of the lost data, ops blew away the rest of the existing data on the system.

So for the past 6 weeks, a few folks have been trying to patch the system back together again (reverse engineering from the latest RDF dump, I suppose). But 6 weeks is a very long outage. Add in the massive AOL layoffs last week, and it's not clear if there's even any left over there who cares. Even if some form of the ODP editing system is brought back, the likelihood of continued existence within AOL seems extremely doubtful.

dmoz doesn't exactly operate on a model of transparency, to say the least, so they have been keeping the details of what happened private. Perhaps they're concerned about an exodus of the remaining editors, or gleeful proclamations of death from the SEM industry. The remaining ODP editors will probably be mad at me for discussing this, but they get mad at me whenever I talk about the ODP....ironic! :-) Hey guys, it's 2006, open up.

...

What do you do when you get an email like this?

To: "Rich Skrenta"
Date: Tue, 14 Jul 1998 18:47:15 -0700
Subject: Infoseek and NewHoo

Rich,

I just got off the phone with Steve Kirsch, Infoseek's founder and Chairman of the Board. We are very much interested in purchasing the technology, content, and founders of NewHoo. This is our preferred option, but we would certainly consider discussing other partnering opportunities if this doesn't work out.

We think that the best way to continue the process would be for you to name a price range for a possible purchase, including the appropriate market and financial information justifying that price.

Next, we can continue our discussions if there is enough interest on this side.

Regards,
Scott

We launched NewHoo in June, 1998. Within 4 months we had the CTO of LookSmart saying we wanted to quit and join us, an acquisition offer from Infoseek, a $5M funding offer from Lycos, an angel funding offer being brokered by the Venture Law Group, and an acquisition offer from Netscape. We took the Netscape offer; it was a great strategic fit, since they had a lot of traffic to pour on the directory, and were willing to give the data away for free.

Unfortunately, as with many (most?) acquisitions, the hopeful little product was eventually lost within the sprawling org.

In a 2003 talk, I predicted that the server would get lost in AOL ops, and, deprived of any staff who understood how it worked, it would just crash one day, and that would be it.

My (edited) reply to a dmoz meta editor who contacted me about the extended outage:

Not sure if you all have been following the drama going on within AOL, but I doubt they have any attention for dmoz at all at this point, less even than usual. In fact, my guess is that everyone involved in the management chain there over dmoz for the past 6 years is now gone.

http://www.brianalvey.com/2006/12/15/just-add-drama/
http://valleywag.com/tech/aol/fucking-way-222195.php

So regardless of the tactics of whether specific front-line people in AOL ops can get the machine running again or not, I doubt that the environment there will be very good in the longer term. All of the folks there who had been championing product-lead growth are now gone. One possible outcome is that Time Warner is slimming AOL down for an eventual spin-out. A more cynical take is that they're going to deliberately torture the org first, as payback for the destruction in Time/Warner value following the AOL merger (this idea was put forward in an NY Times story a few months ago).

I do think it's a great time for a new directory to emerge, and human editing, if supported by sufficient technical automation to make them sufficiently productive, could be a powerful model. Bob Keating's ideas around building a faceted directory are spot-on IMO.

However, I maintain my belief that, without a monetary engine -- in other words, without making the directory a business at some level -- dependence on corporate patronage will eventually leave it weak and understaffed again. One option I might suggest is to look at something like Jimmy Wales' new Wikia service, and see if it could fit the bill, at least at some level. If so, the dmoz editors could move over there and start building again.

WikiPedia is another model to consider. It seems to have depended on patronage, and has probably been limited in the past by resource constraints. Modest advertising (e.g. adsense/adwords on search) on dmoz could easily have supported a staff of 10-20 full time employees, as well as hosting costs. Call it a nonprofit foundation, but you need the entity and some money coming in to pay for things like...proper ops (gosh you could have that from Rackspace for a monthly fee, including backups :-).

But unlikely to be possible within AOL, I'm afraid. I ran a scan of the forums to estimate active editorship...I count approx 4000 recent posters to the forums, given the old 50% measurement that suggests about 8-10k active editors -- plenty to build something fairly interesting again in a relatively short time.

...

In any case, if I can be helpful in any way, let me know.

-- Rich

Update:
I spoke to Bob Keating yesterday and apparently my post shook things up a bit inside of AOL with respect to the ODP. He credited it with getting them to finally assign a sysadmin back to dmoz, which hasn't had a dedicated SA for some time, part of the reason this outage was so long.

I was also contacted by Jimmy Wales of Wikipedia/Wikia, who very much would like to rescue dmoz and give it a good home.

So this post has directly lead to the server being fixed as well as a significant offer of help from a major industry figure.

AOL just had an impressive re-org. I actually briefly worked with Ron Grant while I was there. He's a scarily effective thinker and negotiator and frankly scares the living bejeezus out of me. You have something broken and rotting like AOL, you want some bold moves to try to fix it. Ron's the right guy for that.

Similarly I think the ODP is suffering from its closed, stultifying culture. There needs to be a re-org within the editor culture itself before the ODP will be able to truly move forward. Fire the handful of metas at the core of this rot and have a general housecleaning. Institute term limits for the senior ODP positions; that works great in politics to clean out the old corrupt guys and make way for fresh blood.

Andrew Goodman with the best year-end recap so far: a month-by-month survey of the most surpising web milestones of 2006.

Fred Wilson experiments with RSS ad distribution in feeds. You gotta hand it to the guy for using his blog as a platform for experimentation and learning. Not content to just slurp it up from the business press, Fred is a lean-forward VC. Nice. :-)

Brad Templeton proposed a Zero User Interface backup solution. I think 3X the cost on the disks is going to be a big discouragement, but I really like the idea of Zero User Interface design. Google search is zero user interface: DWIM

Joel Spolsky: "Every few days some crappy software I can't even remember installing pops up noisy bulletins asking me if I want to upgrade something or other. I could not care LESS. I'm doing something. Leave me alone! I'm sure that the team at Sun Microsystems who just released this fabulous new version of the Java virtual machine have been thinking about the incremental release night and day for months and months, but the other 5,000,000,000 of us here on the planet really don't give a flying monkey. You just cannot imagine how little I want to spend even three seconds of my life thinking about whether or not to install that new JVM."

David Naylor's new social network crossed with a speed dating site TickMe.

Tim Bray: Referrer stats proving Slashdot in decline, Reddit on the ascent.

• I whistle like a madman. Often Christmas carols, year-round. I'm not usually even aware I'm doing it. Off-key, not pleasant or tuneful, and usually fairly loud. I've been heard blocks away. I drove my mother insane with my whistling.

• I worked for 2 weeks for a telephone survey firm in college. That was by far the worst job I've ever had. I had to phone people at dinnertime and take them through 15-minute surveys. We were supposed to lie about how long the survey would take. The worst was when someone, screaming obscenties, would hang up during the final few minutes. A floor manager would then come by and scream at me. "You could have kept them on the line!"

• The best job I ever had was being a lifeguard. Getting a tan and hanging out by the pool was great, but the most novel part of my job involved pool cleaning Sunday mornings. It was a big pool, and you could't just stand on the side with the pole to clean the bottom. So I had to wear a Jules Verne style compressed air contraption to walk around the bottom of the pool with the vacuum thingy. The boss would shut off the air hose to let you know your time was up (you couldn't stay down too long or you'd go silly).

• I studied Mandarin for 3 years in college. I really enjoyed studying the language, but this was to pass a 2 year language requirement. True to my teacher's stern prediction, since graduating 'Ni Hao Ma' is about all that's left.

• I collect old computer books. Especially stuff about OS and programming language design. For some reason I haven't been able to get my hands on a copy of Computer Lib/Dream Machines, although Bill Danielson lent me his copy once so I could read it.

I tag Greg, Brad, Susan, Arin, Mike.

...hmmm, too much I- me- my- in this post...it will never make it past the Topix narcissism filter. :-)

Update: Great followup, including a survey of other commentary on this talk, by Greg Linden.

I came across a gripe that topix forums don't support plus signs in their email addrs in my daily topix vanity scan... John Reinke has come up with a way of forwarding gmail accounts to each other with disposable suffixes so that he can shut off a poisoned email target if it starts to get spam. Clever. I get way too much spam; my old personal addresses are essentially unusable at this point, having been around since about 1991, posted on usenet, the web, etc. I have been putting off doing something about this but figure I'll eventually come up with a solution. The problem is that I want to recover use of these old addresses, not just protect brand new ones (I usually use mailinator for throwaway registrations). So either I write some magical content or connection based spam detector (pita), upgrade to the latest & greatest off-the-shelf stuff, (have done before, pita to maintain), or do some kind of aggressive whitelist. Hmmm. Our new VP ops says he has a custom home-grown system he uses on his personal mail that he's going to deploy at topix. Maybe if that works well I'll see if I can use it on my private stuff too. Gosh, seems like we could have a whole new round of spam fighting startups. The last crop did well, but ... there's still spam, so clearly we're not done. :-)

Of course I'm not the first person to point out how off-kilter web measurements are.

A modest proposal

Let's look at a search referral traffic the way a site owner would.

I picked a basket of medium-to-large websites and looked at the inbound search traffic percentages using Hitwise. I included Topix in this mix, both because it's a representative content site, and also because I could double-check the Hitwise numbers against our own server logs and 3rd party measurements from Google Analytics. As it turns out, the relative inbound referral ratios agreed between Hitwise, Google Analytics and our own server stats.

The results:

What I did

I did a simple average of the percentages instead of a weighted average, to offset the chance that a particular site was being unduly favored by a particular engine. (It doesn't look like that is happening though; both Yahoo and Google favor Wikipedia and IMDB in their top organic outbound referrals, so they seem to be sending traffic to the same kinds of places in their listings). These numbers probably undercount Ask, because they were below the top inbound referrer cutoff for some of these sites.

I'm not a professional analyst, and my approach here is pretty back-of-the-napkin. Still, it confirms what those of us in the search industry have known for a long time.

The New York Times, for instance, gets nearly 6X as much traffic from Google as it does from Yahoo. Tripadvisor gets 8X as much traffic from Google vs. Yahoo.

Even Yahoo's own sites are no different. While it receives a greater fraction of Yahoo search traffic than average, Yahoo's own flickr service gets 2.4 times as much traffic from Google as it does from Yahoo.

My favorite example (not included in the above stats): According to Hitwise, Yahoo blogger Jeremy Zawodny gets 92% of his inbound search traffic from Google, and only 2.7% from Yahoo. :-)

"We see little to stop Google from reaching 70 percent market share eventually; the question, really, comes down to, 'How long could it take?" -- RBC Capital Markets analyst Jordan Rohan.

Welcome to the future, we're already there. To paraphrase an old industry saying about IBM...

Google's not the competition, Google's the environment.

Talk about suffering through... Every year end we get these top-10 lists out of the PR departments at search engines. (Our top search at Topix is 'obituaries', as it always is).

If you've ever looked at raw search logs and tried to tally them you know how noisy that data can be. You have to filter out all the bogus robot hits, normalize per source & partner (think a static href link into a search, like piggly wiggly appearing on a high traffic page. So do you de-dup by referrer as well?) You start throwing out the edgy stuff and playing with stripping out quotes and coalescing typos and misspellings and before you know it you can have whatever list you want when you're done.

At Netscape I found that the code used to tally the reports actually discarded the search that more people did than any other search. It was top by a huge margin too. No not sex, yahoo, porn, britney or anything like that.

It was the null search. People just clicking on the search box without having typed anything. Of course the empty string was stripped out during the log tally run...but inadvertently discarding a key piece of user behavior data.

We had an error page which came up when that happened...we quickly replaced it with a friendly page of instructions. Doesn't seem like Yahoo and Google do anything at all with that search now, MSN does a page turn onto a null result page.

In the spirit of the holidays, I will let the mysql thing go...this time.

From: Rich Skrenta
Sent: Wednesday, December 20, 2006 2:31 PM
To: xxxxxx.xxxxxx@oracle.com
Cc: XXXXXX
Subject: RE: 30 minutes in January?

We don't use any databases here. I hate databases. If anyone bought
Oracle here I'd fire them. Someone brought up mysql last week for
a little stat-counter and I'm probably going to fire them after
the holidays.

http://blog.topix.net/archives/000045.html
--
Rich Skrenta
CEO, Topix.net


From: XXXXXX XXXXXX [mailto:xxxxxx.xxxxxx@oracle.com]
Sent: Wednesday, December 20, 2006 1:35 PM
To: Topix.net, RichSkrenta
Cc: XXXXXX
Subject: 30 minutes in January?

Rich,

I'm the local Oracle field rep and wanted to set some time to
make an introduction and to get to know your priorities for '07.
I happened by last week and XXXXXX suggested I follow up via email.
I work with local startups regarding:

Database/Infrastructure
Application Integration and Middleware
Business Intelligence/Reporting

We could explore these areas or others depending on how you feel time
would be best spent.

Please let me know if you see a 30 minute slot on your calendar in
January that would be good for a meeting.

Have a great holiday season. I look forward to speaking with you soon.

XXXXXX

XXXXXX XXXXXX
Technology Sales - Silicon Valley North
(XXX) XXX - XXXX office/fax
(XXX) XXX - XXXX cell
xxxxxx.xxxxxx@oracle.com

I've been reading too much Anonymous Lawyer...

My father was a plastic surgeon. He took me to ..work.. once and I watched him re-attach someone's index finger that had been cut off on a circular saw. I watched the whole operation, and after the finger was bandaged up at the end, I passed out. I think I was 11.

My crystal radio didn't work. I couldn't solder neatly. I got blobs all over the circuit boards and invisibly ruined components by overheating them. "Do you want to be a doctor like your dad?" Wet stuff was even deeper into the physical world. That wasn't for me. So I went into software.

Watch Sawstop's hot dog demo.

I'm often neurotically distracted by safety-related things, like seatbelts. I have a pal who hated to wear his seatbelt. He had 100 rationalizations for not putting it on. He smoked a lot too, and was trying to quit. I told him to keep smoking and put his seatbelt on, he'd come out ahead statistically. I bet a lot of kids with parents who work in the ER hear a lot of brain-scarring accident stories while they're growing up.

I've always liked goats. They seem so gentle and thoughtful, and it's amusing how they'll actually try to eat your pants and stuff. Goats are cool.

I looked into the whole goat-as-pet thing, since I have a fairly large backyard, although it is quite steep -- perfect! But taking care of a goat is a lot of work, they're social animals, they want to be around other goats, there are various goat-issues that you need a proper vet to assist with, and who knows if the city even allows goats (although a neighbor has a chicken coop, so perhaps I could get away with it.)

I thought of donating a goat to my kids school. I could visit the goat, but then the kids and hopefully some of the other parents would have to take care of it. I'm sure the kids would like a goat.

But what I ended up doing was buying a goat through Heifer International. Hopefully my goat will be helpful to someone who can really make good use of it.

Go buy a goat for someone right now. (You only have a few more days for those '06 tax deductions, and you won't feel like giving on the 26th!) You can also give chickens, pigs, water buffalo, cows, and even crazy stuff like a hive of bees. See their full catalog.

Since the earliest Christmas I can remember, I have been disturbed by the suspicious resemblance between Heat Miser in The Year without a Santa Claus, and Burgermeister Meisterburger in Santa Claus is Coming to Town. I thought this was some straightforward shadiness on the part of Rankin/Bass, the production company behind the stop-motion holiday shows; sort of a meta-deception behind the puppetry evil. I started poking around in the Bugermeister's background and came across the career of a remarkable actor, as well as a chain of connected childhood flashbacks.

Burgermeister Meisterburger was voiced by Paul Frees. WikiPedia says, "like Mel Blanc, he was known in the industry as 'The Man of a Thousand Voices'". As I scanned the list of his credits, I nearly fell out of my chair. He had voiced, in an uncredited role, Colossus himself in Colossus: The Forbin Project.

Colossus is one of the most overlooked sci-fi classics. Produced prior to 2001: A Space Odyssey, it was considered too depressing to release by studio execs, and shelved until HAL grabbed the insane killer computer first-mover advantage.
Ship early, ship often.

He also voiced K.A.R.R., the evil twin to David Hasselhoff's good-robot car K.I.T.T. in an episode of Knight Rider.

Bryn's aside:

Back in August when John Mark Karr "confessed" to killing JonBenet Ramsey I notice that the AdSense on Topix.net when I searched for "Karr" was all about Knight Rider ring tones and David Hasselhoff. After some searching I discovered that KARR was the evil twin of KITT (the true star of Knight Rider). So either AdSense buyers are very thorough or Google keyword targeting is very advanced. Time for Turbo-Boost.

But that's not all.

Frees also voiced the original Star Wars trailer. Bob says

I was intrigued by this man's ability to be the voice of authority. His voice is so strong and commanding that people would bring him in to lend credibility to the worst examples of writing.

This trailer has the worst copywriting I've ever heard. "It's the story of a boy, a girl, and a universe." "A million years in the making, and it's coming to your theater this summer." and, my favorite, "Somewhere in space, this may all be happening right now."

He sounded a lot like the "Ghost Host" from the Haunted mansion. When I listen to it, I can almost remember some voiceovers for ads as well...maybe Levi's jeans? I'm sure he did a lot of ads, too bad there isn't a site for discovering these.

I remember that Levi's ad! It was on Bob Abel's demo tape. I can remember the voiceover, it did sound just like the star wars trailer voice. Same time period too...could very well have been Frees. Bob Abel did some amazing graphics stuff, including Sexy Robot, which was a superbowl ad for ... cans. As in, paid for by a canned-food industry association. Crazy.

Those graphics look primitive now. When I saw them in 1986 they were still eye-popping. I think Able had something to do with Tron too. Tron never looked good, unfortunately, not even when it first came out.

Paul Frees did the voice-over for your childhood

If you grew up in the 70's, this guy's voice was the secret ingredient in your childhood. Your alter-parent from the TV. The voice behind your personal voice-over track. Buy this. Watch this now. I command you. Just like Colossus. The voice of the commercial state, just like 1984 but instead of run by the govt it's run by cereal companies.

I got so distracted with Frees that I never followed up on the Heat Miser. Will have to save that for next Christmas Eve's post. :-) Merry Christmas!

Mike and I were poking around Yelp to grok their model.

Mike: "So they got a DB of the restaurants from Acxiom or someone, made landing pages and collect community on them..."

Rich: "What do they have in Google?"

> site:yelp.com

Mike: "270k, doesn't seem like a lot, there's gotta be more restaurants than that."

Rich: "Google 'restaurant masterfile'"

Mike: "Restaurant Owners Masterfile, 385k restaurants, close..."

('Yelp' ... where did that name come from? Yet-another-listings-provider?.. But 'yelp' sounds better that 'yalp'...nah)

Paypal sure seems like it was a successful startup founder factory...

"No, they're behind the pine cone."

This exchange struck me as odd. Even in context I had to think about it.

...

How did Google know what I wanted?

How did they they associate the query with the address, and the intent to want to see a map?

I tried a couple of other likely locations, but wasn't able to make the map pop up.

Their conclusions:

• Don't try to squeeze your web page and make it more compact. There is little benefit in squeezing your pages since many visitors will scroll down below the fold to see your entire page.
• Since visitors will scroll all the way to the bottom of your web page, make life easier for them and divide your layout into sections for easy scanning.
• Minimize your written text and maximize images, visitors usually don't read text - they scan web pages.
• Encourage your visitors to scroll down by using a cut-off layout.

When AOL acquired ICQ, the AOL designers tried to get the ICQ folks to shorten their pages. AOL usability guidelines forbid scrolling, but ICQ had pages that went on for miles and miles. They were crazy and just never ended, but their users obviously loved the service and loved the feel around the whole product so it worked for them. I've seen debates around how long pages should "optimally" be still being played out on design blogs. Nice to see some real user data about how sites are actually being used.

Browser: Do not try and find the fold. That's impossible. Instead, only try to realize the truth.
Designer: What truth?
Browser: There is no fold.
Designer: There is no fold?
Browser: Then you'll see, that it is not the fold that matters, it is only yourself.

Santa Cruz, California

Not pictured: post-traumatic squid disorder incident.