« What fraction of searches are porn? | Main | About the name 'Blekko' »

Why Search?

I've gone and founded a search startup... you can read it about it in this write-up in TechCrunch. But I get asked - why do search?

Simple - the idea that the current state-of-the-art in search is what we'll all be using, essentially unchanged, in 5 or 10 years, is absurd to me.

The web is big. Really, really big. It's literally billions and billions of pages. It's Carl Sagan big. And it's doubling in size every year or two.

So the idea that what you can see in positions 1-3 above the fold on Google are the sum of what the web has to say about every possible query is crazy.

And yet they have 85%+ market share, and little effective competition. At the same time there is such a fabulous business in search. It's the highest monetization service on the web, by far. Why does this Coke have no Pepsi?

Having just spent 5 years in the media space, I've come away with the idea that editorial differentiation is possible. But the editorial voice of a search engine is in the index...so it has to be algorithmic editorial differentiation.

Google and it's copy-tition were designed 10 years ago. But the web has changed significantly in the past decade. Google was built to index a web that no longer exists... a web where people still engaged in social linking behavior, for one thing.

But at the end of the day, founding a startup has to be about personal motivation. My roots go back to os internals, networking, algorithms, and product boot-up strategies. Basically, trying to make algorithmic sense of the vastness of the web is a difficult but really interesting problem. So is tilting at the biggest brand on the web. It's all just plain fun, which ideally should be the point of working. ;)


Listed below are links to weblogs that reference Why Search?:

» Blekko will Google Konkurrenz machen from Marcus Schuler
Konkurrenz für Google? Wieder versucht einer, das schier unmögliche: Rich Skrenta, der Mann, der den ersten Computervirus schrieb, startet ein Suchmaschinenprojekt. Es heißt Blekko. Get the latest Flash Player to see this player. [Javas... [Read More]

» The Google Challengers: 2008 Edition from Search Engine Land: News About Search Engines & Search Marketing
Rich Skrenta -- who, aside from creating the first computer virus, is more notable to search as a cofounder of the Open Directory Project and the Topix news search engine -- has announced he's founded a search start-up. A stealth one, as TechCrunch put... [Read More]

» Ричард Скрента создаёт новый поисковик from DMOZ блог
Ричард Скрента, создатель первого компьютерного вируса, один из основателей DMOZ и Topix, занимается одним из крупнейших проектов за свою ка... [Read More]

Comments (46)


Congrats on emergence from cloaking! Now, if only someone would release a massively scalable data mining platform as an open source project within the next 30 days... :-) http://hypertable.org

It's quite bold what you're planning to do, wish you all the best, and I would be extremely happy if it worked out and managed to challenge Google.

But 2009 for a public release is such a long time...

Peter Da Vanzo:

Hey Rich - great to hear.

Why *not* search....

"Google was built to index a web that no longer exists."

So true. However, Google added blogs, product, video, and news to the main search, which enhanced the results. single web pages results still use the decade old PageRank.

Hey Rich,

Congrats on your new endeavor!! Sounds really exciting and challenging. Best of luck and here's to a great 2008! =)


Came here from digg, I think it's a great idea and you're definitely bang on about what's wrong with Google (I've said much the same thing myself - though usually in arguments with people who think it's fine for paid links to influence search results).

Good luck with it, I'll keep an eye out for you!


Looking forward to seeing how Blekko is going to tackle the search market. Despite Google's dominance, it's encouraging to see new players such as Wikia and Blekko - competition can only be a good thing.


Sounds exciting. Looking forward to seeing what you come up with.

The idea of a different way of approaching search is really appealing, and I hope you see success with it.


Good luck with the search algorithm. I'm sure that your first year will be to figure out the algorithm to which to set your taxonomy on. Good luck. I agree that the Modern Web is upon us and Pagerank is outdated or changing. I find search different now. What I want is good navigation and that includes everything: sites, information, audio, video, blogs,..all in the proper context.

I'm looking forward to seeing your product.


nice! i was guessing it was search =)


Hi Rich,

Good luck and I hope you do well. I agree that there is a lot more innovation left to do.

If I had to guess what you were going to do- it would be a cross between Kosmix and what you originally intended to at Topix.net (see snippet of article below). Co-incidentally, I had read your OJR interview a day before the TechCrunch article.


We started off with a high-level idea of creating a Web page for every person, place or thing in the world. When you go to Google and do a search, you get 10 Web sites and they're all different. You type in "Madonna" and you get 10 Madonna pages. And if you click and go to each one, each one is authored by someone else. You might go to a GeoCities site, you might go to all these official sites.

We had the idea that by using new ideas in the artificial intelligence community, you could actually read the entire Web and parse it and understand it and produce a regular page. So rather than have 10 different pages, you could produce a summary page about whatever you typed. Sort of a Google News per topic, whether it was Madonna, or Billings, Montana, or presidential elections or whatever it was.

susan mernit:

Can't wait to see this smarter and more relevant algo! very good to hear you're back at it with this one.

Kudos, Rich. Your premises are on track and I can't wait to see you and your team tackle the much harder challenge (harder than describing the problem, I mean) of solving that enormously complex search problem you describe so well.

- Andrew

I knew it. Rich, we'd love to help you out with yoru crawler :-)


After Newhoo and Topix, will Blekko make it three for three? You sure have the search cred to pull it off. It's great to see you tackle another 'small' problem. One could argue that Yahoo and Microsoft are the Pepsi and 7UPs of search. Regardless, a fresh take on search is always interesting to watch. Look forward to seeing the beta. Good luck!

Congrats Rick. 2008 is going to be an interesting year...

I agree that a "search engine" has not changed in over eight years and that system or index is corrupted. It is corrupted because of the reliance on a alogrithim and the gaming of the system.

In my eyes and (note: that these are very small eyes as I am not a SEO expert) the context of the overall site is not captured as well as basis of its connections to other websites.

I see one hurdle you will need to overcome. Teaching a old dog (general public) a new trick (new search function). As stated earlier we have been using Google for 5 to 10 years and other search engines who rely on presenting sites matching our keywords. And if you remember, the general public took a while to catch on on how to use Google and its radical at the time feature of keyword searching. This re-education of the general public I envision will be one of your hurdles.

I look forward to seeing what you crank out.


elyfa :

"Why does this Coke have no Pepsi?"

Makes sense. people are satisfied with the current state of search and Google results, just as a few dozen years, people were totally happy with their transportation (horses)- then came the car and it has created a huge industry and eased our lives.

Congrats, Rich. Looking forward to trying out the new search. Update for ya: I'm moving out there in the next three weeks or so.

Congrats Rich, great to hear people are realizing that the (search) emperor is wearing a very skimpy outfit :)

IMHO Google main weapon nowadays is the perception that the resources required for full web search are so humongous that nobody stands a chance going after them. They clearly cultivate this perception .

The scale required is obviously very big, but not nearly as insurmountable as one might think. For anyone interested, there are some data in a comment exchange between Kevin Burton and myself. Everyone is invited to add to it.

It is great to see that search innovation is alive and well, looking forward to the next wave, of course, we are going to make our contribution too, :)

The way we
- do searches ,
- interpret them , and
- present them
hasnt changed since the advent of Google.

As you rightly said , its been 'copy-itions'. Am sure that vizualization has a big key to the next generation of search engines.

All da best


goodluck with your endeavor on this market dominated by some, looking into the success of Blekko.

Any chance for a better search is alright in my books! Good luck with "Blekko" Rich!


The skeptical side of me asks, what are you trying to sell to Google Rich? Is this another Wiki marketing move? Fess up and we'll leave you alone.

I would love to see Beckko or Gekko or whatever knock down Google (hate Google with a passion) but son, you chose the wrong name! Hire some branding experts because you started with the wrong foot. What are those 2 K in the name? Also, too many similar brands out there.

I don't know but I think this is just another marketing scheme fueled by blogs.

Rich, I don't think you got what it takes to be the Pepsi of Google. A watery Safeway "Select" brand just a few buy, maybe. You are all about "niche engines", aren't you? Didn't you say a few months ago that going head to head combat with Google is a sore loser? Or did you change your mind and want now to take on Google head to head?

Woohoo! Congrats on the new venture...


Hi Rich!!!

I was part of your gnuhoo and still part of of chefmoz. Go get em! You are the best and someone I will forever look up to. Need any help let me know. I'd be proud to stand by your side!! :)

Thanks, Bett

Terry Phillips:

Well I hope it doesn't turn into the pile of crap that DMOZ became. You allowed it to fall into the hands of a load of petty dictators.

Don't give up the day job!

Hey! I found your robot crawling my site and came here. I was writing about this the other week - how Google is seen as the end-all be-all of searching the internet, but that there's just so much more that can be done. Best of luck w/the project! You've got your work cut out for you.

I found your robot crawling my site and had to stop by for a look as I wasn't sure what it was. Best of luck with your project and I hope that your search can take a chunk out of the domination that's currently taking place in the search market.

A page for every place on earth?

I think you're on to something there! ;)


Good Luck Rich! You have done wonders in the past but this is a pretty big bite you're trying to take. My advice: If you are going to go head to head with Google the sure fire way to beat them is in customer communication...if you can pull that off you can surely make it.

no one:

I found this after your robot came to my site. Now, the question for me is why should I let you use my bandwidth to develop your search engine?

Since I don't expect you to send any real traffic my way the only thing I have to consider is the hope that you might be better than Google. But being better doesn't just mean sending me more hits. There is something bigger. Google, Yahoo, and the rest operate in a completely amoral, greedy fasion. Yes, they should make as much money as they can -- BUT, that still has to come second to right and wrong. They have no problem helping authoritarian goverments (e.g., China) imprison people and the U.S. government spy on it's own citizens as it marches towards authoritarianism. Google in particular has demonstrated a complete disdain for people's right to privacy.

Even when money isn't the motivation, too many people pray at the alter of computer techno-cool and think that just because they don't care about their own privacy they have a right to violate that of others.

Until I see a clear statement that Beckko will be different in this regard I don't have any incentive to allow it access to my site. I'll keep checking back, and if I see such a statement, then things will change.

Good luck ..

i seen your webcrawler info in my website than i follow it :)) now i understand you want to make a big big search engine .. :P

good luck again ..

Rich - came here via mention on RWW. Interesting quest, best of luck!

So is search the only way you're going to go about this, or dynamic clustering of results, navigation, etc.?

We're currently finishing up our next research piece, the Market IQ on Findability - and in the "timing is everything" mode, would be curious on your feedback on my post earlier today.


Mostly talking about the enterprise, but when trying to find something (content, people, data, etc.), search is only one method of many.

Hi Rich, best of luck to you in your pursuit. Google is a worthy opponent to tackle, sizewise. While it's true they are operating in yesterday's paradigm, the fact that they keep adding satellite operations (free blog bandwidth, video, etc.) give them an anaconda grip on the whole web.

What's interesting to me is that a cult of "Google haters" hasn't developed, as happened with Microsoft as they began to completely dominate the OS world. Perhaps because there's no price tag attached for the next forced upgrade? Food for thought...

Hi Rich,

Love the 'Coke vs Pepsi'. I completely agree. Though Google does has deep pocket companies like Microsoft angling to land the next big buzz in search. Much of the non-tech oriented population would probably be surprised to learn that there are other companies in the search business with the 'possible' exception of Yahoo.
Good luck with the new endeavor. If you need some legwork(research,etc..) contact me. I am a db(sql) guy with asp.net/c# background and enjoy massive undertakings.

Saw that you indexed some of my site recently! Good luck and I'll be sure to check it out when your online.

First of all, congrats Rich!
I also came here by fallowing your crawler..

It looks to me that you are in the right place the right time..

In my personal opinion, Google HAS been doing a lot of development, but in many different markets..
Check out site,blogs,maps,apps,mail,android etc etc etc..

I feel that Google has been doing lately what most of the small companies do in there first steps.. "They just do almost about anything!"

And this is something that i have been thinking for a very long time..
One think though..
If you check out some SEO Masters forums and blogs you will see that everyone is talking about some new "era" of google's algorithms..
Is it true? Is it coming ?? Are they tiered of there job ? :P


Good luck and all the best!


Just seen Scoutjet Indexing one of my sites, Good Luck with Blecko, Hope the David and Goliath thing works out :-)

Good Luck Rich! I have just translater your article "About ScoutJet web crawler" to Chinese ant put it on my website, I hope more and more people to see your growing scoutjet crawler, and hope you can beat google.
wait for your good news, hope you can make it.

I think it's great that you're taking on Google. You just crawled my website. Google is slow and archaic and I'm tired of seeing the same 3 companies who are paying for links, which is against google policy, in the top 3 spots all the time!

Here's a simple suggestion on creating a better search engine: don't broaden the search after the 2nd or 3rd page- try digging deeper. It's so aggravating on Google that after a few pages of not finding what I'm looking for, that it begins bringing up totally abstract sites. Possibly even a "Stay Focused" button on the subsequent pages.


Why not search,Rich?
Search engine is so powerful, I don't know what could we do without Internet and Google,Baidu...


Noticed your scoutjet program had crawled my site, and the name caught my eye. Following the link I found this article and was intrigued! I think you are on to something. Yes, you are tilting at a very large windmill, but I am rooting for the underdog in this case!
Good luck and Good Fortune!


Yes you may use my web site to crawl and hopefully you are the next Google.
Indexing should be free and the same for all on the web. Charge for advertising but leave the search for all to be equal.
Good Luck.
Richard N

Post a comment

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)


This page contains a single entry from the blog posted on January 2, 2008 9:52 PM.

The previous post in this blog was What fraction of searches are porn?.

The next post in this blog is About the name 'Blekko'.

Many more can be found on the main index page or by looking through the archives.

Powered by
Movable Type 3.33