Simple - the idea that the current state-of-the-art in search is what we'll all be using, essentially unchanged, in 5 or 10 years, is absurd to me.
The web is big. Really, really big. It's literally billions and billions of pages. It's Carl Sagan big. And it's doubling in size every year or two.
So the idea that what you can see in positions 1-3 above the fold on Google are the sum of what the web has to say about every possible query is crazy.
And yet they have 85%+ market share, and little effective competition. At the same time there is such a fabulous business in search. It's the highest monetization service on the web, by far. Why does this Coke have no Pepsi?
Having just spent 5 years in the media space, I've come away with the idea that editorial differentiation is possible. But the editorial voice of a search engine is in the index...so it has to be algorithmic editorial differentiation.
Google and it's copy-tition were designed 10 years ago. But the web has changed significantly in the past decade. Google was built to index a web that no longer exists... a web where people still engaged in social linking behavior, for one thing.
But at the end of the day, founding a startup has to be about personal motivation. My roots go back to os internals, networking, algorithms, and product boot-up strategies. Basically, trying to make algorithmic sense of the vastness of the web is a difficult but really interesting problem. So is tilting at the biggest brand on the web. It's all just plain fun, which ideally should be the point of working. ;)
Comments (41)
Rich,
Congrats on emergence from cloaking! Now, if only someone would release a massively scalable data mining platform as an open source project within the next 30 days... :-) http://hypertable.org
Posted by Ethan Stock | January 2, 2008 11:28 PM
Posted on January 2, 2008 23:28
It's quite bold what you're planning to do, wish you all the best, and I would be extremely happy if it worked out and managed to challenge Google.
But 2009 for a public release is such a long time...
Posted by Baher | January 3, 2008 12:41 AM
Posted on January 3, 2008 00:41
Hey Rich - great to hear.
Why *not* search....
Posted by Peter Da Vanzo | January 3, 2008 1:23 AM
Posted on January 3, 2008 01:23
"Google was built to index a web that no longer exists."
So true. However, Google added blogs, product, video, and news to the main search, which enhanced the results. single web pages results still use the decade old PageRank.
Posted by Fares Al-Osaimi | January 3, 2008 2:53 AM
Posted on January 3, 2008 02:53
Hey Rich,
Congrats on your new endeavor!! Sounds really exciting and challenging. Best of luck and here's to a great 2008! =)
Posted by Patrick Lee | January 3, 2008 3:11 AM
Posted on January 3, 2008 03:11
Hello,
Came here from digg, I think it's a great idea and you're definitely bang on about what's wrong with Google (I've said much the same thing myself - though usually in arguments with people who think it's fine for paid links to influence search results).
Good luck with it, I'll keep an eye out for you!
Dave
Posted by Dave Clayton | January 3, 2008 5:36 AM
Posted on January 3, 2008 05:36
Looking forward to seeing how Blekko is going to tackle the search market. Despite Google's dominance, it's encouraging to see new players such as Wikia and Blekko - competition can only be a good thing.
Posted by Mark Evans | January 3, 2008 8:01 AM
Posted on January 3, 2008 08:01
Sounds exciting. Looking forward to seeing what you come up with.
The idea of a different way of approaching search is really appealing, and I hope you see success with it.
Cheers.
Posted by Bill | January 3, 2008 8:58 AM
Posted on January 3, 2008 08:58
Good luck with the search algorithm. I'm sure that your first year will be to figure out the algorithm to which to set your taxonomy on. Good luck. I agree that the Modern Web is upon us and Pagerank is outdated or changing. I find search different now. What I want is good navigation and that includes everything: sites, information, audio, video, blogs,..all in the proper context.
I'm looking forward to seeing your product.
Posted by John Furrier | January 3, 2008 9:45 AM
Posted on January 3, 2008 09:45
nice! i was guessing it was search =)
Posted by Arin | January 3, 2008 10:02 AM
Posted on January 3, 2008 10:02
Hi Rich,
Good luck and I hope you do well. I agree that there is a lot more innovation left to do.
If I had to guess what you were going to do- it would be a cross between Kosmix and what you originally intended to at Topix.net (see snippet of article below). Co-incidentally, I had read your OJR interview a day before the TechCrunch article.
=====================
http://www.ojr.org/ojr/stories/050802Glaser/
We started off with a high-level idea of creating a Web page for every person, place or thing in the world. When you go to Google and do a search, you get 10 Web sites and they're all different. You type in "Madonna" and you get 10 Madonna pages. And if you click and go to each one, each one is authored by someone else. You might go to a GeoCities site, you might go to all these official sites.
We had the idea that by using new ideas in the artificial intelligence community, you could actually read the entire Web and parse it and understand it and produce a regular page. So rather than have 10 different pages, you could produce a summary page about whatever you typed. Sort of a Google News per topic, whether it was Madonna, or Billings, Montana, or presidential elections or whatever it was.
=====================
Posted by Nik | January 3, 2008 10:11 AM
Posted on January 3, 2008 10:11
Can't wait to see this smarter and more relevant algo! very good to hear you're back at it with this one.
Posted by susan mernit | January 3, 2008 11:08 AM
Posted on January 3, 2008 11:08
Kudos, Rich. Your premises are on track and I can't wait to see you and your team tackle the much harder challenge (harder than describing the problem, I mean) of solving that enormously complex search problem you describe so well.
- Andrew
Posted by Andrew Goodman | January 3, 2008 1:47 PM
Posted on January 3, 2008 13:47
I knew it. Rich, we'd love to help you out with yoru crawler :-)
Kevin
Posted by Kevin Burton | January 3, 2008 4:10 PM
Posted on January 3, 2008 16:10
After Newhoo and Topix, will Blekko make it three for three? You sure have the search cred to pull it off. It's great to see you tackle another 'small' problem. One could argue that Yahoo and Microsoft are the Pepsi and 7UPs of search. Regardless, a fresh take on search is always interesting to watch. Look forward to seeing the beta. Good luck!
Posted by Dariusz Paczuski | January 3, 2008 6:17 PM
Posted on January 3, 2008 18:17
Congrats Rick. 2008 is going to be an interesting year...
Posted by Kyle Shank | January 3, 2008 10:44 PM
Posted on January 3, 2008 22:44
I agree that a "search engine" has not changed in over eight years and that system or index is corrupted. It is corrupted because of the reliance on a alogrithim and the gaming of the system.
In my eyes and (note: that these are very small eyes as I am not a SEO expert) the context of the overall site is not captured as well as basis of its connections to other websites.
I see one hurdle you will need to overcome. Teaching a old dog (general public) a new trick (new search function). As stated earlier we have been using Google for 5 to 10 years and other search engines who rely on presenting sites matching our keywords. And if you remember, the general public took a while to catch on on how to use Google and its radical at the time feature of keyword searching. This re-education of the general public I envision will be one of your hurdles.
I look forward to seeing what you crank out.
chris
Posted by chris | January 3, 2008 11:32 PM
Posted on January 3, 2008 23:32
Go, Blekko! :-)
Posted by Marius Chitosca | January 3, 2008 11:33 PM
Posted on January 3, 2008 23:33
"Why does this Coke have no Pepsi?"
Makes sense. people are satisfied with the current state of search and Google results, just as a few dozen years, people were totally happy with their transportation (horses)- then came the car and it has created a huge industry and eased our lives.
Posted by elyfa | January 4, 2008 8:52 AM
Posted on January 4, 2008 08:52
Congrats, Rich. Looking forward to trying out the new search. Update for ya: I'm moving out there in the next three weeks or so.
Posted by Rocky | January 4, 2008 6:46 PM
Posted on January 4, 2008 18:46
Congrats Rich, great to hear people are realizing that the (search) emperor is wearing a very skimpy outfit :)
IMHO Google main weapon nowadays is the perception that the resources required for full web search are so humongous that nobody stands a chance going after them. They clearly cultivate this perception .
The scale required is obviously very big, but not nearly as insurmountable as one might think. For anyone interested, there are some data in a comment exchange between Kevin Burton and myself. Everyone is invited to add to it.
It is great to see that search innovation is alive and well, looking forward to the next wave, of course, we are going to make our contribution too, :)
Posted by Borislav Agapiev | January 6, 2008 4:12 AM
Posted on January 6, 2008 04:12
The way we
- do searches ,
- interpret them , and
- present them
hasnt changed since the advent of Google.
As you rightly said , its been 'copy-itions'. Am sure that vizualization has a big key to the next generation of search engines.
All da best
Posted by Venkat | January 6, 2008 7:31 AM
Posted on January 6, 2008 07:31
goodluck with your endeavor on this market dominated by some, looking into the success of Blekko.
Posted by Leo | January 7, 2008 12:51 AM
Posted on January 7, 2008 00:51
Any chance for a better search is alright in my books! Good luck with "Blekko" Rich!
Posted by Payday Land | January 7, 2008 10:09 AM
Posted on January 7, 2008 10:09
The skeptical side of me asks, what are you trying to sell to Google Rich? Is this another Wiki marketing move? Fess up and we'll leave you alone.
I would love to see Beckko or Gekko or whatever knock down Google (hate Google with a passion) but son, you chose the wrong name! Hire some branding experts because you started with the wrong foot. What are those 2 K in the name? Also, too many similar brands out there.
I don't know but I think this is just another marketing scheme fueled by blogs.
Rich, I don't think you got what it takes to be the Pepsi of Google. A watery Safeway "Select" brand just a few buy, maybe. You are all about "niche engines", aren't you? Didn't you say a few months ago that going head to head combat with Google is a sore loser? Or did you change your mind and want now to take on Google head to head?
Posted by JJ | January 7, 2008 11:47 AM
Posted on January 7, 2008 11:47
Woohoo! Congrats on the new venture...
Posted by Jeremy Zawodny | January 14, 2008 3:48 PM
Posted on January 14, 2008 15:48
Hi Rich!!!
I was part of your gnuhoo and still part of of chefmoz. Go get em! You are the best and someone I will forever look up to. Need any help let me know. I'd be proud to stand by your side!! :)
Thanks, Bett
Posted by LdyDragonfly | April 1, 2008 1:29 AM
Posted on April 1, 2008 01:29
Well I hope it doesn't turn into the pile of crap that DMOZ became. You allowed it to fall into the hands of a load of petty dictators.
Don't give up the day job!
Posted by Terry Phillips | April 2, 2008 10:58 PM
Posted on April 2, 2008 22:58
Hey! I found your robot crawling my site and came here. I was writing about this the other week - how Google is seen as the end-all be-all of searching the internet, but that there's just so much more that can be done. Best of luck w/the project! You've got your work cut out for you.
Posted by The Tarheel Forum | April 4, 2008 7:29 PM
Posted on April 4, 2008 19:29
I found your robot crawling my site and had to stop by for a look as I wasn't sure what it was. Best of luck with your project and I hope that your search can take a chunk out of the domination that's currently taking place in the search market.
Posted by Age Related Macular Degeneration Support | April 21, 2008 7:05 PM
Posted on April 21, 2008 19:05
A page for every place on earth?
I think you're on to something there! ;)
Posted by Duncan Farnworth | April 22, 2008 8:09 AM
Posted on April 22, 2008 08:09
Good Luck Rich! You have done wonders in the past but this is a pretty big bite you're trying to take. My advice: If you are going to go head to head with Google the sure fire way to beat them is in customer communication...if you can pull that off you can surely make it.
Posted by d360 | May 6, 2008 8:32 PM
Posted on May 6, 2008 20:32
I found this after your robot came to my site. Now, the question for me is why should I let you use my bandwidth to develop your search engine?
Since I don't expect you to send any real traffic my way the only thing I have to consider is the hope that you might be better than Google. But being better doesn't just mean sending me more hits. There is something bigger. Google, Yahoo, and the rest operate in a completely amoral, greedy fasion. Yes, they should make as much money as they can -- BUT, that still has to come second to right and wrong. They have no problem helping authoritarian goverments (e.g., China) imprison people and the U.S. government spy on it's own citizens as it marches towards authoritarianism. Google in particular has demonstrated a complete disdain for people's right to privacy.
Even when money isn't the motivation, too many people pray at the alter of computer techno-cool and think that just because they don't care about their own privacy they have a right to violate that of others.
Until I see a clear statement that Beckko will be different in this regard I don't have any incentive to allow it access to my site. I'll keep checking back, and if I see such a statement, then things will change.
Posted by no one | June 16, 2008 1:20 PM
Posted on June 16, 2008 13:20
Good luck ..
i seen your webcrawler info in my website than i follow it :)) now i understand you want to make a big big search engine .. :P
good luck again ..
Posted by nakliyat | June 16, 2008 4:07 PM
Posted on June 16, 2008 16:07
Rich - came here via mention on RWW. Interesting quest, best of luck!
So is search the only way you're going to go about this, or dynamic clustering of results, navigation, etc.?
We're currently finishing up our next research piece, the Market IQ on Findability - and in the "timing is everything" mode, would be curious on your feedback on my post earlier today.
See:
http://www.biztechtalk.com/2008/06/re-enterprise-s.html
Mostly talking about the enterprise, but when trying to find something (content, people, data, etc.), search is only one method of many.
Posted by Dan Keldsen | June 18, 2008 8:38 PM
Posted on June 18, 2008 20:38
Hi Rich, best of luck to you in your pursuit. Google is a worthy opponent to tackle, sizewise. While it's true they are operating in yesterday's paradigm, the fact that they keep adding satellite operations (free blog bandwidth, video, etc.) give them an anaconda grip on the whole web.
What's interesting to me is that a cult of "Google haters" hasn't developed, as happened with Microsoft as they began to completely dominate the OS world. Perhaps because there's no price tag attached for the next forced upgrade? Food for thought...
Posted by Kelly Smith | July 27, 2008 6:30 AM
Posted on July 27, 2008 06:30
Hi Rich,
Love the 'Coke vs Pepsi'. I completely agree. Though Google does has deep pocket companies like Microsoft angling to land the next big buzz in search. Much of the non-tech oriented population would probably be surprised to learn that there are other companies in the search business with the 'possible' exception of Yahoo.
Good luck with the new endeavor. If you need some legwork(research,etc..) contact me. I am a db(sql) guy with asp.net/c# background and enjoy massive undertakings.
Posted by J. P. Piesco | September 2, 2008 1:11 PM
Posted on September 2, 2008 13:11
Saw that you indexed some of my site recently! Good luck and I'll be sure to check it out when your online.
Posted by Omnicognic | September 11, 2008 1:49 AM
Posted on September 11, 2008 01:49
First of all, congrats Rich!
I also came here by fallowing your crawler..
It looks to me that you are in the right place the right time..
In my personal opinion, Google HAS been doing a lot of development, but in many different markets..
Check out site,blogs,maps,apps,mail,android etc etc etc..
I feel that Google has been doing lately what most of the small companies do in there first steps.. "They just do almost about anything!"
And this is something that i have been thinking for a very long time..
>
One think though..
If you check out some SEO Masters forums and blogs you will see that everyone is talking about some new "era" of google's algorithms..
Is it true? Is it coming ?? Are they tiered of there job ? :P
Anyway..
Good luck and all the best!
Posted by pikos | November 9, 2008 5:37 AM
Posted on November 9, 2008 05:37
Just seen Scoutjet Indexing one of my sites, Good Luck with Blecko, Hope the David and Goliath thing works out :-)
Posted by Sean | February 20, 2009 6:26 AM
Posted on February 20, 2009 06:26
Good Luck Rich! I have just translater your article "About ScoutJet web crawler" to Chinese ant put it on my website, I hope more and more people to see your growing scoutjet crawler, and hope you can beat google.
wait for your good news, hope you can make it.
Posted by yaohaibin | February 23, 2009 7:17 PM
Posted on February 23, 2009 19:17