That said it's brand new and we all want to kick the tires.

Search engines are built out of a lot of layered systems. One part can be working great but be subverted by another part that has a gap. Like any product there are always bugs to be fixed and improvements to be made. So launch day isn't the final word on relevance. But it's interesting to survey a variety of results to poke around.

• Overall the navigational results seem very strong.

• Bing is doing aggressive title rewriting to boost perceived relevance. Google has done some of this for a while - note the title change on the same url based on the query - [skrentablog] vs. [rich skrenta].

  The "Skrenta, Rich" title came from dmoz.

  Bing is going farther. Sometimes it makes the result look better than Goog's, e.g. [san carlos art and wine fair]. But others are odd, like result #3 for [mike arrington]. That funny-looking title looks like it came from anchortext.

• Bing's indexing of *.blogspot.com seems really limited. For instance [radish king] doesn't turn up radishking.blogspot.com. Site:blogspot.com on bing returns an estimate of just 560k results. Compared to Google (340m) and Yahoo (230m), Bing's blogspot index seems tiny. Other blogspot sites I've gone looking for are missing too. I wonder if this is some kind of rank or index penalty given the large amount of blogspot spam, or if there is some other issue with their crawl.

• [michael arrington] vs. [mike arrington]. TechCrunch is #2 for Michael Arrington, but is way down at the bottom of the page for Mike Arrington. This seems to be the fault of the section-ized results; it's under a heading called "Mike Arrington Blog". As others have noted I'm not a big fan of sections or universal search style sections on result pages. It's unfortunate to see a strong result for the query get pushed that far down.

• Bing, like Google, returns Dogpile and AltaVista for [search engine]. (Yahoo looks like they manually pinned a couple of results for this query.)

Overall the few bugs I've seen are relatively minor issues in the scheme of the entire product and I'm sure will eventually be addressed by the Bing engineers. It's so cool to have a powerful new engine out with interesting results. Kudos, Microsoft!

]]> http://www.skrenta.com/2009/06/bingram_betahoo_poking_at_a_fe.html http://www.skrenta.com/2009/06/bingram_betahoo_poking_at_a_fe.html main Mon, 01 Jun 2009 16:17:03 -0800 Topix becomes the #1 online property for Gannett, Tribune and McClatchy.

Congrats to the Topix team on the fantastic recent site growth!

]]> http://www.skrenta.com/2009/04/topix_passes_usa_today_to_beco.html http://www.skrenta.com/2009/04/topix_passes_usa_today_to_beco.html main Tue, 21 Apr 2009 08:54:08 -0800

When you have several hundred servers in a cluster, knowing the state and health of all of them can be a challenge. Traditional pager alert systems can often either log too many events, which makes people tune them out, or they miss non-fatal but still serious server sickness, such as degraded disk/cpu/network performance or subtle application errors.

This becomes especially true when the cluster and application are designed for high availability. If the application is doing its best to hide server failures from the user, it's often not apparent when a serious problem is developing until the site fails in a more public or obvious way.

We called these "analog failures" at Topix. There was a fairly complicated chain of processing for incoming stories that had been crawled. Crawl, categorize, cluster, dedup, roboedit, push to front ends, and push to incremental search system. Once an engineer mistakenly deleted half of the sources from our crawl, and it took us a disturbingly long time to notice. The problem was that, while overall we had half as many stories on the site, most pages still had new stories coming in, so we didn't notice that anything was wrong.

Sometimes a server has a messed up failure, like its networking card starts losing 50% of its packets, but stuff is still getting through. Or a drive is in the process of failing, and its read/write rate is 10% of normal, but it hasn't failed enough to be removed from service yet. The cpu overheated and is running at a fraction of its normal speed. There seem to be limitless numbers of unusual ways that servers can fail.

At blekko, there are dozens of stats we'd ideally like to track per host:

• How full are each of the disks?
• Are there any SMART errors being reported from the drives?
• Are we getting read or write errors?
• What is the read/write throughput rate? Sometimes failures degrade the rate substantially, but the disk continues to function
• What is the current disk read latency?
• Is packet loss occurring to the node?
• What is the read/write network throughput?
• What is the cpu load?
• How much memory is in use?
• How much swap is being use?
• How big is the kernel's dirty page cache?
• What are the internal/external temperature sensors reading?
• How many live filesystems are on the host vs. dead disks?

Others stats pertain to our cluster datastore:

• How many buckets are on each host?
• Is the host above or below goal for its number of buckets?
• What is the outbound write lag from the host?
• What is the maximum seek depth for a given path/bucket?
• Do we have three copies of every bucket (R3)?
• If we're not at R3, how many bucket copies are occurring?
• For running mapjobs, what is their ETA + read/write/error rate?
• Are the ram caches fully loaded?
• Are we crawling/indexing, what is the rate compared with historical?

The first step is to start putting the stats you want to be able to see into a big status table. But at 175 hosts, the table is kind of long, and it's hard to spot developing problems in the middle of the table.

So we have been experimenting with mapping system stats onto different visualizations, so we can tell at a glance the overall state of hundreds of servers, and spot minor problems before they grow.

A table with 175 rows is pretty long, but you can fit 175 squares into a very small picture. This table shows overall disk usage by host. The color of the tile shows the disk usage: red is 90%, orange is 80%, yellow is 70%, blue is below 60%. Dead filesystems on the node are represented by grey bars inside the tile. The whole grid is sorted worst-to-best, so it's easy to see the fraction of hosts at a given level of usage.

Our datastore uses a series of buckets (4096 in our current map) to spread the data across the servers. Each bucket is stored three times. If we have three copies of every bucket, we're at "R3". This is the standard healthy state of the system.

Because fetch/store operations will route around failures, it's not at all apparent from the view of the application if some buckets do not have three copies, and the cluster is degraded. So we have a grid of the buckets in our system, color coded to show whether there are 0/1/2/3 copies of the bucket.

In the above picture, the set of buckets in red have only 1 copy. The yellow buckets have 2 copies, and the green have three. We have a big monitor with this display in our office, if it ever shows anything but a big green "3" folks notice and can investigate.

For variety we've experimented with other ways to show data. This display is showing the fraction of a path in our datastore which has been loaded into the ram cache. Ram cache misses will fall back to disk, so it's not necessarily apparent to the user if the ram cache isn't loaded or working. But the disk fetch is much slower than the ram cache, so it's good to know if some machines have crashed and the ram cache isn't at 100%.

Other parts of the display are standard graphs for data aggregated across all of the servers. These are super useful to spot overall load issues.

We're still experimenting with finding the best data to collect and show. But the ambient displays so far are a big win. Obvious issues are immediately visible to everyone in our offfice. And people will walk by and look at the deeper graphs and sometimes spot issues. Taking the data from being something where you would have to proactively type a cli command or click around on some web forms, to displays that engineers will stop and look at for a few minutes on their way to/from getting a coffee or soda has been big improvement in our awareness and response to cluster issues. ]]> http://www.skrenta.com/2009/04/blekkos_ambient_cluster_health.html http://www.skrenta.com/2009/04/blekkos_ambient_cluster_health.html main Thu, 09 Apr 2009 11:39:49 -0800 ]]> http://www.skrenta.com/2009/04/bryn_turned_me_into_a_muppet.html http://www.skrenta.com/2009/04/bryn_turned_me_into_a_muppet.html main Wed, 08 Apr 2009 11:51:51 -0800 That's what the print newspaper is, of course. Why on earth would you print all that stuff out? Over a hundred pages, most of which you're not going to read, with the crease down the middle of the front page photo, story jumps everywhere, a carbon-footprint disaster to produce, distribute and recycle. It's absurd.

Back in 1980 newspapers were the main way that bytes flowed into people's homes. Radio and TV for audio/video, but the newspaper delivered the bytes that were read like the text-based web.

I once worked out some rough back-of-napkin estimates on the number of text bytes in the paper. It was only delivered once during the day, but if you average the bytes across the entire 24 hour period it came out to be about the rate of a 300 baud modem. The newspaper was the internet.

It was mostly one way - except for all those classified ads and the letters to the editor. It was really a lot more like AOL, since it was centrally controlled and edited.

But it did represent the sole text byte pipe into the home. And so it contained every content vertical, all in one package. National news, world news, local community sections. Little league scores and the NFL. Weather, stock tables, TV listings, home sales. Advertising, both national, local and personal. Games and political commentary and the police blotter. Everything.

Fortified by the high cost of the printing press and the limited radius of delivery trucks there was a natural local monopoly to these things. And indeed, they were a wonderful business, a so-called license to print money. Huge fortunes were made.

That's all over now of course. The subsidy that classifieds supplied for bureaus in distant cities is gone. The class of professional reporters as we know them is going to be smaller and funded differently.

I was at the TechCrunch office welcoming party last night, and was struck by how unassuming the offices were. This was the big move up, of course. They were still unpacking after moving out of Mike Arrington's house. But it was a small office with a few desks scattered around, a handful of computers. I've toured the massive AP newsroom, rebuilt in 2004 to cater to every desire of a journalist. The Reuters newsroom had pods that look like they were inspired by Norad in Wargames, with circular banks of monitors around central stations, all showing live feeds or charts from various sources. The old Mercury News offices were vast.

TechCrunch was a modest affair by comparison. So this is where it all happens..., I thought. This is what the modern business press looks like now.

Get used to it. ]]> http://www.skrenta.com/2009/03/the_news_medium_has_a_message.html http://www.skrenta.com/2009/03/the_news_medium_has_a_message.html main Sat, 14 Mar 2009 07:49:18 -0800 describes a paper about finding spam simply by inspecting the returned http headers:

In our proposed approach, the [crawler] only reads the response line and HTTP session headers ... then ... employs a classifier to evaluate the headers ... If the headers are classified as spam, the [crawler] closes the connection ... [and] ignores the [content] ... saving valuable bandwidth and storage.

We were able to detect 88.2% of the Web spam pages with a false positive rate of only 0.4% ... while only adding an average of 101 [microseconds] to each HTTP retrieval operation .... [and saving] an average of 15.4K of bandwidth and storage.

After running web crawls for the past year and finding all manner of spam, I have to say I'm skeptical this technique would really catch much spam on the actual web. Among the top 10 http header features they identify as spam-predictors are:

• Accept-Ranges: bytes
• Content-Type: text/html; charset=iso-8859-1
• Server: Fedora
• X-powered-by: php/4
• 64.225.154.135

These are pretty standard-looking headers. Let's look at some actual spam though and see if we can see anything funny.

$ curl -I http://www.fancieface.com/
HTTP/1.1 200 OK
Date: Sat, 22 Nov 2008 19:13:11 GMT
Server: Apache/1.3.26 (Unix) mod_ssl/2.8.12 OpenSSL/0.9.6b
Last-Modified: Tue, 21 Oct 2008 11:51:10 GMT
ETag: "2081cc-ba62-48fdc22e"
Accept-Ranges: bytes
Content-Length: 47714
Content-Type: text/html

Very spammy site, but totally vanilla heaaders. How about some rolex watch spam:

$ curl -I http://superjewelryguide.com/300.html
HTTP/1.1 200 OK
Date: Sat, 22 Nov 2008 17:48:26 GMT
Server: Apache
X-Powered-By: PHP/5.2.6
Content-Type: text/html

Again, pretty vanilla. Plus this technique isn't going to work at all for spam hosted within trusted domains. Here's some cialis spam smeared onto a my.nbc.com page:

$ curl -I http://my.nbc.com/blogs/GaryRobinson/main/2008/10/13/cialis-cheapest-cialis-pills-here
HTTP/1.1 200 OK
Server: Apache/2.2.0 (Unix) DAV/2 PHP/5.1.6
X-Powered-By: PHP/5.1.6
Wirt: (null)
Content-Type: text/html
Expires: Sat, 22 Nov 2008 19:16:33 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Sat, 22 Nov 2008 19:16:33 GMT
Content-Length: 0
Connection: keep-alive
Set-Cookie: pers_cookie_insert_nbc.com_app1_prod_80=1572983360.20480.0000;
        expires=Sat, 22-Nov-2008 23:16:33 GMT; path=/

but very fishy headers! :-)

It's incredibly difficult to get a high quality random sample of the web. You can't factor crawler strategy bias out of the sample, and any small sample is not necessarily going to very representative.

If the researchers did find good coverage with quirky headers and even individual ip addresses, I suspect that the crawl they're using may be over-weighted in pages from a few servers that spewed out a lot of urls/virtual hosts.

]]> http://www.skrenta.com/2008/11/detecting_spam_from_http_heade.html http://www.skrenta.com/2008/11/detecting_spam_from_http_heade.html main Sat, 22 Nov 2008 11:22:51 -0800 Unfortunately there was also a bmw 540 that I decided was too indulgent to buy and passed on. Afterward I kicked myself -- it would have been free. I would have exercised some netscape options I had to buy it. I held onto them, eventually they declined in value until they were worthless. I should have bought the car!

I saw a joke circulating at the time that beer would have yielded a better return than some stocks. The beer bottles could be returned for the 5 cent deposit, but stocks became worthless. Plus you would get to drink the beer.

Now we're going through it again, but even worse. The banker line now is that it's not the return on your capital that you should be worried about, it's the return of your capital.

I just got a state of California tax refund check. Normally it's ineffecient to pay too much withholding, essentially lending the government your money interest-free until tax time. In this case though it turned out to be a decent investment. :-| ]]> http://www.skrenta.com/2008/11/thank_heaven_for_tax_refunds.html http://www.skrenta.com/2008/11/thank_heaven_for_tax_refunds.html main Fri, 21 Nov 2008 11:15:31 -0800 We have an open office. So when they enter, 11 people immediately look up at them. This can apparently be somewhat intimidating, based on their flummoxed reactions. They usually ask for a business card so they can call us later. I sometimes offer them mine, since my card doesn't have a phone number on it. Then they beat a hasty retreat.

Lately we've been trying a new tactic - not acking their presence when they come in. There's no receptionist (of course), and it's not clear who they should attempt to speak with. None of us really want to listen to their pitch or take their flier anyway, so playing the game of chicken with the other folks in the office sort of emerged as a default behavior. Who will be the first to crack at their nervousness, make eye contact, and thus become the dupe left holding the flier or handing out their business card?

I almost feel sorry for them. Almost! ]]> http://www.skrenta.com/2008/11/cold_call_cold_response.html http://www.skrenta.com/2008/11/cold_call_cold_response.html main Fri, 14 Nov 2008 19:45:54 -0800

It's hard being a campaign worker.
We're completely at the mercy of our candidate.
We do all the work, and the candidate gets all the credit.
We ring doorbells, and make the posters, and build up the candidate's image.
And then he says something stupid, and ruins everything we've done.

The next time I do any campaigning, it's gonna to be for myself!

      -- Lucy, You're (not) elected, Charlie Brown

But we've been here before. Check out these WWI gov't posters.

"Don't waste paper - a pound of paper wasted is a pound of fuel wasted"

"Keep the home garden going"

Check out all the detailed instructions in that one. Public education indeed.

More posters... ]]> http://www.skrenta.com/2008/10/retro_conservation_advertising.html http://www.skrenta.com/2008/10/retro_conservation_advertising.html main Wed, 29 Oct 2008 03:40:00 -0800 blogging is dead it must be time to start Skrentablog up again. Apologies for letting the blog go dormant the last little while, I've had my head down in technology. Quick update: 200 servers, 11 employees, lots of code. Crawl, index, test, repeat.

We hired a naming firm to come up with a better name than 'blekko', they did a great job. Down to two candidates. Testing them.

We built a wicked cluster platform to run our stuff. It's kind of like bigtable from the top-down api view but is an integrated design, vs. the layered impedance mismatches with stuff like gfs/chubby. No masters, all swarm algos. We crawl/index/serve into structured storage. It's very fast, has integrated mapjobs, and is really easy to program on top of. I'll post more details about it in the future.

More posts to come, I promise. ]]> http://www.skrenta.com/2008/10/whats_up_rich.html http://www.skrenta.com/2008/10/whats_up_rich.html main Thu, 23 Oct 2008 08:43:42 -0800 Search is an absolutely fascinating problem to work on for a bunch of reasons. For one thing you have to scale the thing before getting the first user. You can't just start with a server or two and add more when the users come. Step 1 is to copy the internet onto your cluster. Step 2 is to analyze it..

The componentry is remarkably deep.

Search is like 7 hard problems wrapped into a stack. Distributed systems, html analytics, text analytics/semantics, anti-spam, AI/ML, frontend/UI. And scale... Apart from the sexy high end algos there are also the boring 10-year old system libraries and off-the-shelf tools that crack under stress and sometimes need a look. You open the hood and wonder how the thing ever worked in the first place...

Plus there is always something fresh and new every day mining through the vast sordidness of the many billions of pages on the web. You expect to be amazed at the endless varieties of crazy porn domains and new approaches to webspam. But there are equal horrors in the small, finding pathological charset issues, previously-undiscovered abominable server implementations, psychopathic website owners. The web is a reactive fuzz test.

I know there are some great coders out there reading this blog who would have blast working on some of the pieces here that need to get built. This is a great opportunity to join an experienced team early building a big system from the ground up. If you think you might be interested, send me an email and we can chat.

fyi our interviews always have coding tests. Primarily we are looking for folks who love to write code and are good at it. :) ]]> http://www.skrenta.com/2008/05/blekko_is_hiring.html http://www.skrenta.com/2008/05/blekko_is_hiring.html main Thu, 01 May 2008 12:11:09 -0800 The legend says that once upon a time there was a General Motors. This General Motors, GM for short, had a car and a brand for every need, along the plan developed by the great Alfred Sloan prior to the Second World War. There were Chevrolets for regular folk, Pontiacs for the cautious old people (and, thanks to John Z. Delorean's development of the 1964 GTO, for angry young people as well), Buicks and Oldsmobiles for doctors and successful businessmen, and Cadillacs at the very top, for the most successful men in the land.
...
It would have stayed that way forever, but one day a mysterious yet important man at GM had a mysterious yet important idea: Executives should drive cars from their own division!

Which leads to every division of GM building their own version of the Cadillac.

Read more: How Fake Luxury Conquered The World

(thanks Bryn for the tip) ]]> http://www.skrenta.com/2008/05/how_fake_luxury_conquered_the.html http://www.skrenta.com/2008/05/how_fake_luxury_conquered_the.html main Thu, 01 May 2008 11:19:55 -0800 Wikipedia page about zero. Zero has a rich and interesting history and there are many other potentially reasonable results.

But I was surprised to see MSN search had demoted their good results below some crappy ones from MSDN:

Lame! Falling into an inferior lex position and a lower overall relevance page to boost their own network results...give em credit for being old school. :)

...

I found my bug on Yahoo Search. I had tried a lot of smaller engines first because I didn't think a major would have this bug. You can't search for 0 on Yahoo. You can search for all the other numbers, but not 0 ...

Why?.. Because 0 is false. It suggests Yahoo is using a scripting language to front their search form, and a programmer did something like if ( $query ) rather than if ( $query ne '' ).

]]> http://www.skrenta.com/2008/04/microsoft_bias_in_msn_search_r.html http://www.skrenta.com/2008/04/microsoft_bias_in_msn_search_r.html main Thu, 24 Apr 2008 07:45:00 -0800

Hypertable is an open source, high performance, distributed database modeled after Google's Bigtable. It differs from traditional relational database technology in that the emphasis is on scalability as opposed to transaction support and table joining. Tables in Hypertable are sorted by a single primary key. However, tables can smoothly and cost-effectively scale to petabytes in size by leveraging a large cluster of commodity hardware. Hypertable is designed to run on top of an existing distributed file system such as the Hadoop DFS, GLusterFS, or the Kosmos File System (KFS). One of the top design objectives for this project has been optimum performance. To that end, the system is written almost entirely in C++, which differentiates it from other Bigtable-like efforts, such as HBase. We expect Hypertable to replace MySQL for much of Web 2.0 backend technology. In this presentation, Doug will give an architectural overview of Hypertable. He will describe some of the key design decisions and will highlight some of the places where Hypertable diverges from the system described in the Bigtable paper.

More details. ]]> http://www.skrenta.com/2008/04/hypertable_architecture_talk_w.html http://www.skrenta.com/2008/04/hypertable_architecture_talk_w.html main Tue, 22 Apr 2008 12:51:56 -0800