« I'm shocked, shocked to hear about the secret Wikipedia cabal | Main | Multi-paned search UI in testing at Google »

PageRank wrecked the web

Two years later and rel=nofollow is still bugging folks.

Google needs YOUR help

It's still bugging me, too. It doesn't make any sense.

Bad linking hurts everybody

Google couldn't seriously be asking webmasters to tag which of their links were going to affect pagerank vs. the ones they'd sold. Could they?

Let's pick up all the trash in the world

That would be like asking everyone in the world to please be nice so the old algorithm will still work.

Do a random act of internet kindness

If I close my eyes and wish really hard I can bring back the golden age of the 1999 web. Back when links still indicated site quality.

THINK before you LINK

Back when spam was simpler, and G wasn't party to both sides of the transaction.

PageRank stands for PR

The toolbar pagerank display is disconnected from the real topic-sensitive pagerank used in the SERPS. Google can cut your PR in half but your SERPs don't change. It's a message, but what does it mean?

NOFOLLOW if you're PAID, or PAY the cost

Why would they want us to think that these things mattered?

If you don't toe the line, we'll ban you. You'll be sorry.

We can't actually ban the Washington Post or the Stanford Daily though. But we're going to threaten you to make you shape up.

Don't say untrue things about people

On one hand it seems an oddly utopian world view, not a pragmatic one.

Help Google by only publishing quality links

What happened to all the genius researchers building Skynet with their 1 million servers? What's all the AI for if they can't do a better job of tagging web pages than asking users to do it for them?

You mean they can't even detect TextLinkAds on a page, and have to resort to this weird business threat model instead?

The web of spam

Links used to be for human navigation.

Google made them count for money and they're ruined now.

Nofollow isn't going to put it back the way it was.

PageRank wrecked the web

Google is the cause of all of this.
and Google is going down with it.

TrackBack

Listed below are links to weblogs that reference PageRank wrecked the web:

» Blekko vs. Google - I like it !@ from WebMetricsGuru
In The Next Google Search Challenger: Blekko Michael Arrington wrote about an issue that's close to my heart - Google has been the origin of  much of the Search Problems that it's been trying to solve, and penalizing other bloggers... [Read More]

» Why Search? from Skrentablog
I've gone and founded a search startup... you can read it about it in this write-up in TechCrunch. But I get asked - why do search? Simple - the idea that the current state-of-the-art in search is what we'll all... [Read More]

» En attendant qu'une bonne idée jaillisse... from blog.jeanlancon.com
Supplanter le moteur de recherche de Google avec un capital de départ de seulement 2 millions de dollars ? C'est le défi qu'entend relever Blekko (source : Techcrunch ). Je me garderai bien de crier au fou ! ou de [Read More]

» How bad is Page Rank? Can someone compete with Google on Web Search? from The Googlization of Everything
Rich Skrenta, co-founder of news site Topix and the Open Directory Project, writes that PageRank wrecked the web. I am not sure I follow his argument. But it's interesting. Over at Smartmobs, Marius Chitosca writes about Skrenta's new project: a... [Read More]

Comments (26)

Anon:

The word on the street is that your new startup is search. I'm guessing from this post that the rumor is true.

If everything I heard is true, your company needs a new name though! :-)

Can't wait to see what you're cooking up !!

Rich, I truly believe any successful system (be it eBay, Amazon, Usenet, Wikipedia, DMOZ, or government spending) will attract people who try to optimize for that system or even game it. When Google came onto the scene with its new way of ranking search results in 1999/2000, it was inevitable that people would try to optimize for Google and link-based reputation.

Tools like rel=nofollow give site owners a method to decide whether to flow PageRank at a link-level of granularity. For a very good example of why site owners might want that link-level ability, see my "oompa loompa dating site" example at http://www.mattcutts.com/blog/quick-comment-on-nofollow/

Of course, there have always been ways to create links that search engines wouldn't follow (e.g. make the link go through a url redirector that is robot.txt'ed out).

> You mean they can't even detect TextLinkAds on a page, and have to resort to this weird business threat model instead?

Isn't it more of a two-pronged attack-- some algorithms plus user input? How is this different than incorporating reddit or digg style +1/-1 buttons on every link?

A feedback loop on clickthrough ought to solve most of this. Surely Google is measuring search engine result page clickthrough satisfication at some level, even without the +1/-1 buttons?

ctr/time-on-site data is tantalizing. It seems like everything you'd want to know about relevance would be in there. But extracting the signal is tough. There tends to be too much noise (think spam sites that sucker enough non-sophisticated users in to get usage, fake usage via javascript/spyware tricks, etc). Not to mention the feedback issues. If this stuff worked, DirectHit would have been a success back in 1998, pre google pagerank.

You have massive sites like Wikipidea doing nofollow, even though their outbound links are generally high quality. Blog platforms will default to nofollow regardless of the quality of the individual blog's posting community or the level of moderation on the site.

Nofollow changes a subfraction of the pagerank graph in some quasi-random way. Some good links are zapped, some bad ones are caught... But the net effect on SERPS - does anyone seriously believe this is increasing overall rank quality?

If not, why is G pushing this stuff?

"Isn't it more of a two-pronged attack-- some algorithms plus user input?"

Jeff Atwood, we definitely use both of those prongs. For example, Google has accepted spam reports from the public since November 2001. But of course that doesn't mean that Google only can tackle spam based on user reports. Instead, the majority of our efforts are based on algorithms and other robust, scalable ways to tackle webspam. As a result, it's pretty rare to run across off-topic porn or similar webspam in Google these days. But as spammers try to find new attacks (e.g. malware on cheap .cn domains), Google tries to respond and make sure that our algorithms adapt as well.

Rich, is it true that you're working on a startup in the search space? If so, that's very cool. :)

Jake:

What honestly bugs me is when sites start nofollowing completely legitimate links. Wikipedia has the excuse of being user edited. However, so many major news sites are reluctant to give out links. They use some weird JavaScript redirection, or even don't make hyperlinks (thanks Linkification!). It's insane.

Al:

The idea behind the nofollow was pointless, not enough people were ever going to back it (in the scope of the internet) to make it worth while. I agree that Google and co should be able to algorithmically determine whatever they want on their own, without the help of web site owners.

However, what I'm really seeing is that everyone is pissed with Google, are pissed because they got into the pie with a much bigger knife before the next guy. They now have the control that everyone else could only dream of having and now everyone is pissed off.

Correct me if I'm wrong but there isn't a problem with them doing anything they want; last time I checked they were their own company. If you or anyone else on the internet choose to require their business to make your business work then you need to learn to play by their rules, no?

KP:

I just created my first a website without taking care of the seo stuff. It was such a relief. Okay, it might gonna be tougher to get the site spreaded, but hey, who wants to do the crawl in a sea of cheese anyway?

I would really like to add a "made for humans" or "100% seo free" button now. Does something like that exist?

Random fact: "Google wants YOUR help" returns 2330 results. Of course, "Google wants to help YOU" returns 14400...

Pagerank didn't wreck it, it was that little toolbar thingy that did :) Seriously, link analysis is just too valuable to ignore, and even though it seems like the artificial link spammers are ruining things, they are potentially doing just the opposite. They are helping. Link spammers find holes and exploit them, and while I get mad at them and want them to stop, they do have the benefit of forcing the engines do a better job of bush-hogging the ever-growing amount of linkjunk on the web, and by trusting a shrinking number of truly authoritative sources, Google ends up producing better results. Pagerank was inevitable, as were the many flavors of link spam it spawned. The evolving algo will spot link spam just as easily as they used to spot hidden keywords. They already are.


I have to agree with Rich's comment above...

99% of website owners don't know HTML well (note that I say site owners, and not web developers or SEO's or the like. I'm referring to the actual owners of the site). They either had their neighbor's kid design a website for them, or hired the local web guy, or somehow scraped together a little bit of know-how and created a basic website themselves. These people do not know the ins and outs of good site architecture (neither did I until recently). And they certainly don't know what the heck rel="nofollow" means. But, they do know that they need links. So out they go, to assemble a links page. And they ask their plumber, their neighbor, their grandma, and anyone else to link to them in exchange for a link on their link page.

Then, there's 1% of site owners who are savvy and out there building 1000+ websites a day out there to game the Google system. They are attacking the several of the high paying keywords, and the Google results are a bit saturated. As a result, we get the PR hit of 2007, and the nofollow.

Still, 99% of the website owners out there don't know what nofollow is, much less basic HTML. And Google is asking for the savvy 1% to start using nofollow, so they please would not affect their rankings. Will they do it?

Eventually, programmers will catch wind of this, and blog networks will catch wind of this, and will want to design to the Google standards, and overnight every Wordpress and Blogger site will have rel="nofollow" defaulted in all their link codes. The masses of users will know no difference. And everyone will have nofollows behind every link, unknowingly.

What happens when nobody is getting credit for being linked to? How will Google's algorithm still be valid?

I was at Pubcon, and listened to all of Matt's comments on nofollow and paid links. But, I'm still confused as to when a site is to use nofollow or not. This confusion is unfair to site owners, and questions why Google is putting the onus on site owners to correct their SERPs.

rel=nofollow was not created to manage PageRank, but to deal with a problem of link spam in open collaborative websites.

It was designed to allow web masters freedom from policing their collaborative sites, while at the same time leaving the sites open to edit by the world. This was necessary for wikis and blogs to ensure a high level of interaction without editor intervention to approve a change or post.

That not enough sites have implemented rel=nofollow and do not adequately police their sites is a problem that has been exploited by malicious web sites for the last 2-3 months so that they obtained a high PageRank until Google responded and removed the links to the malicious pages.

Even you must do policing on new posters as you do no appear to use rel=nofollow

Rich-

You're right on PR and Google's FUD are wrecking the web. Webmasters are scared into not linking out to *other* websites in fear Google will drop them or their rankings. I miss the days when linking out was a good thing!

Google needs to pull the plug on the toolbar.... this would be a good first step.

Stuart

Is this a rule that will be applied fairly across all properties, or will larger sites be provided with more flexibility. In other words, has Yahoo!! been torched for effectively selling links from its directory? What about DMOZ, where I can still get a submission by paying an editor $300 on the side? Seems to me that if you're going to try to level the playing field, do it right. This looks like an approach of least resistance/cost rather than a real investment in building a better relevancy algorithm thats not based on link architecture alone.

Andy Denton, you break webmasters into two groups, which I would define as Black Hats (1%) and clueless. Maybe a bit unfair, but close.

First, a lot of less experienced site owners will learn about rel="nofollow" when they begin selling links. Second, I respectfully suggest there's a third group in there somewhere, and most of those commenting at sites like this belong to that third group.

"I'm still confused as to when a site is to use nofollow or not. This confusion is unfair to site owners"

Couldn't agree more. Not only has it been a moving target, but it's overlap with Robots.txt techniques for adjusting link flow within a site is confusing. But I expect it will be cleared up. Google has become incredibly responsive. And a lot of the FUD seems to me to stem more from people reactions than from what Google is putting out there.

Rich Skrenta, to me, rel="nofollow" really seems a tool for (1) some webmasters selling links to improve their rank in the SERPs, for others (2) to adjust link flow within their site, and for others (3) to discourage spammers from posting on their sites. And, oh yes, (4) it helps Google prevent the SERPs from being sorted by money.

1-3 look pretty good to me. Discouraging spammers, like Wikipedia did, is always a two-edged sword, regardless of the tools used.

I'm not sure webmasters can help Google that much in (4), but users still click links to go where they want, right? Happens at Wikipedia all the time. So what if links are unclear "signals" for Google? No one thinks rel="nofollow" is going to solve that problem, that's the mission of the Google algorithm.

Tinker:

Google Lost in Jungle of "World Wild Links"
======================================

Search Reality : Google MFA sites are thoroughly cluttering the Internet with poor quality information and google is the "problem maker" than solution provider.

Research Reality : 6 figure salary and stomach full of free food does not create innovation much less artificial intelligence. It is pretty good idea for "buying off" prospective competitors.

AdNonsense Reality : Most of the adnonsense links are blended into the publisher sites to look like "site navigation links" this will produce lots of accidental clicks. Accidental clicks don't convert! It will take another year or 2 for dumb advertisers to understand this reality.

Google is Dispensable : Today if google disappears still the WWW will work perfectly and websites will still attract visitors. Webmasters who generate quality content are truly "Masters of the Web" and no need to fear Matt cutts and his "hand rank" minions..... errr.... web quality team!

Andy Denton said: "overnight every Wordpress and Blogger site will have rel="nofollow" defaulted in all their link codes."

They already do - I've had to do some considerable legwork to disable nofollow in Movable Type, and you need a plugin for Wordpress.

Simple game theory...

1. Realize that determining true relevance via link analysis isn't quite up to snuff.
2. Release some information about how link relevance CAN be detected and tell people to change their behavior.
3. While people debate the IF and the WHY, quietly work on determining relevance with the external focus misdirected.
4. Roll out some incremental updates based on some spam reports and algo detection that focuses more on irrelvant link detection than on paid link detection.
5. Enjoy increased relevancy of organic listings and the higher revenue that ensues on the paid listings because of that higher relevance.

So what's the problem with this all? I like all the traffic types coming in; in order to get that traffic on a couple of sources I have to jump through a few hoops. Big deal. So long as the requirements cost less than the expected revenue from ranking, I'll meet the requirements. With search being dynamic as it is, the requirements will always change, and thus, so will the sites's tactics looking to be found.

Rick:

I screwed myself using the no follow. I have a network of sites - all of the sites are listed via a link on each site.

I finally added the 'no follow' so that google wouldn't ding me by thinking they were paid.

A week later all of the sites dropped down in pagerank.

Page Rank didn't wreck the Web. People that want to game Google's system and greed are currently undermining the systems that organize the Web. But the Web isn't "wrecked". I went to Google today at least three times and I found what I was looking for everytime. If you want to do well in Google, then adhere to their rules. If you don't, then do as you please. But Google doesn't owe anyone anything. It's their search engine that gives Webmasters traffic. The Webmasters didn't pay Google for their traffic. It was free. If you want to pay for the traffic via PPC, you don't have to use rel=nofollow, you can buy whatever links you want, etc and Google will still give you the traffic. If you want free traffic, you have to play by the rules that help them. Seems like a pretty easy game to play to me...lol. So they can rank people however they want. If users don't like it. They will go someone else. So far though, users generally seem to like Google. People have predicted the demise of Google for years and so far, they are only getting bigger.

So, if I promise to never use "nofollow" tags on my blog - Internet Hunger - again, will you give me a link from your blog? Positive reinforcement is the key, in my opinion. :)

Matt Cutts: "When Google came onto the scene with its new way of ranking search results in 1999/2000, it was inevitable that people would try to optimize for Google and link-based reputation."

Michael Martinez: Actually, Matt, as I have pointed out many times through the years, people were heavily involved in link manipulation for other search engines before anyone had ever heard of Google.

Link farms were developed solely to help sites stay in Inktomi's primary index (which was purged and rebuilt every month). Inktomi's "paid inclusion" was viewed by the Webmastering community as a scammy practice at the time, much like Google's "use rel='nofollow' and report paid links" program is viewed as scammy.

Matt Cutts: "Tools like rel=nofollow give site owners a method to decide whether to flow PageRank at a link-level of granularity."

Michael Martinez: But it's not Webmasters' responsibility to determine for search engines where the PageRank should flow. You guys need to stop being lazy and just do that for yourselves.

When Larry Page and Sergey Brin naively claimed that their system couldn't be gamed, they were already behind the curve and had no clue as to what they were doing.

You guys still look like people at Google don't know what they are doing. You need to stand down on this issue and focus on RELEVANCE.

Let's see Google start promoting the MOST RELEVANT RESULTS to the top of its search results for a change.

In a free market the laws of economics always drive behavior. If page rank is the main criteria for elevation in organic search then every website owner will push and pull all the algorithmic levers she can to gain an advantage. This raises the question about internet search being a free market. It seems the keeper of the organic search algorithm can simply change its variables to preserve its paid search revenue growth.

geri:

I have read hundreds of posts on this subject, however, your perspective is refreshing... thanks... If I did Sphinns or Diggs this one would be used.

Brad Collins:

I completely agree. PageRank is based on academic citation counting -- a practice which has corrupted academia to its core.

It always struck me that the whole concept behind backrub was little more than a clever undergrad insight rather than an earth shattering revelation about the nature of the Web.

Folks,

Well, rel=nofollow / rel=noindex can be good! Not really sure if the bots actually respect them or not, but here is a real scenario in which I have used the above two. I have a website, (newjoblist do com), it has a back-end(robot), front-end(site) & a sync-er (updates the front-end-db from the back-end-db).

The site is just a vertical job search engine and yes, there are hundreds of them out there.

Now, I have three columns, left, center & right.
While all columns appear on all pages, the center one keeps being update with new job links.

So, if I find a job for the city of San Fransisco, then I tag it with San Fransisco, California, USA.



Now if you search for a job in San Fran, you'll see that job link. You would also see the job link if you searched for jobs in California or the United States.

I thought that this would probably cause the G,Y,M & C robots to see and tag this job as duplicate content as the title page changes based on the location of your search and is considered as a new page but the content remains the same.

So, I decided to tag a job link with rel="index" only at the city level. If city is empty, then tag at state level and if state is empty then tag at Country level.

And if there is a job in San Fran, if you search for the city of San Fran, the tag would be rel="index", and on any other pages it would be rel="noindex".

As you see I used rel="noindex" to avoid the "duplicate content" issue.

Now, I only keep a link to the job as I consider the description of the job to be the property of the content provider. But I don't want the G bot leave my site using this job link since I want it to index the rest of the links.

And, I keep tagging all the job links that redirect to the content provider's website as rel="nofollow".

Therefore, I would appreciate to have a say in whether a link should be followed or not, regardless of the PageRank issue/importance.

With all that said, Google and others make money selling you ads and making you paying for them. So, it doesn't matter whether you pay to post or post to get paid, they make money. Now, if the big guys rank properly, then who would pay them for ads? No one. Site owners would pay to make their content better to get better ranks. right? So, they (G,Y,M) keep changing the game (algorithm) and it doesn't matter what you do as a webmaster, you are always on the losing side and have to adapt. (You would think so!)

All & all, if rel="nofollow" is respected and works the way it should be, it would definitely be welcomed as it seems to be a solution for my problem.

What do you think?

Ciao,

Val
(http://newjoblist.com)

Post a comment

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)

About

This page contains a single entry from the blog posted on December 10, 2007 4:34 PM.

The previous post in this blog was I'm shocked, shocked to hear about the secret Wikipedia cabal.

The next post in this blog is Multi-paned search UI in testing at Google.

Many more can be found on the main index page or by looking through the archives.

Powered by
Movable Type 3.33