Money Changes Everything (A Parable of Google)
Before Google, if you wanted to find something on the Internet you most likely looked on Yahoo. If not Yahoo, then you probably went to one of the AltaVista-like search engines, but by the time Google hit the scene all the search engines were so cluttered with advertising as to be nearly useless.
What made Yahoo useful was not that it was complete (it wasn't), but that for many topics searching the Yahoo directory was likely to give you highly relevant web pages. Part of this relevance was due to the fact that Yahoo was compiled by people, not machines, and people are very good at judging whether other people will find a web site relevant. Then, as now, advertiser-supported web sites could earn more money if they had more visitors, so a listing in Yahoo was valuable.
Unfortunately, using people to compile a web directory doesn't scale well. You can only add to the directory as fast as your staff can review and classify web sites, and obscure searches are likely to come up empty-handed.
Then Google came along and changed everything.
Google's initial brilliance was discovering that it was possible to create a search algorithm which could provide relevant results for almost any search term without using human editors. Even better, Google didn't try to clutter its search page with blinking banner ads and pitches to use other Google services, a practice which every other search engine engaged in enthusiastically.
Internet users switched to Google in droves, abandoning not just other search engines, but Yahoo's human-maintained index as well. Google's Page Rank algorithm wasn't perfect, but it certainly was pretty darn good, especially when nobody was trying to artificially increase their Page Rank.
But where there's Internet traffic, there's money, and when the bulk of Internet searches shifted to Google there was an economic incentive to outwit Page Rank.
"Search Engine Optimization" existed before Google, but the popularity of Google pushed SEO into a whole new league. It was no longer sufficient to merely include the desired search terms in the text of a web page. Techniques like link farms were invented to artificially drive up a site's Page Rank to ensure it would be at the top of Google search results. This would lead to substantially higher revenue through ad and merchandise sales. Google responded by continually tweaking the Page Rank algorithm to counter tactics used to raise Page Rank.
But the real turn to the bizarre came with the advent of Google's AdSense, which allowed almost any web site to display advertising from Google and get paid based on the results.
This led to a whole new beast: call it the "Google Trap." These web pages exist for the sole purpose of driving traffic to Google-sponsored advertisements, and get most of their visits through Google searches. What little content exists on the page is there just to fool Page Rank into providing a higher ranking: the page itself is often useless.
Had Google remained a small academic project which only a few people used, its original Page Rank algorithm would probably still be providing the best search results. But the irony is that Google, through its AdSense program, is providing people with the economic incentive to try to outwit Google's own Page Rank algorithm. And since people are smarter than computers, they will always find a way. This costs Google money, since it must continually update Page Rank to stay ahead; and there's also evidence that Google actively removes sites from its results which are blatantly trying to manipulate Page Rank - a presumably manual and labor-intensive process.
The lessons?
What works well on a small scale when no money is involved doesn't work on a large scale when people have an incentive to beat the system.
Success creates its own set of challenges.
You can never take people completely out of the equation.