Thursday, July 15, 2010

Our op-ed: Regulating what is “best” in search?



Google’s Marissa Mayer wrote in the Financial Times today about the impact for consumers of governments potentially regulating search results. Because the article is behind the FT’s paywall, we thought we’d share the complete text here (also, check out search analyst Danny Sullivan’s take on this issue).

Do not neutralise the web’s endless search
By Marissa Mayer

Published: July 14 2010

Think about the word “jaguar” – what comes to mind? The animal? The car? A sports team? Now ask yourself: what is the best piece of literature ever written about jaguars? What about the best piece of literature ever written containing the word jaguar?

How do you define what is best? What characteristics and attributes should be taken into account? Which should not? There is a debate brewing, reported in the Financial Times this week, about whether standards are needed to ensure fairness – or what is “best” – in internet search results.

Search engines use algorithms and equations to produce order and organisation online where manual effort cannot. These algorithms embody rules that decide which information is “best”, and how to measure it. Clearly defining which of any product or service is best is subjective. Yet in our view, the notion of “search neutrality” threatens innovation, competition and, fundamentally,your ability as a user to improve how you find information.

When Google was launched in 1998, its fundamental innovation was the PageRank algorithm. It was a new and helpful tool in helping users decide which was the best information available – and one of many hundreds that have since been deployed by search engines to improve the ranking and relevance of their results.

Yet searching the web has never been more complex. Type “World Cup” into Google today and you will see millions of returns, ranging from recent news articles to images of players. Often the answer is not a web page: sports scores, news, pictures and tweets about matches are included. Such results stem from an upgrade in Google’s technology launched in 2007, which made it possible to include media such as maps, books, or videos on a results page. Our goal is to provide our users with the best and most effective answer. Consider the search “how to tie a bowtie”. Answers to these types of searches benefit from the inclusion of different media (diagrams, videos), sometimes from a Google service (books, maps).

To make matters more difficult, a quarter of all daily searches on Google have never been seen before. Each presents a new challenge, so our engineers need constantly to improve and update our algorithms. On average, we make one or two changes every day. But even then they sometimes require a more hands-on approach. For example, we occasionally have to flag malicious programmes manually, removing links to child pornography or spam sites.

The world of search has developed a system in which each search engine uses different algorithms, and with many search engines to choose from users elect to use the engine whose algorithm approximates to their personal notion of what is best for the task at hand. The proponents of “search neutrality” want to put an end to this system, introducing a new set of rules in which governments would regulate search results to ensure they are fair or neutral.

Here the practical challenges would be formidable. What is fair in terms of ordering? An alphabetical listing? Equally, new results will need to be incorporated – new web pages, but also new media types such as tweets or audio streams. Without competition and experimentation between companies, how could the rules keep up? There is no doubt that this will stifle the advance of the science around search engines.

Abuse would be a further problem. If search engines were forced to disclose their algorithms and not just the signals they use, or, worse, if they had to use a standardised algorithm, spammers would certainly use that knowledge to game the system, making the results suspect.

But the strongest arguments against rules for “neutral search” is that they would make the ranking of results on each search engine similar, creating a strong disincentive for each company to find new, innovative ways to seek out the best answers on an increasingly complex web. What if a better answer for your search, say, on the World Cup or “jaguar” were to appear on the web tomorrow? Also, what if a new technology were to be developed as powerful as PageRank that transforms the way search engines work? Neutrality forcing standardised results removes the potential for innovation and turns search into a commodity.

We know that Google plays an important role in accessing information. We also welcome scrutiny and want to ensure everyone understands how we work. Yet we believe the best answer for a particular search changes constantly. It changes because the web changes, because users’ expectations and tastes evolve and because the media never stay still. Yet proponents of search neutrality are effectively saying that they know what is “best” for you. We think consumers should be able to decide for themselves – with an array of internet search engines to choose from, each providing their very best.

The writer is vice-president of search product and user experience at Google.

29 comments:

Erik said...

Great post. A reason why Google is still #1!

TBB said...

@ Eric
Agreed...
I definitely do not want regulation,
If the government regulates search results, it's "Goodbye freedom, welcome China"... if you get what I mean...

倫惟倫惟 said...
This comment has been removed by a blog administrator.
James said...

I believe that some form of search neutrality can be achieved. For me, the problem is highlighted by Google's sometimes dominant share of the market, and the consistent handing over of the 'number one spot' to the same website, even for general terms. For years, in my own corner of the net being online 'games' the top has always gone to the biggest player which is Miniclip. So far as I can see Miniclip is the biggest player because it gets listed first. You get the point. I don't want to pick on Miniclip in particular who provide a good service (although other online games sites are growing much faster) but it serves to prove a point. I wonder how it is for people working in 'flights' or 'hotels' or 'loans'. My experience of being a little guy wanting to break through makes me wonder how "Yet we believe the best answer for a particular search changes constantly." can really be true on Google. Could not a random rotation be achieved, at least for general search terms? I worry that the status quo stifles innovation, which I know Google would hate to be a part of.

Garp said...

@James
miniclip gets to the top of the rankings in no small part because so many places link to them. That's a good measure of reliability as far as search engine algorithms are concerned, especially if those links are coming from other reputable sites. That's a fair way of measuring too.
If I search for online games I'm most likely to want to see a site that people most like, because there are more likely to be Games there I'll want to play.

Breaking through to the top in such a field would be very tricky, and to do so you need to think outside of the search engine. It's really not the search engines fault, all a search engine can ever be (and remain accurate/useful) is a reflection of the internet and how people perceive it. The same applies to Bing, Yahoo and any of the other search engines.

SEO is a means of gaming the search engine, but a lot of the tips that can be gleaned from it are important for advertising your service anyway. At the bare bones you need people talking about your service. Link farms are ugly things and I would not recommend using them in the slightest (liable to see you black listed once they're identified) but that they ever work is a clear indication that "your link in more places" is one way to push yourself up the ranks.

Consider getting in direct contact with sites that review or report on online games, elevator pitch your site to them, explaining why yours is anything better than the competition, and try to get some coverage. Consider other major blogs like TechCrunch. If you can get a link from a reputable source your PageRank will improve nicely, and a favourable review will work wonders.

Shell out some money for AdWords, too. Sponsored search results are popular for good reason. Good marketing is an investment, and it's essential if you're going to get the ball rolling for your site and drive up traffic. Even if you can only afford a day or two, just having your link right up there will see a surge in traffic.

In a dream world you want a killer app. Kongregate came up on my radar not through Google but through Desktop Tower Defense which I was playing a lot of at the time, as were a good number of people. When they released a new version on Kongregate I went and took a look, and since became a regular user of the site. I would bet they saw a huge surge in traffic when they sorted out that deal with DTD.

MANOJ TIRKEY said...

Why search should remain unregulated:

1. It is a private service and not a public good. So the yard stick should be different here as far as regulations are concerned.

2. It is a FREE service, so if anybody is not happy with the results, s(he) may shift to any other service without the need to break any contract or anything of the sort. So the yard stick should be different here as far as "search neutrality" is concerned.

3. Google's search service isn't a strangle-hold in the same way as an operating system installed on a computer. Another search service is just a mouse click away. So people should stop harassing Google with frivolous, one-sided, agenda-driven, and verbose arguments.

Kristenh said...

@james and the miniclip argument.

Sorry but that is so selfish, you are saying this form of censorship and control should go to somebody like the US government because of your personal problem with not being #1?...

Work harder, don't complain about being the little guy. It's up to you to make moves within the market. Produce your own innovative game, build things people WAN'T to link to so you can work on your SEO, do things others don't. Don't just complain that you can't beat the big guy.

I've taken the place i work from below page 200 to middle of the first page for a MONSTER term within 5 months. Competing with various huge players, some backed by companies like HP.

My point is that sitting back and complaining won't get you anywhere, nor will changing the rules. Politicians love to change the rules whenever something happens.. one person dies in a certain type of accident, so all of a sudden there is a law banning whatever... its hopeless and builds a nanny state.

Amber said...

I'm mind boggled that anyone would ever believe that having the government regulate search results would remotely constitute equality or a better user experience.

Needless to say, that opening the door to goverment regulated search result algorithyms would also be the beginning of the United States censoring the information that "we the people" have access to -

This is like someone saying that the government should regulate which magazine placement in retail stores - and lets take it a step further - product placement on store shelves.

Did this generation read Farenheit 451, Animal Farm, and 1984?

sonicdeath said...

Google could be less evil here by randomizing the top 30 page listings and encouraging users to dig deeper in the listing stack.

This is ultimately more fair, reduces gaming of the system, and improves users ability to really find what they are looking for - not what was just easy.

Greg Stein said...

More than just three options... Jaguar is also a classic gaming system!

Vasant said...

It's ideal to let Google keep their algorithm proprietary. After all, they've provided a viable alternative to the iPhone using the open source Android operating system.

David Shellabarger said...

@sonicdeath
Really? You want to take the answer I'm actually looking for from the front page and and put it on page 3?
That's a terrible idea. How many millions of dollars would Americans, as a whole, waste digging though 2 extra pages of search results?
Why would you not put the best result at the top of the list?

Ashwin said...

I think the fear-monging about regulation is dangerous at least as much as the prospect of a government controlled search.

Lets first acknowledge that "information" is an important utility like oil, water or power.

Though Google is a public company, its a profit making enterprise. While its insane to regulate its innovation, its the duty of the government to at least debate whether there needs to be a search body that ensures right information is available to people without any bias.

An important world utility provided by a private enterprise, has to be reviewed and regulated. Period.

The issues is graver with Fb. It owns all our "likes" and is a completely private owned firm.

I've written in the same lines @ http://ashwinramasamy.com/?p=175

- Ashwin

freetheweb said...

The notion of 'neutrality' is nonsense - but all search engines should be transparent about the algorithms they are using - or at least the assumptions baked into them. There should be transparency about the code that confines our information environment, so that different versions of what is 'best' and what is 'most important' can be conceived, but also challenged.

June Stormcrow said...

Government has no place regulating the internet beyond monitoring illegal activity.

Rinse, wash, repeat until it sinks in.

Dave said...

This article links to Danny Sullivan's take on things where he makes an analogy to the New York Times - sarcastically stating that perhaps it should have its algorithms (editorial policy) regulated too.

However - he misses the fact that newspapers (and journalism in general) *IS* quite heavily regulated. Newsapapers can be investigated and held accountable for things they print. They can easily be sued for libel. They have to adhere to certain standards imposed by various regulatory bodies. There are independent bodies people can complain to.

Google has NONE of this - it has free reign to do whatever it likes. This means (that in the UK at least) it has control over what 90 percent of people see on the internet. This is simply too much power for one unregulated corporation for weild.

Julian said...

This looks like an attempt to focus the debate on regulation of google around the search ranking algorithm.
The most concerning aspect of search which has the potential to harm competition and consumers is the conflict of interest in one company providing the de facto ordering of relevant sites for any search (with its dominant share of searches), and extracting a rent from sites wanting traffic for being shown to be relevant for a given search.
If google's businesses of search and displaying ads around search were separate(d), they would be free of this conflict of interest.

TunerTools.com said...

Interesting to see Google finally espouse free market capitalism, to recognize that regulation stifles innovation and to join the rest of us who run businesses in realization that the absolute best and fastest way to force product improvement is by allowing free choice among consumers to decide which product or service succeeds and which will fail.

One Dollar | One Vote

Welcome to the Free World Google. Now if you'd only drop your socialist tendencies toward politics, perhaps we can get back to improving the world, once transaction at a time.

Cheers,
Lloyd

contact said...

The anti-monopoly laws generally create some necessity for balancing the benefits of secrecy and openness. In practice, what we can expect is that there will be some employees on the government payroll who will be of high integrity who monitor google and other monopolies from the inside to insure that they don't use their monopoly power to gain dominance in other areas. It's simply a game of politics at present to determine the details of such an arrangement. These posts are simply positioning. Ultimately, there will be increased governmental oversight, and some reduced rate of innovation, balanced by increased confidence that google is not breaking anti-monopoly laws.

Martin said...

While I am not a proponent of government intervention-one just need look at what government has done for small business let alone the economic chaos we're experiencing now-I do believe that something needs to be done to keep the search engines honest in making sure that organic searches do not take a back seat for sponsored ads. What seems clear is that since Google is no longer a privately held company, the ideology upon which Google was founded has changed dramatically. Makes sense when you think about it, since those that have a vested interest in Google want to see a return on their money. At the same time time, it's no secret that sponsored ads remain Google's bread and butter. My concern is that, whereas before these ads appeared only on the right-hand side of the SERPs, they're now appearing above the organic search results on the left-hand side of the page and, in many instances, it has become more than just one or two ads. If this continues to flourish to the point where the organic SERPs begin on the 2nd page, then I do believe that something needs to be done. Unfortunately, I don't have the answer. Do You?

dofollowuntacticcal000 said...

If the government are calling for regulation then surely they must also include every major player online and off? You just can't streamline Google because its methods are safe and effective in their ability to perform and out-do, it is nonetheless a business that should be safeguarded with the respect it deserves. If Google are abiding by the policies set down by government, then why would regulation be a call? All businesses are accountable for their activities just like its user's, we should have a more in depth arrangement for how we govern security online instead. If Google have not contravened any By-Laws then it would an impeachment to regulate it.

PicnicOutdoors said...

Oh-oh someone said government "control" on a US board - as the US still allows most of its population to have no healthcare because they will submit them to death panels this is a non-starter.
No-one, I hope, wants government control of much in their lives but we also do not want our search results skewed by the provider, because it owns, for example, an airline, a travel company, a etc etc to pump those results to #1 if they just could be garbage.
I have no problem with Google but I do not believe "Do no Evil" - if you do you know where the swamp land is.

miau said...

all those murdoch's guys and all the wall street monsters should make their very own search engine, so all this bullshit about blamig google could stop

Handmade Furniture said...

Can you imagine the kind of lobbying that would be going on?

Christopher said...

@martin
If organic results start coming up on the second page of Google, and the first page is all ads, people will either get very fast at moving to the second page, or they'll start using an alternative search engine. What reasonable action could you possibly mean by "something needs to be done?" You want somebody (the government) to go to Google with guns and tell them how it's going to be? "we're so sick of using your product, which you offer to us for free, which we only use in the first place because it's so great, that we're going to take your money, or put you in a box, or shoot you (progressively, depending on how much you resist us) unless you change it to fit with what we want." Wow. Great job. Ever think that maybe the people in government that allow such ideas to even be suggested should be put in a box or shot? That's what we do with other organized criminals...

David Eldridge said...

It is not the state's responsibility to ensure that we have equally valuable search results. Will they guarantee the quality of search results by legislating a minimal quality requirement on content providers? What will you do when your site is removed for failing to meet those requirements? This is infringement upon our freedom of speech (in both the actual and potential case). Not all results are equal. Google is not forcing us to use the service at the point of a sword, like the state could force Google to do such-and-such thing at the point of a sword (or under threat of fine and imprisonment). The state does not stand here to sand all the corners off of the edges of the world and to ensure that we all have a fair shot. Everything they touch becomes more complicated and costly. Government intervention and regulation is a series of breakfixes placed atop each other like a piece of duck tape atop a piece of duck tape atop a piece of duck tape atop.... Really, it's time for the government to 'do no evil'.

Andres Torrubia said...

" If search engines were forced to disclose their algorithms and not just the signals they use, or, worse, if they had to use a standardised algorithm, spammers would certainly use that knowledge to game the system, making the results suspect."

Marissa - Does google even disclose the signals used in the algorithm/s? Your sentence above suggests that you do.

Are you referring to the ~200 signals that some googlers have hinted take place in the rankings?

Andres Torrubia said...

" If search engines were forced to disclose their algorithms and not just the signals they use, or, worse, if they had to use a standardised algorithm, spammers would certainly use that knowledge to game the system, making the results suspect."

Marissa - Are you referring to the ~200 signals that some googlers have hinted take place in the ranking algorithm?

Has Google ever disclosed such list?

Your sentence above suggests that you do.

johnk said...

I'm all for the government regulating search results and it doesn't have anything to do with regulating Google. Let the government build, maintain and regulate their own search engine, and give consumer's a choice. I wonder how many people would be opting for the government site? I'm sure that would be a cost effective way to handle the controversy, or more likely a terrible waste of tax payer's money.