Citizens should have a right to privacy online. And governments have an obligation to keep their citizens safe. Finding the right balance between privacy and security is a delicate balancing act. Europe’s recent experience with data retention holds interesting lessons for everyone concerned with this balance.

In the aftermath of the Madrid bombings in 2004, the European Council adopted a Declaration on Combating Terrorism, which stated the need for rules on the retention of communications traffic data by European service providers for the first time. In some European countries, the ability to monitor communications was perceived as a practical priority in helping law enforcement agencies prevent and investigate terrorist acts. In April of 2004, the UK, Sweden, Ireland and France put forward a proposal for a Framework Decision calling for the retention of a wide variety of data for between 12 and 36 months.

However, for some politicians, the idea of adopting wide-ranging measures, requiring providers of telecommunications and Internet services to retain details of calls and electronic communications for periods of time beyond their pure operational needs, was not entirely justified. Indeed, for a while European privacy rights appeared to have the upper hand and the European Union institutions seemed to listen to the objections of the European Parliament’s Committee on Civil Liberties, Justice and Home Affairs.

According to the calculations of this group of European Members of Parliament, if all the traffic data covered by the proposal did indeed have to be stored, the network of a large Internet provider would accumulate up to 40,000 terabytes – the equivalent of four million kilometers worth of paper files -- or about 10 stacks of files each reaching from Earth to the moon. But others pointed out that even the slowest terrorist would figure out that he could simply avoid his communications being traced by using a non-European service provider. Nonetheless, the political pressure continued, and the European Commission went on to propose a directive on data retention in September 2005.

The rest is history… and now law. Although the European Parliament’s Committee on Civil Liberties, Justice and Home Affairs succeeded at introducing some amendments aimed at softening the effect of the proposal, an unprecedented data retention directive was adopted by the European Council on 15 March 2006. This directive imposes retention obligations between six months and two years in relation to accessible data generated or processed as a consequence of a communication or a communication service.

On paper, the aim behind the directive is simple and proper: to harmonise data retention rules across the EU and to ensure that the necessary information is available for the purpose of the investigation, detection and prosecution of serious crime. Unfortunately, the simplicity pretty much ends there. For a start, using the words “directive” and “harmonisation” in the same sentence is often an oxymoron, especially when a directive is cobbled together as a compromise between conflicting ideological positions.

On a practical level, the likelihood of seeing a consistent implementation of the rules across the EU is effectively zero. The timing of the implementation – due by September 15, 2007 – will certainly vary. 16 of the 27 EU Member States have already declared that they will delay the implementation of data retention of Internet traffic data for an additional period of 18 months, as permitted by the directive. The compulsory retention period for each type of data will also vary from country to county (e.g. Germany has proposed 6 months, the UK 12 months, and the Netherlands 18 months). The interpretation of other key elements, such as “serious crime," “competent national authorities,” or “electronic communications services” will be different across jurisdictions too.

These uncertainties impact on the justification for any privacy intrusions. Is a country more democratic than its neighbour because of its shorter retention period? Or do the citizens of that country face a greater security risk for the same reason? If there is something about the data retention directive that can be called into question is its proportionality – not necessarily in terms of financial cost to service providers, but in terms of privacy and anonymity loss. And what will Internet companies do in practice, especially if they operate one data architecture that cannot vary from one country to another: apply the longest retention period, or the shortest, or some “average”?

The data retention directive is of course just part of the picture. Several other initiatives provide additional evidence of the fact that traditional concepts of Internet privacy are in turmoil. One example was a proposal by the German government to complement its anti-terrorism measures by prohibiting the use of anonymous email accounts, by mandating that service providers verify the identity of their account holders.

Thankfully, the German government has recently retracted this proposal. Nonetheless, the idea continues to appeal to many: to make sure that every single e-mail user can be tracked down to an identifiable individual, so that the police can locate the terrorist behind the e-mail with the bomb-making instructions attachment, to take the most blatant possible example. The issue once again is whether this threat to anonymity on the Internet will be effective in making the world a safer place. Or will it do nothing to catch your average technology-savvy terrorist while eroding yet another layer of Internet privacy?

So, against this background, what is Google doing? We have recently announced a new policy to anonymize our search server logs after 18 months (we’re the first in our industry to have taken this step). We’re trying to get the balance right too, between privacy and other goals (like security, fraud prevention, and search improvements). People want to be free as much as they want to be safe. That’s true online too.