As the Prime Minister’s former press secretary, Andy Coulson was never likely to flee the country and hide in the Peruvian desert. But the Scottish police didn’t take any chances when they arrested him – they drove him to Glasgow, rather than let him fly up himself. To add to the drama, motorcycle outriders were there to meet him at the border and his capture trumpeted on the police’s Facebook page. In all the excitement, they seem to have overlooked a rather important fact: no crime had been committed. On Monday, a judge pointed this out and the case against him collapsed.

As Lord Burns explained to the bemused jury, the perjury charge had been a red herring all along. If a witness testifies to being teetotal, when in fact he’s an incurable drunk, it’s a lie. But it’s only perjury if it’s relevant to the accused’s case. Coulson was, originally, in court five years ago to discuss his newspaper’s (accurate) exposé of the debauchery of a politician named Tommy Sheridan. Hacking had no relevance to Sheridan’s fondness for swingers’ clubs, the judge ruled, so Coulson had no case to answer. Just as his defence lawyer had argued. The crime, it turns out, wasn’t a crime after all.

Andy Coulson and Rebekah Brooks at News of the World

This fits a trend. The past few years have seen dozens of journalists arrested for paying government officials for stories. But jury after jury has decided that this isn’t a crime either, so the journalists have been cleared. The much-hyped trial of Rebekah Brooks, who preceded Coulson as editor of the News of the World, ended in her being acquitted of all charges. The same with Charlie Brooks, her husband. Bob Bird, her former Scottish counterpart, had the charges against him dropped earlier this week.

The investigation into journalists has now cost the taxpayer more than £35 million – more than the investigation into the Stephen Lawrence case or the July 7 terrorist atrocity. We have seen judicial inquiries, dawn raids and never-ending investigations. And for what? In case after case, juries have decided that there was no offence. Once, we were told this was the crime of the century. Slowly, a slightly different picture is emerging: one of a justice system that succumbed to a political feeding frenzy.

This story starts five years ago, when Rupert Murdoch sought to buy the shares in Sky that he did not already own. He wanted to make it far stronger, capable of bringing greater competition to the BBC – which, of course, was horrified. As were his rivals, who signed a letter asking the then Business Secretary, Vince Cable, to block the deal. The BBC’s director-general signed the letter and, in so doing, made the BBC an actor in this drama. Given that the BBC website now has more readers than any newspaper in Britain, the corporation has an amazing ability to decide what counts as a scandal.

Newspapers tend not to fight each other, because of the cycle of violence that’s certain to ensue. But Coulson’s position in No 10 changed this: he wove a thread through Murdoch’s business empire to the heart of David Cameron’s government. So when the hacking scandal reopened, the stakes were a great deal higher. Suddenly, this was about the Prime Minister’s judgment, his cosiness with media moguls and a supposedly corrupt establishment. As papers went to war, Ed Miliband joined in, bizarrely posing as the only man brave enough to sock it to Rupert Murdoch.

I knew Coulson well; he hired me to write for the News of the World shortly before the first hacking scandal led to his resignation. There’s no doubt about his complicity: he was found guilty of one count of conspiring to intercept voicemails. But I felt sympathy for him then, and do now. This was the failing of an entire industry, not just one newspaper – it looked as if he was being blamed for a far wider problem. And the law was never as clear as the prosecutors made out.

At the time, even policemen thought it was legal to hack into mobile phones and listen to old voicemails. An MI5 training exercise asked new recruits if a warrant was needed to intercept a letter. It was a trick question: the answer is “yes” if it was sealed, but “no” if it had been opened. The same, it was assumed, applied to voicemails. The legal fuzziness continued throughout this whole scandal. Is it really illegal for journalists to pay a whistleblower who works in a government department? Prosecutors decided that it was only by interpreting a 13th-century offence of “misconduct in public office”. To journalists, this was entirely new.

Take Clodagh Hartley, a former political correspondent with The Sun. She had cultivated (and, yes, paid) a government source – and brilliantly used their relationship to reveal all manner of government waste. Together, they exposed a £1.3 million bill to promote a government website (by paying celebrities) and Ed Balls blowing £24 million by moving a schools quango to Coventry. These are precisely the sort of stories that the public have the right to know about – the stories journalists have a duty to uncover.

^{The Sun's Whitehall editor Clodagh Hartley (Photo: ANDREW COWIE/AFP)}

But in the post-hacking atmosphere, police saw journalistic corruption everywhere. Lawyers, too. Judge Charles Wide, who presided over the conviction of a journalist at the Old Bailey six months ago, was later found by the Lord Chief Justice to have “misdirected” the jury. Throughout his trial, Duncan Larcombe, The Sun’s former royal correspondent, said he had done nothing wrong and “still have to be convinced why I’m sitting here”. A jury agreed: he and three of his colleagues were acquitted three months ago. When four more journalists were acquitted later, the trials of nine others were abandoned.

The fuss created about journalistic corruption – and the money spent investigating it – now stand in grotesque contrast to hard evidence of actual criminality. We have had the most expensive investigation in British history, and for what? It is, of course, crucial that Britain has a justice system capable of following the evidence wherever it leads. But it’s also crucial that Britain has a justice system capable of standing up to political hysteria, rather than becoming part of it. Keir Starmer, the former director of public prosecutions, was so taken with political drama that he became a Labour MP at the last election.

^{Vince Soodin was on bail for 989 days. No charge (Photo: Christopher Pledger)}

For Coulson, it is finally over. I understand that he has not quite forgiven David Cameron for saying that he had been “lied” to – a word with particular potency, given the perjury trial he was facing. If he writes a book, it’s unlikely to be one from which the Prime Minister emerges well. But no one emerges well from this imbroglio, which has exposed the ugliest side of journalism. It has also exposed how keen politicians of all parties were to try to introduce state licensing of newspapers when they thought they had the chance, seeking to end 350 years of press freedom.

Clodagh Hartley, a young mother, spent more than two years wondering if she’d go to jail for reporting the news and doing her job. She was one of 5,500 who have spent at least six months on police bail without charge, trapped in a legal system that is ruining the lives of the innocent. The new Justice Secretary, Michael Gove, will soon be drafting a British Bill of Rights and planning wider legal reform. The fiasco of the past few years will have given him plenty to reflect upon.

CORRECTION: This article originally stated that phone hacking was the work of "shady private investigators" rather than reporters. In fact, journalists have also been convicted of phone hacking offences. We are happy to make this clear. The article has been amended accordingly.