Forty years ago this week, President Richard Nixon defined his country’s new drug policy with a military metaphor that stuck.

“Public enemy number one in the United States is drug abuse,” he declared. “In order to fight and defeat this enemy, it is necessary to wage a new, all-out offensive.” A trillion dollars and seven presidents later, the war is still being fought, and lost, with catastrophic results.

In its relentless prosecution of addicts, dealers and drug producers, the US has no peer. Prohibition has failed all over the world, but the consequences are most evident in the land of the free.

There are more than 500,000 people in US prisons for drug offences, a tenfold increase since 1980. Drug overdose has become the leading cause of death in the 35-54 age group. Four in five drug arrests are for possessing small quantities, with no intent to sell. Whole communities have been criminalised: more than half of the African-American men in Chicago have a charge sheet, which disqualifies them from public housing and student loans.

Even adjusting for inflation, Nixon’s $100 million annual budget for the “War on Drugs” has multiplied 50 times: the Obama administration has asked Congress for $26.2 billion next year. Despite all this money being thrown at the problem, narcotics are cheaper and more widely available than ever, according to the government’s own National Drug Threat Assessment.

‘When 1.9 million kids go to bed at night with one of their parents in prison, the thing they’re claiming to protect - the children - is what they’re harming’

Ian Bezman was a bright but insecure child. At 14, he started to self-medicate. “First it was marijuana, then mixed with PCP, then over the years it was meth, heroin, cocaine,” says his mother, Suzanne Riordan.

After a brief stint in rehab, the state of California showed little interest in treating his addiction, preferring to lock him up. “He was in juvenile detention, then jail a number of times, never for more than a few months, always for possession,” Riordan says. “He was a very sensitive young man, with pretty fragile self-esteem. Over several years, he cut deeply into his arms, all the way to the bone. Jail was hard on him.”

Bezman came out of jail in 2005 drug free and determined never to go back. He found a good job, working with the carpenter’s union on the Ronald Reagan Memorial Library. His girlfriend became pregnant and he was excited by the prospect of being a father. Although checking in with his probation officer every week was a hassle and he worried about being sent to prison for a minor violation, he seemed to be gaining some stability. Then, late one night, during a drinking session at his foreman’s house, he did a line of cocaine. Five days later, he failed a mandatory drug test. “He was extremely upset and frightened,” Riordan says. “He disappeared that night and police found his body three weeks later.”

US prisons hold around 2.3 million people, meaning that a country with 5% of the world’s population has 25% of its prisoners.

The shift towards punishment and away from rehabilitation in the “corrections” system has been stunningly counter-productive. According to the Bureau of Justice, half the prisoners released this year are expected to be back inside by 2014. Last year, expenditure on prisons was $68bn. The Supreme Court recently ruled that California’s prisons are so overcrowded that the constitutional rights of inmates are being violated, ordering the state to release or transfer 32,000 people.

American prisons are notoriously violent places. According to a Bureau of Justice Statistics survey, an estimated 70,000 prisoners are sexually assaulted each year. In 2005, Bryson Martel testified to a congressional committee that during his nine months in Arkansas state prison he was raped repeatedly. His punishment for the crime of cashing a forged cheque, to pay for crack cocaine, was the HIV virus, contracted at knifepoint.

Richard Van Wickler, who runs Cheshire County jail in New Hampshire, argues that sending non-violent addicts to prison is a colossal waste of money and human life. He says: “62% of our prison population receives prescription medication. 32% have been diagnosed with a mental illness. Here in New Hampshire, if you’re an addict and you’re trying to get help, it’s almost impossible, so the only place left for you to go is jail, which is the most expensive option.”

VAN Wickler is a member of Law Enforcement Against Prohibition (LEAP), a group that believes all drugs should be legalised, taxed and regulated. Neill Franklin, the organisation’s executive director, is a retired Maryland State Police Major who spent more than three decades fighting the war on drugs. He started speaking out against it after a close friend, working undercover, was murdered by the cocaine dealer he was trying to bust.

At a recent workshop in a juvenile detention centre, Franklin asked the kids, almost all of whom were African-American, what would happen to their neighbourhoods if drugs were legal. “The number one answer was, ‘We would have no money’ – because that’s what they see as employment,” he says.

The drug war has a disproportionate impact on minorities: more than half of the 50,000 people arrested for marijuana possession in New York last year were black or Latino.

Franklin’s home town, Baltimore, is best known in Britain as the setting for The Wire, a television series that portrays a violent, dysfunctional city, in which black communities are abandoned to addiction, incarceration and civil war between rival drug gangs. “I liken that show to a documentary where the names have been changed,” Franklin says. The only part that isn’t realistic, he offers, is the episode with a decriminalised zone, nicknamed Hamsterdam, where pushers and junkies do as they please. In real life, a needle exchange programme was the only experimental approach Baltimore’s mayor was prepared to sanction.

Although LEAP’s manifesto – cannabis in corner shops, heroin and cocaine at official dispensaries – is no longer considered to be as radical as it once was, most of the alternatives to prohibition that have been tried, in Portugal and Switzerland, for instance, have involved decriminalising drugs, rather than regulating their sale.

Earlier this month, the Global Commission on Drug Policy – a panel including former UN Secretary General Kofi Annan, 11 former presidents, and Ronald Reagan’s Secretary of State, George Shultz – released a report that declared “the war on drugs has failed” and called for fundamental policy reforms. Although the eminent names made the findings difficult to dismiss, the Obama administration rejected their central premise. “Making drugs more available – as this report suggests – will make it harder to keep our communities healthy and safe,” said drug policy spokesman Rafael Lemaitre.

SEVEN years ago, on the campaign trail, Obama called the drug war an “utter failure,” although he added that he did not favour legalising marijuana. In office, he has reduced the disparity between sentences for crack and powder cocaine possession and ended a ban on federal funding for needle exchange programmes, but he is otherwise following the same policies as his predecessors.

Every president since Nixon has endorsed the war on drugs. Bill Clinton spent $1.3bn on Plan Colombia, which financed paramilitary crackdowns and sprayed cocaine crops with herbicide. George W Bush signed the Merida Initiative, which committed $1.4bn in aid to Mexico, with the specific aim of breaking up the narcotics business.

Obama has doubled down, sending billions more, without loosening the grip of cartels or stemming the violence. In the four years since Mexican president Felipe Calderón announced a military offensive against the traffickers, at least 40,000 people have been killed.

Terry Nelson, who spent most of his career in the US Customs Service and Department of Homeland Security disrupting drug routes in Central America, is scathing about the effectiveness of the cross-border efforts, noting that cocaine production increased in the Plan Colombia era. “Even if you manage to wipe it out in Colombia and Peru and Bolivia, this stuff will grow in sub-Saharan Africa,” he says.

Nelson is a lifelong Republican, unlike most in the legalisation movement, but he says conservatives are coming around. At a recent party meeting, the organiser told him that regulating marijuana should be on the manifesto for the next election. Former Speaker Newt Gingrich has formed a group called Right On Crime, which calls for non-violent offenders to be sent to treatment, not prison.

“The war on drugs means the destruction of the American family,” Nelson says. “When 1.9 million kids go to bed at night with one of their parents in prison, the very thing they’re claiming to protect – the children – is what they’re harming.”

There are signs that public opinion is changing. In a recent Rasmussen poll, 42% of respondents said marijuana should be legalised, while 45% disagreed, with 13% unsure. In Congress, a marijuana legalisation bill is being prepared, but it stands no chance of passing.

If the law does change it will be too late for Ian Bezman and too late for Jeff Cullen, another troubled young man who died of an overdose, not long after leaving jail.

His mother, Denise, is a member of Moms United To End The War On Drugs. “Addicts have nowhere to turn, so it’s just a cycle, in and out, to keep the money going in the prison industry,” she says. “They look at drug users as ‘those people,’ even though they are often good kids, from loving families. There are so many casualties of this war. Every day I get a new call from somebody who has lost someone.”