Undercover policing by the Metropolitan police faces scrutiny from a judge-led inquiry, the home secretary has announced, in a major move revealing the true extent of ministers' collapsed confidence in the police.
The Metropolitan police's Special Demonstration Squad (SDS) will face its biggest ever probe after the government openly questioned a number of convictions obtained through undercover police work.
It follows a review of the Metropolitan police's handling of the murder which David Cameron has already called "profoundly shocking".
Like the Home Secretary, I find the conclusions of the Stephen Lawrence review profoundly shocking. It's important we have a full inquiry.
— David Cameron (@David_Cameron) March 6, 2014
A new offence of 'police corruption' is also set to be introduced after Ellison found officers were deployed into activist groups that sought to influence the Lawrence family.
!I cannot even imagine how a law-abiding citizen who is supposed to be protecting the community could try and find dirt on a grieving family," Stephen's father Neville Lawrence said.
Former undercover officer Peter Francis had alleged last year that he had been ordered to find information which could be used to discredit the Lawrence family.
Ellison said there was insufficient evidence to either confirm or reject his allegations.
But his review suggested a further inquiry could uncover the extent to which attempts were made to discredit the Lawrence family by using undercover officers to investigate them.
He also found that one of the undercover officers deployed into activist groups which had sought to influence the Lawrence family met with a senior Met officer seconded to the Lawrence review team.
Ellison concluded "the opening of this channel of communication completely improper".
Responding to his report, May said: "I think the greatest possible scrutiny is now needed into what has taken place.
"So, given the gravity of what has now been uncovered, I have decided that a public inquiry led by a judge is necessary to investigate undercover policing and the operation of the SDS. Only a public inquiry will be able to get to the full truth."
Francis told the Guardian newspaper: "When the full truth comes out about the police’s work and activities, across the UK, against political campaigns and protests since 1968, I think they will be very shocked."
Ellison's probe established one officer, Detective Sergeant John Davidson, was corrupt.
He found that allegations made against Davidson by a former police detective in 1998, together with other intelligence about the officer, were not revealed by Sir William Macpherson's inquiry – but that this cover-up was the Met's fault.
"Ellison finds this lack of disclosure was a significant failure by the Metropolitan police," May told MPs.
Today's report stated: "The inquiry was clearly most troubled about the possible motives behind the investigative deficiencies of DS Davidson."
It found Davidson's dealings with 'James Grant', an informant, were "highly unsatisfactory" and noted "a number of dealings with potential witnesses that resulted in little evidence being obtained, and which were conducted in an inappropriate manner".
The Met's record-keeping was "a cause of real concern", the review added, noting "mass shredding" of key evidence had taken place in 2003. A hard drive containing relevant data was only discovered in 2013.
May told MPs: "The totality of what the report shows is deeply troubling."
"In policing as in other areas, the problems of the past have a danger of infecting the present, and can lay traps for the future. Policing stands damaged today.
"Trust and confidence in the Metropolitan police, and policing more generally, is vital. A public inquiry, and the other work I have set out, are part of the process of repairing the damage."
May has asked the National Crime Agency to determine how best to investigate outstanding lines of enquiry identified by Ellison.
The Home Office's permanent secretary will investigate the extent to which the department was aware of the SDS' undercover policing work in the 1990s.
Meanwhile Ellison will begin a fresh review investigating the extent to which any cases where convictions could have been made on the basis of undercover policing work need reconsidering.
Shadow home secretary Yvette Cooper offered the opposition's support to the public inquiry but called for the Independent Police Complaints Commission to be replaced with a stronger watchdog.
"Families should never have to wait 20 years to get to the truth," she said.
"The institutional failure to get to the truth and to justice over so many years continues to cast a long shadow over policing and the criminal justice system.
"It is only through the determination of the Lawrence family that this new shocking information has been uncovered now.
"For the sake of the Lawrence family and for the sake of confidence in the vital and much valued work the police do, we need forensic pursuit of the truth about what went wrong as well as every sinew strained to get justice."