The woman Eric Pickles told to adjust her medication was "utterly unreasonable", the communities secretary has told Politics.co.uk.
Pickles came under fire from mental health charities after a recording of him emerged telling the survivor of alleged child abuse to "adjust your medication".
Teresa Cooper, a constituent of the Brentwood and Ongar MP, secretly recorded an exchange with Pickles in which she angrily accused him of ignoring evidence about a care home.
She is one of a number of women who have claimed they were drugged while teenagers at the Kendall House care home in Kent in the late 1970s and early 1980s - and went on to have children with birth defects.
The recording, made public by the Independent newspaper, features Cooper resisting an attempt by Pickles to get her to sign a statement putting her allegations into writing. Pickles tells her: "You change your views, you give me different stories..."
Cooper replies: "It's been proved publicly, we've got all this stuff. Nothing has changed on the Kendall House abuse, only you have ignored it."
Pickles, speaking over Cooper, tells her to "adjust your medication". She replies: "I am not on medication, but you need to be."
Speaking at the conclusion of the Conservative party conference yesterday, Pickles told Politics.co.uk he had a "long history with that particular constituent".
"Her behaviour was utterly unreasonable," he explained.
"She recorded me surreptitiously. I was giving genuine advice... and her reaction was to do 50 tweets and broadcast it to the world.
"I'm very sympathetic with her, she has a difficult life. But she makes a number of serious allegations about people and I think, it's very important she puts that allegation in writing and she puts her name to it."
He claimed Cooper had denied all knowledge of making an initial statement to his office.
Labour has attacked Pickles for "thoughtless, stigmatising" behaviour, while mental health charities have also spoken out against the communities and local government secretary.
"Even if they were taken out of context, they could be seen as thoughtless and hurtful, and such personal comment should never be made," Marjorie Wallace of Sane said.
Sensitivities about mental health issues are running high at the moment after supermarkets Tesco and Asda were forced to apologise for selling Halloween outfits of mental health patients.
Even Britain's most senior politicians are unclear about what is now politically correct. David Cameron said Labour's tax policy was "nuts" before adding: "I am going to get into a huge argument with the mental health lobby, that is not what I want."
Wallace of the mental health charity Sane took a more relaxed view, however.
"This treading on egg shells doesn't help reduce stigma and can even increase it making mental health a no-go area, leaving those who suffer more isolated," she added.