Chris Grayling is facing the prospect of being flooded with books, after the Howard League asked its supporters to mail them to him.
The books will be sent with a request that they be forwarded on to a prison, thereby avoiding the ban on prisoners being directly sent books from members of the public.
"You have indicated that you consider sending in books to be a security concern because they could be used to smuggle in drugs, although there has never been any evidence of this being the case," Howard League chief executive Frances Crook wrote to the justice secretary.
"Both the Prison Governors Association and the Prison Officers Association have refuted the contention that there is a security issue with the sending in of books from reputable stores or from families."
The letter was sent to Grayling along with a copy of Fyodor Dostoevesky's Crime and Punishment, which campaigners held up in a silent protest during his appearance in front of a committee of MPs earlier this month.
The Society of Authors and English PEN are also supporting the campaign and have urged supporters to send in books for prisons via the justice secretary.
The prison book ban was part of Grayling's draconian new prison regime which was initiated in November last year.
It came amid a significant decline in prison staff numbers, leading some to predict rioting similar to that which hit HMP Ranby in Nottinghamshire this weekend.
Ministers say prisoners can still buy books with money they make in prison or use the library.
But the small wages prisoners receive make book purchases an extremely expensive option and library attendance is often very difficult due to staff cut-backs.
Penal experts have warned that any obstacle to prisoners reading makes rehabilitation less likely.
There was a small ray of light for penal reform campaigners yesterday, when the Ministry of Justice agreed to reverse its ban on steel-string guitars in prison.
The U-turn followed an outcry from high profile musicians, including folk guitarist Billy Bragg and former Smiths member Johnny Marr.
"As an incentive to engage in rehabilitation, individual access to steel-strung guitars can really help the atmosphere on a prison wing," Bragg said.
"I've had a number of projects involving guitars on hold which now will be able to go ahead, and will allow those using music in prisons to get on with this important work."
Labour MP Kevin Brennan said: "I started the campaign after prisoners wrote to me explaining how they had saved from their small prison wages to buy guitars and how therapeutic learning to play the guitar had been for them before the ban.
"If we want to reduce reoffending we need to support purposeful activities like learning to play an instrument."