David Cameron appeared to subtly shift his position on an EU referendum this lunchtime, as he suggested an in-or-out vote would only take place in the event of a "Tory-only government".

The prime minister's language at a Cameron Direct Q&A in Eastleigh suggested only a Conservative government with an overall majority would be sufficient to guarantee the vote.

He told an audience of by-election voters at the event: "To get an EU referendum you need to vote for a Tory-only government."

That clashes with his pledge at the end of last month's Bloomberg event unveiling his new Europe policy, when he said a referendum would take place "if I am prime minister" – whether in coalition or not.

The Liberal Democrats were vociferous in their opposition to Cameron's Europe speech, with their foreign affairs spokesperson Martin Horwood even calling the prime minister's approach "crazy".

Nick Clegg has subsequently indicated he would be prepared to accept an in-or-out referendum on Europe as part of potential coalition negotiations with the Conservatives after the next general election.

That suggests Cameron's "Tory-only" comment may be nothing more than a piece of Eastleigh electioneering.

But it prompted instant concerns among some Conservative backbenchers, who would rather the prime minister ends all uncertainty about the issue by legislating now for a referendum in the next parliament.

Leading eurosceptic John Baron told politics.co.uk he believed Cameron may be changing the terms of the referendum.

"This is why I am pushing hard the idea of bringing legislation into this parliament enabling the EU in-out referendum  to take place in the next," he said.

"It would tick a number of boxes. It would address people's mistrust of politicians making promises about matters European – too many have been broken in the past.

"It would oblige all parliamentarians to make their views known.

"And it would be difficult for any prime minister after the next general election to repeal a popular piece of legislation."