Senior Conservative MPs plan to make a formal complaint of bias to John Bercow amid suggestions the Speaker personally "hates" David Cameron.
Members of the Tories' 1922 committee are expected to speak to Bercow, after he intervened earlier this week to prevent the prime minister referring to Ed Miliband as a "conman".
Bercow ordered Cameron to withdraw what he described as his "unparliamentary" remark about the Labour leader.
Cameron had accused Miliband of attempting the "politics of the conman" during exchanges at PMQs on Wednesday.
Bercow interrupted to say: "The word 'conman' is frankly unparliamentary. The prime minister is a man of great versatility in the use of language. It's a bit below the level."
One Conservative MP told the Daily Mail the party had finally run out of patience with Bercow.
"It was the first time I have seen the vast majority of Tory MPs outraged," they said.
"Bercow’s face was a face of rage. There was simply contempt in his eyes. You shouldn’t address the prime minister like that."
"This was anger and dislike directed at the PM. It is clear that he hates him personally. We knew he could be biased but this went way beyond what was acceptable."
Conservative MPs have long had a difficult relationship with Bercow, who was elected as Speaker due to widespread support from the Labour benches.
It reached a low point in 2010, when Tory MP Simon Burns referred to him as a "sanctimonious dwarf".
Under parliamentary rules, MPs are not allowed to accuse each other of being liars, although they are allowed to accuse others of "misleading" the House.
It emerged yesterday that the phrase "con man" has only been used once before in the Commons.
In 2001, the then-chancellor Gordon Brown was answering a question about Britain and the euro when a Conservative MP interrupted with the shout of "con man!"
That Conservative MP was John Bercow. Like Cameron, he was quickly forced to withdraw his remark.