Oliver Dowden goes after the Lib Dems and Labour inflict defeats on the Conservatives in Devon and West Yorkshire.
The Conservatives have lost two parliamentary seats in by-elections, with the Liberal Democrats and Labour delivering blows to Boris Johnson.
In Tiverton and Honiton, Devon, the Lib Dems overturned a huge Tory majority to win by more than 6,000 votes.
And, in Wakefield, West Yorkshire, Labour took back a seat it had lost to the Conservatives in 2019.
The results follow months of criticism of the prime minister over parties in Downing Street during lockdown.
They also come amid soaring inflation and concerns over the cost of living, and follow a narrower-than-expected vote of confidence by Tory MPs in the prime minister earlier this month.
In Tiverton and Honiton, where former MP Neil Parish quit after he was found watching pornography in the House of Commons, the Lib Dems took 22,537 votes, beating the Conservatives by 6,144.
At the last general election, the Tories had a 24,239 majority in the mostly rural constituency.
In his victory speech, the Lib Dems’ new MP Richard Foord said: “Tonight, the people of Tiverton and Honiton have spoken for Britain.
“They’ve sent a loud and clear message: It’s time for Boris Johnson to go, and go now.”
Party leader Sir Ed Davey added: “The Liberal Democrats have made political history with this stunning win. It is the biggest by-election victory our country has ever seen.”
A Conservative source called the result in Tiverton and Honiton “disappointing but not unexpected”, telling the BBC the party was confident it could retake the seat at the next general election.
And in Wakefield, where a Labour victory had been largely expected, the party’s candidate, Simon Lightwood, won by 4,925 votes.
The previous MP, Imran Ahmad, resigned after being convicted of sexually assaulting a 15-year-old boy.
Reacting to his party’s win in Wakefield, Labour leader Sir Keir Starmer said it showed the “country has lost confidence in the Tories”.
“This result is a clear judgement on a Conservative Party that has run out of energy and ideas. Britain deserves better,” he added.
But Mr Johnson, who is attending a Commonwealth leaders’ summit in Rwanda, suggested before the results that it would be “crazy” for him to quit if the Conservatives lost Wakefield and Tiverton and Honiton, saying mid-term by-elections were “never necessarily easy for any government”.