But former Home Secretary Sajid Javid says he is confident his successor is doing everything she can.
Priti Patel’s “incompetence” in dealing with small boats crossing the Channel is “dangerous”, Labour has said.
More than 24,700 people have made the crossing from France to the UK by boat so far this year to seek asylum or to immigrate – almost three times the total of 8,500 in 2020.
Shadow home secretary Nick Thomas-Symonds accused the government of “comprehensive failure” on the issue.
But ex-Home Secretary Sajid Javid said his successor was doing all she could.
He also questioned why people were making the dangerous journey, telling the BBC’s Andrew Marr programme: “You have to ask the question, if you are a genuine asylum seeker, why have you not claimed asylum already if you’re in a safe country [like France]?”
The government has launched a review into how to prevent boats from crossing the 21-mile stretch of water from Calais, with the now-Health Secretary Mr Javid saying it wanted to “bust the business model” of people smugglers, who were “ultimately responsible” for the problem.
Record numbers of people have made the crossing this year, with more than 1,000 making the journey in a single day earlier this month.
Home Secretary Ms Patel has promised tougher action, but the numbers have continued to rise.
And it is understood Prime Minister Boris Johnson wants more to be done.
The UK signed a deal with France in July, pledging €62.7m (£54m) during 2021-22 to help the country increase police patrols along its coastline, boost aerial surveillance and increase security infrastructure at ports.
But Labour’s Mr Thomas-Symonds accused his government counterpart of being “more interested in a diplomatic spat [with France] than she is in a workable deal”.
The shadow home secretary said his party would reintroduce the Dubs scheme – offering safe passage to certain asylum seekers – as well as increasing patrols further away from the coast and reinstating overseas aid funding.
He also said a Labour government would find a replacement for the Dublin agreement, which allows EU countries to transfer asylum seekers back to the first member state they were proven to have entered – an agreement the UK left after Brexit.
Between January 2019 and October 2020, 231 migrants who crossed the Channel were returned to mainland Europe using the Dublin agreement.
But Immigration Minister Tom Pursglove told MPs last week that only five had been returned to mainland Europe so far in 2021.
Mr Thomas-Symonds told Andrew Marr: “There are thousands of lives being risked in the English Channel on a day-to-day basis and the home secretary’s incompetence on this matter is dangerous.”
But Mr Javid said leaving the Dublin agreement had made little difference to the UK’s ability to send asylum seekers back to other countries.
“Hardly any country in Europe enforced the Dublin agreement,” the health secretary said.
“I can tell you that because I was home secretary at the time trying to enforce it and almost every country in Europe ignored it.”
The minister said new agreements were needed with the EU, but he said Ms Patel had already signed some with countries like India.
He added: “I would say that the pandemic has made returning people across the world, across asylum systems, much harder, and we do have to take that into account as well.”
Mr Javid also said the home secretary “is doing everything that she can and she hasn’t taken anything off the table” to tackle the issue.