David Cameron's claim that it is no longer "fair" to call Britain's overseas territories 'tax havens' is being challenged by campaigners today, as a joint ministerial summit takes place in London.

A coalition of charities is angrily accusing the government of hypocrisy over its approach to tax-dodging by companies who use places like the Virgin Islands, Bermuda and the Caymans to disguise their profits, launder money and even finance terrorism.

"I do not think it is fair any longer to refer to any of [Britain's] overseas territories or crown dependencies as tax havens," the prime minister said in September.

"They have taken action to ensure that they have fair and open tax systems."

At the G8 summit in June the overseas territories agreed to sign up to the OECD Multilateral Convention, which allows governments to swap tax information.

But Britain has agreed to establish a public list showing who owns companies and the charities want overseas territories to agree to do the same.

Doing so would make it easier for the governments of developing countries to work out how much they are owed.

"Tax havens are places that offer extreme secrecy and low tax rates – and the UK's overseas territories still do both," Action Aid's Chris Jordan said.

Claire Godfrey of Oxfam added: "The meeting this week is a golden opportunity for Britain's overseas territories to take a big step towards shaking off the shameful status of being 'tax havens' once and for all.

"Only when all tax havens have agreed to build a fair and transparent global tax system will the world's poorest countries be able to reap the taxes they're due."

War On Want has focused its criticisms on Google, which it claims funnelled £6 billion – four-fifths of its global revenue – through Bermuda in 2012.

Bermuda has no corporate income tax, no registers of company ownership and no requirement to file company accounts.

"Cameron declares there are too many tax havens, too many places where people and businesses manage to avoid paying taxes," tax campaigner Murray Worth said.

"Yet, at the same time, he refuses to eradicate Britain's own global network of tax havens.

"Rather than ignoring the facts and pretending these places aren't tax havens, the government should be using this summit to finally tackle the UK's devastating network of tax havens that enable the global scourge of tax avoidance and evasion."

There is now thought to be a total of £20 trillion held by rich individuals in what War On Want calls 'secrecy jurisdictions'.

If this money was taxed it would generate as much as £180 billion in revenue – double the amount developed countries spend on overseas aid.

The Foreign Office said the joint ministerial council taking place today would cover international financial services regulation, but its main focus was on economic growth and diversification.

"Jobs and economic growth are fundamental to building safe, successful and sustainable territories," minister Mark Simmonds said.

"This year's [summit] will focus on agreeing practical steps for the UK and pverseas territories to work together to achieve that goal."