The Archbishop of Canterbury has admitted he was "irritated" after it emerged the Church of England has tens of thousands of pounds invested in Wonga.
But the Rt Rev Justin Welby insisted "life is complicated" as he underlined his determination to do more than just "sit on the sidelines" over payday lenders.
Welby, who took over from Rowan Williams as the Archbishop of Canterbury at the start of this year, had declared his aim of pushing payday lender Wonga out of business in a magazine interview earlier this week.
But the Financial Times newspaper revealed the Church has £75,000 invested in US venture capital firm Accel Partners, which led Wonga's 2009 fundraising.
Welby admitted this morning he was eight out of ten embarrassed by the revelation, which had left him "irritated for a few minutes".
"But these things happen," he told the Today programme, pointing out the Church's total investments exceed £5 billion.
"It shouldn't happen, it's very embarrassing but these things do happen. We have to find out why and make sure it doesn't happen again."
At present investment rules set by the church commissioners allow investment in a company where less than 25% of its business is in money-lending.
"My own view is this is too high a level," Welby said. He pledged a review of this and other caps - including the three per cent limit on the church investing in companies which sell pornography.
"We are going to have to review these levels and how we do it," Welby accepted.
"If you invest in a hotel chain, a lot of hotel chains sell pornography in their hotel room.
"Do you therefore not invest in any hotel chains at all? I've been living in this [world] for many years. I have no illusions about this."
Welby wants the Church to explore the viability of setting up credit unions which could rival Wonga and other payday lenders.
He was forced to concede such a credit union would struggle to offer interest levels of less than 70%, however.
Working with partners like housing associations, using cheaper premises and more efficient management could cut the interest rate down to the historic 25% rates used by credit unions in the past.
"If we can't get it right we won't do it," he said.
"It's better to have a go rather than stand and wring your hands and say 'it's all terrible'."
The revelations come just days after the very first UK ban on payday loan advertising was initiated in Plymouth, which has prohibited all such advertising from bus shelters, billboards and publicly available computers in the city.
The £2 billion payday loan industry was accused by its regulator of widespread irresponsible lending earlier this year.
At an industry summit in Whitehall last month lenders were told they could face tighter controls, including limits on the number of loans taken out and a cap on the total cost of credit.