The top dozen European investment banks have taken just 20.7 percent of global investment banking fees so far this year, down almost a third from a peak of 29 percent in 2003, according to Thomson Reuters data. Their share has fallen every year since 2005.

The top eight U.S. banks, meanwhile, had 35.7 percent of the market so far this year, their highest since 2007, although still below their 44.3 percent peak in 2001 before boutique firms began absorbing a larger share of mergers deal revenue.

The Thomson Reuters data covers bond underwriting, equity underwriting, loans and M&A income, but not trading income.

For some European banks, like UBS, cutting back in investment banking could be a good sign: they are reducing a business where profitability was weak to focus on areas where they have expertise, like wealth management. But others, like Deutsche Bank, may face tougher questions: if investment banking is not their strength, what is?

The shrinking European banks are putting more focus on profitability than in the past, which investors and analysts said was welcome, but there is unease about whether other businesses will be hurt by the smaller scale.

"If they are making better use of their capital it's got to be a good thing. But for anybody who believes in critical mass and the importance of scale then it's a bad thing, especially when you start to withdraw from the biggest market in the world," said Chris Wheeler, analyst at Atlantic Equities in London.

After years of chasing revenues around the world, investment banks, especially Europe's, are cutting offices and products due to tougher rules and capital regulations.

"There was this whole dream to be global masters of the universe, flow monsters – but is it really relevant to be global?" said one senior investment banker, who asked not to be identified to avoid appearing to comment on behalf of his firm.

"Being global is brainwashing... The environment has changed, those who resisted most are having to do it now in pain. You have to trade growth for higher profitability," he said.

Investors now perceive running a global investment bank as "too complex to manage, too risky and too expensive," he said.

HSBC (>> HSBC Holdings plc), Europe's biggest bank, became the latest last week to scale back global investment banking ambitions, with plans to shrink the division by a third to boost profitability.

Tellingly, it said 70 percent of its revenues were made in markets where it held a top five position.

New bosses at Deutsche Bank (>> Deutsche Bank AG) and Credit Suisse (>> Credit Suisse Group AG) are also expected to cut back hard. Barclays (>> Barclays PLC), UBS (>> UBS Group AG) and Royal Bank of Scotland (>> Royal Bank of Scotland Group plc) already have.

The top positions for fees have long been dominated by the big five U.S. banks - JPMorgan (>> JPMorgan Chase & Co.), Bank of America Merrill Lynch (>> Bank of America Corp), Goldman Sachs (>> Goldman Sachs Group Inc), Morgan Stanley (>> Morgan Stanley) and Citigroup (>> Citigroup Inc). From Europe, Only Credit Suisse and UBS have briefly broken into the top five since the turn of the century.

Investment banking fees are not the only area where Europe is losing ground. U.S. banks have also increased their share in equities trading. In fixed income, currencies and commodities they have grabbed about 7 percentage points of market share from European rivals since 2010, led by gains by JPMorgan and Citi, Wheeler estimated.

U.S. banks' return on equity is lower than before the crisis, but at 12 percent is still almost three times higher than in Europe, according to consulting firm EY.

MIND THE REVENUE GAP

The twin challenge for Europe's banks is to make their investment banks profitable and make up for any revenue shortfall in other areas.

Deutsche Bank under new CEO John Cryan potentially faces the biggest challenge. The bank has ranked 6-8th in investment banking fees in the last decade, leaving it as Europe's last challenger as a top tier global firm after Barclays' retreat.

But with a return on equity of just 2.7 percent last year and shares trading at barely half of their book value, investors are growing impatient.

Cryan has limited options to fall back on, as German retail banking is an unattractive market and Deutsche is not a powerhouse in wealth management.

By contrast, UBS's big retreat from fixed income trading in 2012, and a tumble to 12th position in investment banking fees this year from 5th in 2008, allowed it to fall back on its core wealth management strength. Its shares have rallied 75 percent since its shift and trade at a 40 percent premium to book value, one of the highest bank valuations.

Bankers said the industry is likely to remain in flux for several years, although the declining trend in Europe appears well defined and unlikely to reverse soon.

"The question people are still asking is what is the impact of stepping back on other businesses. Does stepping back (in fixed income) impact M&A or equities or the investing clients?" Wheeler said.

(Editing by Peter Graff)

By Anjuli Davies and Steve Slater