NEW YORK (Reuters) - Royal Bank of Canada (>> Royal Bank of Canada) plans to boost its share in the U.S. investment banking market by tapping opportunities with existing mid-sized and large clients and filling gaps left by European banks that are scaling back, one of its key executives said.

With European banks like Deutsche Bank (>> Deutsche Bank AG), Barclays Plc (>> Barclays PLC) and Credit Suisse (>> Credit Suisse Group AG) toning down their investment banking strategies and in some cases playing a less aggressive role in the deal flow, Canadian banks are expanding their presence in U.S. capital markets.

U.S. investment bank Morgan Stanley (>> Morgan Stanley) said last year that it planned to cut up to 25 percent of its fixed-income jobs.

"We've got strong momentum. Some banks are having difficulties, and we’re well-positioned to capitalize on opportunities that arise," said Blair Fleming, head of RBC Capital Markets in the United States.

RBC, Canada’s biggest bank, has broken into the top ten in U.S. investment banking rankings. It advised on Dell Inc's $24.4 billion deal to go private in 2013 and the leveraged buyout of U.S. security company ADT Corp by private equity firm Apollo Global Management LLC (>> Apollo Global Management LLC) this year. (http://reut.rs/1UBoblc)

RBC stepped up investments in the U.S. market since the financial crisis and is seeing benefits of that push as the brand becomes more well known and the company targets bigger deals. It also expects its recent $5 billion acquisition of City National, a Los Angeles-based bank focusing on high-net worth clients, to open doors.

As the Canadian market becomes fairly saturated, the United States has become central to RBC's global strategy.

"In the U.S., we're 3 percent against a $40 billion fee pool," Fleming said in an interview, referring to the company's market share in U.S. investment banking. "That fee pool won't likely grow dramatically, but we feel we can take market share from 3 percent to 3.5 percent to 4 percent."

About 60 percent of RBC's capital markets revenue and net income is from the United States. Its U.S. capital markets revenue is double the capital markets revenue from Canada.

While RBC wants to grow, it is not eyeing a top 5 position.

"Breaking into the top 5 would require a lot of capital and taking significant risk – this isn’t something we aspire to," Fleming said. "Overall 7 to 10 is a reasonable spot for us to occupy."

The company is also looking to increase the proportion of revenue that comes from M&A to about 25 percent, he said.

"We've accelerated our growth of larger M&A opportunities," said Vito Sperduto, head of U.S. mergers & acquisitions at RBC. "For example, the number of deals that have been $5 million or higher in M&A fees has been consistently growing every year."

RBC is also looking to take advantage of top talent and could make key hires as the European banks change gears.

RBC, which has roughly doubled its bankers since the financial crisis, has hired bankers from Goldman Sachs (>> Goldman Sachs Group Inc), Citigroup (>> Citigroup Inc), Morgan Stanley (>> Morgan Stanley) and Bank of America (>> Bank of America Corp). Its U.S. capital markets staff headcount since 2009 has increased about 42 percent to 2,700.

"We provide a bulge bracket experience for the employee, but with a boutique mentality," Fleming said.

To be sure, the environment for investment banks is becoming more challenging. The year has seen few initial public offerings, companies with big debt piles are struggling to raise funding, trading is tough as investors are risk-adverse and the M&A levels are not expected to reach the records set last year.

Fleming said the higher level of regulatory demands will increase costs for RBC and noted taxes are higher than in Canada.

Still, RBC is looking to improve the unit's return on equity and bring it closer to where it is in Canada, he said.

Other Canadian banks have also been trying to crack the U.S. investment banking market in recent years. BMO has expanded aggressively since the financial meltdown, and TD said last year it was looking to double its U.S. capital markets business in three to four years.

(Reporting by John Tilak; Editing by Chizu Nomiyama)

By John Tilak