TORONTO (Reuters) - Canada's largest bank and some of its most influential fund managers plan to set up a new stock exchange to challenge the dominant TMX Group Ltd (>> TMX Group Ltd), one that would limit the role of controversial high-frequency trading strategies.

The new exchange operator, Aequitas Innovations Inc, will be backed by Royal Bank of Canada (>> Royal Bank of Canada) and other institutions not involved in the 2012 takeover of TMX, which operates the Toronto Stock Exchange.

Its founders said on Tuesday that the new exchange, expected to launch in late 2014, will cater to retail and institutional investors who they believe have been short-changed by predatory high-frequency trading practices.

Once launched, Aequitas would be a direct competitor to the TMX Group, which also operates Canada's main small-cap and derivatives exchanges. TMX handles roughly 80 percent of equity trading by value in Canada. The exchange operator said high- frequency trading makes up around 15 percent of its volume.

High-frequency traders use sophisticated algorithms to trade thousands of shares in a millisecond with the aim of earning a profit from market making and price imbalances. But many players, including some large fund managers, have criticized their impact on markets.

"I would call it a grassroots reaction from some key market stakeholders saying we need choice, more choice in the marketplace," said Jos Schmitt, chief executive officer of Aequitas, who headed TMX rival Alpha Group before it was taken over by the larger exchange operator.

Schmitt said the long-term goal is for the new exchange to grab 20 percent of the Canadian equity trading market.

Aequitas also plans to launch a secondary market for shares of private companies, which it said will increase funding opportunities for start-ups and give early investors in these companies an exit option.

Schmitt said Aequitas would shy away from a now-common fee structure in which one side of a trade - typically an algorithmic trader - receives a rebate for placing an order. Exchanges often cover the cost of these with higher fees on the other side of the trade, in many cases retail or institutional brokerages.

Tom Caldwell, the chairman of Toronto's Caldwell Securities and TMX rival Canadian National Stock Exchange, said he does not see Aequitas as a big challenger, but expects they will come out with some innovations.

"If I had to stereotype this - I think it is a big company solution for big companies. So I think the primary focus is going to be around going head-to-head with the TMX by providing better filter-out systems for some of the abuses of high-frequency trading," he said.

TMX SHARES SLIP

Asked to comment on the potential rival, TMX said in a statement it was prepared and well-positioned to compete. Its shares were off 1.4 percent at C$43.20 on Tuesday afternoon in a broadly rising market.

"It might put some pressure on TMX shares closer to launch as the news flow around it picks up," said Andrey Omelchak, a portfolio manager with Montrusco Bolton, which owns 271,000 shares in TMX. "But they are not in a position at all to affect TMX's business in any way that I can envision."

BMO Nesbitt Burns analyst John Reucassel wrote in a note to clients that Aequitas would likely not hinder the TMX's growth prospects in clearing, derivatives, listings and market data.

He also said that market share gains for the new entrant would be limited by the absence of the big investment dealers who back TMX Group.

TMX is controlled by a consortium of Canadian banks and other financial institutions that thwarted a takeover bid by London Stock Exchange Group (>> London Stock Exchange Group Plc). The consortium, known as Maple Group, then combined the Canadian exchange operator with Alpha and a trading clearinghouse.

RBC, the country's biggest lender, was one of the few Canadian banks not involved in the consortium, as it had advised the London Stock Exchange on its bid.

In addition to RBC, Aequitas is backed by Barclays Plc (>> Barclays PLC), Canadian mutual fund managers CI Financial Corp (>> CI Financial Corp) and IGM Financial Inc (>> IGM Financial Inc.), pension fund PSP Public Markets, and Investment Technology Group Inc (>> Investment Technology Group).

"This is the first marketplace in existence that has more investors around the boardroom table than it does brokers," ITG Canada's CEO, Nick Thadaney, said in an interview.

(Additional reporting by Bhaswati Mukhopadhyay in Bangalore; editing by Jeffrey Hodgson and Matthew Lewis)

By Alastair Sharp and Euan Rocha