Wall Street bank JPMorgan Chase & Co risks being the biggest loser. It was sole adviser to Abbvie and had been set to earn $45 million in fees, a substantial portion of which were to be paid on completion, according to Abbvie's deal prospectus.

Shire's advisers -- Goldman Sachs, Morgan Stanley, Citi, Deutsche Bank and Evercore -- were set to share a fee pool of between $70 and $90 million, according to estimates from Thomson Reuters/Freeman Consulting.

It is not known how much could still be payable to advisors. Bankers will typically earn about 10 percent of the fee total if a deal fails, according to the consulting firm.

For example, British water company Severn Trent paid 19 million pounds ($30.4 million) in fees to its legal and financial advisers when it fended off a takeover bid by a Canadian-led consortium of investors in 2013.

"If a deal is hostile, defence advisers tend to get full fees," Lam Nguyen, director at Freeman told Reuters.

"But even though this deal was not hostile, the target's advisers might be liable to share a piece of the break-up fee but it really depends case-by-case", he added.

JPMorgan, Deutsche Bank, Goldman Sachs and Morgan Stanley declined to comment. Evercore and Citi could not immediately be reached for comment.

Abbvie was also set to pay $220 million in financing-related transaction fees, according to the prospectus, including $137 million for the establishment of an undrawn bridge facility arranged by JPMorgan.

AbbVie has recommended shareholders vote against the proposed $55 billion takeover of Shire after a change in U.S. tax rules.

The U-turn hands a major scalp to the U.S. Treasury, which has been fighting to make tax-avoiding acquisitions more difficult and could deter further dealmaking.

Abbvie is also liable to pay a break-up fee of about $1.64 billion if AbbVie's shareholders follow the advice and reject the transaction.

The Shire episode has also fuelled doubts about whether Pfizer, which abandoned a $118 billion bid for AstraZeneca in May after its offer was rejected, will ever make another run at its British rival.

Global investment banking fees in the first nine months of the year were at their strongest level since 2007, totalling $68.8 billion as a string of mega deals drove deal volumes up 60.1 percent to $2.66 trillion, Thomson Reuters data shows.

JPMorgan topped the global investment banking league table during the period with $5 billion in fees or 7.2 percent of overall wallet-share. (1 US dollar = 0.6257 British pound)

(Editing by Keith Weir)

By Anjuli Davies