KUALA LUMPUR (Reuters) - Malaysia will scrap a proposed $20 billion (13 billion pounds) merger that would have created the country's biggest bank, after CIMB Holdings (>> CIMB Group Holdings Bhd) and RHB Capital (>> RHB Capital Bhd) failed to agree on new deal terms, people familiar with the matter said.

The merger would have formed a banking group with assets of about $190 billion, eclipsing Malayan Banking Bhd (Maybank) (>> Malayan Banking Berhad) and making it Southeast Asia's fourth-biggest bank. The state-backed deal was part of Malaysia's ambitious plan to promote its firms as regional champions.

The scrapping of the deal is a major blow for CIMB Chairman Nazir Razak, the brother of Malaysian Prime Minister Najib Razak. Nazir spearheaded the bank's expansion over the past 16 years, snapping up a domestic rival, lenders in Indonesia and Thailand and businesses owned by Royal Bank of Scotland (>> Royal Bank of Scotland Group plc).

CIMB shares had tumbled more than 26 percent between Monday and the launch of the deal in early October, as investors questioned CIMB's ability to extract synergy from the deal.

"We have been cautious on the mega-merger proposal, especially on synergy creation vis-a-vis the high merger cost," Tan Ei Leen, analyst with Kuala Lumpur-based Affin Hwang Capital, wrote in a note to clients on Monday.

The CIMB share price fall dimmed the deal's appeal, prompting RHB to seek an improved share swap ratio, and possibly cash, people familiar the with the situation said. A central bank directive against CIMB retrenching people for 24 months after the merger also posed difficulties.

On Tuesday, CIMB rallied as much as 11.4 percent as investors breathed a sigh of relief that the deal was finished. RHB gained 1.9 percent, while RHB was up 0.2 percent. Malaysia's benchmark share index <.KLSE> was up 0.4 percent.

An announcement cancelling the deal could come as early as Wednesday evening after CIMB board meeting, two people familiar with the matter said.

RHB said there were no developments in relation to the merger, while CIMB and MBSB declined to immediately comment.

The collapse of the deal raises concerns about Malaysian bank governance after state investor 1Malaysia Development Bhd (1MBD) [TERRN.UL] missed the repayment of a 2 billion ringgit ($563 million) bridge loan that was due end-December.

The bank troubles come as Malaysia's economy faces sharp falls in oil-related revenue, prompting Bank of America Merrill Lynch economists to warn last month the government might be forced to cut its gross domestic product growth forecast for 2015 to 4.6 percent from 5 percent in 2015.

(Additional reporting by Praveen Menon, Al-Zaquan Amer Hamzah and Saeed Azhar; Writing by Denny Thomas; Editing by Stephen Coates)

By Yantoultra Ngui