LONDON (Reuters) - Royal Bank of Scotland (>> Royal Bank of Scotland Group plc) is close to settling a costly and potentially embarrassing case alleging it misled shareholders during a 12 billion pound fundraising at the height of the financial crisis, sources familiar with the talks said on Monday.

The state-owned British bank is in talks to settle with a group of investors to end legal proceedings that started five years ago and have been unprecedented in English legal history for their size and complexity.

Some large shareholders want to settle, meaning smaller retail investors would struggle to fund the legal fees needed for the case to proceed, sources involved in the group said.

"It is going to be hard for us to keep going," said one source, who asked not to be identified.

RBS, and the RBoS shareholder action group which represents around 9,000 private investors and 20 larger institutional claimants, declined to comment.

Trevor Hemmings, a multimillionaire businessman whom Reuters previously reported is one of the main financial backers of the claim, is advocating accepting the settlement offer according to two sources with knowledge of the situation.

A spokesman for Hemmings declined to comment on Monday.

The civil trial brought by the investors was due to open at the High Court in London on Monday but was adjourned for a day to allow the settlement talks to continue.

The plaintiffs allege former executives gave a misleading picture of the bank's financial health ahead of the cash call in 2008. Months after the cash call, RBS had to be rescued by the government with a 45.8 billion pound bailout.

RBS, which remains more than 70 percent state-owned, denies any wrongdoing over the 2008 rights issue and says its former bosses did not act illegally.

Jonathan Nash, a lawyer representing the claimants, appealed in court for an adjournment saying the two parties were in settlement talks and wanted longer to strike a deal.

"We are involved in settlement discussions and we are hopeful of making progress," Nash said.

The sources said RBS Chief Executive Ross McEwan was directly involved in talks over the weekend and that the bank had offered more than 80 pence for each RBS share held, though it was not clear if any investors have accepted the offer.

A settlement at that price would cost RBS "in the tens of millions of pounds", a third source familiar with the matter said.

The bank has settled with 87 percent of the investors who originally brought the case but the others have so far rejected its offers and say they were determined to go to court.

By doubling the amount on offer, RBS is close to a sum the remaining investors would accept, one of the sources said, indicating that they might settle if RBS raises its offer to 100 pence per share.

That represents half of the 200 pence per share investors paid at the time of the rights issue.

The outstanding group represents about 9,000 retail shareholders and 20 institutional investors. The large investors include U.S. bank Wells Fargo (>> Wells Fargo & Co), the Boeing (>> Boeing Co) pension fund, Bank of America Merrill Lynch (>> Bank of America Corp) and local British council pension funds.

(Editing by David Goodman/David Clarke/Alexander Smith)

By Andrew MacAskill and Lawrence White