U.S. and European banks have paid $230 billion in litigation costs since 2009 and could pay out another $70 billion by the end of 2016, mostly from the 20 largest European banks, they said in a research note on Tuesday.

European banks have paid out about $104 billion so far and the $52 billion they still have to pay, much of it related to foreign exchange trading and U.S. mortgage mis-selling, could restrain how much they pay in dividends, the analysts said.

The fines and compensation in the last five years are related to practices that include alleged manipulation of benchmark interest rates and mis-selling of mortgages in the United States and insurance in Britain.

Regulators fined six banks $4.3 billion in November after traders tried to manipulate foreign exchange markets.

"FX settlements underscore (the) need to prove culture and business models are transformed before returns and payouts can rise," analyst Huw van Steenis said in a note.

RBS, majority owned by the UK government, will have to pay another $10.6 billion on top of the $12.6 billion already paid or provisioned for, Morgan Stanley estimated.

The analysts predicted Barclays could have to pay another $8.3 billion, HSBC $7.7 billion, Lloyds $6.1 billion and Germany's Deutsche Bank $5.1 billion.

They estimated that future litigation costs for European banks would include $7.5 billion related to alleged foreign exchange rigging, $6.5 billion from interest rate benchmarks Libor and Euribor and $9.4 billion related to U.S. mortgages.

U.S. banks are more advanced in their litigation payouts, the analysts said. Five major U.S. banks have paid out $128 billion and are forecast to incur another $18 billion.

JPMorgan analysts this week also said British banks faced additional litigation provisions. They forecast the big four banks faced 15.1 billion pounds ($22.8 billion) of extra provisions for litigation in the next two years, to add to 11.6 billion pounds of reserves they already have set aside for such payouts.

($1 = 0.6615 pounds)

(Reporting by Steve Slater. Editing by Jane Merriman)