Ross McEwan had said several times this year he expected to reach a settlement in 2017, a key step in allowing the bank to return to full-year profit in 2018. Analysts have estimated the bank could pay up to $12 billion to settle the case.

"There are diminishing chances we settle in the year," McEwan said in the interview.

RBS had no further comment.

McEwan has been trying to clean up RBS's balance sheet and end an array of legal cases so the UK government can sell the more than 70 percent stake in the bank it obtained via a 46 billion pound ($60 billion) bailout during the financial crisis.

The case with the U.S. Department of Justice (DoJ) is the last major such problem remaining, after RBS in July paid the U.S. Federal Housing Finance Agency $5.5 billion to settle similar claims.

RBS has not made an annual profit since 2007. It has forecast it will in 2018, contingent on settling with the DoJ.

PROFITS JEOPARDIZED

If the deal is delayed into 2018, as is now likely, it could jeopardize the bank's plan to return to profit at a time when Britain's government is preparing to offload its stake in RBS.

Jefferies analyst Joe Dickerson said he expected the bank will need to make an additional provision $2.5 billion to cover the settlement. If shifted into 2018, that would wipe out much of the 3.2 billion pounds of profit the bank is forecast to make, he continued.

However he added that if the bank got some certainty on the figure it could still make the provision before its full-year results on Feb. 23. The bank already has $3.1 billion set aside.

Talks with the DoJ initially stalled due to staffing changes in the U.S. government following the election of President Donald Trump.

McEwan said there had been no substantial discussions with the department when reporting the bank's third quarter results in October, but that he remained hopeful of settling this year and some preliminary conversations had taken place.

This had seemed optimistic to some. Dickerson said most market participants had been expecting a settlement some time between now and the first half of 2018.

"It would be positive if this deal could be settled within that timeframe from the standpoint of returning capital to shareholders," he said.

RBS shares were up 1.5 percent by 1052 GMT, against a broader 2.8 percent climb in the STOXX European banks index <.SX7P>.

(Reporting by Emma Rumney and Lawrence White; Editing by Gareth Jones and Mark Potter)

By Emma Rumney and Lawrence White