LONDON (Reuters) - Royal Bank of Scotland Group reported its first annual profit in a decade on Friday, but the milestone was bitter-sweet for the British bank which had wanted to resolve a multibillion-dollar misconduct case in 2017.

The bank is hopeful the case with the U.S. Department of Justice (DoJ) for mis-selling toxic mortgage backed securities can be settled in the coming months, RBS chairman Howard Davies said.

"That is our hope," Davies said at a presentation for journalists. "We continue to do whatever we can to try to advance a settlement but we do not have any news to offer... today on that."

RBS shares were down 4.2 percent at 270.2 pence by 1400 GMT on the lack of information about a case which some analysts estimate could mean a charge of up to $12 billion.

The bank's 752 million pound ($1.05 billion) profit beat a company-provided average of analyst forecasts for a 592 million pound loss, with a number having factored in a provision for the DoJ fine.

Chief Executive Ross McEwan said the DoJ settlement was out of the bank's control but it could now begin to think about resuming dividends or buying back shares nearly a decade on from its 45.5 billion pound state bailout during the financial crisis.

"With many of our legacy issues behind us, the investment case for this bank is much clearer and the prospect of returning any excess capital to shareholders is getting closer," he said.

McEwan, a New Zealander who has run RBS since 2013, later told reporters that he plans to stay at the bank until it has settled the U.S. case and resumed payouts, which would likely take the form of a small dividend initially.

The DOJ issue weighs on RBS's share price and complicates the government's plan to sell down its 71 percent stake.

DIFFICULT READING

RBS has in recent months been at the centre of renewed political furore over the mistreatment of thousands of business customers during and after the financial crisis.

This has hamstrung McEwan's efforts to rebuild the bank's image and he admitted on Friday that a confidential report into the bank's conduct, recently made public and detailing behaviour by RBS staff, made for "difficult reading".

The CEO's pay fell in 2017 as a result of the bad publicity around the bank's brand image.

RBS took 764 million pounds of provisions in the fourth quarter for conduct issues such as its mis-selling of payment protection insurance, which came in at 175 million pounds. Another 492 million pounds went to other cases related to the bank's mortgage-backed security mis-selling.

Restructuring costs were 531 million pounds for the quarter and 1.6 billion for the year. RBS said it would invest 1.5 billion pounds more than previously forecast in the next two years, around 800 million pounds of which will go towards digitisation.

"Investors are likely to be frustrated by the 1.5 billion pounds of incremental restructuring costs needed to deliver accelerated investment spend," Joe Dickerson, an analyst at Jefferies who has a 'buy' rating on RBS shares, said.

RBS said it had beaten its overall cost reduction target of 750 million pounds in 2017, taking out 810 million pounds.

(Reporting by Emma Rumney and Lawrence White; Editing by Alexander Smith and Keith Weir)

By Emma Rumney and Lawrence White