Rock-bottom interest rates reduce insurers' investment returns, raising the risk of them having to eat into capital reserves to pay policyholders.

With yields on some government bonds -- staple investments for insurers -- turning negative recently, the International Monetary Fund (IMF) warned last month of a "high and rising" danger to weaker mid-sized European life insurers.

Calling for urgent action to tackle the issue, it cited 2014 data from the European insurance regulator showing 24 percent of the region's insurers may not be able to meet new minimum capital requirements if interest rates remain low for a protracted period of time.

Insurance stocks have until recently found favour with investors, hitting 14-year highs this year, partly because of their high dividends. But that has started to change.

"Expectations for dividends and buybacks have become more restrained in the last few months," said Andrea Williams, European equities fund manager at Royal London Asset Management.

Insurance watchdog chief Gabriel Bernardino told the Reuters Regulation Summit that insurers should exercise caution over dividends as they strive to meet new capital rules akin to those imposed on banks after the financial crisis.

The hit to solvency levels could mean some mutual insurers reliant on a single area of insurance need capital injections, according to Paul Traynor, international head of insurance at BNY Mellon.

The STOXX Europe 600 Insurance index <.SXIP> currently offers a dividend yield of 4.3 percent, against 3.4 percent for the broader FTSE Eurofirst <.FTEU3> index.

While no large listed insurers are expected to ask shareholders for more cash, any stress among smaller peers could damage confidence in the sector as a whole, the IMF said.

REVERSING COURSE

Some analysts are speculating that as the year goes on, listed insurers' capital reserves over and above those required by impending so-called Solvency II rules will shrink. The rules are due to go live in 2016.

As a result, many investors that had piled into insurance stocks as a safer, income-offering way of playing the financial sector -- given the regulatory and litigation uncertainties facing banks -- are now reversing course.

The European insurance index <.SXIP> has dropped 9 percent from its 14-year high since the IMF's report, while the banking sector has fallen only 3 percent <.SX7P>.

    Although the big insurers are expected to deliver on promises of cash made so far, hopes for greater returns in the future appear to have run their course, analysts said.

    "The more difficult environment and falling interest rates in particular have caused solvency ratios to decline, bringing them closer to the point where companies might be wary of paying out additional money," said Berenberg analyst Peter Eliot.

Barclays last week cut its rating on the European insurance sector to "negative" from "neutral," citing "significant and growing capital and earnings headwinds".

    "Earnings and solvency headwinds should combine to stall or even shrink dividends," it said.

    Underpinning concerns for future payouts is the state of the government bond markets, with many offering next to nothing to investors. Ten-year German Bunds have sold off in recent weeks, but are still paying yields of only 0.6 percent .

    Europe's regulator, the European Insurance and Occupational Pensions Authority, and national insurance supervisors may be wary of letting insurers pay out too much of their capital to shareholders while Solvency II beds down, analysts said.

    Big insurers such as Allianz (>> Allianz SE), Axa (>> AXA) and Generali (>> Assicurazioni Generali SpA) had hoped Solvency II would free up capital, for example by fully taking into account the diversification of their businesses as reflected in in-house capital models.

    But EIOPA has already told national supervisors to make sure those internal models include capital cushions against possible losses for sovereign bond holdings, an indication of the tight rein the watchdog will keep on insurer capital.

    "It's very unlikely that the regulators are going to be lenient," Berenberg's Eliot said.

The capital figures insurers publish as Solvency II draws nearer could be weaker than many investors expect, with regulators highlighting the lengthy liabilities to policyholders in countries such as Germany and Sweden.

Credit rating agency Moody's expects companies' final Solvency II ratios for the start of 2016 to be lower than their previous estimates.

JP Morgan calculated them at around 150-170 percent of Solvency II requirements rather than the 170-200 percent the market was expecting.

    And that difference is important.

Allianz, Europe's biggest insurer, cheered investors last year by saying it would raise its dividend payout ratio to 50 percent of earnings from 40 percent, but added the caveat that its Solvency II ratio must be sustainably above 160 percent. It was 192 percent at the end of March.

(Editing by Mark Potter)

By Jonathan Gould and Carolyn Cohn