TOKYO (Reuters) - When Prime Minister Shinzo Abe vowed at the World Economic Forum in Davos to take a "drill bit" to the "solid rock" of vested interests in reforming Japan's economy, executives at companies such as General Electric (>> General Electric Company) and IBM (>> International Business Machines Corp.) paid attention.

In the months that followed the January appearance at Davos, blue-ribbon panels sought out GE and IBM for advice on how to give companies more flexibility in hiring and pay.

But the labor market deregulation to be announced on Tuesday, as part of an economic package aimed at boosting investment in Japan and lifting growth, stops far short of the sweeping change foreign business leaders had sought.

That marks a setback at a time when investor optimism about the pace of "Abenomics" reforms has ebbed. The Nikkei <.N225> stock index, a closely watched indicator for Abe's closest advisors which rose more then 50 percent in 2013 as investors cheered the first blast of Abenomics, is down 5 percent this year.

"Changes in labor regulations will be incremental. It's a step forward, but it's not that big," said Robert Feldman, chief economist at Morgan Stanley MUFG in Tokyo.

The story of how Abe's promised deregulation faltered on job market reforms when confronted by resistance from Japan's bureaucracy and political controversy illustrates the challenge for Abe's "Third Arrow", a growth strategy intended to boost private investment and compliment earlier programs of fiscal stimulus and monetary expansion by the Bank of Japan.

Although Abe's reform panel sought ideas from companies such as Ikea [IKEA.UL] and Danone (>> DANONE) in recent months, the crucial decisions were made behind closed doors by about half a dozen Tokyo bureaucrats over a few days in June, according to interviews with 11 people who participated in the process.

The result: a pledge to end compulsory overtime allowance for workers earning more than the equivalent of $100,000 per year, a change affecting less than 4 percent of Japan's workers.

The more contentious question of how and whether to make it easier for companies to fire workers was left unanswered with the government pledging to find a "place for debate" on the issue next year. Both rules on overtime and firing will be further discussed in a council at Japan's Ministry of Health, Labour and Welfare, where unions have a strong voice and have made it clear they will fight to prevent change.

"It's true that right now it is only limited to the group which supports competitiveness the most, but I think it is still meaningful to offer them to be able to work according to new rules," said Makoto Murayama, the labor ministry official in charge of negotiations.

"We can't go ahead looking only at the economic impact, but also have to protect the workers and gain the understanding of the citizens."

LONG HOURS, LOW PRODUCTIVITY

At a London event targeting foreign investors in May, Abe seemed to promise a lot. "Japan's population is declining. In order to grow, we must raise productivity," he said in answer to a question. "We need to reform labor market regulation in order to make working conditions more flexible."

Abe's comments suggested that he was aiming at an overhaul of Japan's lifetime employment system. That entrenched practice gave workers secure jobs until retirement and was a feature in Japan's astonishing economic rise in the 1960s and 1970s. But it has been unraveling since an asset bubble burst in the 1990s and proponents of reform argue that a freer job market would lift growth and incomes.

"This is a system that was built up over 30 or 40 years bit by bit," said Junji Annen, Chuo University Law School professor who advises on one of Abe's reform panels. "No one thinks it can go on like this, but they are scared to change it."

The Japanese work long hours, but remain far less productive than workers in the United States or the European Union. Rules on firing are complicated and the outcome is uncertain for employers. In part because of the unpredictable costs of restructuring in a downturn, employers are reluctant to hire full-time. Almost 40 percent of workers are in a temporary class with limited protections and fewer benefits.

Between the start of the year and April, representatives of Abe's economic reform team sought ideas from foreign companies such as Procter and Gamble [PGEO.UL] and Unilever (>> Unilever plc) on the changes they wanted to see in labor regulation.

In March, John Rice, GE's vice chairman, appealed directly to an Abe-appointed panel on foreign investment to give employers more flexibility and make it easier for employees to move between companies.

"If labor mobility is high, productivity would go up and we could take on the challenge of managing through the business cycle," Rice said, according to a transcript of the Japanese translation.

IBM, which has clashed with Japanese unions over its dismissal of workers who received below-average appraisals, was also consulted, officials involved in the process said.

GE confirmed its executives talked to the government and provided the transcript of Rice's appearance at the panel. IBM Japan's spokesman Takeo Tamagawa declined to provide the details of consultations, but said that two main issues discussed with the government were the flexibility of working arrangements and a working culture in which workers were evaluated based on performance, not the time spent at work.

'NO OVERTIME'

Japan's rigid labor laws allow almost all workers to claim overtime if they work more than eight hours a day or 40 hours a week, with a narrow exception for executives and some occupations.

Labor unions say the rules prevent abuses and "karoshi", the coined-in-Japan term for death by overwork.

"The people who die by overwork in Japan are mostly white collar workers," said Yosuke Minaguchi, a lawyer specializing in labor law. "These are not laborers toiling in factories but they also need protection."

For his part, Abe has wanted to take on overtime rules since his first brief stint as prime minister in 2006 and 2007, officials involved say.

But a plan to carve out a "white collar exemption" for overtime in Abe's first term met strong opposition and contributed to a decline in his support ratings. This time, advisers say, he wanted to make sure that any reforms would not be characterized as an attempt to strip hard-working voters of overtime that could cost him support.

"The prime minister had one request. To clarify to the public the difference between this proposal and the previous one," said Futoshi Nasuno, a Ministry of Economy, Trade and Industry official who prepared drafts of the policy proposals.

After the consultations that included IBM and GE, the advisory panel proposed two kinds of exemptions from overtime allowance – one for mid-level managers and creative workers and another based on income for high performers earning more than about $100,000 per year, a threshold taken from a precedent in the United States.

Then, in late April, Asahi Shimbun, Japan's second-largest newspaper, ran a front page story that raised questions about whether the exemption was so broad it would amount to a "no overtime pay" policy. The leak of the plan put the advisory panel on the defensive, officials involved said.

Wary of a public backlash, Trade ministry officials from the government's reform taskforce decided to narrow their demand to seek an exemption only for those earning more than $100,000. The exemption was so limited that officials decided not to even run an economic simulation of what it would mean for growth or income.

Around the same time, trade ministry officials who were pushing reform were running up against resistance from the labor ministry on the wording of the final policy plan.

"We were pushing to get specific dates and implementation periods for our projects into the bill. We asked for concrete words like 'establish' and 'create'. The labor ministry would come back with 'consider' or 'study' and vague commitments for implementation," Akiko Sugahara, who joined the government reform taskforce from business lobby group Japan Association of Corporate Executives told Reuters.

Foreign investors had asked for changes to Japan's labor law, which gives broad discretion to judges in settling dismissal cases. As a result, labor-related cases take on average, twice as long as other civil cases, court data show.

In the end, the Abe plan commits only to further debate on a "dispute resolution system" by stakeholders. Labor unions are already girded for a fight.

"We don't need this new system," Nobuyuki Shintani, director at the Japanese Trade Union Confederation, Japan's most powerful labor union, told Reuters.

(Editing by Kevin Krolicki and Alex Richardson)

By Antoni Slodkowski