(Reuters) - Family Dollar Stores Inc (>> Family Dollar Stores, Inc.) rejected a $9 billion buyout offer from Dollar General Corp (>> Dollar General Corp.) that it said could run foul of competition law, opting instead for a smaller bid from Dollar Tree Inc (>> Dollar Tree, Inc.) that guarantees a job for its CEO.

Family Dollar, the second-largest dollar store in the United States, said it believed a deal with its larger rival would be unlikely to win antitrust approval, despite a promise by Dollar General to close up to 700 stores.

"We will not jeopardize the Dollar Tree deal for a transaction with Dollar General that has a high likelihood of not closing due to antitrust considerations," said Ed Garden, an independent director on Family Dollar's board and co-founder of Trian Fund Management LP, a large shareholder.

Trian Fund Management, led by activist investor Nelson Peltz, owned 7.3 percent stake in Family Dollar as of July 27.

The formal rejection came a day after Dollar General Chief Executive Rick Dreiling questioned whether his counterpart at Family Dollar, Howard Levine, was serving his own interests in supporting Dollar Tree's $8.5 billion offer.

As friction escalated, Levine responded on Thursday by saying that Dreiling's letter "contained blatant mischaracterizations and did nothing to address the antitrust issues in Dollar General's proposal."

Dollar stores have been a popular choice for U.S. consumers in a weak economy. Competition to woo penny-pinching customers has intensified, particularly as Wal-Mart Stores Inc (>> Wal-Mart Stores, Inc.) has opened more small-format stores.

The U.S. market for dollar stores grew 46 percent to $48.2 billion between 2008 and 2013 and is expected to grow 18 percent in the next five years, according to Euromonitor International.

A combined Dollar General-Family Dollar would have nearly 20,000 stores across 46 U.S. states and annual sales in excess of $28 billion.

Dollar Tree and Family Dollar combined would have 13,000 stores across the United States and Canada and $18 billion in annual sales - enough to vault it ahead of Dollar General as the biggest discount retailer in North America.

Diana Moss, vice president of the pro-competition American Antitrust Institute, said a union between Dollar General and Family Dollar would create a company that controls the market.

"That's going to raise some eyebrows," she said. "I can see this will get some heavy scrutiny."

Rahul Sharma, managing director of investment advisory Neev Capital, said antitrust issues were not insurmountable given the much bigger size of retailers such as Wal-Mart and Kroger Co (>> The Kroger Co.), which also cater to low- and middle-income consumers.

Combined, the three dollar chains' annual sales total about $35 billion – less than a tenth of Wal-Mart's $473 billion.

"It is not clear to us why the antitrust concerns could not be resolved via methods such as store divestitures," said S&P Capital IQ analyst Efraim Levy. "It might be that antitrust was not the full reason."

'UNFRIENDLY'

Certainly, the Dollar Tree cash-and-stock bid is friendlier to Family Dollar's management. Levine, son of Family Dollar's founder, would remain CEO of the Matthews, North Carolina-based company were it to be bought by Dollar Tree.

Were Dollar General to buy Family Dollar, Levine is expected to lose his job, although this has not been confirmed.

"The Dollar General offer, while it clearly makes sense in the form of synergies, was unfriendly to Family Dollar," said Joan Storms, retail analyst at Wedbush Securities Inc.

Activist investor Carl Icahn, who has whittled down his stake in Family Dollar to 3.6 percent as of July 30 after buying 9.4 percent, has also questioned whether Levine's future could have swung the decision to favor Dollar Tree.

There are reasons to argue that Dollar General and Family Dollar might be an easier fit. Both offer goods at multiple price points, while Dollar Tree sticks to a $1 or less format.

Family Dollar's presence is biggest in Texas and the eastern United States. Texas is also home to more Dollar General stores than any other state - more than 1,000 stores, or about a tenth of the total.

But divesting 700 stores, as Dollar General has promised, might not be simple, a former antitrust lawyer in the federal government, who declined to be identified, told Reuters.

"The wild card," he said, would be finding "a purchaser who is a viable competitor."

Most analysts said they expected a renewed offer from Dollar General.

"This may be a negotiating tactic for (Family Dollar) to get a better price," said Claudia Higgins, antitrust partner at Kaye Scholer LLP. "Antitrust is very often a part of the negotiating picture."

Shares of Dollar General were down 0.2 percent at $63.63 in afternoon trading on the New York Stock Exchange, while Family Dollar's shares were down 0.3 percent at $79.54.

Shares of Dollar Tree, which also reported a 2.6 percent fall in quarterly profit as its costs rose, were down 1.4 percent at $54.25 on the Nasdaq.

(Writing by Robin Paxton; Editing by Savio D'Souza and Kirti Pandey)

By Sruthi Ramakrishnan, Devika Krishna Kumar and Lawrence Hurley