(Reuters) - Family Dollar Stores Inc's (>> Family Dollar Stores, Inc.) shareholders approved the discount retailer's deal to be bought by Dollar Tree Inc (>> Dollar Tree, Inc.), scuppering a higher hostile offer from larger rival Dollar General Corp (>> Dollar General Corp.).

About 74 percent of the total outstanding shares of Family Dollar, the No.2 U.S. discount retailer, were voted in favor of the deal on Thursday.

Family Dollar accepted its smaller rival's cash-and-stock offer of $8.5 billion in July and later rejected a $9.1 billion all-cash offer from Dollar General, citing antitrust concerns. The biggest U.S. discount retailer then took its offer directly to Family Dollar shareholders.

Dollar Tree later threatened to walk away after Family Dollar shareholders twice postponed voting to approve the deal.

"... today was the drop dead day. Shareholders got nervous," Edward Jones analyst Brian Yarbrough said.

The Dollar Tree deal was the right one for shareholders as it has a better chance of passing regulatory muster, Family Dollar Chief Executive Howard Levine said, showing little emotion in discussing the sale of the company his father founded.

Dollar General CEO Rick Dreiling called the vote a loss for consumers and Family Dollar shareholders.

Levine said he expects the Federal Trade Commission to approve the deal by the end of March.

If approved, the deal will give Dollar Tree more than 13,000 stores across the United States and Canada, and more than $18 billion in annual sales.

Dollar General, which has about 11,500 stores, had revenue of $17.5 billion for the year ended Jan. 31, 2014.

Dollar Tree said it would stick to selling food items, household products and party goods at $1, while Family Dollar stores would continue to have multiple price points.

While Dollar Tree has suburban stores across most U.S. states, Family Dollar and Dollar General have a large presence in low-income neighborhoods.

They have been facing competition from the smaller format stores started by retailers such as Wal-Mart Stores Inc (>> Wal-Mart Stores, Inc.) and Target Corp (>> Target Corporation).

The combination would help Dollar Tree increase its clout with vendors, S&P Capital IQ analyst Levy Efraim told Reuters.

Family Dollar shares were up 1.3 percent at $76.50, while Dollar Tree rose 3.8 percent to $70.38.

Dollar General shares rose 4 percent to $69.90 after the company said Dreiling would continue as CEO through January 2016, unless a successor was appointed earlier. He was slated to step down at the end of May.

(Editing by Joyjeet Das)

By Sruthi Ramakrishnan and Ken Otterbourg