The SEC has not released any public explanation of the decision, which was disclosed on Friday by Falcone's publicly traded company Harbinger Group Inc.

But a person familiar with the matter said commissioners voted 3-1 to reject the proposed settlement during a closed-door meeting on Thursday due to concerns it was too weak.

Voting against it were SEC Chair Mary Jo White, an independent, and Democrats Luis Aguilar and Elisse Walter, the person said.

Among the concerns were the fact that the proposed settlement did not seek to bar Falcone from serving as an officer or director of a public company.

The SEC last year charged Falcone with market manipulation and other violations.

Republican Commissioner Troy Paredes voted for the settlement, while Daniel Gallagher, the other Republican commissioner, did not participate in the closed-door meeting.

The rejection of the settlement comes as White has said she intends to be more aggressive in dealing with settlements and to take more cases to trial.

On Friday the SEC initiated a blockbuster case against hedge fund mogul Steven A. Cohen by charging him with failing to supervise two employees who are accused of insider trading.

C. Evan Stewart, a partner at Zuckerman Spaeder who is not connected to the case, said it appeared the proposed Falcone settlement would have been effective in some regards but would have allowed Falcone to maintain control over investor funds.

"That mixed message really is why I thought it would not be well-received several months ago. It appears that consideration carried the day," Stewart said.

Falcone announced in May he would pay $18 million to settle two SEC lawsuits accusing him of market manipulation, giving preferential treatment to certain investors and borrowing cash from his own fund to pay his personal taxes.

The lawsuits were not filed against Harbinger Group Inc, the publicly traded investment company where Falcone is chairman and chief executive officer. Instead, they targeted his hedge fund Harbinger Capital Partners and other related funds.

Falcone manages roughly $3.1 billion in assets. He notified his hedge fund investors on Friday of the SEC's rejection in an email but did not provide any details beyond what was in Harbinger Group's regulatory filing, according to an investor in the fund who did not want to be identified due to fear of a reprimand from Falcone.

While the dollar amount of the settlement was relatively small, the deal would have required Falcone to return money to his hedge fund investors and would have effectively prohibited him from starting a new hedge fund for the next two years.

It would, however, have permitted Falcone to remain CEO of the Harbinger Group, a rare instance of leniency; such settlements usually include lifetime bans on being either a director or officer in a public company, according to legal experts.

"Historically it's uncommon for the commissioners to reject settlements," said Ron Geffner, a partner at Sadis & Goldberg who is not connected to the case. He said the commission could begin to reject deals more frequently.

"Since Madoff, and since the 2008 recession, Congress, other members of the government and the public have scrutinized the SEC's behavior and in terms of settlements or the outcome of certain lawsuits more than ever," he said.

Falcone's lawyer Matthew Dontzin did not immediately respond to a request for comment. Falcone did not respond to email and phone messages seeking comment.

The government had asserted that Falcone manipulated the price of some bonds his fund had invested in and at the height of the financial crisis, he let select investors get out while denying that same opportunity to others.

The SEC also said Falcone illegally loaned himself $113 million from the fund to pay his taxes, leaving investors unable to access their own money. Falcone eventually repaid the loan.

In March, Forbes listed Falcone as having $1.2 billion in personal wealth.

The SEC's vote took place a week after Harbinger Capital's bankrupt wireless communications firm LightSquared Inc announced plans to go to trial with its lenders over whether Dish Network Chairman Charlie Ergen's acquisition of big chunks of its loan debt violated an agreement pertaining to how LightSquared can restructure.

(Reporting by Emily Flitter in New York and Sarah N. Lynch in Washington; Additional reporting by Svea Herbst-Bayliss and Sarah N. Lynch; Editing by Matthew Goldstein, Phil Berlowitz and Tim Dobbyn)

By Emily Flitter and Sarah N. Lynch

Stocks treated in this article : DISH Network Corp, Harbinger Group Inc