The negotiated price is around $46 (29.51 pounds) per share, said one of the people who wished to remain anonymous because they are not permitted to speak to the media. That would value the deal at $6.55 billion.

Discussions are ongoing and the terms could still change, the sources said on Sunday.

The deal would be one of the largest leveraged buyouts so far this year, after Michael Dell teamed up with private equity firm Silver Lake Partners LP to take Dell Inc private for $24.4 billion. Silver Lake accounts for only a quarter of the equity in that deal.

Buoyant debt markets have encouraged private equity to consider larger deals, which in turn call for larger equity checks and make buyout firms more open to teaming up.

E-mails to BMC were not immediately returned. Bain and Golden Gate officials declined to comment.

Houston, Texas-based BMC Software shares ended trading at $45.42 on Friday. It competes with Oracle Corp, SAP AG, CA Inc and Compuware Corp, and has been under pressure from Paul Singer's activist hedge fund Elliott Associates LP to sell itself since last year.

BMC currently trades in line with its peers at around 11.5 times projected earnings, according to Thomson Reuters data. At an offer price of $46 per share, it would be slightly above the average at 11.6 times projected earnings.

Elliott, a seasoned investor in the software sector, has a 9.7-percent stake in the company and has argued that BMC's management was neglecting a huge opportunity to expand into Internet-based business software, a market dominated by the likes of Salesforce.com Inc.

The world's largest providers of software for enterprises, including Oracle, SAP and Microsoft Corp, have already begun investing heavily in that market. Since Elliott first announced an activist stake in BMC last May, the software company's shares have jumped over eight percent from $42 a share.

Elliott is a shareholder in business software company Attachmate Corp, alongside Golden Gate, Thoma Bravo and Francisco Partners.

BMC is due to announce first-quarter earnings on May 7. The company said on April 8 that it would reduce its workforce following a company-wide operational review. Those job cuts will result in pretax charges of approximately $33 million to $38 million for severance and related termination costs, it added.

(Additional reporting by Jessica Toonkel; Editing by Marguerita Choy and Stephen Coates)

By Greg Roumeliotis