Shares of BMC jumped as high as 9 percent on the news and were trading up 4.1 percent at $45.71 in early afternoon trading, giving the Houston, Texas-based company a market value of around $6.6 billion.
KKR & Co LP and TPG Capital LP have formed a consortium, the people said on condition of anonymity because the process is confidential. Bain Capital LLC and Golden Gate Capital have separately also teamed up for the auction, the people added.
Thoma Bravo LLC is participating in a third buyout consortium, one of the people said. None of the people disclosed how much the private equity firms would be willing to pay for BMC.
The process is now past the first rounds of bids and management presentations are taking place, the people said, adding that final bids are expected in the next few weeks.
A BMC Software spokesman declined to comment. KKR, TPG, Golden Gate, Bain and Thoma Bravo also declined to comment.
BMC, which competes with Oracle Corp, SAP AG, CA Inc and Compuware, was under pressure from Paul Singer's activist hedge fund Elliott Management to sell itself last year.
The company eventually said it had weighed strategic options and had decided to buy back $1 billion in stock.
BMC executives admitted in 2012 that the company had been "late" to latch onto an industry shift toward Internet-based software, also known as "software-as-a-service".
Activist investor Elliott Management argued last year that management was neglecting a huge opportunity to use their large installed base to expand into Internet-based business software, a market then dominated by the likes of Salesforce.com Inc. The world's largest providers of software for enterprises, including Oracle, SAP and Microsoft, had already begun investing heavily in that market.
BMC needed a board with a fresh approach in order to keep up, Elliott argued. The investment firm also pointed out significant scope to trim headcount and create a more efficient business.
Elliott owned 9.6 percent of BMC as of January 30, according to a regulatory filing.
WHAT'S ITS FUTURE?
In January, BMC forecast a lower-than-expected profit for 2013 after reporting third-quarter results below Wall Street estimates due to lower license bookings at its two main divisions: enterprise services management and mainframe service management businesses.
At the end of trading on Wednesday, BMC's enterprise value stood at 6.2 times its projected 12-month earnings before interest, tax, depreciation and amortization, compared to a 7.1 times average for its peer group, according to Thomson Reuters data.
"The buyout of BMC is unlikely to happen at a materially higher level from the current (share) price ... BMC's business has been suffering as customers are concerned about its future," Mizuho Securities USA Inc analysts wrote in a note on Thursday.
Elliott Management had signed a standstill agreement with BMC last summer that is set to expire on March 23 and prevented the New York-based hedge fund from making a bid or nominating board directors without the target's prior consent.
In a regulatory filing on Wednesday, BMC extended by two weeks the time window for shareholders to submit board nominees and items for the annual meeting. The period to submit proposal will now run from April 6 to May 6.
Rival Compuware Corp, which has rejected a $2.3 billion bid from Elliott, is also exploring a sale and talking to buyout firms to gauge takeover interest, people familiar with the matter said last month.
Elliott is an investor alongside Golden Gate, Thoma Bravo and Francisco Partners in business software company Attachmate Corporation.
(Reporting by Greg Roumeliotis, Nadia Damouni and Soyoung Kim in New York; Editing by Maureen Bavdek, Tim Dobbyn, Bernard Orr and Nick Zieminski)
By Greg Roumeliotis, Nadia Damouni and Soyoung Kim
Stocks treated in this article : BMC Software, Inc.
, CA, Inc.
, Compuware Corporation
, salesforce.com, inc.
, KKR & Co. L.P.
, Microsoft Corporation
, Oracle Corporation
, SAP AG