The 51-year-old banker, widely seen as the natural successor to Goldman Chief Executive Officer Lloyd Blankfein, once again finds himself caught up in another public relations fiasco involving the storied Wall Street investment bank.
On Wednesday, Cohn and Blankfein were cast as the villains in a scathing op-ed in The New York Times, in which London-based derivatives salesman Greg Smith said he was quitting the firm because he was sick of watching his colleagues "ripping off" clients on a regular basis.
Smith blamed Blankfein and Cohn for having "lost hold of the firm's culture" and letting it become something "toxic and destructive."
The angry rant quickly became a Wall Street sensation, bringing to mind Rolling Stone writer Matt Taibbi's comparison of the firm to a money-sucking vampire squid "wrapped around the face of humanity."
It also was a reminder that if Cohn does succeed Blankfein, the image problems that Goldman has fought to overcome these past few years may not go away anytime soon. That's in part because both Blankfein and Cohn come from the trading side of the Wall Street firm and are seen as standing arm-in-arm on big decisions for the past six years.
Indeed, on Wednesday there was little separation between the two men. In a letter to the firm's employees, in which they took issue with Smith's assertions, they said Goldman is committed to "the work it does on behalf of our clients."
William Cohan, a former investment banker and author of "Money and Power: How Goldman Sachs Came to Rule the World," said it appears Goldman's board remains in Blankfein's and Cohn's camp and sees public attacks like Smith's as misguided and irrelevant distractions.
"I think the board is of the view that the firm is just being targeted unfairly for political reasons and this is just something that is going to have to blow over," said Cohan. "For some reason they're on this Gary kick, probably because Lloyd has done such a great job of eliminating other people."
To be sure, Blankfein has not indicated any plans to step down anytime soon and there are plenty of rivals for the top job. In recent months, some of the most prominent contenders like Vice Chairmen J. Michael Evans and Michael Sherwood, seem to have been eclipsed by Cohn.
According to people familiar with the matter, Cohn has been in charge of implementing strategy around cost cutting and regulatory reform -- dubbed by some internally as "the new Goldman" -- and some members of his posse, including Isabelle Ealet and Justin Gmelich, have recently been promoted to higher positions within the trading business.
Still, the furor over the Smith op-ed was the second time in the past few weeks that Goldman found itself fending off allegation it puts profits ahead of clients.
Recently, a prominent Delaware judge lambasted Goldman for advising El Paso Corp on its sale to Kinder Morgan Inc when the investment bank's private equity arm had a 19.1 percent stake in Kinder Morgan and two employees on its board.
"The record suggests that there were questionable aspects to Goldman's valuation of the spinoff and its continued revision downward that could be seen as suspicious in light of Goldman's huge financial interest in Kinder Morgan," Judge Leo Strine wrote in a February 29 decision that allowed the deal to move forward.
Top Goldman brass were not blind to such criticism: while the two companies were still in talks, Blankfein called El Paso CEO Douglas Foshee and said Goldman was "very sensitive to the appearance of conflict."
But he and Cohn allowed the investment banking division to advise El Paso anyway, collecting an estimated $20 million fee in addition to its sizeable stake in Kinder Morgan, which has climbed 17.3 percent since the time the deal was announced in October, from $4.4 billion to $5.4 billion.
The two executives oversaw a similar conflict in 2010 related to buying and selling shares of pre-IPO tech darling Facebook in private transactions.
A unit in Goldman's investment management division passed on buying Facebook shares, saying the price was too rich, though the bank bought a $450 million stake itself with plans to sell another $1.5 billion worth of shares to wealthy clients. After the transactions drew scrutiny, Goldman limited investments to its foreign clients.
May be the most serious public lashing Goldman got came from in 2010 when the firm was accused by U.S. securities regulators and a Senate subcommittee of selling faulty subprime-mortgage linked securities to investors.
Goldman paid a $500 million fine to settle a lawsuit filed by the Securities and Exchange Commission over its marketing of collateralized debt obligations. And endured a public tongue lashing from Sen. Carl Levin over another CDO deal.
Goldman's strategy so far has been to keep a low profile and hope the uproar blows over. Indeed, some Goldman customers are saying Smith's critique isn't a big deal for them.
One longtime customer of Goldman's wealth management business said just about everything in Smith's letter was true. But he said you could say the same thing about any other big Wall Street investment firm.
This customer, who didn't want to be named, said Wall Street firms like Goldman long ago stopped being client centric after they went from private partnerships to publicly traded companies, which forces executives to focus ever more on the bottom line.
(Reporting By Lauren Tara LaCapra; edited by Matthew Goldstein, Jennifer Ablan, Edward Tobin)
By Lauren Tara LaCapra