Virgin Media, the No. 2 pay-TV group in Britain behind Murdoch's satellite group BSkyB, has a valuation including debt of around $20 billion.
A deal would give Liberty entry to one of Europe's biggest telecom markets and allow it to apply lessons learned as a pay-TV and broadband provider in 11 other European countries.
Malone's Liberty could offer one of the sternest challenges to Murdoch as cable groups across the region start to assert their authority over traditional telecoms firms with the offer of super-fast broadband and pay-television.
Malone, whose group has 19.6 million customers, came up against Murdoch a decade ago when Murdoch's News Corp. and Liberty Media vied for control of DirecTV Group, the largest U.S. satellite TV broadcaster.
The stand-off ended when both sides backed off. News Corp. sold its one-third stake in DirecTV to Malone's group and Malone sold 16 percent of News Corp. that Liberty had acquired, giving the Murdochs fuller control over News Corp.
Buying Britain's second-biggest pay-TV company would put Malone up against Murdoch's satellite group BSkyB, which leads the British pay-TV market with 10.7 million customers compared with Virgin's 4.9 million.
It could also leave the combined group weighed down with heavier debts.
Virgin Media's bonds widened and the cost of insuring its debt rose on expectations that more debt would have to be raised to finance a deal. It is also rated higher than Liberty Global, which could impact its credit profile.
Virgin Media's main listing is in the United States but it has a smaller one in London, where the shares were up 14 percent in early trading.
News Corp owns 39 percent of BSkyB. Murdoch tried to buy the rest in 2010 but had to drop the $12 billion bid due to public outrage over a phone hacking scandal. Shares in BSkyB were down 1 percent against a FTSE 100 Index which was up 0.3 percent.
Virgin Media, which confirmed the talks with Liberty in a brief statement, has a market value of $10.6 billion and $9 billion of debt.
It made its first annual profit in 2011 after a turnaround engineered by Chief Executive Neil Berkett and a takeover could make Virgin more aggressive in the British market. Virgin reports its 2012 results on Wednesday.
"Virgin Media confirms that it is in discussions with Liberty Global, a leading international cable company, concerning a possible transaction," it said.
Analysts at Espirito Santo said they believed a fair enterprise value for Virgin Media would be around $24 billion, although they questioned how Liberty would pay for it.
Espirito put Liberty's net debt at 5 times its core earnings.
Virgin Media was formed by the merger of cable groups Telewest and NTL and mobile operator Virgin Mobile in 2006. It uses the brand of Richard Branson's Virgin Group and the British entrepreneur retains a 3 percent stake.
The company sells cable TV, telephony and broadband and also competes with a television service from BT called BT Vision and online offerings such as Lovefilm.
Its biggest shareholders are Capital World Investors which own 14.6 percent and Capital Research Global Investors which own 10.9 percent, according to Reuters data.
Virgin Media spent the first few years of its existence engaged in a long legal with BSkyB over access to content.
Berkett shunned that approach after he took over in 2008, settling the dispute and slowly building up Virgin Media's customer base by focusing on a superior broadband offering.
Its share price has responded in turn, soaring almost 160 percent since March 2008.
A deal to become part of the wider Liberty group would not necessarily result in huge synergies because there are few links between cable networks in different countries, but it could help produce savings in buying equipment.
Liberty has recently increased its stake in Belgian operator Telenet to 58 percent and owns other consumer brands across Europe including UPC, Unitymedia, Kabel BW and VTR.
It also owns a content and channels group called Chellomedia.
(Reporting by Kate Holton; Additional reporting by Josie Cox; Editing by James Davey and Tom Pfeiffer)
By Kate Holton