The regulator's action comes when the British government is facing pressure over a housing shortage in parts of the country and rising prices have put home ownership out of the reach of many people.
The Financial Conduct Authority said it can be challenging for people to compare a vast array of home loans on offer with no consistent terminology for fees and charges. There are also concerns about the quality and suitability of mortgage advice.
The FCA is also looking at the impact of new rules it introduced last year, which require tougher checks by lenders on the ability of borrowers to pay back a loan.
"There is also anecdotal evidence, subject to much media interest, that some types of consumers are becoming unfairly trapped and unable to access more suitable deals," the FCA said.
"We are keen to ascertain how prevalent this problem is, and the extent to which the regulatory regime, the way in which it has been interpreted by firms, or other factors have played a role," it added.
Lloyds (>> Lloyds Banking Group PLC), Santander UK (>> Banco Santander, S.A.), Barclays (>> Barclays PLC) and Nationwide (>> Nationwide Building Society) account for more than half of Britain's 7 million mortgage holders.
Conditions for granting a home loan have become stricter following the financial crisis. There are fears that higher interest rates could make it even harder for young people to buy their first property.
Paul Smee, director general of the Council of Mortgage Lenders, an industry body, said the review is an opportunity to look at the effect of regulation, as well as market practice, on lenders and their customers.
"It's also essential in delivering the kind of environment in which reputable lenders of all shapes and sizes can thrive," Smee said.
The FCA is aiming to gather views before launching any full-fledged market study next year. Its review covers a wide range of home loans, including lifetime mortgages, shared ownership, buy-to-let, second charge mortgages and bridging loans.
It will also scrutinise activities and relationships within the sector, such as between lenders, brokers, valuation services and estate agents to see if they inhibit choice.
Britain has sought to boost competition more generally in banking but few of the new "challengers banks" offer home loans as their total deposits remain relatively small.
The FCA said it wanted to find out more about how access to funding for mortgage lending was affecting new entrants.
"We have a particular interest in the scope for newer types of funding models, such as peer-to-peer lending or crowdfunding, to overcome the constraints potentially imposed by other, more traditional, funding models," the watchdog said.
The FCA, which has powers to force changes in how the companies it regulates behave, said it would publish a feedback statement in the first quarter of 2016 and set out any further action.
"These views, together with evidence from the FCA's wider programme of work on mortgages, will help inform any future FCA work on this key sector of the economy, including any future competition market study," Christopher Woolard, FCA director of strategy and competition, said.
(Reporting by Huw Jones; Editing by Jane Merriman and Keith Weir)
By Huw Jones