Get the Picture: How AI Helps Bing Search for Images Within Images

Search just got smarter, thanks to AI and NVIDIA GPUs.

Microsoft's Bing now lets you search for images within images. You can even buy items you find there.

[Attachment] Let's say you're a 'Fast and Furious' fan, and want to trick out your ride with the gear you've seen on the big screen. Or you're remodeling your living room, and see a photo with a glittering chandelier that would add just the elegance you crave.

Whatever has caught your eye in a photo, just draw a box around it. Bing's Visual Search then displays photos similar to your selection, where to buy it and, in many cases, what it will cost.

'We want to go way beyond the search box,' said Meenaz Merchant, who leads the Visual Search Group at Bing.

Merchant and his team will demonstrate Visual Search this week in Honolulu at IEEE's Computer Vision and Pattern Recognition (CVPR) conference, the premier annual computer vision event. If you're attending the conference, you can also attend a keynote talk on July 23 by Microsoft's Harry Shum, executive vice president for the company's Artificial Intelligence and Research Group.

How Bing Knows a Louis Vuitton

This new type of search is for more than shopping. You can upload your own photos or select anything online - an apple pie, a waterfall, a hotel - and Bing Image Search returns similar photos and tags that describe the search engine's notion of what's in the picture. Once you select a lookalike image, you may see useful information like recipes for apple pies, the location of the waterfall or the name of the hotel.

Visual Search is powered by deep learning so the more it's used, the more accurate it will become. For now, the related images sometimes look visually similar but aren't really the same thing. Search for that sporty yellow dress Emma Stone wore in 'La La Land' and you'll get more yellow dresses, but they're all evening gowns.

While Bing isn't the first to introduce search within an image, its advantage, said Merchant, is the vast collection of images from the Bing search index.

'If someone's holding a Louis Vuitton handbag, we can identify it because we've seen thousands of images of these,' he said.

The Future of Search

Merchant and his team used images from the Bing search index, along with our GPUs, to train the deep learning algorithm for Visual Search. All of the images had been identified or, in deep learning parlance, labeled.

Researchers provide a detailed technical explanation of how Bing Image Search works and the image understanding technologies behind it in this blog post. In addition to the image search on its website and mobile app, Bing released an API so developers can build Visual Search into their apps.

Merchant noted his team continues work to make searching easier. 'We're working to get better at this,' he said.

Soon, Bing Image Search will help you pick objects without needing to draw a box, Merchant said. He said Bing also plans to connect the objects identified to the Bing Satori and Web graph knowledge repository so when the search engine identifies a celebrity, for example, you can get detailed information about that person. When it's a product, Bing will show where you can buy it.

'Visual search is a fraction of text search today,' Merchant said. 'Now that everyone is carrying a high-resolution camera, it's going to explode over the next couple of years and maybe even surpass text.'