Living the Meme: AI as Funny as Humans for Generating Image Captions

It's possible to get grad-school credit for writing memes. At least if you use deep learning to do it.

Just ask Lawrence Peirson.

The 23-year-old is pursuing a theoretical astrophysics Ph.D. at Stanford, but decided to enroll in a couple AI courses this year. He and classmate E. Meltem Tolunay came up with a neural network that captions memes for a class project, now published in a whitepaper aptly titled 'Dank Learning.' ('Dank,' for the uninitiated, is a synonym for 'cool.')

There are lots of examples of training deep learning models to produce literal captions for an image - for example, accurately captioning an image as 'man riding a surfboard' or 'child with ice cream cone.' With memes, Peirson's challenge was to see if a neural network could go beyond literal interpretation and create humorous captions.

Though he was initially skeptical that the memes would be funny, Peirson found that the deep learning model produced 'some quite interesting and original humor.'

Attaining Deep Meme

The deep learning network captioned this meme, a variation on the popular advice animals template.To collect training data for the deep learning model, Peirson scraped around 400,000 user-generated memes from the website memegenerator.net. The site provides meme templates and allows users to come up with their own captions.

The dataset included around 3,000 base images, each with many different captions. Since the input data was crowdsourced, there was a wide range in quality of meme captions the deep learning model processed.

'With 400k memes, most aren't going to be that funny, but at least they teach the system what a meme is, what joke is relevant,' he said.

Internet memes have circulated around the web for years, with a strong foothold in websites like Reddit, Facebook, 9GAG and Quick Meme. The most popular can get more than 2 million unique captions created.

Memes often reference pop culture, current events or esoteric bits of a particular internet subculture. (Peirson runs a meme page called 'The specific heat capacity of europium at standard temperature and pressure.')

As a result, they imbibe both the good and bad of digital culture - the paper notes a bias in the training data towards expletive, racist and sexist memes. Peirson sees the need to filter these out in future work, but points out that 'it's a big problem in natural language processing in general,' not one specific to memes.

The deep learning model was programmed in CUDA and used an NVIDIA TITAN Xp GPU. Peirson and Tolunay tried using both unlabeled data and data labeled with the meme title (for example, success kid or trollface), but saw no significant difference in meme quality.

'They're very funny in a 'it sort of makes sense, but not really' way,' Peirson said. 'Memes lend themselves to that kind of humor.'

One Does Not Simply Declare a Meme Dank

To evaluate the deep learning model's success, the collaborators calculated a perplexity score, which checks whether the neural network can identify clear patterns in the data. They calculated this metric for a few hundred memes with preset formats, such as the Boromir meme, which always begins with the phrase 'one does not simply.'

But the true test of a meme is whether it's funny.

In a qualitative survey, Peirson and his co-author presented people with a human-generated and a deep learning-generated meme, side by side. They asked two questions: whether the subject thought the meme was created by a human or computer, and how the subject would rate the meme's humor.

The data shows the deep learning memes 'were fairly indistinguishable from the real memes,' Peirson says.

They also investigated how the neural network would caption images that were not among the templates in the training dataset. In these cases, the algorithm infers patterns from the unknown image based on what it has seen in the training data. Peirson even showed the deep learning system a photo of his own face to test this, with entertaining results.

Memes often go viral, and it seems meme-themed whitepapers are no exception. Peirson says he was 'extremely surprised' by the media coverage and wide interest in the project. A complementary mobile app, also titled Dank Learning, will soon be available on the App Store.

This project, he says, has given him a new perspective on how powerful memes can be. Millions of users worldwide consume memes daily on social media sites.

Peirson sees the potential for a strong AI to produce memes 'at a whim' on current events to influence public opinion - or for advertisers to use memes for brand awareness: 'Having this go viral is an incredible way to market.'