News / Toronto

Karaoke-bot designed at UofT sings you a merry Christmas... sort of

Computer-vision program has amazing composition powers - and a few glitches.

Can you write a better Christmas carol about this photo than a robot?

Hang Chu via Vimeo

Can you write a better Christmas carol about this photo than a robot?

It’s the techno-idealists dream: A future where artificial intelligence is everywhere and robots listen to humans and speak back to us in a language we understand.

It’s not as far-fetched as it sounds. Thanks to a new AI project from three researchers at the University of Toronto, robots are learning the language of music.  

They built a bot that can look at a photo and compose a little song about it on the spot.  Here’s part of a carol it wrote about a picture of a Christmas tree:

The best Christmas present in the world is a blessing …
I’m glad to meet you
I can hear the music coming from the hall
A fairy tale
A Christmas tree
There are lots and lots and lots of flowers.

It’s no Mariah Carey-worthy hit, but it could almost pass for generic mall music — at least until the kooky tangent at the end.

The bot analyzed melodies and lyrics from pop songs and video games, along with a collection of photos and their captions, said Sanja Fidler, a UofT machine learning professor and one author of a forthcoming paper on the subject.

Using complex algorithms, it then taught itself to match visual cues in the photos with words, and, in a second step, set those words to music.

Fidler was quick to explain the program is a prototype. “I don’t think it sounds particularly awesome,” she said. (It should be said: There are no flowers in the photo).

The project builds on an earlier bot that can look at a picture of a beach, a mountain or the like and generate lyrics about it in the style of Taylor Swift.

In a month or so, Fidler’s team — which also includes computer science professor Raquel Urtasun and grad student Hang Chu, the lead author of the paper — hopes to release a browser-based demo. Anyone will be able to upload a photo and have the bot sing a song about it.

One day you might be able to feed a robot a photo of a special moment and have it compose a ballad for your beloved on the spot. And the karaoke machine of the future might make up songs as well as sing them.

“Everyone can be an artist,” Fidler said.

But as of today, the bot would flunk a Turing test: There’s no mistaking it for a human composer.

“Music conveys your emotions. You cannot get that from a machine,” Fidler said.  “We’re always hoping for that level of performance, but we aren’t there yet.”

More on