Thanks to a new deep learning system called Caffe2Go, Facebook now lets users of its mobile app transform videos with artwork-inspired special effects in real time. The platform enables such on-the-fly style transfers by dramatically improving how efficiently artificial intelligence (AI) processes images and videos.
With the help of Caffe2Go, users of Facebook’s iOS or Android app can quickly change the look of their photos or videos in the style of Van Gogh’s paintings, for example (pictured above). Facebook engineers took on the challenge of developing a way to do that in real time on mobile devices only a few months ago.
Facebook CTO Mike Schroepfer described the development of Caffe2Go and other aspects of Facebook’s long-term AI plans today during a talk at the Web Summit conference in Lisbon. Those plans are part of the company’s wider ambitions to advance three technologies — AI, connectivity and virtual reality — over the next few years.
AI ‘in the Palm of Your Hand’
“Just three months ago we set out to do something nobody else had done before: ship AI-based style transfer running live, in real time, on mobile devices,” Schroepfer wrote in a blog post today. “This was a major engineering challenge, as we needed to design software that could run high-powered computing operations on a device with unique resource constraints in areas like power, memory and compute capability.”
The result of those efforts — Caffe2Go — uses an AI image- and video-processing model that is 100 times more efficient than previous versions, enabling users to apply special effects in real time via Facebook’s mobile app.
The use of high-efficiency deep neural networks allows some mobile devices to apply artificial intelligence to images at speeds of less than 1/20th of a second, according to Facebook research scientists Yangqing Jia and Peter Vajda. That’s significantly faster than the time it takes for a typical human eye blink: 1/3 of a second.
“This is all happening in the palm of your hand, so you can apply styles to videos as you’re taking them,” Schroepfer noted, adding that such advances also enable users to take advantage of gesture-based controls or facial recognition techniques. “With Caffe2Go, AI has opened the door to new ways for people to express themselves.”
Aiming for ‘Computers with Common Sense’
“Artificial intelligence has made a significant impact on computer science, but it’s been mostly limited to big data centers that are miles or maybe hundreds of miles away from people using AI-powered services,” Jia and Vajda wrote today on Facebook’s Code blog. “Caffe2 is also our first industrial-strength deep learning platform that ships at full speed on four platforms with the same set of code: server CPU, GPU, iOS, and Android.”
They added that Facebook plans to open-source parts of the AI framework for Caffe2Go over the coming months so developers can work on their own deep-learning-based applications for deployment across a range of devices.
Facebook is also continuing research into ways to help computers better understand context so they can glean human-like meaning from images and unstructured data, such as Wikipedia entries, as well as become better at predicting future actions, Schroepfer said.
“When our research succeeds in teaching computers all the abilities I outlined above — context, knowledge about the world, reasoning and predicting — these will add up to something like what we call common sense,” he said. “And when computers have common sense they can interact with us in better, more natural ways, from surfacing the most relevant information for us and assisting us with tasks to enabling whole new ways for people to connect.”
