In a lengthy blog post about Facebook's ten-year plan to "accelerate innovation and power new experiences with AI," chief technology officer Mike Schroepfer mentioned that one update coming to the company's mobile apps will be a "style transfer" tool that turns normal photos and videos into works of art using "high efficiency neural networks," all running directly on iOS and Android smartphones.

Facebook's impending update will, according to Schroepfer, run entirely on the user's smartphone and not be dependent on the content to be sent to servers, creating long load times and frustrating users. The CTO described this as the most demanding and "technically difficult" hurdle to clear in the process of adding the feature to the company's mobile apps, but Schroepfer said the company has done just that, and the result is a deep learning platform called "Caffe2Go."

facebook-art-filters

Just three months ago we set out to do something nobody else had done before: ship AI-based style transfer running live, in real time, on mobile devices. This was a major engineering challenge, as we needed to design software that could run high-powered computing operations on a device with unique resource constraints in areas like power, memory and compute capability. The result is Caffe2Go, a new deep learning platform that can capture, analyze and process pixels in real time on a mobile device.

We found that by condensing the size of the AI model used to process images and videos by 100x, we’re able to run deep neural networks with high efficiency on both iOS and Android. This is all happening in the palm of your hand, so you can apply styles to videos as you’re taking them.

Schroepfer said that the alternative of sending the content to data centers to be analyzed and filtered was "not ideal for letting people share fun content in the moment." In addition to basic image and video filtering capabilities, the new deep-learning platform will also feasibly be able to understand gesture controls when taking a selfie, for instance. In today's blog post, an example is given of a user swiping right and left between various artistic filters for a selfie (all running live, in real-time) and even snapping a picture when the user smiles.

As a comparison, the update sounds largely similar to Prisma, an app that launched over the summer and impressed many with its ability to turn photos, and eventually videos, into stylized images. Originally, Prisma used a server-side combination of neural networks and artificial intelligence to apply the different filters to user photos, but an update also introduced offline image processing, meaning users could use some of the app's filters to alter their images right from their smartphones.

For Facebook, the announcement follows a year of video-first announcements from the company, most recently embodied in CEO and co-founder Mark Zuckerberg's plan to make the camera more prominent in the app. No specific timeline was given for a possible launch of the new features on Facebook's mobile apps, but the company is clearly looking to lay the groundwork for its future, calling its new AI initiative, along with virtual reality, "new technologies that will shape the next decade."

Top Rated Comments

jmh600cbr Avatar
65 months ago
Facebook - the startup eater
Score: 3 Votes (Like | Disagree)
Zorn Avatar
65 months ago
Wow prisma filters! Sounds like a cool feature no one asked for. When will it stop looking like mud on the iPad Pro that's now a year old?
Score: 2 Votes (Like | Disagree)
-BigMac- Avatar
65 months ago
Yeh man keep beefing and slowing that sluggish app up. Why not a build in a music player?
Score: 1 Votes (Like | Disagree)
OldSchoolMacGuy Avatar
65 months ago
Prisma just added the ability to apply filters to Facebook live video on the iPhone 7 and 6S today.
Score: 1 Votes (Like | Disagree)
Hiran Avatar
65 months ago
But I deleted the official Facebook app and installed Friendly for iOS when the official app lost messenger feature. Mark Zuckerberg is dumb and arrogant as hell.
Score: 1 Votes (Like | Disagree)
smacrumon Avatar
65 months ago
Facebook is out to lunch. Their product is dated, the design is 2009 with a side of stale fries.
Score: 1 Votes (Like | Disagree)

Related Stories

studio buds family

Beats Studio Buds Debuting Today With Active Noise Cancellation, Stemless Design, and More for $150

Monday June 14, 2021 8:00 am PDT by
We've seen a lot of teasers about the Beats Studio Buds over the past month since they first showed up in Apple's beta software updates, and today they're finally official. The Beats Studio Buds are available to order today in red, white, and black ahead of a June 24 ship date, and they're priced at $149.99. The Studio Buds are the first Beats-branded earbuds to truly compete with AirPods...
macos catalina legacy system extension alert

Apple Begins Warning Users That 'Legacy System Extensions' Won't Work With a Future Version of macOS

Wednesday March 25, 2020 9:53 am PDT by
Apple has shared a new support document that indicates kernel extensions — which it calls "legacy system extensions" — will not be compatible with a future version of macOS because they "aren't as secure or reliable as modern alternatives."System extensions are a category of software that works in the background to extend the functionality of your Mac. Some apps install kernel extensions, which...
General Spotify Feature

Spotify Pauses Plans to Add AirPlay 2 Support to iOS App [Update: Spotify Clarifies]

Friday August 6, 2021 9:07 am PDT by
See update at bottom of article Spotify this week confirmed that its plans to add AirPlay 2 support to its iOS app have been placed on indefinite hiatus. In an online discussion forum post, a Spotify representative said the streaming music service had been working on supporting AirPlay 2, but the company has paused the efforts "for now" due to "audio driver compatibility issues." The...
apple privacy

Apple Publishes FAQ to Address Concerns About CSAM Detection and Messages Scanning

Monday August 9, 2021 1:50 am PDT by
Apple has published a FAQ titled "Expanded Protections for Children" which aims to allay users' privacy concerns about the new CSAM detection in iCloud Photos and communication safety for Messages features that the company announced last week. "Since we announced these features, many stakeholders including privacy organizations and child safety organizations have expressed their support of...
youtube apple tv

YouTube Discontinuing 3rd-Generation Apple TV App, AirPlay Still Available

Wednesday February 3, 2021 3:09 pm PST by
YouTube is planning to stop supporting its YouTube app on the third-generation Apple TV models, where YouTube has long been available as a channel option. A 9to5Mac reader received a message about the upcoming app discontinuation, which is set to take place in March.Starting early March, the YouTube app will no longer be available on Apple TV (3rd generation). You can still watch YouTube on...
m1 macbook air

Kuo: Mini-LED MacBook Air Coming in Mid-2022

Thursday July 22, 2021 7:48 pm PDT by
Apple will release a new version of the MacBook Air around the middle of 2022, Apple analyst Ming-Chi Kuo said today in note to investors seen by MacRumors. The upcoming MacBook Air will feature a 13.3-inch mini-LED display, which would make it the second Mac to gain mini-LED technology after the 2021 MacBook Pro, which is rumored to include a mini-LED display and is expected to launch later ...
Apple Films Tom Hanks Finch First Look

Apple Original Film 'Finch' Starring Tom Hanks to Premiere November 5

Thursday August 12, 2021 8:52 am PDT by
Apple today announced that the original film "Finch," starring Tom Hanks in its titular role, will premiere on Apple TV+ on Friday, November 5 and shared a first-look image from the film. The first look at "Finch," shared by Apple. The film, which is anticipated to be an awards season contender, revolves around a man, a robot (played by "Get Out" actor Caleb Landry Jones), and a dog that form ...
whatsintheboxiphonexs

Apple's iPhone XS, XS Max and XR Won't Ship With Lightning to 3.5mm Headphone Jack Adapter

Wednesday September 12, 2018 12:58 pm PDT by
Since the headphone jack was removed from the iPhone with the launch of the iPhone 7, Apple has bundled iPhones with a Lightning to 3.5mm Headphone Jack Adapter for customers who continue to have 3.5mm headphones. With the launch of the iPhone XS, iPhone XS Max, and iPhone XR, Apple is discontinuing this practice and will not include the Lightning to 3.5mm Headphone Jack Adapter. On the...
macbook air m1 first benchmark

Apple Silicon M1 Chip in MacBook Air Outperforms High-End 16-Inch MacBook Pro

Wednesday November 11, 2020 4:43 pm PST by
Apple introduced the first MacBook Air, MacBook Pro, and Mac mini with M1 Apple Silicon chips yesterday, and as of today, the first benchmark of the new chip appears to be showing up on the Geekbench site. The M1 chip, which belongs to a MacBook Air with 8GB RAM, features a single-core score of 1687 and a multi-core score of 7433. According to the benchmark, the M1 has a 3.2GHz base...
iphone 13 teal with text

Apple Begins Preparation for iPhone 13 Production Ahead of Fall Launch

Monday June 28, 2021 3:29 am PDT by
We're just a few months away from when Apple is expected to reveal the 2021 iPhone, dubbed the "iPhone 13." In preparation for its launch, it has been pulling in shipments of different components needed to produce the new iPhones, according to a report from DigiTimes. In years past, Apple released its latest iPhone lineup, alongside a new Apple Watch, during a September event at Apple Park....