In a lengthy blog post about Facebook's ten-year plan to "accelerate innovation and power new experiences with AI," chief technology officer Mike Schroepfer mentioned that one update coming to the company's mobile apps will be a "style transfer" tool that turns normal photos and videos into works of art using "high efficiency neural networks," all running directly on iOS and Android smartphones.

Facebook's impending update will, according to Schroepfer, run entirely on the user's smartphone and not be dependent on the content to be sent to servers, creating long load times and frustrating users. The CTO described this as the most demanding and "technically difficult" hurdle to clear in the process of adding the feature to the company's mobile apps, but Schroepfer said the company has done just that, and the result is a deep learning platform called "Caffe2Go."

facebook-art-filters

Just three months ago we set out to do something nobody else had done before: ship AI-based style transfer running live, in real time, on mobile devices. This was a major engineering challenge, as we needed to design software that could run high-powered computing operations on a device with unique resource constraints in areas like power, memory and compute capability. The result is Caffe2Go, a new deep learning platform that can capture, analyze and process pixels in real time on a mobile device.

We found that by condensing the size of the AI model used to process images and videos by 100x, we’re able to run deep neural networks with high efficiency on both iOS and Android. This is all happening in the palm of your hand, so you can apply styles to videos as you’re taking them.

Schroepfer said that the alternative of sending the content to data centers to be analyzed and filtered was "not ideal for letting people share fun content in the moment." In addition to basic image and video filtering capabilities, the new deep-learning platform will also feasibly be able to understand gesture controls when taking a selfie, for instance. In today's blog post, an example is given of a user swiping right and left between various artistic filters for a selfie (all running live, in real-time) and even snapping a picture when the user smiles.

As a comparison, the update sounds largely similar to Prisma, an app that launched over the summer and impressed many with its ability to turn photos, and eventually videos, into stylized images. Originally, Prisma used a server-side combination of neural networks and artificial intelligence to apply the different filters to user photos, but an update also introduced offline image processing, meaning users could use some of the app's filters to alter their images right from their smartphones.

For Facebook, the announcement follows a year of video-first announcements from the company, most recently embodied in CEO and co-founder Mark Zuckerberg's plan to make the camera more prominent in the app. No specific timeline was given for a possible launch of the new features on Facebook's mobile apps, but the company is clearly looking to lay the groundwork for its future, calling its new AI initiative, along with virtual reality, "new technologies that will shape the next decade."

Top Rated Comments

jmh600cbr Avatar
73 months ago
Facebook - the startup eater
Score: 3 Votes (Like | Disagree)
Zorn Avatar
73 months ago
Wow prisma filters! Sounds like a cool feature no one asked for. When will it stop looking like mud on the iPad Pro that's now a year old?
Score: 2 Votes (Like | Disagree)
-BigMac- Avatar
73 months ago
Yeh man keep beefing and slowing that sluggish app up. Why not a build in a music player?
Score: 1 Votes (Like | Disagree)
OldSchoolMacGuy Avatar
73 months ago
Prisma just added the ability to apply filters to Facebook live video on the iPhone 7 and 6S today.
Score: 1 Votes (Like | Disagree)
Hiran Avatar
73 months ago
But I deleted the official Facebook app and installed Friendly for iOS when the official app lost messenger feature. Mark Zuckerberg is dumb and arrogant as hell.
Score: 1 Votes (Like | Disagree)
smacrumon Avatar
73 months ago
Facebook is out to lunch. Their product is dated, the design is 2009 with a side of stale fries.
Score: 1 Votes (Like | Disagree)

Popular Stories

iPhone 14 Pro Purple Front and Back MacRumors Exclusive

iPhone 14 Pro Renders Highlight Multiple Design Changes

Wednesday May 25, 2022 8:56 am PDT by
Leaker Jon Prosser today shared ostensibly accurate renders of the iPhone 14 Pro, providing the most accurate look yet at what the device could look like when it launches later this year. In the latest video on YouTube channel Front Page Tech, Prosser revealed renders of the iPhone 14 Pro made by Apple concept graphic designer Ian Zelbo, highlighting a range of specific design changes...
iPad Pro USB C Feature Coral

Deals: Apple's iPad Pro Reaches Up to $449 Off in Amazon's Latest Sales

Wednesday May 25, 2022 10:09 am PDT by
Amazon is marking down a wide variety of 11-inch and 12.9-inch iPad Pro models this week, with prices starting as low as $749.00 for the 11-inch tablet. You'll find the full list of sales below, all of which can be found on Amazon. Note: MacRumors is an affiliate partner with some of these vendors. When you click a link and make a purchase, we may receive a small payment, which helps us keep...
apple account card

Wallet App Now Supports Apple Account Cards on iOS 15.5

Wednesday May 25, 2022 5:01 pm PDT by
Apple appears to have recently updated the Wallet app to allow users to add an Apple Account Card, which displays the Apple credit balance associated with an Apple ID. If you receive an App Store or Apple Store gift card, for example, it is added to an Apple Account that was previously visible in the App Store and Apple Store apps. As of today, the Apple Account balance can also be added to...
iphone 13 pro max display bleen

iPhone 14 Max Reportedly Weeks Behind Schedule [Updated]

Thursday May 26, 2022 7:25 am PDT by
The iPhone 14 Max is currently behind schedule by around three weeks, according to Haitong International Securities analyst Jeff Pu. Yesterday, Nikkei Asia reported that at least one iPhone 14 model was three weeks behind schedule due to the impact of lockdowns on Apple's supply chains in China, but it was not clear which iPhone 14 model this related to. Now, Pu has clarified that the model...
iPhone 13 Always On Feature

iPhone 14 Pro Screen Refresh Rate Upgrade Could Allow for Always-On Display

Tuesday May 24, 2022 7:23 am PDT by
Last year's iPhone 13 Pro models were the first of Apple's smartphones to come with 120Hz ProMotion displays, and while the two iPhone 14 Pro models will continue to feature the technology, their screens could well boast expanded refresh rate variability this time round. To bring ProMotion displays to the ‌iPhone 13 Pro models‌, Apple adopted LTPO panel technology with variable refresh...
Apple Tap to Pay iPhone

Apple Stores Rolling Out iPhone-to-iPhone Contactless Payments Starting Today

Wednesday May 25, 2022 6:54 am PDT by
Apple in February unveiled a new "Tap to Pay on iPhone" feature that will allow compatible iPhones to accept payments via Apple Pay, contactless credit and debit cards, and other digital wallets, with no additional hardware required. Apple began testing the feature at its Apple Park Visitor Center earlier this month, and now Bloomberg's Mark Gurman has tweeted that the feature will begin...
apple tv 4k design green

Apple Releases tvOS 15.5.1 for Apple TV HD and Apple TV 4K

Wednesday May 25, 2022 9:42 am PDT by
Apple today released tvOS 15.5.1, a minor update to the tvOS operating system that first launched in September 2021. tvOS 15.5.1 comes about 10 days after the launch of tvOS 15.5. tvOS 15.5.1 can be downloaded over the air on the Apple TV through the Settings app by going to System > Software Update. ‌‌‌‌‌‌Apple TV‌‌‌‌‌‌ owners who have automatic software updates...