Craig Federighi Explains How Scribble Feature in iPadOS 14 Was Developed

Scribble is one of the most notable new features in iPadOS 14, and in a new interview with Popular Mechanics, Apple SVP of software engineering Craig Federighi has revealed more details about how it was developed and how it works on iPad.

apple ipados 14 scribble 062220 inline
With Scribble in ‌iPadOS 14‌, you can use an ‌Apple Pencil‌ to write in any text field, and the written text is then converted to typed text. The feature means you can use the ‌‌Apple Pencil‌‌ without having to swap over to a keyboard when you need to compose an email, write a Calendar event, visit a URL, send an iMessage, or conduct a search.

Federighi explained to Popular Mechanics how Scribble's early development relied on extensive data gathering involving analyzing how people from all over the world write things down.

"When it comes to understanding [handwriting] strokes, we do data-gathering. We find people all over the world, and have them write things," says Craig Federighi, senior vice president of software engineering at Apple. "We give them a Pencil, and we have them write fast, we have them write slow, write at a tilt. All of this variation."

That methodology is distinct from the comparatively simple approach of scanning and analyzing existing handwriting. Federighi says that for Apple's tech, static examples weren't enough. They needed to see the strokes that formed each letter. "If you understand the strokes and how the strokes went down, that can be used to disambiguate what was being written."

The job of integrating stroke-based recognition with character and word prediction is all done on-device, something that machine learning and raw processing power makes possible, explained Apple's software chief.

"It's gotta be happening in real time, right now, on the device that you're holding," Federighi says. "Which means that the computational power of the device has to be such that it can do that level of processing locally."


Thanks to Scribble and ‌iPadOS 14‌, an ‌iPad‌‌ can be used almost entirely with touch and tools, and there's little need to type. If you're someone who prefers writing by hand, ‌Scribble‌ introduces tools that only enhance the experience. Check out the hands-on video above to see Scribble in action.

Related Roundups: iOS 14, iPadOS 14
Related Forums: iOS 14, iPad Accessories

Top Rated Comments

Abazigal Avatar
8 months ago

Everyday, thousands of MacBook screens are touched and people are bewildered that nothing happens. Touchscreens have been standard for over a decade now - it’s bizarre that there are computers costing thousands of dollars that don’t have touchscreens.

The fact that it’d be used so little didn’t stop Apple from including the touchbar - why on earth haven’t they made their products “Just Work” by including a touchscreen yet?

Seriously - have any kid who hasn’t used a Mac before use a MacBook for a few hours. I guarantee they’ll touch the screen at least once and consider it broken when nothing happens. Literally every phone, tablet, and non-Apple-or-budget laptop has a touchscreen - it’s just a given that a MacBook should, too.
I have read and reread your rationale as to why Macs ought to have touchscreens and it seems like your argument boils down to "just because".

And honestly speaking, I haven't exactly seen touchscreen PCs take the world by storm either. Yes, they are there, and I suppose having a feature you hardly use is better than not having it, but it feels like this is there purely to tick off a checkbox, not so much because it legitimately improves the end user experience.
Score: 13 Votes (Like | Disagree)
chucker23n1 Avatar
8 months ago

inb4 people say "steve jobs rolling in his grave" and bringing up the stylus quote...
The stylus quote was clearly a shorthand of "who wants to rely on a stylus exclusively to interact with the UI, when you already have ten fingers?". Using a stylus to write handwritten text or draw something is quite different from that.
Score: 11 Votes (Like | Disagree)
bob122989 Avatar
8 months ago
Newton technology all the way. I’ve been surprised they haven’t had it on iPhone and iPad since day one.
Score: 11 Votes (Like | Disagree)
Harvey Zoltan Avatar
8 months ago
I would not be surprised if it's Apple Pencil support and not touch support that comes to the AS Macs.
Score: 7 Votes (Like | Disagree)
ArtOfWarfare Avatar
8 months ago

You may well be right but many tech commentators seem to be convinced that touch screen Macs are coming. Hopefully not too much longer before we know for sure.
Everyday, thousands of MacBook screens are touched and people are bewildered that nothing happens. Touchscreens have been standard for over a decade now - it’s bizarre that there are computers costing thousands of dollars that don’t have touchscreens.

The fact that it’d be used so little didn’t stop Apple from including the touchbar - why on earth haven’t they made their products “Just Work” by including a touchscreen yet?

Seriously - have any kid who hasn’t used a Mac before use a MacBook for a few hours. I guarantee they’ll touch the screen at least once and consider it broken when nothing happens. Literally every phone, tablet, and non-Apple-or-budget laptop has a touchscreen - it’s just a given that a MacBook should, too.
Score: 6 Votes (Like | Disagree)
ouimetnick Avatar
8 months ago
It’s amazing how much more there is to this than just meets the eye. I generally don’t think about on device processing and learning. I’ve been using Scribble since iPadOS 14 came out and it’s really useful for converting notes into text.
Score: 6 Votes (Like | Disagree)

Top Stories

tracking disabled ios 14 5

Analytics Suggest 96% of Users Leave App Tracking Disabled in iOS 14.5

Friday May 7, 2021 1:51 am PDT by
An early look at an ongoing analysis of Apple's App Tracking Transparency suggests that the vast majority of iPhone users are leaving app tracking disabled since the feature went live on April 26 with the release of iOS 14.5. According to the latest data from analytics firm Flurry, just 4% of iPhone users in the U.S. have actively chosen to opt into app tracking after updating their device...
macbook colors 3d black bezels

Prosser: Next MacBook Air Could Come in Colors Similar to iMac

Friday May 7, 2021 6:55 am PDT by
According to Apple leaker Jon Prosser, Apple's upcoming release of the MacBook Air will feature various colors, similar to the colors in the newly released 24-inch iMac. In the latest video of his YouTube channel Front Page Tech, Prosser says the same source who accurately provided him information on the first Apple silicon iMac coming in colors has told him that he recently saw a prototype...
tile amazon sidewalk integration

Tile to Leverage Amazon Echo and Ring Devices to Better Compete With AirTags

Friday May 7, 2021 2:07 pm PDT by
Amazon today announced that it is teaming up with Tile to add Amazon Sidewalk integration to Tile's Bluetooth trackers. Amazon Sidewalk, for those unfamiliar, is a network of Amazon Bluetooth devices that's designed to improve the connectivity of devices like the Ring and Amazon Echo. Tile will now be joining Amazon Sidewalk, and through this integration, Amazon Echo and Ring devices will be ...
Top Stories 59 Feature

Top Stories: Epic Games vs. Apple, Hidden AirTag Mode, Apple Music Hi-Fi, Colorful MacBook Air?

Saturday May 8, 2021 6:00 am PDT by
While we wait for the newly introduced iMac, iPad Pro, and Apple TV models to launch later this month, this week saw the kickoff of the big Epic Games v. Apple trial, with lots of juicy tidbits coming out as the two sides make their arguments. This week also saw some rumors about a Hi-Fi tier for Apple Music, more biometric sensing capabilities for Apple Watch, and timing for the...
zoom app icon

Apple Gave Zoom Access to Special API to Use iPad Camera During Split View Multitasking

Sunday May 9, 2021 2:00 am PDT by
Zoom, a hallmark platform used by millions during the global health crisis, has been given access to a special iPadOS API that allows the app to use the iPad camera while the app is in use in Split View multitasking mode. This case of special treatment was first brought to attention by app developer Jeremy Provost, who, in a blog post, explains that Zoom uses a special API that allows the...
snapchat dark mode

Snapchat Rolls Out Dark Mode on iOS

Wednesday May 5, 2021 1:17 am PDT by
Nearly two years following the release of iOS and iPadOS 13, which included native, built-in, and systemwide dark mode, Snapchat, one of the world's most prominent social media networks, has finally rolled out a dark mode theme for iOS users. Snapchat began testing a dark mode theme of its app design late last year with a small group of iOS users. Now, Snapchat says that as of this week, it...
tile sticker e1570533758981

Tile CEO: 'We Welcome Competition From Apple, But We Think It Needs to Be Fair'

Tuesday May 4, 2021 9:51 am PDT by
Just after Apple announced its AirTags, Tile CEO CJ Prober relayed his concerns about competing with Apple in the tracking space, and said that Tile would ask Congress to investigate Apple's business practices specific to Find My and item trackers. Prober this week did an interview with Bloomberg, where he further expanded on Tile's complaints about Apple and why he feels that Tile is...
fortnite apple logo 2

Epic CEO Tim Sweeney Admits App Store's 30% Cut Is Similar to Consoles, Would Have Accepted Special Deal With Apple

Tuesday May 4, 2021 1:54 pm PDT by
Apple's legal battle with Epic Games is continuing on, and during the second day of the trial, Epic Games' CEO Tim Sweeney continued his testimony against Apple. Sweeney was grilled by Apple's lawyers, and made several points seemingly favorable to Apple. In addition to mentioning how he prefers Apple's iPhone and values Apple's privacy policies that he's aiming to dismantle, Sweeney...
signal instagram ads3

Signal Shares the Instagram Ads Facebook Doesn't Want You to See

Wednesday May 5, 2021 1:29 am PDT by
Encrypted messaging app Signal has had a series of Instagram ads blocked from the social media platform, after it attempted to show users how much data the Facebook-owned company collects about them and how it's used to push targeted ads. In a blog post, Signal described how it generated the ads to show users why they were seeing them, simply by declaring upfront the information that the...
iphone 12 preorder purple

Apple Begins Transition to Randomized Serial Numbers With Purple iPhone 12

Wednesday May 5, 2021 9:17 am PDT by
MacRumors previously reported about Apple's plan to switch to randomized serial numbers for future products starting in early 2021, and this transition has now started with the new purple iPhone 12 model in multiple countries. With assistance from Aaron Zollo, host of the YouTube channel ZolloTech, we can confirm that the purple iPhone 12 released last month has a new 10-character serial...