Craig Federighi Explains How Scribble Feature in iPadOS 14 Was Developed

Scribble is one of the most notable new features in iPadOS 14, and in a new interview with Popular Mechanics, Apple SVP of software engineering Craig Federighi has revealed more details about how it was developed and how it works on iPad.

apple ipados 14 scribble 062220 inline
With Scribble in iPadOS 14, you can use an ‌Apple Pencil‌ to write in any text field, and the written text is then converted to typed text. The feature means you can use the ‌‌Apple Pencil‌‌ without having to swap over to a keyboard when you need to compose an email, write a Calendar event, visit a URL, send an iMessage, or conduct a search.

Federighi explained to Popular Mechanics how Scribble's early development relied on extensive data gathering involving analyzing how people from all over the world write things down.

"When it comes to understanding [handwriting] strokes, we do data-gathering. We find people all over the world, and have them write things," says Craig Federighi, senior vice president of software engineering at Apple. "We give them a Pencil, and we have them write fast, we have them write slow, write at a tilt. All of this variation."

That methodology is distinct from the comparatively simple approach of scanning and analyzing existing handwriting. Federighi says that for Apple's tech, static examples weren't enough. They needed to see the strokes that formed each letter. "If you understand the strokes and how the strokes went down, that can be used to disambiguate what was being written."

The job of integrating stroke-based recognition with character and word prediction is all done on-device, something that machine learning and raw processing power makes possible, explained Apple's software chief.

"It's gotta be happening in real time, right now, on the device that you're holding," Federighi says. "Which means that the computational power of the device has to be such that it can do that level of processing locally."


Thanks to Scribble and iPadOS 14, an ‌iPad‌‌ can be used almost entirely with touch and tools, and there's little need to type. If you're someone who prefers writing by hand, ‌Scribble‌ introduces tools that only enhance the experience. Check out the hands-on video above to see Scribble in action.

Related Forums: iOS 14, iPad Accessories

Top Rated Comments

Abazigal Avatar
25 months ago

Everyday, thousands of MacBook screens are touched and people are bewildered that nothing happens. Touchscreens have been standard for over a decade now - it’s bizarre that there are computers costing thousands of dollars that don’t have touchscreens.

The fact that it’d be used so little didn’t stop Apple from including the touchbar - why on earth haven’t they made their products “Just Work” by including a touchscreen yet?

Seriously - have any kid who hasn’t used a Mac before use a MacBook for a few hours. I guarantee they’ll touch the screen at least once and consider it broken when nothing happens. Literally every phone, tablet, and non-Apple-or-budget laptop has a touchscreen - it’s just a given that a MacBook should, too.
I have read and reread your rationale as to why Macs ought to have touchscreens and it seems like your argument boils down to "just because".

And honestly speaking, I haven't exactly seen touchscreen PCs take the world by storm either. Yes, they are there, and I suppose having a feature you hardly use is better than not having it, but it feels like this is there purely to tick off a checkbox, not so much because it legitimately improves the end user experience.
Score: 13 Votes (Like | Disagree)
chucker23n1 Avatar
25 months ago

inb4 people say "steve jobs rolling in his grave" and bringing up the stylus quote...
The stylus quote was clearly a shorthand of "who wants to rely on a stylus exclusively to interact with the UI, when you already have ten fingers?". Using a stylus to write handwritten text or draw something is quite different from that.
Score: 11 Votes (Like | Disagree)
bob122989 Avatar
25 months ago
Newton technology all the way. I’ve been surprised they haven’t had it on iPhone and iPad since day one.
Score: 11 Votes (Like | Disagree)
Harvey Zoltan Avatar
25 months ago
I would not be surprised if it's Apple Pencil support and not touch support that comes to the AS Macs.
Score: 7 Votes (Like | Disagree)
ArtOfWarfare Avatar
25 months ago

You may well be right but many tech commentators seem to be convinced that touch screen Macs are coming. Hopefully not too much longer before we know for sure.
Everyday, thousands of MacBook screens are touched and people are bewildered that nothing happens. Touchscreens have been standard for over a decade now - it’s bizarre that there are computers costing thousands of dollars that don’t have touchscreens.

The fact that it’d be used so little didn’t stop Apple from including the touchbar - why on earth haven’t they made their products “Just Work” by including a touchscreen yet?

Seriously - have any kid who hasn’t used a Mac before use a MacBook for a few hours. I guarantee they’ll touch the screen at least once and consider it broken when nothing happens. Literally every phone, tablet, and non-Apple-or-budget laptop has a touchscreen - it’s just a given that a MacBook should, too.
Score: 6 Votes (Like | Disagree)
ouimetnick Avatar
25 months ago
It’s amazing how much more there is to this than just meets the eye. I generally don’t think about on device processing and learning. I’ve been using Scribble since iPadOS 14 came out and it’s really useful for converting notes into text.
Score: 6 Votes (Like | Disagree)

Related Stories

iOS 15 General Feature Blue

Apple Releases iOS 15.3.1 and iPadOS 15.3.1 With Security Updates and Bug Fixes

Thursday February 10, 2022 10:10 am PST by
Apple today released iOS 15.3.1 and iPadOS 15.3.1, two minor updates to the iOS and iPadOS operating systems released in September 2021. iOS and iPadOS 15.3.1 come two weeks after the release of iOS and iPadOS 15.3. The iOS 15.3.1 and iPadOS 15.3.1 updates can be downloaded for free and the software is available on all eligible devices over-the-air in the Settings app. To access the new...
iOS 15

Apple Seeds Fourth Betas of iOS 15.4 and iPadOS 15.4 to Developers [Update: Public Beta Available]

Tuesday February 22, 2022 10:07 am PST by
Apple today seeded the fourth betas of upcoming iOS 15.4 and iPadOS 15.4 updates to developers for testing purposes, with the new software coming one week after Apple seeded the third betas of iOS 15.4 and iPadOS 15.4. Developers can download iOS 15.4 and iPadOS 15.4 through the Apple Developer Center or over the air after the proper profile has been installed on an iPhone or iPad. iOS...
iOS 15

Apple Seeds Release Candidate Version of iOS 15.4 and iPadOS 15.4 to Developers and Public Beta Testers

Tuesday March 8, 2022 11:19 am PST by
Apple today seeded the release candidate versions of upcoming iOS 15.4 and iPadOS 15.4 updates to developers for testing purposes, with the new software coming one week after Apple seeded the fifth betas of iOS 15.4 and iPadOS 15.4. The RCs represent the final versions of the updates that will be released to the public next week. Developers can download iOS 15.4 and iPadOS 15.4 through the...
iOS 15

Apple Seeds Fifth Betas of iOS 15.4 and iPadOS 15.4 to Developers [Update: Public Beta Available]

Tuesday March 1, 2022 10:04 am PST by
Apple today seeded the fifth betas of upcoming iOS 15.4 and iPadOS 15.4 updates to developers for testing purposes, with the new software coming one week after Apple seeded the fourth betas of iOS 15.4 and iPadOS 15.4. Developers can download iOS 15.4 and iPadOS 15.4 through the Apple Developer Center or over the air after the proper profile has been installed on an iPhone or iPad. iOS...
ipados14scribblecopyastext

iPadOS 14 Apple Pencil Features Expand to French, German, Italian, Portuguese, and Spanish Languages

Tuesday July 6, 2021 4:30 pm PDT by
Apple has expanded several iPadOS 14 Apple Pencil features to additional languages, improving Apple Pencil functionality for those who write in French, German, Italian, Portuguese, and Spanish. According to Apple's iOS and iPadOS Feature Availability page, these languages can now be used when copying handwriting as text and there's also data detector support. That means if you write...
maxresdefault

Hands-On With Universal Control in iPadOS 15.4 and macOS Monterey 12.3

Thursday January 27, 2022 3:01 pm PST by
Apple today released new developer betas of iPadOS 15.4 and macOS Monterey 12.3, and the updated software adds the long awaited and highly anticipated Universal Control feature. We tested out Universal Control, and even in an early beta form, Universal Control is an easy to use and convenient multi-device control option. Subscribe to the MacRumors YouTube channel for more videos. To use...
applepencilipadair

Deals: Apple Pencil 2 Drops to $99.99 on Verizon ($29 Off)

Thursday October 28, 2021 6:35 am PDT by
Verizon today is discounting the Apple Pencil 2 to $99.99, down from $129.00. This deal has been applied automatically and does not require a special promo code. Note: MacRumors is an affiliate partner with Verizon. When you click a link and make a purchase, we may receive a small payment, which helps us keep the site running. This sale is a match of one of the lowest prices that we've ever ...
Writing Session History ulysses

Ulysses Gains Enhanced Blog Publishing and Session History Features

Tuesday August 3, 2021 12:02 am PDT by
Popular writing app Ulysses today reached version 23, and this update improves its blog publishing features as well as the way session histories and writing goals are calculated. For a while now, Ulysses has offered a writing goal feature to help writers foster a writing habit. The goal feature allows them to set a daily target – 500 words, say – and monitor their progress towards...

Popular Stories

iPhone 14 Dummies 1 Feature

Everything Rumored for Apple's September Event: iPhone 14, Apple Watch Pro and More

Friday August 12, 2022 2:34 pm PDT by
There's just about a month to go until Apple holds its annual September event focusing on new iPhone and Apple Watch models. We thought we'd take a quick look at everything that's rumored for Apple's September event to give MacRumors readers a rundown on what to expect when the first fall event rolls around. iPhone 14 The iPhone 14 can probably be described more as an "iPhone 13S" because...
iPhone 14 Pro Purple Rear Flat MacRumors Exclusive

iPhone 14 Pro Predicted to Start With Increased 256GB Storage Alongside Rumored Price Increase

Wednesday August 10, 2022 11:14 am PDT by
Earlier today, analyst Ming-Chi Kuo claimed iPhone 14 Pro models will be more expensive than iPhone 13 Pro models. Kuo did not reveal exact pricing, but he said that the average selling price of all four iPhone 14 models will increase by about 15% overall. While higher prices would be disappointing for customers, it is possible the iPhone 14 Pro and iPhone 14 Pro Max will offer increased...
iPhone 14 Pro Lineup Feature Silver

Kuo: Apple to Increase Prices of iPhone 14 Pro Models

Wednesday August 10, 2022 8:22 am PDT by
Apple plans to increase the prices of iPhone 14 Pro models compared to iPhone 13 Pro models, according to analyst Ming-Chi Kuo. Kuo did not reveal exact pricing for the iPhone 14 Pro models. However, in a tweet today, he estimated that the average selling price of the iPhone 14 lineup as a whole will increase by about 15% compared to the iPhone 13 lineup. In the United States, the iPhone...
iPhone 14 Pro Purple Front and Back MacRumors Exclusive feature

iPhone 14 Is Just a Few Weeks Away: Three Tips to Prepare for the New iPhone

Wednesday August 10, 2022 4:08 am PDT by
The launch of the new iPhone 14 is just a few weeks away, meaning millions of iPhone customers will soon upgrade their existing iPhone or perhaps get an iPhone for the first time. Exclusive MacRumors iPhone 14 Pro renders by graphic designer Ian Zelbo Whether upgrading from an older model or this is your first iPhone, we've rounded up a few tips to help you prepare for the next flagship...
z fold 4 1

Hands-On With Samsung's Latest Foldable Smartphones, the Galaxy Z Fold and Z Flip

Friday August 12, 2022 12:46 pm PDT by
Samsung this week launched its fourth-generation foldable devices, the $1,000 Galaxy Z Flip and the $1,800 Galaxy Z Fold. Though there's no sign of a comparable Apple foldable device on the horizon, rumors suggest prototypes are in the works, so we thought we'd take a look at Samsung's newest smartphones to see what Apple needs to measure up to when a foldable iPhone does come out. Subscribe ...
iPhone 14 Pro Purple Rear Flat MacRumors Exclusive

iPhone 14 Pro: Analyst Believes Storage Will Continue to Start at 128GB Despite Rumored Price Increase

Friday August 12, 2022 7:04 am PDT by
It wouldn't be the month before an iPhone launch without a few back-and-forth rumors, with the latest debate concerning iPhone 14 Pro storage. iPhone 14 Pro render by Ian Zelbo While research firm TrendForce forecasted that iPhone 14 Pro models could start with an increased 256GB of storage, Haitong International analyst Jeff Pu today forecasted that the upcoming Pro models will continue to...