Craig Federighi Explains How Scribble Feature in iPadOS 14 Was Developed

Scribble is one of the most notable new features in iPadOS 14, and in a new interview with Popular Mechanics, Apple SVP of software engineering Craig Federighi has revealed more details about how it was developed and how it works on iPad.

apple ipados 14 scribble 062220 inline
With Scribble in iPadOS 14, you can use an ‌Apple Pencil‌ to write in any text field, and the written text is then converted to typed text. The feature means you can use the ‌‌Apple Pencil‌‌ without having to swap over to a keyboard when you need to compose an email, write a Calendar event, visit a URL, send an iMessage, or conduct a search.

Federighi explained to Popular Mechanics how Scribble's early development relied on extensive data gathering involving analyzing how people from all over the world write things down.

"When it comes to understanding [handwriting] strokes, we do data-gathering. We find people all over the world, and have them write things," says Craig Federighi, senior vice president of software engineering at Apple. "We give them a Pencil, and we have them write fast, we have them write slow, write at a tilt. All of this variation."

That methodology is distinct from the comparatively simple approach of scanning and analyzing existing handwriting. Federighi says that for Apple's tech, static examples weren't enough. They needed to see the strokes that formed each letter. "If you understand the strokes and how the strokes went down, that can be used to disambiguate what was being written."

The job of integrating stroke-based recognition with character and word prediction is all done on-device, something that machine learning and raw processing power makes possible, explained Apple's software chief.

"It's gotta be happening in real time, right now, on the device that you're holding," Federighi says. "Which means that the computational power of the device has to be such that it can do that level of processing locally."


Thanks to Scribble and iPadOS 14, an ‌iPad‌‌ can be used almost entirely with touch and tools, and there's little need to type. If you're someone who prefers writing by hand, ‌Scribble‌ introduces tools that only enhance the experience. Check out the hands-on video above to see Scribble in action.

Related Forums: iOS 14, iPad Accessories

Top Rated Comments

Abazigal Avatar
17 months ago

Everyday, thousands of MacBook screens are touched and people are bewildered that nothing happens. Touchscreens have been standard for over a decade now - it’s bizarre that there are computers costing thousands of dollars that don’t have touchscreens.

The fact that it’d be used so little didn’t stop Apple from including the touchbar - why on earth haven’t they made their products “Just Work” by including a touchscreen yet?

Seriously - have any kid who hasn’t used a Mac before use a MacBook for a few hours. I guarantee they’ll touch the screen at least once and consider it broken when nothing happens. Literally every phone, tablet, and non-Apple-or-budget laptop has a touchscreen - it’s just a given that a MacBook should, too.
I have read and reread your rationale as to why Macs ought to have touchscreens and it seems like your argument boils down to "just because".

And honestly speaking, I haven't exactly seen touchscreen PCs take the world by storm either. Yes, they are there, and I suppose having a feature you hardly use is better than not having it, but it feels like this is there purely to tick off a checkbox, not so much because it legitimately improves the end user experience.
Score: 13 Votes (Like | Disagree)
chucker23n1 Avatar
17 months ago

inb4 people say "steve jobs rolling in his grave" and bringing up the stylus quote...
The stylus quote was clearly a shorthand of "who wants to rely on a stylus exclusively to interact with the UI, when you already have ten fingers?". Using a stylus to write handwritten text or draw something is quite different from that.
Score: 11 Votes (Like | Disagree)
bob122989 Avatar
17 months ago
Newton technology all the way. I’ve been surprised they haven’t had it on iPhone and iPad since day one.
Score: 11 Votes (Like | Disagree)
Harvey Zoltan Avatar
17 months ago
I would not be surprised if it's Apple Pencil support and not touch support that comes to the AS Macs.
Score: 7 Votes (Like | Disagree)
ArtOfWarfare Avatar
17 months ago

You may well be right but many tech commentators seem to be convinced that touch screen Macs are coming. Hopefully not too much longer before we know for sure.
Everyday, thousands of MacBook screens are touched and people are bewildered that nothing happens. Touchscreens have been standard for over a decade now - it’s bizarre that there are computers costing thousands of dollars that don’t have touchscreens.

The fact that it’d be used so little didn’t stop Apple from including the touchbar - why on earth haven’t they made their products “Just Work” by including a touchscreen yet?

Seriously - have any kid who hasn’t used a Mac before use a MacBook for a few hours. I guarantee they’ll touch the screen at least once and consider it broken when nothing happens. Literally every phone, tablet, and non-Apple-or-budget laptop has a touchscreen - it’s just a given that a MacBook should, too.
Score: 6 Votes (Like | Disagree)
ouimetnick Avatar
17 months ago
It’s amazing how much more there is to this than just meets the eye. I generally don’t think about on device processing and learning. I’ve been using Scribble since iPadOS 14 came out and it’s really useful for converting notes into text.
Score: 6 Votes (Like | Disagree)

Related Stories

Apple Pencil 25 off

Deals: Pair Your New iPad Pro With the Apple Pencil 2 for $103.99 ($25 Off)

Friday April 30, 2021 6:41 am PDT by
If you're shopping for accessories to pair with your new iPad Pro, Verizon is discounting the Apple Pencil 2 to $103.99, down from $129.00. This is currently the best sale on the accessory, and it's the best deal that we've seen in a few weeks. Note: MacRumors is an affiliate partner with Verizon. When you click a link and make a purchase, we may receive a small payment, which helps us keep...
iOS 14 on iPhone feature emergency

Apple Seeds Second Betas of iOS and iPadOS 14.6 to Developers

Friday April 30, 2021 10:17 am PDT by
Apple today seeded the second betas of new iOS and iPadOS 14.6 updates to developers for testing purposes, one week after seeding the first iOS and iPadOS 14.6 updates. iOS and iPadOS 14.6 can be downloaded through the Apple Developer Center or over the air after the proper profile has been installed on an iPhone or iPad. iOS and iPadOS 14.6 appear to be more minor changes focusing on...
Apple iPadPro iPadOS15 springboard widgets 060721 big

Apple Introduces iPadOS 15: First Look at New Features

Monday June 7, 2021 10:59 am PDT by
Apple today unveiled iPadOS 15, its next-generation operating system for iPad that introduces a slew of new features like widgets on the Home Screen, an iPhone-style App Library, new multi-tasking features, and more. Here's a rundown of what to expect. Widgets Like iOS 14, iPadOS 15 lets users place widgets anywhere on their Home Screen pages and among their apps. Widgets are also...
maxresdefault

Apple Releases iOS 14.5 and iPadOS 14.5 With Apple Watch iPhone Unlocking, Dual-SIM 5G Support, Apple Maps Accident Crowdsourcing and Tons More

Monday April 26, 2021 9:59 am PDT by
Apple today released iOS and iPadOS 14.5, marking the fifth major updates to the iOS and iPadOS 14 operating systems that were initially launched in September. iOS 14.5 comes three months after the release of iOS 14.4, and a month after the release of iOS 14.4.2, an update that introduced security fixes. Subscribe to the MacRumors YouTube channel for more videos. The iOS and iPadOS 14.5...
scribble ipados 14 5 beta

iPadOS 14.5 Beta Adds Apple Pencil Scribble Support for German, French, Spanish, Italian and Portuguese

Friday February 19, 2021 4:29 pm PST by
Apple in iPadOS 14 introduced an Apple Pencil feature called "Scribble," which lets users write in any text field on the iPad, with the handwritten text then converted automatically into typed text. At launch, Scribble was limited to the English and Chinese languages, but with iPadOS 14.5 beta that's available to developers and public beta testers, Apple is expanding Scribble support to...
iOS 14 on iPhone feature emergency

Apple Seeds First Betas of iOS and iPadOS 14.6 to Developers

Thursday April 22, 2021 1:29 pm PDT by
Apple today seeded the first betas of new iOS and iPadOS 14.6 updates to developers for testing purposes, with the new software available just ahead of when Apple plans to release iOS 14.5. iOS and iPadOS 14.6 can be downloaded through the Apple Developer Center or over the air after the proper profile has been installed on an iPhone or iPad. There is no word yet on what's included in...
14

Apple Seeds Eighth Betas of iOS 14.5 and iPadOS 14.5 to Developers [Update: Public Beta Available]

Tuesday April 13, 2021 10:06 am PDT by
Apple today seeded the eighth betas of upcoming iOS 14.5 and iPadOS 14.5 updates to developers for testing purposes, with the new beta updates coming one week after Apple released the seventh iOS and iPadOS 14.5 betas. iOS and iPadOS 14.5 can be downloaded through the Apple Developer Center or over the air after the proper profile has been installed on an iPhone or iPad. iOS 14.5 is the...
apple pencil 3 leak

Apple Again Rumored to Be Readying Third-Generation Apple Pencil

Thursday April 15, 2021 4:51 am PDT by
Apple is preparing to release a third-generation Apple Pencil, according to a leaker on the Chinese social networking platform Weibo, suggesting it could conceivably appear alongside new iPads at Apple's upcoming "Spring Loaded" April event. Image via Mr White. The claim comes from Weibo account "Uncle Pan Pan," which doesn't have much of a track record for sharing accurate Apple information, ...

Popular Stories

Upcoming Products 2022 Feature

Gurman: Apple Preparing 'Widest Array of New Hardware Products in Its History' for Fall

Sunday January 23, 2022 10:32 am PST by
Apple is working on a number of new products that are set to launch this fall, and Bloomberg's Mark Gurman says that it will be "the widest array" of new devices that Apple has introduced in its history. In his latest "Power On" newsletter, Gurman explains that Apple is working on four new flagship iPhones (iPhone 14, iPhone 14 Max, iPhone 14 Pro, and iPhone 14 Pro Max), an updated low-end Ma...
macbook pro 14 16 2021

Three Months After Launch, Apple Still Struggling to Meet Demand for Redesigned 14-Inch and 16-Inch MacBook Pro

Monday January 24, 2022 7:12 am PST by
Three months after their launch, the 14-inch and 16-inch MacBook Pros continue to experience high demand and seemingly short supply, with shipping dates for both models stretching into multiple weeks in several of Apple's key markets. In the United States, the baseline 14-inch MacBook Pro with the M1 Pro chip is estimated to ship in three to four weeks, promising an arrival by at least...
General Dropbox Feature

macOS 12.3 Will Include Cloud Storage Changes Affecting Dropbox and OneDrive

Tuesday January 25, 2022 3:31 pm PST by
Dropbox today announced that users who update to macOS 12.3 once that software version becomes available may temporarily encounter issues with opening online-only files in some third-party apps on their Mac. In a support document and an email to customers, Dropbox said it is actively working on full support for online-only files on macOS 12.3 and will begin rolling out an updated version of...
Questionable Design Decisions

Apple's Most Questionable Design Decisions in Recent Memory

Sunday January 23, 2022 2:59 am PST by
Apple has always emphasized the depth of thought that goes into the design of its products. In the foreword to Designed by Apple in California, a photo book released by the company in 2016, Jony Ive explains how Apple strives "to define objects that appear effortless" and "so simple, coherent and inevitable that there could be no rational alternative." But every once in a while even Apple...
Apple Watch Red Yellow Green Feature 1

Apple Launches Black Unity Braided Solo Loop With 'Unity Lights' Watch Face

Wednesday January 26, 2022 6:05 am PST by
Apple today announced the Black Unity Braided Solo Loop for the Apple Watch, as well as a new downloadable watch face, to celebrate Black History Month. Following the launch of the limited edition Black Unity Apple Watch Series 6 and Sport Band in 2021, Apple today launched the Black Unity Braided Solo Loop as part of its celebrations for Black History Month this year.Apple is launching a...
att gigabit internet

AT&T Bringing $180/Month 5-Gigabit Internet to 70 Cities

Monday January 24, 2022 9:20 am PST by
AT&T today announced the launch of upgraded AT&T Fiber plans, which support speeds of up to 5 Gigabits for some customers. There are two separate plans, one "2 GIG" plan and one "5 GIG" plan, available to new and existing AT&T Fiber subscribers. According to AT&T, the new plans are available to nearly 5.2 million customers across 70 metro areas including Los Angeles, Atlanta, Chicago, San...
intel vs m1 max chip purple

Benchmarks Confirm Intel's Latest Core i9 Chip Outperforms Apple's M1 Max With Several Caveats

Wednesday January 26, 2022 8:56 am PST by
Benchmark results have started to surface for MSI's new GE76 Raider, one of the first laptops to be powered by Intel's new 12th-generation Core i9 processor. Intel previously said that its new high-end Core i9 processor is faster than Apple's M1 Max chip in the 16-inch MacBook Pro and, as noted by Macworld, early Geekbench 5 results do appear to confirm this claim, but there are several...
ios 15

Apple Releases iOS 15.3 and iPadOS 15.3 With Fix for Safari Bug That Leaks Browsing Activity

Wednesday January 26, 2022 10:00 am PST by
Apple today released iOS 15.3 and iPadOS 15.3, the third major updates to the iOS and iPadOS 15 operating systems that were released in September 2021. iOS and iPadOS 15.3 come almost two weeks after the release of iOS and iPadOS 15.2.1, minor bug fix updates. The iOS 15.3 and iPadOS 15.3 updates can be downloaded for free and the software is available on all eligible devices over-the-air in ...