Craig Federighi Explains How Scribble Feature in iPadOS 14 Was Developed

Scribble is one of the most notable new features in iPadOS 14, and in a new interview with Popular Mechanics, Apple SVP of software engineering Craig Federighi has revealed more details about how it was developed and how it works on iPad.


With Scribble in ‌iPadOS 14‌, you can use an ‌Apple Pencil‌ to write in any text field, and the written text is then converted to typed text. The feature means you can use the ‌‌Apple Pencil‌‌ without having to swap over to a keyboard when you need to compose an email, write a Calendar event, visit a URL, send an iMessage, or conduct a search.

Federighi explained to Popular Mechanics how Scribble's early development relied on extensive data gathering involving analyzing how people from all over the world write things down.

"When it comes to understanding [handwriting] strokes, we do data-gathering. We find people all over the world, and have them write things," says Craig Federighi, senior vice president of software engineering at Apple. "We give them a Pencil, and we have them write fast, we have them write slow, write at a tilt. All of this variation."

That methodology is distinct from the comparatively simple approach of scanning and analyzing existing handwriting. Federighi says that for Apple's tech, static examples weren't enough. They needed to see the strokes that formed each letter. "If you understand the strokes and how the strokes went down, that can be used to disambiguate what was being written."

The job of integrating stroke-based recognition with character and word prediction is all done on-device, something that machine learning and raw processing power makes possible, explained Apple's software chief.

"It's gotta be happening in real time, right now, on the device that you're holding," Federighi says. "Which means that the computational power of the device has to be such that it can do that level of processing locally."

Subscribe to the MacRumors YouTube channel for more videos.

Thanks to Scribble and ‌iPadOS 14‌, an ‌iPad‌‌ can be used almost entirely with touch and tools, and there's little need to type. If you're someone who prefers writing by hand, ‌Scribble‌ introduces tools that only enhance the experience. Check out the hands-on video above to see Scribble in action.

Related Roundups: iOS 14, iPadOS 14

Top Rated Comments

(View all)
Avatar
3 weeks ago


Everyday, thousands of MacBook screens are touched and people are bewildered that nothing happens. Touchscreens have been standard for over a decade now - it’s bizarre that there are computers costing thousands of dollars that don’t have touchscreens.

The fact that it’d be used so little didn’t stop Apple from including the touchbar - why on earth haven’t they made their products “Just Work” by including a touchscreen yet?

Seriously - have any kid who hasn’t used a Mac before use a MacBook for a few hours. I guarantee they’ll touch the screen at least once and consider it broken when nothing happens. Literally every phone, tablet, and non-Apple-or-budget laptop has a touchscreen - it’s just a given that a MacBook should, too.

I have read and reread your rationale as to why Macs ought to have touchscreens and it seems like your argument boils down to "just because".

And honestly speaking, I haven't exactly seen touchscreen PCs take the world by storm either. Yes, they are there, and I suppose having a feature you hardly use is better than not having it, but it feels like this is there purely to tick off a checkbox, not so much because it legitimately improves the end user experience.
Score: 13 Votes (Like | Disagree)
Avatar
3 weeks ago


inb4 people say "steve jobs rolling in his grave" and bringing up the stylus quote...

The stylus quote was clearly a shorthand of "who wants to rely on a stylus exclusively to interact with the UI, when you already have ten fingers?". Using a stylus to write handwritten text or draw something is quite different from that.
Score: 11 Votes (Like | Disagree)
Avatar
3 weeks ago
Newton technology all the way. I’ve been surprised they haven’t had it on iPhone and iPad since day one.
Score: 11 Votes (Like | Disagree)
Avatar
3 weeks ago
I would not be surprised if it's Apple Pencil support and not touch support that comes to the AS Macs.
Score: 7 Votes (Like | Disagree)
Avatar
3 weeks ago


You may well be right but many tech commentators seem to be convinced that touch screen Macs are coming. Hopefully not too much longer before we know for sure.

Everyday, thousands of MacBook screens are touched and people are bewildered that nothing happens. Touchscreens have been standard for over a decade now - it’s bizarre that there are computers costing thousands of dollars that don’t have touchscreens.

The fact that it’d be used so little didn’t stop Apple from including the touchbar - why on earth haven’t they made their products “Just Work” by including a touchscreen yet?

Seriously - have any kid who hasn’t used a Mac before use a MacBook for a few hours. I guarantee they’ll touch the screen at least once and consider it broken when nothing happens. Literally every phone, tablet, and non-Apple-or-budget laptop has a touchscreen - it’s just a given that a MacBook should, too.
Score: 6 Votes (Like | Disagree)
Avatar
3 weeks ago
It’s amazing how much more there is to this than just meets the eye. I generally don’t think about on device processing and learning. I’ve been using Scribble since iPadOS 14 came out and it’s really useful for converting notes into text.
Score: 6 Votes (Like | Disagree)

Top Stories

iPhone 12 Pro in Graphite and iPhone 12 in Blue Shown Off in Unboxing Videos

Monday October 19, 2020 8:20 am PDT by
While the iPhone 12 Pro does not launch until Friday, we now have an early unboxing video of the device courtesy of Twitter account DuanRui, providing a closer look at the shiny new flat-edge design and sleek Graphite color option. Ben Geskin re-uploaded the unboxing video to YouTube, which we've embedded below: Geskin has also uploaded an unboxing video of the iPhone 12 in Blue: ...

Kuo: iPhone 12 Pro Demand Higher Than Expected

Sunday October 18, 2020 10:39 pm PDT by
TF International Securities analyst Ming-Chi Kuo released a research note this morning detailing what he's seen with the volume of iPhone 12 and iPhone 12 Pro pre-orders in the first weekend of sales. Kuo had previously indicated that Apple's estimated shipment allocations for the new iPhone models placed the iPhone 12 at the top with 40-45% of inventory allocation (up from 15-20%). However, ...

Apple's New MagSafe Charger and Cases Begin Arriving to Customers

Saturday October 17, 2020 10:10 am PDT by
Apple's new MagSafe charger and cases have begun arriving to some customers earlier than expected, and images of the accessories have started to surface on Twitter. The photos provide a first look at the products in real-world use. As of writing, some MagSafe cases are also available for pickup at select Apple Stores in countries like the United States, Canada, and Germany. Filip...

HomePod Mini Cable is Non-Detachable, Ends With USB-C Connector for Use With Included 20W Power Adapter

Friday October 16, 2020 12:45 pm PDT by
While not detailed in the tech specs, MacRumors can confirm that Apple's new HomePod mini features a non-detachable power cable that ends with a USB-C connector for use with the 20W power adapter included in the box. With the switch to USB-C, the HomePod mini could potentially be powered by a wider range of devices and peripherals, ranging from MacBooks to USB-C battery packs with enough...

Samsung Mocks Apple for Ditching Power Adapters With iPhone 12 Lineup

Thursday October 15, 2020 11:51 am PDT by
Samsung on its social channels is mocking Apple for removing the power adapter from the iPhone 12 lineup and other iPhone models, pointing out the fact that the Samsung Galaxy smartphones continue to ship with a power adapter. "Included with your Galaxy," reads a Samsung Facebook post that features a picture of a power adapter. Apple notably is no longer providing power adapters or...

New Google App Feature Lets You Hum a Song to Search for It

Saturday October 17, 2020 4:05 am PDT by
Google has added a new feature to its Search app that allows you to hum a song that's stuck in your head, and then use the company's machine learning algorithm to try and identify it. In the Google app or using the Google Search widget, tap the mic icon and say "what's this song?" or click the "Search a song" button. Then start humming the tune for 10-15 seconds. When you're done, the...

Brazilian Certifications Suggest iPhone 12 Mini Features 2,227mAh Battery and iPhone 12 Has 2,815mAh Battery

Friday October 16, 2020 1:08 pm PDT by
Apple's iPhone mini has the shortest battery life out of all the iPhones in the iPhone 12 lineup due to its small size, but Apple has not provided public information about the battery's capacity. A regulatory filing from Brazil, however, suggests the iPhone 12 mini has a battery capacity of 2,227mAh. The same regulatory information says the iPhone 12 features a 2,815mAh battery, which is...

iPhone 12 Pro Pre-Orders Already Selling Out With Delivery Times Pushing Into November

Friday October 16, 2020 6:35 am PDT by
Apple today opened pre-orders for the 6.1-inch models of the iPhone 12 and iPhone 12 Pro through its website and the Apple Store app, and estimated delivery times are already slipping into November for select configurations in the United States. Customers ordering a SIM-free/Pacific Blue/128GB version of the iPhone 12 Pro, for example, are already facing an estimated delivery window of...

Hands-On With Apple's MagSafe Charger for iPhone 12

Monday October 19, 2020 11:54 am PDT by
Alongside the new iPhone 12 models, Apple introduced a MagSafe charger that attaches to the back of the iPhones using magnets embedded both in the charger and in the iPhone. It allows for speedier charging and paves the way for a portless iPhone in the future. MagSafe chargers are shipping out and are in some Apple retail locations now, and we picked one up to check it out. Subscribe to the ...

Apple Offering Free AirPods With iPhone 11 Purchase in India as Part of Diwali Celebration

Friday October 16, 2020 12:35 pm PDT by
Apple today launched a new Diwali promotion in India that will see the company providing customers with a set of AirPods with the purchase of any iPhone 11 model. The new iPhone 12 models are not part of the promotion. Apple is offering the standard AirPods With Charging Case free with purchase, but customers can choose to upgrade to the AirPods with Wireless Charging Case or the AirPods Pro....