Third-Party Devs Will Be Able to Access iPadOS Apple Pencil Latency Improvements for Art Apps

Apple in iPadOS introduced some performance improvements between the iPad Pro and the Apple Pencil, cutting latency from 20ms to 9ms with the new software.

Third-party developers who make apps that use the ‌Apple Pencil‌ will also be able to take advantage of some of these latency improvements, Apple software development chief Craig Federighi confirmed last week.

ipadproapplepencil
Federighi shared the information in a response to an email sent by Artstudio Pro developer Cladio Juliano, who tweeted what Federighi had to say last week. The info was highlighted today in a tweet by developer Steve Troughton-Smith.

In the email, Federighi explains that third-party developers have had access to predicted touches via UIKit since iOS 9, and with iOS 13, developers will receive the "latest and greatest" touch prediction advancements in minimizing PencilKit drawing latency.

Federighi explains just how Apple introduced the latency improvements, and he points out that there's a small gap of 4ms that developers won't have access to at the current time because Apple didn't have a way to safely expose the capability to developers. From Federighi's email:

Note that we achieve low latency through a combination of several techniques: Metal rendering optimizations, touch prediction, and mid-frame event processing. Third-party developers can achieve similar low-latency drawing experiences by taking advantage of Metal rendering and touch prediction best practices covered in the WWDC Sessions I've referenced below.

With these you can achieve nearly all of the improvements you've seen in PencilKit drawing with your own renderer. (There does remain a small gap: 4 ms of our improvement comes from a technique called mid-frame event processing; we are looking for ways to expose this capability to third party engines in the future, but for this year this one was only safely achievable through tight integration within our frameworks).

For developers, the WWDC sessions Federighi suggests include PencilKit, Adopting Predicted Touches, and Metal Performance Optimization.

In a nutshell, the information shared by Federighi confirms that third-party apps that take advantage of the ‌Apple Pencil‌ will be getting some of the same latency improvements that we'll be seeing when using the ‌Apple Pencil‌ within native functions like Markup.

The ‌Apple Pencil‌ latency improvements are built into iPadOS, the version of iOS 13 that is designed to run on the iPad. All of Apple's current iPads support the ‌Apple Pencil‌. ‌iPad Pro‌ models work with the ‌Apple Pencil‌ 2, while the 6th-generation ‌iPad‌, iPad mini, and iPad Air work with the original ‌Apple Pencil‌.

Top Rated Comments

thisisnotmyname Avatar
26 months ago
I would have expected that was automatically exposed from the OS. I'm a bit surprised that they had to explicitly make some of those capabilities available to third party developers.
Score: 12 Votes (Like | Disagree)
cocky jeremy Avatar
26 months ago
I mean... duh?
Score: 6 Votes (Like | Disagree)
cmaier Avatar
26 months ago
I would have expected that was automatically exposed from the OS. I'm a bit surprised that they had to explicitly make some of those capabilities available to third party developers.
It is, if you use the appropriate control. But developer’s may want to integrate it into their own canvas or controls. In which case it is harder to expose it since things you do in your own code can interfere with the ability of the pen code to get the cycles it needs from the GPU and CPU.
Score: 4 Votes (Like | Disagree)
nexusrule Avatar
26 months ago
How nice of Apple. You would think they would limit functionality improvements to their own apps.
I think you don’t know how development works. When you start creating code you can’t always abstract it in a way that’s usable by third party devs through an API. What Federighi meant is right now the code that allow for that part of delay reduction is split between different of Apple software technologies. To be made safely accessible to others devs it needs to be abstracted, made indipendent, because private frameworks can’t be exposed for security reasons. You build these frameworks after you have the working feature, it’s simply impossible to abstract a solution that doesn’t exist. And this sort of work can require a massive rewrite of some parts of the relevant underlying technologies, and it requires time.
Score: 4 Votes (Like | Disagree)
NickName99 Avatar
26 months ago
I love that he gets into such detail. That’s interesting about the 4ms improvement they got using something they apparently can’t expose as a public method without some risk.

Now I’m curious about “mid-frame event processing”, but googling it hasn’t immediately got me anything.
Score: 4 Votes (Like | Disagree)
Cayden Avatar
26 months ago
I love that he gets into such detail. That’s interesting about the 4ms improvement they got using something they apparently can’t expose as a public method without some risk.

Now I’m curious about “mid-frame event processing”, but googling it hasn’t immediately got me anything.
Now I’m not sure so take this with a grain of salt, but as an engineer I’m inclined to believe “mid-frame event processing” means they are updating some pixel information (likely just the pixels associated with the pencil) in between frame updates in which all pixel information is updated and displayed. In other words, in between hardware detections of the pencil location, software would update where it presicts the pencil to be on the next update, and it can start looking for the pencil there instead of looking arbitrarily, mean the location can (usually) be found quicker. What I’m not sure about is if these pixels are actually being updated mid-frame or if the processing is simply keeping this information stored until the next frame is ready to update. I can’t see how the pixels could be updated mid-frame unless they had an individual refresh rate, so I’m inclined to believe the second case. If it’s the second case, it would make sense why Apple doesn’t want to give developers access to this, as this could quickly lead to timing errors between the software and hardware interrupts, such that it would only work within Apple’s framework and not an arbitrary code framework.
Score: 3 Votes (Like | Disagree)

Top Stories

YouTube Picture in Picture Feature

YouTube Says iOS Picture-in-Picture Coming to All US Users

Friday June 18, 2021 9:41 am PDT by
After a long wait, YouTube for iOS is officially gaining picture-in-picture support, allowing all users, non-premium and premium subscribers, to close the YouTube app and continue watching their video in a small pop-up window. In a statement to MacRumors, YouTube says that picture-in-picture is currently rolling out to all premium subscribers on iOS and that a larger rollout to all US iOS...
Top Stories 63 Feature

Top Stories: Beats Studio Buds Announced, Apple Watch Series 7 Rumors, and More

Saturday June 19, 2021 6:00 am PDT by
The Apple news cycle started to move beyond WWDC this week, but that doesn't mean there still wasn't a lot to talk about, led by the official debut of the much-leaked Beats Studio Buds that might give us a hint of what to expect for the second-generation AirPods Pro. With no hardware announcements at WWDC, we also took a look at when we might finally see the long-rumored redesigned MacBook...
ios wifi settings

iOS Bug Causes Specific Network Name to Disable Wi-Fi on iPhones

Sunday June 20, 2021 4:15 am PDT by
A wireless network naming bug has been discovered in iOS that effectively disables an iPhone's ability to connect to Wi-Fi. Security researcher Carl Schou found that after joining a Wi-Fi network with the name "%p%s%s%s%s%n" his iPhone's Wi-Fi functionality was left "permanently disabled." Changing a hotspot's SSID did nothing to correct the problem, with even a reboot failing to make a...
maxresdefault

Video: 20 Annoyances Apple Fixed in iOS 15 and macOS Monterey

Friday June 18, 2021 11:36 am PDT by
With iOS 15 and macOS Monterey, Apple is adding several quality of life improvements, which are designed to address some of the complaints that people have had with these operating systems for years now. Subscribe to the MacRumors YouTube channel for more videos. In our latest YouTube video, we're highlighting some of our favorite "fix" features that address long-running problems in iOS and...
16 inch macbook pro m2 render

When Can We Expect the Redesigned MacBook Pros Now?

Wednesday June 16, 2021 7:11 am PDT by
With no sign of redesigned MacBook Pro models at this year's WWDC, when can customers expect the much-anticipated new models to launch? A number of reports, including investor notes from Morgan Stanley and Wedbush analysts, claimed that new MacBook Pro models would be coming during this year's WWDC. This did not happen, much to the disappointment of MacBook Pro fans, who have been...
space gray magic accessories trio

Apple Stops Selling Magic Accessories in Space Gray

Friday June 18, 2021 9:16 am PDT by
Apple this week stopped selling its Magic Keyboard with Numeric Keypad, Magic Mouse 2, and Magic Trackpad 2 accessories for the Mac in a Space Gray color, around three months after discontinuing the iMac Pro, which also came in Space Gray. Last month, Apple listed the Space Gray accessories as available while supplies last, and the company has now removed the product pages from its website...
m1 v intel thumb

Intel Processor Market Share May Fall to New Low Next Year Due to Apple Silicon

Friday June 18, 2021 2:06 am PDT by
Intel may see its market share fall to a new low next year, in large part thanks to Apple's decision to move away from using Intel processors in its Mac computers and instead use Apple silicon. Apple announced last year that it would embark on a two-year-long journey to transition all of its Mac computers, both desktops, and laptops, to use its own in-house processors. Apple is expected to...
3nm apple silicon feature

Apple Supplier TSMC Readies 3nm Chip Production for Second Half of 2022

Friday June 18, 2021 6:59 am PDT by
Apple supplier TSMC is preparing to produce 3nm chips in the second half of 2022, and in the coming months, the supplier will begin production of 4nm chips, according to a new report from DigiTimes. Apple had previously booked the initial capacity of TSMC's 4nm chip production for future Macs and more recently ordered TSMC to begin production of the A15 chip for the upcoming iPhone 13,...
2021 back t0 school

Apple Launches 2021 Back to School Promotion: Free AirPods With Eligible Mac or iPad Purchase

Thursday June 17, 2021 4:56 am PDT by
Apple today launched its seasonal back-to-school sale for the upcoming school year in the United States and Canada, offering students free AirPods alongside purchases of select Macs and iPad models. Similar to last year's promotion, this year's offer includes free AirPods alongside the purchase of a MacBook Air, MacBook Pro, the new 24-inch iMac, the Mac Pro, Mac mini, and the new M1-powered ...
applecare lower prices

Apple Lowers Prices of AppleCare+ Plans for M1 MacBook Air and MacBook Pro

Thursday June 17, 2021 7:33 am PDT by
Apple today lowered the prices of AppleCare+ plans for MacBook Air and 13-inch MacBook Pro models equipped with the M1 chip. Coverage offered by the plans, as well as accidental damage fees, appear to remain unchanged. In the United States, AppleCare+ for the MacBook Air now costs $199, down from $249. The new price applies to both M1 and Intel-based MacBook Air models, although Apple no...