Third-Party Devs Will Be Able to Access iPadOS Apple Pencil Latency Improvements for Art Apps

Apple in iPadOS introduced some performance improvements between the iPad Pro and the Apple Pencil, cutting latency from 20ms to 9ms with the new software.

Third-party developers who make apps that use the ‌Apple Pencil‌ will also be able to take advantage of some of these latency improvements, Apple software development chief Craig Federighi confirmed last week.

ipadproapplepencil
Federighi shared the information in a response to an email sent by Artstudio Pro developer Cladio Juliano, who tweeted what Federighi had to say last week. The info was highlighted today in a tweet by developer Steve Troughton-Smith.

In the email, Federighi explains that third-party developers have had access to predicted touches via UIKit since iOS 9, and with iOS 13, developers will receive the "latest and greatest" touch prediction advancements in minimizing PencilKit drawing latency.

Federighi explains just how Apple introduced the latency improvements, and he points out that there's a small gap of 4ms that developers won't have access to at the current time because Apple didn't have a way to safely expose the capability to developers. From Federighi's email:

Note that we achieve low latency through a combination of several techniques: Metal rendering optimizations, touch prediction, and mid-frame event processing. Third-party developers can achieve similar low-latency drawing experiences by taking advantage of Metal rendering and touch prediction best practices covered in the WWDC Sessions I've referenced below.

With these you can achieve nearly all of the improvements you've seen in PencilKit drawing with your own renderer. (There does remain a small gap: 4 ms of our improvement comes from a technique called mid-frame event processing; we are looking for ways to expose this capability to third party engines in the future, but for this year this one was only safely achievable through tight integration within our frameworks).

For developers, the WWDC sessions Federighi suggests include PencilKit, Adopting Predicted Touches, and Metal Performance Optimization.

In a nutshell, the information shared by Federighi confirms that third-party apps that take advantage of the ‌Apple Pencil‌ will be getting some of the same latency improvements that we'll be seeing when using the ‌Apple Pencil‌ within native functions like Markup.

The ‌Apple Pencil‌ latency improvements are built into iPadOS, the version of iOS 13 that is designed to run on the iPad. All of Apple's current iPads support the ‌Apple Pencil‌. ‌iPad Pro‌ models work with the ‌Apple Pencil‌ 2, while the 6th-generation ‌iPad‌, iPad mini, and iPad Air work with the original ‌Apple Pencil‌.

Related Forum: iOS 13

Top Rated Comments

thisisnotmyname Avatar
58 months ago
I would have expected that was automatically exposed from the OS. I'm a bit surprised that they had to explicitly make some of those capabilities available to third party developers.
Score: 12 Votes (Like | Disagree)
cocky jeremy Avatar
58 months ago
I mean... duh?
Score: 6 Votes (Like | Disagree)
cmaier Avatar
58 months ago
I would have expected that was automatically exposed from the OS. I'm a bit surprised that they had to explicitly make some of those capabilities available to third party developers.
It is, if you use the appropriate control. But developer’s may want to integrate it into their own canvas or controls. In which case it is harder to expose it since things you do in your own code can interfere with the ability of the pen code to get the cycles it needs from the GPU and CPU.
Score: 4 Votes (Like | Disagree)
nexusrule Avatar
58 months ago
How nice of Apple. You would think they would limit functionality improvements to their own apps.
I think you don’t know how development works. When you start creating code you can’t always abstract it in a way that’s usable by third party devs through an API. What Federighi meant is right now the code that allow for that part of delay reduction is split between different of Apple software technologies. To be made safely accessible to others devs it needs to be abstracted, made indipendent, because private frameworks can’t be exposed for security reasons. You build these frameworks after you have the working feature, it’s simply impossible to abstract a solution that doesn’t exist. And this sort of work can require a massive rewrite of some parts of the relevant underlying technologies, and it requires time.
Score: 4 Votes (Like | Disagree)
NickName99 Avatar
58 months ago
I love that he gets into such detail. That’s interesting about the 4ms improvement they got using something they apparently can’t expose as a public method without some risk.

Now I’m curious about “mid-frame event processing”, but googling it hasn’t immediately got me anything.
Score: 4 Votes (Like | Disagree)
Cayden Avatar
58 months ago
I love that he gets into such detail. That’s interesting about the 4ms improvement they got using something they apparently can’t expose as a public method without some risk.

Now I’m curious about “mid-frame event processing”, but googling it hasn’t immediately got me anything.
Now I’m not sure so take this with a grain of salt, but as an engineer I’m inclined to believe “mid-frame event processing” means they are updating some pixel information (likely just the pixels associated with the pencil) in between frame updates in which all pixel information is updated and displayed. In other words, in between hardware detections of the pencil location, software would update where it presicts the pencil to be on the next update, and it can start looking for the pencil there instead of looking arbitrarily, mean the location can (usually) be found quicker. What I’m not sure about is if these pixels are actually being updated mid-frame or if the processing is simply keeping this information stored until the next frame is ready to update. I can’t see how the pixels could be updated mid-frame unless they had an individual refresh rate, so I’m inclined to believe the second case. If it’s the second case, it would make sense why Apple doesn’t want to give developers access to this, as this could quickly lead to timing errors between the software and hardware interrupts, such that it would only work within Apple’s framework and not an arbitrary code framework.
Score: 3 Votes (Like | Disagree)

Popular Stories

apple card 1

Apple Ending Apple Card Partnership With Goldman Sachs

Tuesday November 28, 2023 3:09 pm PST by
Apple is ending its credit card partnership with Goldman Sachs, according to The Wall Street Journal. Apple plans to stop working with Goldman Sachs in the next 12 to 15 months, and it is not yet clear if Apple has established a new partnership for the Apple Card. Apple and Goldman Sachs will dissolve their entire consumer partnership, including the Apple Card and the Apple Savings account....
iOS 17

Everything New in iOS 17.2 Beta 4

Tuesday November 28, 2023 12:18 pm PST by
Apple is wrapping up development on iOS 17.2, with the update expected to come out in December. While we're getting to the end of the beta testing period, Apple is still tweaking features and adding new functionality. We've rounded up everything new in the fourth beta of iOS 17.2. Default Notification Sound Under Sounds & Haptics, there's a new "Default Alerts" section that allows you to ...
iOS 17

Apple Releases iOS 17.1.2 With Security Fixes

Thursday November 30, 2023 10:12 am PST by
Apple today released iOS 17.1.2 and iPadOS 17.1.2, small updates to the iOS 17 and iPadOS 17 operating systems that Apple introduced in September. iOS 17.1.2 and iPadOS 17.1.2 come a few weeks after the release of iOS 17.1.1, another bug fix update. iOS 17.1.2 and iPadOS 17.1.2 can be downloaded on eligible iPhones and iPads over-the-air by going to Settings > General > Software Update....
Apple 5G Modem Feature Triad

Apple to Discontinue Custom 5G Modem Development, Claim Reports

Wednesday November 29, 2023 4:19 am PST by
Apple is discontinuing in-house modem development after several unsuccessful attempts to perfect its own custom 5G modem chip, according to unconfirmed reports coming out of Asia. According to the operator of news aggregator account "yeux1122" on the Naver blog, supply chain sources related to Apple's 5G modem departments claim that the company's attempts to develop its own modem have...
All New CarPlay Five New Features Article 2

What to Expect From All-New CarPlay, Still Listed as Coming 'Late 2023'

Tuesday November 28, 2023 7:44 am PST by
At WWDC in June 2022, Apple previewed the next generation of CarPlay, promising deeper integration with vehicle functions like A/C and FM radio, support for multiple displays across the dashboard, increased personalization, and more. Apple's website still says the first vehicles with support for the next-generation CarPlay experience will be announced in "late 2023," but it has not shared...
Apple Logo

Apple Discontinued These 5 Products This Year

Monday November 27, 2023 7:03 am PST by
As the end of 2023 nears, now is a good opportunity to look back at some of the devices and accessories that Apple discontinued throughout the year. Apple products discontinued in 2023 include the iPhone 13 mini, 13-inch MacBook Pro, MagSafe Battery Pack, MagSafe Duo Charger, and leather accessories. Also check out our lists of Apple products discontinued in 2022 and 2021. iPhone Mini ...
iOS 17

iOS 17.1.2 Update for iPhone Likely to Be Released This Week

Monday November 27, 2023 8:24 am PST by
Apple will likely release iOS 17.1.2 this week, based on mounting evidence of the software in our website's analytics logs in recent days. As a minor update, iOS 17.1.2 should be focused on bug fixes, but it's unclear exactly which issues might be addressed. Some users have continued to experience Wi-Fi issues on iOS 17.1.1, so perhaps iOS 17.1.2 will include the same fix for Wi-Fi...
iOS 17

28 New Things Your iPhone Can Do in December's iOS 17.2 Update

Friday December 1, 2023 2:57 am PST by
Apple made the first beta of iOS 17.2 available to developers in October. Since then we've seen three more betas, and with each iteration Apple continues to add more new features and changes, many of which users have been anticipating for quite a while. Below, we've listed 28 new things that are coming to your iPhone when the finalized version is publicly released this December. 1. Help...