Third-Party Devs Will Be Able to Access iPadOS Apple Pencil Latency Improvements for Art Apps

Apple in iPadOS introduced some performance improvements between the iPad Pro and the Apple Pencil, cutting latency from 20ms to 9ms with the new software.

Third-party developers who make apps that use the ‌Apple Pencil‌ will also be able to take advantage of some of these latency improvements, Apple software development chief Craig Federighi confirmed last week.

ipadproapplepencil
Federighi shared the information in a response to an email sent by Artstudio Pro developer Cladio Juliano, who tweeted what Federighi had to say last week. The info was highlighted today in a tweet by developer Steve Troughton-Smith.

In the email, Federighi explains that third-party developers have had access to predicted touches via UIKit since iOS 9, and with iOS 13, developers will receive the "latest and greatest" touch prediction advancements in minimizing PencilKit drawing latency.

Federighi explains just how Apple introduced the latency improvements, and he points out that there's a small gap of 4ms that developers won't have access to at the current time because Apple didn't have a way to safely expose the capability to developers. From Federighi's email:

Note that we achieve low latency through a combination of several techniques: Metal rendering optimizations, touch prediction, and mid-frame event processing. Third-party developers can achieve similar low-latency drawing experiences by taking advantage of Metal rendering and touch prediction best practices covered in the WWDC Sessions I've referenced below.

With these you can achieve nearly all of the improvements you've seen in PencilKit drawing with your own renderer. (There does remain a small gap: 4 ms of our improvement comes from a technique called mid-frame event processing; we are looking for ways to expose this capability to third party engines in the future, but for this year this one was only safely achievable through tight integration within our frameworks).

For developers, the WWDC sessions Federighi suggests include PencilKit, Adopting Predicted Touches, and Metal Performance Optimization.

In a nutshell, the information shared by Federighi confirms that third-party apps that take advantage of the ‌Apple Pencil‌ will be getting some of the same latency improvements that we'll be seeing when using the ‌Apple Pencil‌ within native functions like Markup.

The ‌Apple Pencil‌ latency improvements are built into iPadOS, the version of iOS 13 that is designed to run on the iPad. All of Apple's current iPads support the ‌Apple Pencil‌. ‌iPad Pro‌ models work with the ‌Apple Pencil‌ 2, while the 6th-generation ‌iPad‌, iPad mini, and iPad Air work with the original ‌Apple Pencil‌.

Top Rated Comments

thisisnotmyname Avatar
24 months ago
I would have expected that was automatically exposed from the OS. I'm a bit surprised that they had to explicitly make some of those capabilities available to third party developers.
Score: 12 Votes (Like | Disagree)
cocky jeremy Avatar
24 months ago
I mean... duh?
Score: 6 Votes (Like | Disagree)
cmaier Avatar
24 months ago
I would have expected that was automatically exposed from the OS. I'm a bit surprised that they had to explicitly make some of those capabilities available to third party developers.
It is, if you use the appropriate control. But developer’s may want to integrate it into their own canvas or controls. In which case it is harder to expose it since things you do in your own code can interfere with the ability of the pen code to get the cycles it needs from the GPU and CPU.
Score: 4 Votes (Like | Disagree)
nexusrule Avatar
24 months ago
How nice of Apple. You would think they would limit functionality improvements to their own apps.
I think you don’t know how development works. When you start creating code you can’t always abstract it in a way that’s usable by third party devs through an API. What Federighi meant is right now the code that allow for that part of delay reduction is split between different of Apple software technologies. To be made safely accessible to others devs it needs to be abstracted, made indipendent, because private frameworks can’t be exposed for security reasons. You build these frameworks after you have the working feature, it’s simply impossible to abstract a solution that doesn’t exist. And this sort of work can require a massive rewrite of some parts of the relevant underlying technologies, and it requires time.
Score: 4 Votes (Like | Disagree)
NickName99 Avatar
24 months ago
I love that he gets into such detail. That’s interesting about the 4ms improvement they got using something they apparently can’t expose as a public method without some risk.

Now I’m curious about “mid-frame event processing”, but googling it hasn’t immediately got me anything.
Score: 4 Votes (Like | Disagree)
Cayden Avatar
24 months ago
I love that he gets into such detail. That’s interesting about the 4ms improvement they got using something they apparently can’t expose as a public method without some risk.

Now I’m curious about “mid-frame event processing”, but googling it hasn’t immediately got me anything.
Now I’m not sure so take this with a grain of salt, but as an engineer I’m inclined to believe “mid-frame event processing” means they are updating some pixel information (likely just the pixels associated with the pencil) in between frame updates in which all pixel information is updated and displayed. In other words, in between hardware detections of the pencil location, software would update where it presicts the pencil to be on the next update, and it can start looking for the pencil there instead of looking arbitrarily, mean the location can (usually) be found quicker. What I’m not sure about is if these pixels are actually being updated mid-frame or if the processing is simply keeping this information stored until the next frame is ready to update. I can’t see how the pixels could be updated mid-frame unless they had an individual refresh rate, so I’m inclined to believe the second case. If it’s the second case, it would make sense why Apple doesn’t want to give developers access to this, as this could quickly lead to timing errors between the software and hardware interrupts, such that it would only work within Apple’s framework and not an arbitrary code framework.
Score: 3 Votes (Like | Disagree)

Top Stories

samsung experience 1

Samsung's 'iTest' Lets You Try a Galaxy Device on Your iPhone

Thursday April 8, 2021 12:42 pm PDT by
Samsung has launched "iTest," an interactive website experience that's designed to allow iPhone users to test out Android on a Galaxy device, or "sample the other side," as Samsung puts it. Subscribe to the MacRumors YouTube channel for more videos. The iTest website is being advertised in New Zealand, according to a MacRumors reader who came across the feature. Visiting the iTest website on...
sonny 2021 ipad mini pro dummies

Leaked Dummy Units Show iPad Mini 6 With Thick Bezels and Home Button, New iPad Pro Models

Thursday April 8, 2021 2:11 am PDT by
Rumors suggest Apple will release refreshed versions of the iPad mini and iPad Pro models in the first half of this year, potentially as soon as this month, and a new leak today has provided us with a possible preview of what to expect in terms of the devices' overall design and camera prospects. Tech leaker and Apple blogger Sonny Dickson this morning shared images on Twitter showing dummy ...
fake airpods 3

Counterfeit 'AirPods 3' Hit the Market Prior to Official Announcement

Friday April 9, 2021 2:45 am PDT by
Apple is expected to launch the third iteration of AirPods in the third quarter of this year. Rumors and reports suggest the new AirPods will feature an updated design more in line with the AirPods Pro, but lacking in "Pro" features such as active noise cancellation. Despite AirPods 3 not yet being officially announced by Apple, counterfeit products of the unreleased earbuds have already hit ...
iMessage Android featured

Apple's Rationale for Not Bringing iMessage to Android Revealed in Legal Documents

Friday April 9, 2021 2:22 am PDT by
It's no secret that Apple sees iMessage as a big enough selling point to keep the service exclusive to Apple devices, however new court filings submitted by Epic Games in its ongoing lawsuit with the company reveal just how Apple executives have rationalized their decision not to develop a version of iMessage for Android. Apple clearly recognizes the power that iMessage has to keep users...
nba tracking prompt orange

Two-Thirds of iPhone Users Expected to Block Ad Tracking

Friday April 9, 2021 7:19 am PDT by
As many as 68 percent of iPhone users are expected to deny advertisers permission to track them thanks to Apple's App Tracking Transparency feature, in what is beginning to look like a significant blow to the advertising industry (via AdWeek). With the launch of iOS 14.5, apps will have to receive explicit user permission before accessing an iPhone's advertising identifier or IDFA, which is...
iPhone 13 Battery Life Feature

DigiTimes: iPhone 13 Pro Models to Feature 120Hz ProMotion Refresh Rate and 15-20% Less Power Consumption

Friday April 9, 2021 12:52 am PDT by
The two premium "Pro" models of the upcoming iPhone 13 lineup will be equipped with a low-power LTPO display, enabling the iPhones to have a 120Hz refresh rate, according to industry sources cited by Taiwanese publication DigiTimes. According to today's paywalled report, Apple suppliers Samsung and LG Display are in the process of converting parts of their production capacity to produce LTPO ...
ehric

iPhone 12 Mini Missing From Top 5 Best Selling Smartphone List of January 2021

Friday April 9, 2021 4:58 am PDT by
According to market data compiled by Counterpoint Research, Apple's smallest iPhone since the 2016 iPhone SE, the iPhone 12 mini, struggled to obtain a spot in the top five list of best-selling smartphones in January of this year. According to the market data, the iPhone 12 mini came in eighth place for the best-selling smartphone worldwide in the first month of the year. However, the iPhone ...
Top Stories 56 Feature

Top Stories: 'Find My' Expansion, iPhone 13 Pro Mockup, Largest-Ever iMac?

Saturday April 10, 2021 6:00 am PDT by
Apple's AirTags may still be nowhere to be seen nearly two years after signs of them were first discovered, but Apple this week launched its Find My network accessory program that will let third-party devices integrate with the Find My app on Apple's platforms to make it easy to keep track of your items. This week also saw fresh rumors about the upcoming "iPhone 13" and new iMacs, while...
fortnite apple featured

Epic's Case Against Apple in Australia Might Be Over

Friday April 9, 2021 12:21 pm PDT by
Epic Games and Apple are involved in a highly public lawsuit in the United States, but Epic has also sued Apple in other countries, including Australia, in an attempt to boost its chances of getting a favorable ruling. Things aren't quite going Epic's way, however, as the justice overseeing the case today decided that he's going to let the two companies battle it out in the United States. ...
Apple TV Ray Light Teal Film

Apple Wants to Draw Subscribers to Apple TV+ With More Feature Films

Friday April 9, 2021 10:27 am PDT by
Apple is ramping up its efforts to produce feature films, according to a new report from The Information. Apple has a limited number of high profile films already, including "Palmer," "Cherry," and "Greyhound," and sees it as a way to draw more people into Apple TV+. Apple has been telling studio executives that it is aiming to do more than 10 to 12 original movies per year for Apple TV+,...