Third-Party Devs Will Be Able to Access iPadOS Apple Pencil Latency Improvements for Art Apps

Apple in iPadOS introduced some performance improvements between the iPad Pro and the Apple Pencil, cutting latency from 20ms to 9ms with the new software.

Third-party developers who make apps that use the ‌Apple Pencil‌ will also be able to take advantage of some of these latency improvements, Apple software development chief Craig Federighi confirmed last week.

ipadproapplepencil
Federighi shared the information in a response to an email sent by Artstudio Pro developer Cladio Juliano, who tweeted what Federighi had to say last week. The info was highlighted today in a tweet by developer Steve Troughton-Smith.

In the email, Federighi explains that third-party developers have had access to predicted touches via UIKit since iOS 9, and with iOS 13, developers will receive the "latest and greatest" touch prediction advancements in minimizing PencilKit drawing latency.

Federighi explains just how Apple introduced the latency improvements, and he points out that there's a small gap of 4ms that developers won't have access to at the current time because Apple didn't have a way to safely expose the capability to developers. From Federighi's email:

Note that we achieve low latency through a combination of several techniques: Metal rendering optimizations, touch prediction, and mid-frame event processing. Third-party developers can achieve similar low-latency drawing experiences by taking advantage of Metal rendering and touch prediction best practices covered in the WWDC Sessions I've referenced below.

With these you can achieve nearly all of the improvements you've seen in PencilKit drawing with your own renderer. (There does remain a small gap: 4 ms of our improvement comes from a technique called mid-frame event processing; we are looking for ways to expose this capability to third party engines in the future, but for this year this one was only safely achievable through tight integration within our frameworks).

For developers, the WWDC sessions Federighi suggests include PencilKit, Adopting Predicted Touches, and Metal Performance Optimization.

In a nutshell, the information shared by Federighi confirms that third-party apps that take advantage of the ‌Apple Pencil‌ will be getting some of the same latency improvements that we'll be seeing when using the ‌Apple Pencil‌ within native functions like Markup.

The ‌Apple Pencil‌ latency improvements are built into iPadOS, the version of iOS 13 that is designed to run on the iPad. All of Apple's current iPads support the ‌Apple Pencil‌. ‌iPad Pro‌ models work with the ‌Apple Pencil‌ 2, while the 6th-generation ‌iPad‌, iPad mini, and iPad Air work with the original ‌Apple Pencil‌.

Top Rated Comments

thisisnotmyname Avatar
34 months ago
I would have expected that was automatically exposed from the OS. I'm a bit surprised that they had to explicitly make some of those capabilities available to third party developers.
Score: 12 Votes (Like | Disagree)
cocky jeremy Avatar
34 months ago
I mean... duh?
Score: 6 Votes (Like | Disagree)
cmaier Avatar
34 months ago
I would have expected that was automatically exposed from the OS. I'm a bit surprised that they had to explicitly make some of those capabilities available to third party developers.
It is, if you use the appropriate control. But developer’s may want to integrate it into their own canvas or controls. In which case it is harder to expose it since things you do in your own code can interfere with the ability of the pen code to get the cycles it needs from the GPU and CPU.
Score: 4 Votes (Like | Disagree)
nexusrule Avatar
34 months ago
How nice of Apple. You would think they would limit functionality improvements to their own apps.
I think you don’t know how development works. When you start creating code you can’t always abstract it in a way that’s usable by third party devs through an API. What Federighi meant is right now the code that allow for that part of delay reduction is split between different of Apple software technologies. To be made safely accessible to others devs it needs to be abstracted, made indipendent, because private frameworks can’t be exposed for security reasons. You build these frameworks after you have the working feature, it’s simply impossible to abstract a solution that doesn’t exist. And this sort of work can require a massive rewrite of some parts of the relevant underlying technologies, and it requires time.
Score: 4 Votes (Like | Disagree)
NickName99 Avatar
34 months ago
I love that he gets into such detail. That’s interesting about the 4ms improvement they got using something they apparently can’t expose as a public method without some risk.

Now I’m curious about “mid-frame event processing”, but googling it hasn’t immediately got me anything.
Score: 4 Votes (Like | Disagree)
Cayden Avatar
34 months ago
I love that he gets into such detail. That’s interesting about the 4ms improvement they got using something they apparently can’t expose as a public method without some risk.

Now I’m curious about “mid-frame event processing”, but googling it hasn’t immediately got me anything.
Now I’m not sure so take this with a grain of salt, but as an engineer I’m inclined to believe “mid-frame event processing” means they are updating some pixel information (likely just the pixels associated with the pencil) in between frame updates in which all pixel information is updated and displayed. In other words, in between hardware detections of the pencil location, software would update where it presicts the pencil to be on the next update, and it can start looking for the pencil there instead of looking arbitrarily, mean the location can (usually) be found quicker. What I’m not sure about is if these pixels are actually being updated mid-frame or if the processing is simply keeping this information stored until the next frame is ready to update. I can’t see how the pixels could be updated mid-frame unless they had an individual refresh rate, so I’m inclined to believe the second case. If it’s the second case, it would make sense why Apple doesn’t want to give developers access to this, as this could quickly lead to timing errors between the software and hardware interrupts, such that it would only work within Apple’s framework and not an arbitrary code framework.
Score: 3 Votes (Like | Disagree)

Popular Stories

Upcoming Products 2022 Feature

Gurman: Apple Preparing 'Widest Array of New Hardware Products in Its History' for Fall

Sunday January 23, 2022 10:32 am PST by
Apple is working on a number of new products that are set to launch this fall, and Bloomberg's Mark Gurman says that it will be "the widest array" of new devices that Apple has introduced in its history. In his latest "Power On" newsletter, Gurman explains that Apple is working on four new flagship iPhones (iPhone 14, iPhone 14 Max, iPhone 14 Pro, and iPhone 14 Pro Max), an updated low-end Ma...
Questionable Design Decisions

Apple's Most Questionable Design Decisions in Recent Memory

Sunday January 23, 2022 2:59 am PST by
Apple has always emphasized the depth of thought that goes into the design of its products. In the foreword to Designed by Apple in California, a photo book released by the company in 2016, Jony Ive explains how Apple strives "to define objects that appear effortless" and "so simple, coherent and inevitable that there could be no rational alternative." But every once in a while even Apple...
macbook pro 14 16 2021

Three Months After Launch, Apple Still Struggling to Meet Demand for Redesigned 14-Inch and 16-Inch MacBook Pro

Monday January 24, 2022 7:12 am PST by
Three months after their launch, the 14-inch and 16-inch MacBook Pros continue to experience high demand and seemingly short supply, with shipping dates for both models stretching into multiple weeks in several of Apple's key markets. In the United States, the baseline 14-inch MacBook Pro with the M1 Pro chip is estimated to ship in three to four weeks, promising an arrival by at least...
att gigabit internet

AT&T Bringing $180/Month 5-Gigabit Internet to 70 Cities

Monday January 24, 2022 9:20 am PST by
AT&T today announced the launch of upgraded AT&T Fiber plans, which support speeds of up to 5 Gigabits for some customers. There are two separate plans, one "2 GIG" plan and one "5 GIG" plan, available to new and existing AT&T Fiber subscribers. According to AT&T, the new plans are available to nearly 5.2 million customers across 70 metro areas including Los Angeles, Atlanta, Chicago, San...
airpodsinear 1

AirPods Save Woman's Life With Feature Everyone Should Know

Friday January 21, 2022 2:13 am PST by
Apple's AirPods have been credited with saving a woman's life after a potentially fatal fall, People reports. When a 60-year-old florist in New Jersey tripped and hit her head in her studio, she lost consciousness and awoke heavily bleeding. With nobody around to call for help, she realized she had her AirPods in, and used a "Hey Siri" command to call 911. An operator was able to stay on the ...
AirPods Pro Gen 3 Mock Feature ugly yellow

AirPods Pro 2 Could Beat AirPods Max Sound Quality

Monday January 24, 2022 2:15 am PST by
Apple's second-generation AirPods Pro could deliver higher fidelity audio than the AirPods Max over-ear headphones, despite being comparatively small in-ear buds, recent reports have suggested. In a recent note to investors, seen by MacRumors, TF International Securities analyst Ming-Chi Kuo claimed that the second-generation AirPods Pro will offer improved audio quality thanks to support...
iCloud General Feature

Developers Unhappy With Bug Causing iCloud Unreliability

Monday January 24, 2022 10:15 am PST by
A number of developers are upset with an increasingly problematic iCloud server issue that is causing some apps that have implemented iCloud support to fail to sync properly. As outlined on the Developer Forums and on Twitter, there are CloudKit connectivity issues that have been occurring since November. Some users of apps that have iCloud support built in are seeing the following message:...
top stories 2022jan22

Top Stories: Spring Apple Event Rumors, Apple Opposes Sideloading, and More

Saturday January 22, 2022 6:00 am PST by
As we roll into the latter half of January, we're starting to hear more about a potential spring Apple event, which is likely to take place in March or April. There are a number of potential announcements on deck, so an event would be a good opportunity for Apple to get them all out there. We've also been going back and forth on some iPhone 14 rumors, and we've taken a look at a number of...