Apple Preparing for Upcoming Siri Onscreen Awareness Feature With New iOS 18.2 API for Developers

Apple is working on more advanced Siri functionality as part of its Apple Intelligence feature set, and to prepare, it has been providing developers with App Intent APIs so apps will be ready for the new capabilities.

Apple Intelligence General Feature 2
With the latest wave of betas, Apple has a new API that lets developers make onscreen content in their apps available to ‌Siri‌ and ‌Apple Intelligence‌. From Apple's documentation:

When a user asks a question about onscreen content or wants to perform an action on it, Siri and Apple Intelligence can retrieve the content to respond to the question and perform the action. If the user explicitly requests it, Siri and Apple Intelligence can send content to supported third-party services. For example, someone could view a website and use Siri to provide a summary by saying or typing a phrase like "Hey Siri, what's this document about?"

In the iOS 18.2 beta, ChatGPT integration with ‌Siri‌ allows users to ask questions about photos and documents, such as PDFs and presentations, and get information about them. You can, for example, ask ‌Siri‌ "what's in this photo?" and ‌Siri‌ will take a screenshot and hand it over to ChatGPT. ChatGPT then relays what's in the image, and the same feature works for PDFs and other documents.

It does not seem that the iOS 18.2 ChatGPT integration is the onscreen awareness functionality that Apple has planned for ‌Siri‌, but it could be related. Apple describes onscreen awareness as the ability for ‌Siri‌ to understand and take action on things on the screen. If someone texts you an address, for example, you'll be able to say "Add this address to their contact card," and ‌Siri‌ will do it. This functionality is not available in iOS 18.2, and ChatGPT is limited to assessing screenshots, but it is somewhat confusing.

Onscreen awareness, like personal context and in-app actions, is a feature that Apple has planned for ‌Siri‌, but it's probably not something that we're getting this year. Many of the ‌Siri‌ features are coming in a future version of iOS 18, and Bloomberg's Mark Gurman has said we can expect to see them in iOS 18.4, an update set to be released in the spring of 2025.

While there are multiple ‌Siri‌ features that won't be coming until next year, Apple is providing developers with APIs in advance so that developers have several months to prepare and so the features are ready for the public when the updates actually come out.

Related Roundups: iOS 18, iPadOS 18
Related Forums: iOS 18, iPadOS 18

Popular Stories

apple tv 4k new orange

New Apple TV Expected Later This Year With These New Features

Saturday July 12, 2025 3:09 pm PDT by
A new Apple TV is expected to be released later this year, and a handful of new features and changes have been rumored for the device. Below, we recap what to expect from the next Apple TV, according to rumors. Rumors Faster Wi-Fi Support The next Apple TV will be equipped with Apple's own combined Wi-Fi and Bluetooth chip, according to Bloomberg's Mark Gurman. He said the chip supports ...
Apple Watch Ultra 2 Complications

Apple Watch Ultra 3: What to Expect

Sunday July 13, 2025 10:30 am PDT by
The long wait for an Apple Watch Ultra 3 is nearly over, and a handful of new features and changes have been rumored for the device. Below, we recap what to expect from the Apple Watch Ultra 3:Satellite connectivity for sending and receiving text messages when Wi-Fi and cellular coverage is unavailable 5G support, up from LTE on the Apple Watch Ultra 2 Likely a wide-angle OLED display that ...
iPhone 17 Pro in Hand Feature Lowgo

iPhone 17 Pro Coming Soon With These 16 New Features

Friday July 11, 2025 12:40 pm PDT by
Apple's next-generation iPhone 17 Pro and iPhone 17 Pro Max are only two months away, and there are plenty of rumors about the devices. Below, we recap key changes rumored for the iPhone 17 Pro models. Latest Rumors These rumors surfaced in June and July:A redesigned Dynamic Island: It has been rumored that all iPhone 17 models will have a redesigned Dynamic Island interface — it might ...
iphone 16 pro ghost hand

5 Reasons to Skip This Year's iPhone 17 Pro

Thursday July 10, 2025 4:54 am PDT by
Apple will launch its new iPhone 17 series in two months, and the iPhone 17 Pro models are expected to get a new design for the rear casing and the camera area. But more significant changes to the lineup are not expected until next year, when the iPhone 18 models arrive. If you're thinking of trading in your iPhone for this year's latest, consider the following features rumored to be coming...
iOS 26 on Three iPhones

Here's When to Expect the iOS 26 Public Beta

Tuesday July 15, 2025 11:07 am PDT by
Apple previously announced that a public beta of iOS 26 would be available in July, and now a more specific timeframe has surfaced. Bloomberg's Mark Gurman today said that Apple's public betas should be released on or around Wednesday, July 23. In other words, expect the public betas of iOS 26, iPadOS 26, macOS 26, and more to be available at some point next week. Apple will be releasing...
Apple Hornsby

Apple Store Near Sydney Permanently Closing Later This Year

Monday July 14, 2025 6:14 pm PDT by
Apple today said its store at the Westfield Hornsby shopping mall, in Hornsby, Australia, will be permanently closing in October. Apple Hornsby In a statement shared with Australian tech news website EFTM (via Reddit), Apple said that it has decided not to renew its lease at Westfield Hornsby. Apple said all affected retail employees will be given the opportunity to work at Apple's nearby...
iPhone 14 Pro Dynamic Island

iPhone 17 May See 'Significant' Dynamic Island Changes

Monday July 14, 2025 12:11 am PDT by
The iPhone's Dynamic Island experience is set to undergo "significant evolution" over the next few years, according to a new rumor. Earlier this month, a report suggested that the iPhone 17 lineup will feature a redesigned Dynamic Island user interface, but little else was explained about the software changes. Now, the leaker known as "Majin Bu" appears to have corroborated this, commenting ...

Top Rated Comments

smoker68x Avatar
9 months ago
Is this going to be a one time universal permission to Siri, or an individual access per app like camera/microphone/location?
Score: 4 Votes (Like | Disagree)
Fraserpatty Avatar
9 months ago
This is the type of Siri functionality that I have been waiting for. I hope Apple doesn’t disappoint.
Score: 3 Votes (Like | Disagree)
dannys1 Avatar
9 months ago

but it is somewhat confusing.
Is it?
Score: 2 Votes (Like | Disagree)
deeside81 Avatar
9 months ago
Siri still doesn’t know my Scottish accent. It’s absolutely useless and doesn’t work with me. After 10 years they’ve had plenty of time to fix Siri not understanding accents. If I put an American accent on it understands me clearly. Rubbish.
Score: 2 Votes (Like | Disagree)
alexjholland Avatar
9 months ago
Siri still gets confused when I try to turn off the lights.

Forgive me if I'm cynical.
Score: 2 Votes (Like | Disagree)
svish Avatar
9 months ago
Good to know. Should make Siri much more useful.
Score: 2 Votes (Like | Disagree)