Apple Preparing for Upcoming Siri Onscreen Awareness Feature With New iOS 18.2 API for Developers

Apple is working on more advanced Siri functionality as part of its Apple Intelligence feature set, and to prepare, it has been providing developers with App Intent APIs so apps will be ready for the new capabilities.

Apple Intelligence General Feature 2
With the latest wave of betas, Apple has a new API that lets developers make onscreen content in their apps available to ‌Siri‌ and ‌Apple Intelligence‌. From Apple's documentation:

When a user asks a question about onscreen content or wants to perform an action on it, Siri and Apple Intelligence can retrieve the content to respond to the question and perform the action. If the user explicitly requests it, Siri and Apple Intelligence can send content to supported third-party services. For example, someone could view a website and use Siri to provide a summary by saying or typing a phrase like "Hey Siri, what's this document about?"

In the iOS 18.2 beta, ChatGPT integration with ‌Siri‌ allows users to ask questions about photos and documents, such as PDFs and presentations, and get information about them. You can, for example, ask ‌Siri‌ "what's in this photo?" and ‌Siri‌ will take a screenshot and hand it over to ChatGPT. ChatGPT then relays what's in the image, and the same feature works for PDFs and other documents.

It does not seem that the iOS 18.2 ChatGPT integration is the onscreen awareness functionality that Apple has planned for ‌Siri‌, but it could be related. Apple describes onscreen awareness as the ability for ‌Siri‌ to understand and take action on things on the screen. If someone texts you an address, for example, you'll be able to say "Add this address to their contact card," and ‌Siri‌ will do it. This functionality is not available in iOS 18.2, and ChatGPT is limited to assessing screenshots, but it is somewhat confusing.

Onscreen awareness, like personal context and in-app actions, is a feature that Apple has planned for ‌Siri‌, but it's probably not something that we're getting this year. Many of the ‌Siri‌ features are coming in a future version of iOS 18, and Bloomberg's Mark Gurman has said we can expect to see them in iOS 18.4, an update set to be released in the spring of 2025.

While there are multiple ‌Siri‌ features that won't be coming until next year, Apple is providing developers with APIs in advance so that developers have several months to prepare and so the features are ready for the public when the updates actually come out.

Related Roundups: iOS 18, iPadOS 18
Related Forums: iOS 18, iPadOS 18

Popular Stories

iPhone 17 Pro Lower Logo Feature 1

iPhone 17 Pro Coming Soon With These 14 New Features

Monday June 30, 2025 1:08 pm PDT by
Apple's next-generation iPhone 17 Pro and iPhone 17 Pro Max are less than three months away, and there are plenty of rumors about the devices. Apple is expected to launch the iPhone 17, iPhone 17 Air, iPhone 17 Pro, and iPhone 17 Pro Max in September this year. Below, we recap key changes rumored for the iPhone 17 Pro models:Aluminum frame: iPhone 17 Pro models are rumored to have an...
Apple Watch Ultra Night Mode Screen

Apple Watch Ultra 3 Launching Later This Year With Two Key Upgrades

Wednesday July 2, 2025 1:13 pm PDT by
The long wait for an Apple Watch Ultra 3 appears to be nearly over, and it is rumored to feature both satellite connectivity and 5G support. Apple Watch Ultra's existing Night Mode In his latest Power On newsletter, Bloomberg's Mark Gurman said that the Apple Watch Ultra 3 is on track to launch this year with "significant" new features, including satellite connectivity, which would let you...
A18 Pro Chip

New MacBook With A18 Pro Chip Spotted in Apple Code

Monday June 30, 2025 8:05 am PDT by
Apple is developing a MacBook with the A18 Pro chip, according to findings in backend code uncovered by MacRumors. Earlier today, Apple analyst Ming-Chi Kuo reported that Apple is planning to launch a low-cost MacBook powered by an iPhone chip. The machine is expected to feature a 13-inch display, the A18 Pro chip, and color options that include silver, blue, pink, and yellow. MacRumors...
iPhone 17 Pro Lower Logo Magsafe

iPhone 17 Pro's New MagSafe Design Revealed in Leaked Photo

Wednesday July 2, 2025 8:37 am PDT by
The upcoming iPhone 17 Pro and iPhone 17 Pro Max are rumored to have a slightly different MagSafe magnet layout compared to existing iPhone models, and a leaked photo has offered a closer look at the supposed new design. The leaker Majin Bu today shared a photo of alleged MagSafe magnet arrays for third-party iPhone 17 Pro cases. On existing iPhone models with MagSafe, the magnets form a...
macbook air spacegray purple

Apple Planning to Launch Low-Cost MacBook Powered By iPhone Chip

Monday June 30, 2025 3:20 am PDT by
Apple is planning to launch a low-cost MacBook powered by an iPhone chip, according to Apple analyst Ming-Chi Kuo. In an article published on X, Kuo explained that the device will feature a 13-inch display and the A18 Pro chip, making it the first Mac powered by an iPhone chip. The A18 Pro chip debuted in the iPhone 16 Pro last year. To date, all Apple silicon Macs have contained M-series...
iOS 18

Apple Releases Second iOS 18.6 Public Beta

Tuesday July 1, 2025 10:19 am PDT by
Apple today seeded the second betas of upcoming iOS 18.6 and iPadOS 18.6 updates to public beta testers, with the betas coming just a day after Apple provided the betas to developers. Apple has also released a second beta of macOS Sequoia 15.6. Testers who have signed up for beta updates through Apple's beta site can download iOS 18.6 and iPadOS 18.6 from the Settings app on a compatible...
Wi Fi WiFi General Feature

iOS 26 Adds a Useful New Wi-Fi Feature to Your iPhone

Wednesday July 2, 2025 6:36 am PDT by
iOS 26 and iPadOS 26 add a smaller yet useful Wi-Fi feature to iPhones and iPads. As spotted by Creative Strategies analyst Max Weinbach, sign-in details for captive Wi-Fi networks are now synced across iPhones and iPads running iOS 26 and iPadOS 26. For example, while Weinbach was staying at a Hilton hotel, his iPhone prompted him to fill in Wi-Fi details from his iPad that was already...
maxresdefault

Five Features Coming to AirPods Pro 3

Friday June 27, 2025 10:52 am PDT by
Apple hasn't updated the AirPods Pro since 2022, and the earbuds are due for a refresh. We're counting on a new model this year, and we've seen several hints of new AirPods tucked away in Apple's code. Rumors suggest that Apple has some exciting new features planned that will make it worthwhile to upgrade to the latest model. Subscribe to the MacRumors YouTube channel for more videos. Heal...
replay all time playlist apple music

Apple Music Debuts All-New Personalized Playlist

Monday June 30, 2025 7:16 am PDT by
As part of its 10-year celebrations of Apple Music, Apple today released an all-new personalized playlist that collates your entire listening history. The playlist, called "Replay All Time," expands on Apple Music's existing Replay features. Previously, users could only see their top songs for each individual calendar year that they've been subscribed to Apple Music, but now, Replay All...

Top Rated Comments

smoker68x Avatar
9 months ago
Is this going to be a one time universal permission to Siri, or an individual access per app like camera/microphone/location?
Score: 4 Votes (Like | Disagree)
Fraserpatty Avatar
9 months ago
This is the type of Siri functionality that I have been waiting for. I hope Apple doesn’t disappoint.
Score: 3 Votes (Like | Disagree)
dannys1 Avatar
9 months ago

but it is somewhat confusing.
Is it?
Score: 2 Votes (Like | Disagree)
deeside81 Avatar
9 months ago
Siri still doesn’t know my Scottish accent. It’s absolutely useless and doesn’t work with me. After 10 years they’ve had plenty of time to fix Siri not understanding accents. If I put an American accent on it understands me clearly. Rubbish.
Score: 2 Votes (Like | Disagree)
alexjholland Avatar
9 months ago
Siri still gets confused when I try to turn off the lights.

Forgive me if I'm cynical.
Score: 2 Votes (Like | Disagree)
svish Avatar
9 months ago
Good to know. Should make Siri much more useful.
Score: 2 Votes (Like | Disagree)