Apple Researchers Reveal New AI System That Can Beat GPT-4
Apple researchers have developed an artificial intelligence system named ReALM (Reference Resolution as Language Modeling) that aims to radically enhance how voice assistants understand and respond to commands.

In a research paper (via VentureBeat), Apple outlines a new system for how large language models tackle reference resolution, which involves deciphering ambiguous references to on-screen entities, as well as understanding conversational and background context. As a result, ReALM could lead to more intuitive and natural interactions with devices.
Reference resolution is an important part of natural language understanding, enabling users to use pronouns and other indirect references in conversation without confusion. For digital assistants, this capability has historically been a significant challenge, limited by the need to interpret a wide range of verbal cues and visual information. Apple's ReALM system seeks to address this by converting the complex process of reference resolution into a pure language modeling problem. In doing so, it can comprehend references to visual elements displayed on a screen and integrate this understanding into the conversational flow.
ReALM reconstructs the visual layout of a screen using textual representations. This involves parsing on-screen entities and their locations to generate a textual format that captures the screen's content and structure. Apple researchers found that this strategy, combined with specific fine-tuning of language models for reference resolution tasks, significantly outperforms traditional methods, including the capabilities of OpenAI's GPT-4.
ReALM could enable users to interact with digital assistants much more efficiently with reference to what is currently displayed on their screen without the need for precise, detailed instructions. This has the potential to make voice assistants much more useful in a variety of settings, such as helping drivers navigate infotainment systems while driving or assisting users with disabilities by providing an easier and more accurate means of indirect interaction.
Apple has now published several AI research papers. Last month, the company revealed a new method for training large language models that seamlessly integrates both text and visual information. Apple is widely expected to unveil an array of AI features at WWDC in June.
Popular Stories
iOS 18.3 was released last month, so the first iOS 18.4 beta should be coming soon. iOS 18.4 is expected to be a more substantial update for the iPhone, with several new features and changes related to Apple Intelligence and beyond.
Apple's website suggests that iOS 18.4 will be released in April, following beta testing. Below, we outline what to expect from the update so far.
Apple...
If you pay for iCloud storage on your iPhone, Apple has a new perk for you, at no additional cost.
iCloud+ is the official name for Apple's paid iCloud storage plans, which range from 50GB for $0.99 per month to 12TB for $59.99 per month in the United States. iCloud+ plans already come with multiple perks for free, such as Hide My Email and HomeKit Secure Video, and now there is another one...
Apple's next-generation iPhone SE could debut as soon as next week with a launch to follow later in February, reports Bloomberg's Mark Gurman. Apple isn't expected to hold an event for the iPhone SE 4, and will instead unveil the device through a press release.
The iPhone SE 4 is expected to have an iPhone 14-style design, with Apple eliminating the thick bezels and Touch ID Home button of...
Apple is internally testing iOS 18.3.1 for iPhones, according to our website's analytics logs, which have been a consistently reliable indicator of upcoming iOS versions. The software update should be released within the next few weeks.
iOS 18.3.1 should be a minor update that addresses software bugs and/or security vulnerabilities. Apple Intelligence notification summaries for news and...
Apple hasn't refreshed the Apple TV since 2022, but rumors suggest that we're finally going to get an update in 2025. We don't have a full picture of what to expect yet, but we have some hints on what's coming.
Subscribe to the MacRumors YouTube channel for more videos.
Updated A-Series Chip
The current Apple TV 4K uses the A15 Bionic chip that was in the iPhone 13 lineup, and it's time for...
The British government has secretly demanded that Apple give it blanket access to all encrypted user content uploaded to the cloud, reports The Washington Post.
The undisclosed order is said to have been issued last month, and requires that Apple creates a back door that allows UK security officials unencumbered access to encrypted user data worldwide – an unprecedented demand not before...
Last year, we reported that Apple sued its former software engineer Andrew Aude for providing journalists with confidential information about the company's future plans, including details about the Journal app, Vision Pro headset, and more.
As reported by 9to5Mac, the Superior Court of Santa Clara County on Thursday dismissed the lawsuit after Apple and Aude reached an agreement to resolve...
In select U.S. states, residents can add their driver's license or state ID to the Wallet app on the iPhone and Apple Watch, providing a convenient and contactless way to display proof of identity or age at select airports and businesses, and in select apps.
Below, we outline which U.S. states and territories offer the feature, and additional states that have committed to rolling it out in...