Apple Researchers Reveal New AI System That Can Beat GPT-4
Apple researchers have developed an artificial intelligence system named ReALM (Reference Resolution as Language Modeling) that aims to radically enhance how voice assistants understand and respond to commands.
In a research paper (via VentureBeat), Apple outlines a new system for how large language models tackle reference resolution, which involves deciphering ambiguous references to on-screen entities, as well as understanding conversational and background context. As a result, ReALM could lead to more intuitive and natural interactions with devices.
Reference resolution is an important part of natural language understanding, enabling users to use pronouns and other indirect references in conversation without confusion. For digital assistants, this capability has historically been a significant challenge, limited by the need to interpret a wide range of verbal cues and visual information. Apple's ReALM system seeks to address this by converting the complex process of reference resolution into a pure language modeling problem. In doing so, it can comprehend references to visual elements displayed on a screen and integrate this understanding into the conversational flow.
ReALM reconstructs the visual layout of a screen using textual representations. This involves parsing on-screen entities and their locations to generate a textual format that captures the screen's content and structure. Apple researchers found that this strategy, combined with specific fine-tuning of language models for reference resolution tasks, significantly outperforms traditional methods, including the capabilities of OpenAI's GPT-4.
ReALM could enable users to interact with digital assistants much more efficiently with reference to what is currently displayed on their screen without the need for precise, detailed instructions. This has the potential to make voice assistants much more useful in a variety of settings, such as helping drivers navigate infotainment systems while driving or assisting users with disabilities by providing an easier and more accurate means of indirect interaction.
Apple has now published several AI research papers. Last month, the company revealed a new method for training large language models that seamlessly integrates both text and visual information. Apple is widely expected to unveil an array of AI features at WWDC in June.
Popular Stories
Apple today released iOS 18.0.1 and iPadOS 18.0.1, the first updates to the iOS 18 and iPadOS 18 operating systems that debuted earlier in September. iOS 18.0.1 and iPadOS 18.0.1 come two weeks after the launch of iOS 18. The new software can be downloaded on eligible iPhones and iPads over-the-air by going to Settings > General > Software Update. According to Apple's release notes, the...
Apple today released macOS Sequoia 15.0.1, the first update for the macOS Sequoia operating system. The 15.0.1 update comes a week after Apple first released macOS Sequoia 15. Mac users can download the macOS Sequoia update by using the Software Update section of System Settings. According to Apple's release notes, macOS Sequoia 15.0.1 fixes a bug that could cause the Messages app...
Now that it's been two weeks since the iPhone 16 models were released, we've been able to spend enough time with the new devices to share a more in-depth review on their performance, battery life, feature set, and more. Subscribe to the MacRumors YouTube channel for more videos. We've been testing the iPhone 16 Pro and Pro Max, but the gap between the Pro models and the standard iPhone 16...
Amazon is hosting another Prime Day event this year, called Amazon Prime Big Deal Days and offering shoppers the first chance to save on holiday shopping from a major retailer. Similar to the first Prime Day, it will last for two days (October 8-9) and you can already find a large selection of early deals across Amazon's storefront, covering savings on tech, clothing, video games, groceries, and...
It's hard to believe we're already into October with the iPhone 16 launch behind us, but there's lots more still to come from Apple this year on both the hardware and software fronts. We're still expecting a number of Mac and perhaps some iPad updates in the very near future, while Apple Intelligence features are set to begin rolling out with iOS 18.1 and related operating system updates....
Apple is set to release iOS 18.1 in October, bringing the first set of Apple Intelligence features to iPhone 15 Pro and iPhone 16 models. This update marks a significant step forward in Apple's AI integration, offering a new Siri contextually-aware experience and a range of additional capabilities powered by on-device machine learning and large language models. There are a couple of handy new...
Apple is working on a new iPad mini that will "potentially" be released "by the end of 2024," according to a report today from Bloomberg's Mark Gurman. Last month, Gurman reported that Apple had "new iPads in the works," including an upgraded version of the iPad mini. At the time, he said the device was "on deck for Apple's October event" alongside the first M4 Macs. The wording in his...
Apple will announce several new M4 Mac models around the end of October, with the company planning to launch at least some of them as soon as Friday, November 1, according to Bloomberg's Mark Gurman. Writing in his latest Power On newsletter, Gurman said that Apple will launch a new M4 version of its low-end 14-inch MacBook Pro, as well as higher-end 14-inch and 16-inch MacBook Pro models...
