Chatbot-Like Siri Patent Includes Intelligent Image, Video, and Audio Recognition Within Messages

by

A patent application published by the United States Patent and Trademark Office today details a new Apple service where users could make inquiries and talk with the company's AI assistant Siri through Messages (via AppleInsider). The new patent is similar to a filing the USPTO published late last year, but now includes deeper integration with audio, video, and image files.

Similar to chatbots in Facebook Messenger and other texting services, Apple's patent describes a Siri that could perform her current duties without the user having to speak aloud, which could be helpful in certain public situations.

The "Intelligent Automated Assistant in a Messaging Environment" could respond to text, audio, images, and video when sent to it by the user, which Apple said would result in "a richer interactive experience between a user and a digital assistant." The patent gives a few examples of a conversation held between Siri and a user in Messages, with the user asking questions regarding calorie content in food, upcoming meetings, and even asking Siri to text a friend.

siri chatbot 2
Interesting applications include a thread where a user texts Siri a picture of a car or a bottle of wine, and Siri sees the images and can intelligently respond to the user's inquiries about them. For the car, the user asks Siri for details on pricing for a specific model using only an image, and Siri searches the internet and returns the relevant MSRP information.

The bottle of wine image is used as an example to show Siri's memory functions, where a user asks Siri to remember their favorite wine, which she can resurface at a later date. Siri sees the wine image, reads the label, and can then respond to a user's question in text format about the brand and even year it was made.

siri chatbot 6
Other image-related inquiries include "Where is this place?" and "What insect is this?", to which Siri would respond "This is the country Algeria" and "This is an earwig," respectively. Audio and video could also be recognized by Siri, including simple Shazam-like questions related to songs and the content of shared videos.

Apple points out in its patent that thanks to the chronological format of texting, users would be able to "review previous interactions" with Siri, unlike how current Siri conversations disappear immediately after they conclude. Subsequently, Siri would be able to use that history to become smarter and "define a wider range of tasks."

The messaging platform can enable multiple modes of input (e.g., text, audio, images, video, etc.) to be sent and received. As described herein, this can increase the functionality and capabilities of the digital assistant, thereby providing a richer interactive experience between a user and a digital assistant.

A digital assistant in a message environment can thus enable greater accessibility to the digital assistant. In particular, the digital assistant can be accessible in noisy environments or in environments where audio output is not desired (e.g., the library). Moreover, the chronological format enables a user to conveniently review previous interactions with the digital assistant and utilize the contextual history associated with the previous interactions to define a wider range of tasks.

The patent includes a description where Siri would be "a participant in a multi-party conversation," allowing group chats to use Apple's AI simultaneously. Apple gives an example where one user asks Siri to list nearby Chinese restaurants to begin making the group's dinner plans, and then another user responds by asking Siri to whittle down the list to only include the cheapest places. One user's personal Siri can even be asked to remind other participants of the upcoming dinner.

siri chatbot 8
Apple is believed to be working on an "enhanced Siri" that might launch in iOS 11 this fall, but the exact specifications as to what would make the new Siri "enhanced" have never been divulged. A questionable rumor in March stated that deep Siri integration is coming to Messages in iOS 11, but the source of the news -- The Verifier -- doesn't have a previous track record of reporting accurate rumors.

Chatbots are certainly growing in popularity so it wouldn't be too surprising if Apple introduced some kind of text-based Siri interface, particularly considering the multiple patents the company has published on the topic. Still, as with all patents it's best to look at Apple's new filing as an intriguing insight into what the company might be working on for the future, rather than proof of an impending launch.

Top Rated Comments

lincolntran Avatar
48 months ago

So instead of fixing Siri's speech interface Apple wants you to type in queries...

...so different than Googling.

bc lots of people prefer to not talking in to a phone in public places or quite places.
Score: 6 Votes (Like | Disagree)
NT1440 Avatar
48 months ago

So instead of fixing Siri's speech interface Apple wants you to type in queries...

...so different than Googling.

.....are you asserting that somehow working on one feature means that the core of Siri isn't being worked on?
Score: 5 Votes (Like | Disagree)
Scottsoapbox Avatar
48 months ago
So instead of fixing Siri's speech interface Apple wants you to type in queries...

...so different than Googling.
Score: 3 Votes (Like | Disagree)
Scottsoapbox Avatar
48 months ago
. . . . . . . . [image of car]

What should I do with it?

. . . . . . . . How much does it cost?

OK, now playing songs by Lady Gaga.
Score: 3 Votes (Like | Disagree)
lazyrighteye Avatar
48 months ago
Low-hanging Siri jokes aside, this seems interesting. I welcome any traction on Siri development. And while today's Siri makes the thought of Siri inquiries via photo or video seems laughable, the next update to Siri could make this concept more plausible.

As one who tries to interact with Siri as much as possible, this Messages concept could be nice for a couple of reasons.

1. An archived Siri thread.
I'm often frustrated when after conducting a Siri inquiry and then leaving the Siri window to (say) check a link, I can't get back to my initial Siri interaction to review, continue or amend. Even a standalone Siri "app" could be interesting. Could offer a simple/intuitive way for users to "get back" to their Siri-ing via Home button double-click to reveal a Siri slide in the app switcher.

2. The ability to access Siri via text input.
There are definitely scenarios where accessing Siri via voice isn't always desirable. Being able to interact with her via text is a welcome option.

Will be interesting to see what Apple has up their sleeve regarding Siri. Will
Be nice if they update us on Siri dev at WWDC. While it can never been good enough, any development/advancement of Siri will be welcomed.
Score: 2 Votes (Like | Disagree)
lincolntran Avatar
48 months ago

And most people prefer google search to Siri search so...

So now we have 2 choices (and many others) instead of google, choices are good.
Score: 1 Votes (Like | Disagree)

Top Stories

Flat MacBook Air Feature

Bloomberg: Apple Working on 'Thinner and Lighter' High-End MacBook Air With MagSafe, Could Launch in Second Half of 2021

Friday January 22, 2021 3:34 am PST by
Apple is working on a "thinner and lighter" version of the MacBook Air that the company plans to release during the second half of this year at the earliest or in 2022, according to a new report by well-connected Bloomberg journalist Mark Gurman. It will include Apple's MagSafe charging technology and a next-generation version of the company's in-house Mac processors. Apple has discussed...
Top Stories 44 Feature

Top Stories: 'Thinner and Lighter' MacBook Air, Smaller iPhone 13 Notch, iOS 14.4 Incoming

Saturday January 23, 2021 6:00 am PST by
We continued to hear a lot more about Apple's plans for its Mac lineup this week, including word of a high-end redesigned MacBook Air and the return of an SD card slot as part of the upcoming MacBook Pro redesign. It also sounds like Apple has been working on Face ID for Mac, but it won't be appearing in a redesigned iMac this year as originally planned. This week also saw rumors about the...
Apple VR Feature

Bloomberg: Apple's First AR/VR Headset 'Pricey, Niche Precursor' to More Ambitious AR Glasses and Could Launch Next Year

Thursday January 21, 2021 3:27 am PST by
Apple's first virtual reality headset will be a "pricey, niche precursor" to a more ambitious augmented reality product, according to a new report from Bloomberg's Mark Gurman. As a mostly virtual reality device, it will display an all-encompassing 3-D digital environment for gaming, watching video and communicating. AR functionality, the ability to overlay images and information over a view...
iOS 15 icon mock banner

iOS 15 Rumored to Drop Support for iPhone 6s and 2016 iPhone SE

Thursday January 21, 2021 11:58 am PST by
Apple's upcoming iOS 15 operating system, which we expect to see unveiled in June, is rumored to be dropping support for a few of Apple's older iPhones. According to French site iPhoneSoft, iOS 15 will not be able to be installed on the iPhone 6s, the iPhone 6s Plus, or the 2016 iPhone SE, all of which have an A9 chip. The iPhone 6s and 6s Plus were introduced in 2015 and are now more...
2021 mbp sd slot feature2

Bloomberg: Next MacBook Pro to Feature SD Card Reader

Friday January 22, 2021 7:50 am PST by
Last week, reputable analyst Ming-Chi Kuo outlined his expectations for new 14-inch and 16-inch MacBook Pro models later this year, including the return of the MagSafe charging connector, the removal of the Touch Bar, a new flat-edged design, and the return of more ports built into the notebooks for expanded connectivity. A concept of a modern MacBook Pro with an SD card reader Kuo did not...
iPhone 13 Notch Feature

iPhone 13 Rumored to Feature Smaller Notch, Pro Model Cameras to Use Larger Image Sensor

Thursday January 21, 2021 1:38 am PST by
Apple's iPhone 13 series will feature a redesigned Face ID system that will allow for a smaller notch at the top of the screen, according to a new report today. The rumor comes via hit-and-miss Taiwanese industry publication DigiTimes, whose supply chain sources also claim that the ultra wide-angle lens in Apple's next-generation iPhones is due for an upgrade. The next-generation iPhones'...
airpods galaxy buds comparison

Samsung Galaxy Buds Pro vs. Apple AirPods Pro

Friday January 22, 2021 2:34 pm PST by
Samsung in January unveiled new flagship Galaxy S21 smartphones and alongside the new phones, introduced the $200 Galaxy Buds Pro, which are priced at $199 and offer Active Noise Cancellation. Subscribe to the MacRumors YouTube channel for more videos. These new Galaxy Buds Pro are clearly designed to compete with Apple's AirPods Pro, so we thought we'd compare the two sets of earbuds in our...
maxresdefault

Video Demos macOS Catalina Running on iPad Pro via x86 Emulation

Thursday January 21, 2021 11:36 am PST by
A video demonstrating macOS Catalina running on a current 2020 iPad Pro has been shared on YouTube, giving us a look at an interesting hack that has a Mac OS up and working on one of Apple's iPads. There's limited information about how the process of getting macOS Catalina on an iPad Pro works, but it uses x86 emulation and was done through the UTM software that allows virtual machines to...
2019 mac pro side and front

Tim Cook Gifted Donald Trump 'First' 2019 Mac Pro

Wednesday January 20, 2021 5:45 pm PST by
Apple CEO Tim Cook gifted former United States President Donald Trump with the first 2019 Mac Pro that came off of the assembly line in Austin, Texas, according to a financial disclosure report that was released today (via The Verge). "Mac Pro Computer, the first created at the Flex Factory in Austin, Texas," reads the entry, which values the machine at $5,999, the base price for a Mac Pro....
iOS 14

Apple Seeds iOS 14.4 and iPadOS 14.4 Release Candidate to Developers and Public Beta Testers

Thursday January 21, 2021 10:14 am PST by
Apple today seeded the RC version of upcoming iOS 14.4 and iPadOS 14.4 updates to developers for testing purposes, with the new betas coming a week after Apple released the second betas. iOS 14.4 and iPadOS 14.4 can be downloaded through the Apple Developer Center or over the air after the proper profile has been installed on an iPhone or iPad. Paired with the HomePod 14.4 beta that is...