Chatbot-Like Siri Patent Includes Intelligent Image, Video, and Audio Recognition Within Messages

by

A patent application published by the United States Patent and Trademark Office today details a new Apple service where users could make inquiries and talk with the company's AI assistant Siri through Messages (via AppleInsider). The new patent is similar to a filing the USPTO published late last year, but now includes deeper integration with audio, video, and image files.

Similar to chatbots in Facebook Messenger and other texting services, Apple's patent describes a Siri that could perform her current duties without the user having to speak aloud, which could be helpful in certain public situations.

The "Intelligent Automated Assistant in a Messaging Environment" could respond to text, audio, images, and video when sent to it by the user, which Apple said would result in "a richer interactive experience between a user and a digital assistant." The patent gives a few examples of a conversation held between Siri and a user in Messages, with the user asking questions regarding calorie content in food, upcoming meetings, and even asking Siri to text a friend.


Interesting applications include a thread where a user texts Siri a picture of a car or a bottle of wine, and Siri sees the images and can intelligently respond to the user's inquiries about them. For the car, the user asks Siri for details on pricing for a specific model using only an image, and Siri searches the internet and returns the relevant MSRP information.

The bottle of wine image is used as an example to show Siri's memory functions, where a user asks Siri to remember their favorite wine, which she can resurface at a later date. Siri sees the wine image, reads the label, and can then respond to a user's question in text format about the brand and even year it was made.


Other image-related inquiries include "Where is this place?" and "What insect is this?", to which Siri would respond "This is the country Algeria" and "This is an earwig," respectively. Audio and video could also be recognized by Siri, including simple Shazam-like questions related to songs and the content of shared videos.

Apple points out in its patent that thanks to the chronological format of texting, users would be able to "review previous interactions" with Siri, unlike how current Siri conversations disappear immediately after they conclude. Subsequently, Siri would be able to use that history to become smarter and "define a wider range of tasks."

The messaging platform can enable multiple modes of input (e.g., text, audio, images, video, etc.) to be sent and received. As described herein, this can increase the functionality and capabilities of the digital assistant, thereby providing a richer interactive experience between a user and a digital assistant.

A digital assistant in a message environment can thus enable greater accessibility to the digital assistant. In particular, the digital assistant can be accessible in noisy environments or in environments where audio output is not desired (e.g., the library). Moreover, the chronological format enables a user to conveniently review previous interactions with the digital assistant and utilize the contextual history associated with the previous interactions to define a wider range of tasks.

The patent includes a description where Siri would be "a participant in a multi-party conversation," allowing group chats to use Apple's AI simultaneously. Apple gives an example where one user asks Siri to list nearby Chinese restaurants to begin making the group's dinner plans, and then another user responds by asking Siri to whittle down the list to only include the cheapest places. One user's personal Siri can even be asked to remind other participants of the upcoming dinner.


Apple is believed to be working on an "enhanced Siri" that might launch in iOS 11 this fall, but the exact specifications as to what would make the new Siri "enhanced" have never been divulged. A questionable rumor in March stated that deep Siri integration is coming to Messages in iOS 11, but the source of the news -- The Verifier -- doesn't have a previous track record of reporting accurate rumors.

Chatbots are certainly growing in popularity so it wouldn't be too surprising if Apple introduced some kind of text-based Siri interface, particularly considering the multiple patents the company has published on the topic. Still, as with all patents it's best to look at Apple's new filing as an intriguing insight into what the company might be working on for the future, rather than proof of an impending launch.

Top Rated Comments

(View all)
Avatar
40 months ago

So instead of fixing Siri's speech interface Apple wants you to type in queries...

...so different than Googling.

bc lots of people prefer to not talking in to a phone in public places or quite places.
Score: 6 Votes (Like | Disagree)
Avatar
40 months ago

So instead of fixing Siri's speech interface Apple wants you to type in queries...

...so different than Googling.

.....are you asserting that somehow working on one feature means that the core of Siri isn't being worked on?
Score: 5 Votes (Like | Disagree)
Avatar
40 months ago
So instead of fixing Siri's speech interface Apple wants you to type in queries...

...so different than Googling.
Score: 3 Votes (Like | Disagree)
Avatar
40 months ago
. . . . . . . . [image of car]

What should I do with it?

. . . . . . . . How much does it cost?

OK, now playing songs by Lady Gaga.
Score: 3 Votes (Like | Disagree)
Avatar
40 months ago
Low-hanging Siri jokes aside, this seems interesting. I welcome any traction on Siri development. And while today's Siri makes the thought of Siri inquiries via photo or video seems laughable, the next update to Siri could make this concept more plausible.

As one who tries to interact with Siri as much as possible, this Messages concept could be nice for a couple of reasons.

1. An archived Siri thread.
I'm often frustrated when after conducting a Siri inquiry and then leaving the Siri window to (say) check a link, I can't get back to my initial Siri interaction to review, continue or amend. Even a standalone Siri "app" could be interesting. Could offer a simple/intuitive way for users to "get back" to their Siri-ing via Home button double-click to reveal a Siri slide in the app switcher.

2. The ability to access Siri via text input.
There are definitely scenarios where accessing Siri via voice isn't always desirable. Being able to interact with her via text is a welcome option.

Will be interesting to see what Apple has up their sleeve regarding Siri. Will
Be nice if they update us on Siri dev at WWDC. While it can never been good enough, any development/advancement of Siri will be welcomed.
Score: 2 Votes (Like | Disagree)
Avatar
40 months ago

And most people prefer google search to Siri search so...

So now we have 2 choices (and many others) instead of google, choices are good.
Score: 1 Votes (Like | Disagree)

Top Stories

'This App is No Longer Shared' iOS Bug Preventing Some Apps From Opening

Friday May 22, 2020 3:58 pm PDT by
An app bug is causing some iOS users to be unable to open their apps, with affected iPhone and iPad users seeing the message "This app is no longer shared with you" when attempting to access an app. There are multiple complaints about the issue on the MacRumors forums and on Twitter from users who are running into problems. A MacRumors reader describes the issue:Is anyone else experiencing...

Apple Memorial Day Deals: Shop the Best Apple Accessory Sales From Twelve South, eBay, Anker, Mophie, and More

Friday May 22, 2020 6:39 am PDT by
We're now just a few days away from Memorial Day on Monday, May 25, and numerous retailers have opened up discounts in celebration of the holiday. This includes sales on helpful Apple-related accessories like Anker's portable batteries, Beats headphones at eBay, Incase and Incipio's protective iPad and iPhone cases, Mophie's iPhone battery cases, JBL's Bluetooth speakers, and much more. Note:...

Former iOS Chief Scott Forstall Shares Intriguing Story of His Interview With Steve Jobs at NeXT

Friday May 22, 2020 4:01 am PDT by
Former Apple executive and iOS chief Scott Forstall made a rare public appearance this week at Code.org's virtual Code Break event, and in between classes, Forstall shared the intriguing story of how he was hired by Steve Jobs. Forstall revealed that he had been considering working at Microsoft when he went to interview at NexT, the company started by Jobs after he had left Apple. Forstall...

'Apple Glass' Rumored to Start at $499, Support Prescription Lenses, and More

Tuesday May 19, 2020 6:30 am PDT by
Front Page Tech host and leaker Jon Prosser today shared several alleged details about Apple's rumored augmented reality glasses, including an "Apple Glass" marketing name, $499 starting price, prescription lens option, and more. The marketing name will be "Apple Glass" The glasses will start at $499 with the option for prescription lenses at an extra cost There will be displays in both...

Apple's 'Bounce' AirPods Ad Wins 'Best of Advertising' Award

Friday May 22, 2020 10:09 am PDT by
Apple's creative "Bounce" ad designed to highlight the AirPods took top honors in the 99th annual ADC (Art Director's Club) awards for advertising, earning the "Best of Discipline" award along with two Gold Cube awards in the craft in video and branded content categories. Released in June 2019, the ad features a bored man who pulls his AirPods off of their wireless charging pad and then pops ...

Apple's 'AirPods Studio' Over-Ear Headphones Have Reportedly Kicked Off Production

Friday May 22, 2020 7:03 am PDT by
We've been hearing quite a bit recently about Apple's long-rumored over-ear headphones, said to be called "AirPods Studio," and it looks like a launch may be coming in the relatively near future. Artist mockup based on Beats Studio3 Rumors have generally suggested a summer or fall launch for AirPods Studio, with a report earlier this week claiming that suppliers in Vietnam will begin...

Top Stories: Apple Glass and iPhone 12 Rumors, iOS 13.5 Update, and More!

Saturday May 23, 2020 6:00 am PDT by
It was another big week for rumors this week, with a flurry of reports about Apple's augmented reality glasses, the iPhone 12, and Apple's "AirPods Studio" over-ear headphones. This week also saw the release of iOS 13.5, bringing a number of health-related updates to Apple's mobile devices. Subscribe to the MacRumors YouTube channel for more videos. Other topics of interest this week included ...

Apple Releases iPadOS and iOS 13.5 With Exposure Notification API, Face ID Mask Updates, Group FaceTime Changes and More

Wednesday May 20, 2020 10:00 am PDT by
Apple today released iOS and iPadOS 13.5, major updates that come more than a month after the launch of iOS and iPadOS 13.4.1. iOS 13.5 is a major health-related update that brings many features related to the ongoing public health crisis. The iOS and ‌‌iPadOS‌‌ 13.5 updates are available on all eligible devices over-the-air in the Settings app. To access the updates, go to Settings...

Jon Prosser Claims Apple is Working on 'Steve Jobs Heritage Edition' AR Glasses, Gurman Calls Rumor 'Complete Fiction'

Thursday May 21, 2020 4:50 pm PDT by
Apple is working on a limited-edition version of its augmented reality smart glasses that's designed to look like the round, frameless glasses that Steve Jobs was famous for wearing, according to Jon Prosser. Prosser, who runs YouTube show Front Page Tech and who has been sharing a flood of Apple rumors in recent weeks, mentioned the detail in Cult of Mac's latest Cultcast podcast....

T-Mobile and Sprint Offering Free iPhone SE With Trade-In

Thursday May 21, 2020 1:14 pm PDT by
T-Mobile is launching a Memorial Day promotion that will see the company offering a free iPhone SE to customers who trade in an eligible older smartphone in good condition. From Friday to Monday, customers who trade in an existing smartphone can get a free iPhone SE (sales tax still needs to be paid) or up to $500 off a Samsung Galaxy S20. The free iPhone SE will be provided in the form...