Chatbot-Like Siri Patent Includes Intelligent Image, Video, and Audio Recognition Within Messages

by

A patent application published by the United States Patent and Trademark Office today details a new Apple service where users could make inquiries and talk with the company's AI assistant Siri through Messages (via AppleInsider). The new patent is similar to a filing the USPTO published late last year, but now includes deeper integration with audio, video, and image files.

Similar to chatbots in Facebook Messenger and other texting services, Apple's patent describes a Siri that could perform her current duties without the user having to speak aloud, which could be helpful in certain public situations.

The "Intelligent Automated Assistant in a Messaging Environment" could respond to text, audio, images, and video when sent to it by the user, which Apple said would result in "a richer interactive experience between a user and a digital assistant." The patent gives a few examples of a conversation held between Siri and a user in Messages, with the user asking questions regarding calorie content in food, upcoming meetings, and even asking Siri to text a friend.


Interesting applications include a thread where a user texts Siri a picture of a car or a bottle of wine, and Siri sees the images and can intelligently respond to the user's inquiries about them. For the car, the user asks Siri for details on pricing for a specific model using only an image, and Siri searches the internet and returns the relevant MSRP information.

The bottle of wine image is used as an example to show Siri's memory functions, where a user asks Siri to remember their favorite wine, which she can resurface at a later date. Siri sees the wine image, reads the label, and can then respond to a user's question in text format about the brand and even year it was made.


Other image-related inquiries include "Where is this place?" and "What insect is this?", to which Siri would respond "This is the country Algeria" and "This is an earwig," respectively. Audio and video could also be recognized by Siri, including simple Shazam-like questions related to songs and the content of shared videos.

Apple points out in its patent that thanks to the chronological format of texting, users would be able to "review previous interactions" with Siri, unlike how current Siri conversations disappear immediately after they conclude. Subsequently, Siri would be able to use that history to become smarter and "define a wider range of tasks."

The messaging platform can enable multiple modes of input (e.g., text, audio, images, video, etc.) to be sent and received. As described herein, this can increase the functionality and capabilities of the digital assistant, thereby providing a richer interactive experience between a user and a digital assistant.

A digital assistant in a message environment can thus enable greater accessibility to the digital assistant. In particular, the digital assistant can be accessible in noisy environments or in environments where audio output is not desired (e.g., the library). Moreover, the chronological format enables a user to conveniently review previous interactions with the digital assistant and utilize the contextual history associated with the previous interactions to define a wider range of tasks.

The patent includes a description where Siri would be "a participant in a multi-party conversation," allowing group chats to use Apple's AI simultaneously. Apple gives an example where one user asks Siri to list nearby Chinese restaurants to begin making the group's dinner plans, and then another user responds by asking Siri to whittle down the list to only include the cheapest places. One user's personal Siri can even be asked to remind other participants of the upcoming dinner.


Apple is believed to be working on an "enhanced Siri" that might launch in iOS 11 this fall, but the exact specifications as to what would make the new Siri "enhanced" have never been divulged. A questionable rumor in March stated that deep Siri integration is coming to Messages in iOS 11, but the source of the news -- The Verifier -- doesn't have a previous track record of reporting accurate rumors.

Chatbots are certainly growing in popularity so it wouldn't be too surprising if Apple introduced some kind of text-based Siri interface, particularly considering the multiple patents the company has published on the topic. Still, as with all patents it's best to look at Apple's new filing as an intriguing insight into what the company might be working on for the future, rather than proof of an impending launch.

Top Rated Comments

(View all)
Avatar
44 months ago

So instead of fixing Siri's speech interface Apple wants you to type in queries...

...so different than Googling.

bc lots of people prefer to not talking in to a phone in public places or quite places.
Score: 6 Votes (Like | Disagree)
Avatar
44 months ago

So instead of fixing Siri's speech interface Apple wants you to type in queries...

...so different than Googling.

.....are you asserting that somehow working on one feature means that the core of Siri isn't being worked on?
Score: 5 Votes (Like | Disagree)
Avatar
44 months ago
So instead of fixing Siri's speech interface Apple wants you to type in queries...

...so different than Googling.
Score: 3 Votes (Like | Disagree)
Avatar
44 months ago
. . . . . . . . [image of car]

What should I do with it?

. . . . . . . . How much does it cost?

OK, now playing songs by Lady Gaga.
Score: 3 Votes (Like | Disagree)
Avatar
44 months ago
Low-hanging Siri jokes aside, this seems interesting. I welcome any traction on Siri development. And while today's Siri makes the thought of Siri inquiries via photo or video seems laughable, the next update to Siri could make this concept more plausible.

As one who tries to interact with Siri as much as possible, this Messages concept could be nice for a couple of reasons.

1. An archived Siri thread.
I'm often frustrated when after conducting a Siri inquiry and then leaving the Siri window to (say) check a link, I can't get back to my initial Siri interaction to review, continue or amend. Even a standalone Siri "app" could be interesting. Could offer a simple/intuitive way for users to "get back" to their Siri-ing via Home button double-click to reveal a Siri slide in the app switcher.

2. The ability to access Siri via text input.
There are definitely scenarios where accessing Siri via voice isn't always desirable. Being able to interact with her via text is a welcome option.

Will be interesting to see what Apple has up their sleeve regarding Siri. Will
Be nice if they update us on Siri dev at WWDC. While it can never been good enough, any development/advancement of Siri will be welcomed.
Score: 2 Votes (Like | Disagree)
Avatar
44 months ago

And most people prefer google search to Siri search so...

So now we have 2 choices (and many others) instead of google, choices are good.
Score: 1 Votes (Like | Disagree)

Top Stories

Leaker: 'iPhone 12 mini' and iPhone 12 Storage Capacities Start at 64GB, Pro Models at 128GB

Tuesday September 29, 2020 2:31 am PDT by
Rumors suggest Apple's iPhone 12 launch event will be held on October 13, with the more affordable 5.4 and 6.1-inch devices set to ship out ahead of the more expensive 6.1-inch and 6.7-inch Pro devices, and this morning hit-and-miss leaker Jon Prosser has further committed to that date by providing alleged details on Apple's first shipment of finalized iPhone 12 units. Prosser claims the...

Hands-On With iOS 14.2's New Shazam Music Recognition Toggle in Control Center

Monday September 28, 2020 2:35 pm PDT by
Shortly after launching iOS 14, Apple introduced an upcoming iOS 14.2 update, which is now available to developers and public beta testers ahead of a public release that could come at some point in October. Subscribe to the MacRumors YouTube channel for more videos. The iOS 14.2 update mainly focuses on the Control Center, introducing a new Music Recognition toggle that deepens the Shazam...

iPhone 12 'Pro Max' Model to Sport Unique High-End Features

Wednesday September 30, 2020 2:01 am PDT by
The upcoming "iPhone 12 Pro Max" is anticipated to have a number of unique high-end features not found on any other iPhone, such as its screen size, LiDAR scanner, faster 5G, and potentially a higher display refresh rate. The iPhone 12 Pro Max is also expected to be the largest ever iPhone, with a 6.7-inch display. Previously, the largest iPhones have been 6.5-inches in the iPhone XS Max and ...

DigiTimes: 12.9-inch Mini-LED iPad Pro Arriving Early 2021, Mini-LED MacBook Coming Later

Tuesday September 29, 2020 4:18 am PDT by
Apple will launch a 12.9-inch mini LED-backlit iPad Pro in early 2021 and a mini LED-backlit MacBook in the second-half of next year, according to DigiTimes. The Taiwan-based industry publication claims Epistar will supply the over-10,000 mini LEDs used in each iPad Pro tablet. Meanwhile, Apple is expected to recruit Osram Opto as another supplier of mini LEDs for use in a new "high-end"...

iOS 14.2 Suggests Apple Won't Include EarPods in the Box With iPhone 12

Tuesday September 29, 2020 2:19 pm PDT by
Rumors have suggested that Apple's iPhone 12 models will not include power adapters or EarPods in the box, and a minor code tweak in iOS 14.2 seemingly confirms Apple's plan to sell the new devices without EarPods. In iOS 14 and earlier versions of iOS, there's a mention of reducing exposure to RF energy by using the "supplied headphones," which is the same wording that Apple has used for...

iPhone 12 Production Lines at Foxconn's Zhengzhou Factory in China Running '24 Hours a Day'

Tuesday September 29, 2020 3:38 am PDT by
Apple contract manufacturer Foxconn is running its massive Zhengzhou factory in China 24 hours a day to produce the new iPhone 12, according to Chinese media reports. Apple's main iPhone manufacturer in China is said to be cancelling workers' holidays and introducing mandatory overtime with additional bonuses for longer-serving staff, according to information garnered from employees,...

iPhone 12 May Launch Earlier Than Usual in South Korea

Monday September 28, 2020 5:24 am PDT by
The upcoming iPhone 12 lineup may launch earlier than usual in South Korea, reports The Korea Herald. South Korean telecoms firms speaking to The Korea Herald have said that the iPhone 12 lineup will launch ahead of its usual schedule. Normally, the release of new iPhones in South Korea comes about one month after launch in the United States. Last year, the iPhone 11 arrived in South Korea ...

iOS 14.2 Beta 2 Adds New Emoji Characters like Ninja, Pinata, Bubble Tea, Polar Bear and More

Tuesday September 29, 2020 11:22 am PDT by
The second beta of iOS 14.2 introduces the new Emoji 13 characters that Apple previewed earlier this year as part of World Emoji Day. New emoji options include ninja, people hugging, black cat, bison, fly, polar bear, blueberries, fondue, bubble tea, and more, with a list below. Faces - Smiling Face with Tear, Disguised Face People - Ninja, Person in Tuxedo, Woman in Tuxedo, Person...

Epic Games Unlikely to Win Injunction in Ongoing Fortnite Battle With Apple, Jury Trial Possible

Monday September 28, 2020 1:14 pm PDT by
The ongoing legal dispute between Apple and Epic Games continued on today, with a preliminary injunction hearing taking place this morning. We're still waiting to hear the judge's official ruling, but it looks like Epic is not going to be granted an injunction to allow Fortnite back into the App Store as the case unfolds. Many of the arguments that lawyers for Apple and Epic Games made were...

Apple Releases Ninth Beta of macOS Big Sur to Developers

Tuesday September 29, 2020 10:07 am PDT by
Apple today seeded the Ninth beta of an upcoming macOS Big Sur update to developers for testing purposes, a week after releasing the eighth beta and more than two months after the new update was unveiled at the Worldwide Developers Conference. The macOS Big Sur beta can be downloaded through the Apple Developer Center and once the appropriate profile is installed, subsequent betas will be...