Siri on iOS Answered 78.5% of Queries Correctly in Latest Test, Trailing Google Assistant at 85.5%

Five months after performing a test that put the smart speakers of multiple companies in the spotlight to determine how well they performed in various categories, Loup Ventures is back today with an IQ test focused entirely on digital AI assistants. To get the necessary results, the researchers asked Siri, Google Assistant, Alexa, and Cortana 800 questions each on a smartphone, and compared their findings to a previous AI test held in April 2017.

siri iphone x
For Siri in the new test, Apple's AI helper understood 99 percent of the queries and answered 78.5 percent of them correctly. That's an improvement on a similar AI-focused test from April 2017 (66.1 percent of 800 questions answered correctly). While Loup Ventures looked at similar methodologies when testing smart speakers in February, the researchers explain that it's "not worthwhile to compare" the results across these tests since "the use cases differ greatly between digital assistants and smart speakers."

This is particularly true for Siri on HomePod, which performs well in certain areas but is largely limited to the amount of actions it can perform on the speaker itself. This led Apple's HomePod to become relegated to the "bottom of the totem pole" in an AI assistant performance test during Loup Venture's smart speaker research in February, with Siri answering 52.3 percent of 782 total questions correctly, across the same five categories as the new test.

Loup Ventures grades each digital assistant on two metrics: "Did it understand what was being asked?" and "Did it deliver a correct response?" Questions came from five categories, including Local (example: "Where is the nearest coffee shop?"), Commerce ("Can you order me more paper towels?"), Navigations ("How do I get to uptown on the bus?"), Information ("Who do the Twins play tonight?"), and Command ("Remind me to call Steve at 2pm today").

loup ventures july test

Charts via Loup Ventures

Questions were asked of Siri on an iPhone running iOS 11.4, Google Assistant on a Pixel XL, Alexa on the iOS app, and Cortana on the iOS app. Siri's best category was Command (90 percent of questions answered correctly), outperforming all rivals when asked to control aspects of the iPhone, smart home products, Apple Music, and more. Following Command, Siri performed well in Local (87 percent), Navigation (83 percent), and began dipping in Information (70 percent) and Commerce (60 percent).

Google Assistant has the edge in every category except Command. Siri’s lead over the Assistant in this category is odd, given they are both baked into the OS of the phone rather than living on a 3rd party app (as Cortana and Alexa do). We found Siri to be slightly more helpful and versatile (responding to more flexible language) in controlling your phone, smart home, music, etc. Our question set also includes a fair amount of music-related queries (the most common action for smart speakers). Apple, true to its roots, has ensured that Siri is capable with music on both mobile devices and smart speakers.

Google Assistant was the top digital assistant in all categories except Command, with Loup Ventures particularly liking Google's "featured snippets" feature that reads off search results of voice queries and is often "exactly what you're looking for." Both Alexa and Cortana were lesser performers in the test due to the iOS app for each limiting what the assistants can do on an iPhone, unlike Siri's ability to perform tasks all over iOS and not just in one app.

In total, Google Assistant answered 85.5 percent of the 800 questions asked correctly and understood all of them, compared to Siri's 78.5 percent answered correctly and 11 misunderstood. Alexa correctly answered 61.4 percent and misunderstood 13, while Cortana was the "laggard" and correctly answered 52.4 percent and misunderstood 19.

loup ventures july test 2
Over the 15 month period since April 2017, Siri improved by 13 percentage points, with Loup Ventures pointing out that it was "impressed with the speed at which the technology is advancing" for most of the assistants. The researchers went on to explain that many of the issues they had last year were erased by "improvements to natural language processing and inter-device connectivity."

Loup Ventures also sees more improvements coming down the line, particularly for Siri. This fall, the digital assistant will gain new abilities with Apple's upcoming "Siri Shortcuts" feature in iOS 12, which will let users create automations between apps that can be easily triggered through voice commands. The researches say they are "eager to test that feature" when it launches within iOS 12, likely coming in September.

Top Rated Comments

DrJohnnyN Avatar
59 months ago
Siriously still bad.
Score: 22 Votes (Like | Disagree)
Heineken Avatar
59 months ago
Impressive number without any personal data mining.
Score: 20 Votes (Like | Disagree)
ikir Avatar
59 months ago
Siriously still bad.
Absolutely not. Sure whiners will always say that.
Siri was and it is great, few years ago was impossible to dream of speak to your device. I use Siri about every hour for work.
Internet forums users tend to bash everything. Siri also is available on many languages and it is a big plus.
Cortana is quite bad for example, want to bash something? Cortana is there.
Score: 18 Votes (Like | Disagree)
MrGimper Avatar
59 months ago
Absolutely not. Sure whiners will always say that.
Siri was and it is great, few years ago was impossible to dream of speak to your device. I use Siri about every hour for work.
Internet forums users tend to bash everything. Siri also is available on many languages and it is a big plus.
Cortana is quite bad for example, want to bash something? Cortana is there.
You know what's more annoying than "whiners?", people who label anyone who has an opinion or criticism that doesn't align with their own, a "whiner".

Glad you use it that much for work, you must have a job where you have to set a lot of timers :cool:
Score: 17 Votes (Like | Disagree)
Stella Avatar
59 months ago
Impressive number without any personal data mining.
Personal Data mining is irrelevant.

Siri still sucks.
Score: 17 Votes (Like | Disagree)
PickUrPoison Avatar
59 months ago
So Siri’s not nearly the train wreck MacRumors forum posters endlessly and loudly claim it to be.

Never could have guessed...

Most improved, usually beaten by google, usually better than Alexa, and always beats Cortana; a lot of folks here aren’t going to want to hear this. Time to shoot the messenger I suppose.
Score: 15 Votes (Like | Disagree)

Popular Stories

iOS 16

Apple Releases iOS 16.3 With Security Keys for Apple ID, New HomePod Support, Bug Fixes and More

Monday January 23, 2023 10:10 am PST by
Apple today released iOS 16.3, the third major update to the iOS 16 operating system that first came out in September. iOS 16.3 comes over a month after the launch of iOS 16.2, an update that added the Freeform app, Apple Music Sing, Advanced Data Protection, and more. iOS 16‌.3 can be downloaded on eligible iPhones and iPads over-the-air by going to Settings > General > Software Update....
iOS 16

iOS 16.3 for iPhone Launching Next Week With These 4 New Features

Friday January 20, 2023 11:43 am PST by
In a recent press release, Apple confirmed that iOS 16.3 will be released to the public next week. The software update will be available for the iPhone 8 and newer and includes a handful of new features, changes, and bug fixes. Below, we've recapped bigger features in iOS 16.3, including support for physical security keys as a two-factor authentication option for Apple ID accounts, worldwide ...
iPhone 14 Pro Purple Side Perspective Feature Purple

iPhone 15 Pro Expected Later This Year With These 7 Exclusive Features

Tuesday January 24, 2023 4:53 pm PST by
Apple's next-generation iPhone 15 Pro and iPhone 15 Pro Max are expected to be announced in September as usual. Already, rumors suggest the devices will have at least seven exclusive features not available on the standard iPhone 15 and iPhone 15 Plus. An overview of the seven features rumored to be exclusive to iPhone 15 Pro models:A17 chip: iPhone 15 Pro models will be equipped with an A17...
watchOS 9 Feature

Apple Releases watchOS 9.3 With New Watch Face, Bug Fixes

Monday January 23, 2023 10:06 am PST by
Apple today released watchOS 9.3, the third major update to the watchOS 9 operating system that first launched in September. watchOS 9.3 comes over a month after watchOS 9.2, an update that added new Workout functionality and Crash Detection optimizations. watchOS 9.3 can be downloaded for free through the Apple Watch app on the iPhone by opening it up and going to General > Software Update. ...
iPhone trade in

Apple Adjusts Trade-In Values for iPhones, Macs, and More

Wednesday January 25, 2023 9:40 am PST by
After announcing new Mac and HomePod models last week, Apple adjusted its trade-in values for select devices in the United States. iPhone trade-in values decreased by up to $80, and most Android smartphones also went down. Mac trade-in values remained unchanged or increased by up to $40 depending on the model, while some Apple Watch models increased in value and others decreased. Trade-in...
Ventura Macs Feature Blue

Apple Releases macOS Ventura 13.2

Monday January 23, 2023 10:09 am PST by
Apple today released macOS Ventura 13.2, the second major update to the macOS Ventura operating system initially released in October. macOS Ventura 13.2 comes more than a month after macOS Ventura 13.1, an update that added the Freeform app and other changes. The ‌macOS Ventura‌ 13.2 update can be downloaded for free on all eligible Macs using the Software Update section of System...
Mac mini M2 2023

New 256GB Mac Mini and 512GB MacBook Pro Have Slower SSD Speeds Than Previous Models

Tuesday January 24, 2023 1:11 pm PST by
While the new Mac mini with the M2 chip has a lower $599 starting price, the base model with 256GB of storage has slower SSD read and write speeds compared to the previous-generation model with the M1 chip and 256GB of storage. A teardown of the new Mac mini shared by YouTube channel Brandon Geekabit reveals that the 256GB model is equipped with only a single 256GB storage chip, while the...
apple tv 4k red image

Apple Releases tvOS 16.3

Tuesday January 24, 2023 10:10 am PST by
Apple today released tvOS 16.3, the third major point update to the tvOS 16 operating system that originally came out in September. Available for the Apple TV 4K and Apple TV HD, tvOS 16.3 comes six weeks after tvOS 16.2, an update that added Apple Music Sing. The tvOS 16.3 update can be downloaded over the air through the Settings app on the ‌‌‌‌Apple TV‌‌‌‌ by going to System > Software...