Siri on iOS Answered 78.5% of Queries Correctly in Latest Test, Trailing Google Assistant at 85.5%

Five months after performing a test that put the smart speakers of multiple companies in the spotlight to determine how well they performed in various categories, Loup Ventures is back today with an IQ test focused entirely on digital AI assistants. To get the necessary results, the researchers asked Siri, Google Assistant, Alexa, and Cortana 800 questions each on a smartphone, and compared their findings to a previous AI test held in April 2017.

siri iphone x
For Siri in the new test, Apple's AI helper understood 99 percent of the queries and answered 78.5 percent of them correctly. That's an improvement on a similar AI-focused test from April 2017 (66.1 percent of 800 questions answered correctly). While Loup Ventures looked at similar methodologies when testing smart speakers in February, the researchers explain that it's "not worthwhile to compare" the results across these tests since "the use cases differ greatly between digital assistants and smart speakers."

This is particularly true for Siri on HomePod, which performs well in certain areas but is largely limited to the amount of actions it can perform on the speaker itself. This led Apple's HomePod to become relegated to the "bottom of the totem pole" in an AI assistant performance test during Loup Venture's smart speaker research in February, with Siri answering 52.3 percent of 782 total questions correctly, across the same five categories as the new test.

Loup Ventures grades each digital assistant on two metrics: "Did it understand what was being asked?" and "Did it deliver a correct response?" Questions came from five categories, including Local (example: "Where is the nearest coffee shop?"), Commerce ("Can you order me more paper towels?"), Navigations ("How do I get to uptown on the bus?"), Information ("Who do the Twins play tonight?"), and Command ("Remind me to call Steve at 2pm today").

loup ventures july test

Charts via Loup Ventures

Questions were asked of Siri on an iPhone running iOS 11.4, Google Assistant on a Pixel XL, Alexa on the iOS app, and Cortana on the iOS app. Siri's best category was Command (90 percent of questions answered correctly), outperforming all rivals when asked to control aspects of the iPhone, smart home products, Apple Music, and more. Following Command, Siri performed well in Local (87 percent), Navigation (83 percent), and began dipping in Information (70 percent) and Commerce (60 percent).

Google Assistant has the edge in every category except Command. Siri’s lead over the Assistant in this category is odd, given they are both baked into the OS of the phone rather than living on a 3rd party app (as Cortana and Alexa do). We found Siri to be slightly more helpful and versatile (responding to more flexible language) in controlling your phone, smart home, music, etc. Our question set also includes a fair amount of music-related queries (the most common action for smart speakers). Apple, true to its roots, has ensured that Siri is capable with music on both mobile devices and smart speakers.

Google Assistant was the top digital assistant in all categories except Command, with Loup Ventures particularly liking Google's "featured snippets" feature that reads off search results of voice queries and is often "exactly what you're looking for." Both Alexa and Cortana were lesser performers in the test due to the iOS app for each limiting what the assistants can do on an iPhone, unlike Siri's ability to perform tasks all over iOS and not just in one app.

In total, Google Assistant answered 85.5 percent of the 800 questions asked correctly and understood all of them, compared to Siri's 78.5 percent answered correctly and 11 misunderstood. Alexa correctly answered 61.4 percent and misunderstood 13, while Cortana was the "laggard" and correctly answered 52.4 percent and misunderstood 19.

loup ventures july test 2
Over the 15 month period since April 2017, Siri improved by 13 percentage points, with Loup Ventures pointing out that it was "impressed with the speed at which the technology is advancing" for most of the assistants. The researchers went on to explain that many of the issues they had last year were erased by "improvements to natural language processing and inter-device connectivity."

Loup Ventures also sees more improvements coming down the line, particularly for Siri. This fall, the digital assistant will gain new abilities with Apple's upcoming "Siri Shortcuts" feature in iOS 12, which will let users create automations between apps that can be easily triggered through voice commands. The researches say they are "eager to test that feature" when it launches within iOS 12, likely coming in September.

Top Rated Comments

DrJohnnyN Avatar
37 months ago
Siriously still bad.
Score: 22 Votes (Like | Disagree)
Heineken Avatar
37 months ago
Impressive number without any personal data mining.
Score: 20 Votes (Like | Disagree)
ikir Avatar
37 months ago
Siriously still bad.
Absolutely not. Sure whiners will always say that.
Siri was and it is great, few years ago was impossible to dream of speak to your device. I use Siri about every hour for work.
Internet forums users tend to bash everything. Siri also is available on many languages and it is a big plus.
Cortana is quite bad for example, want to bash something? Cortana is there.
Score: 18 Votes (Like | Disagree)
MrGimper Avatar
37 months ago
Absolutely not. Sure whiners will always say that.
Siri was and it is great, few years ago was impossible to dream of speak to your device. I use Siri about every hour for work.
Internet forums users tend to bash everything. Siri also is available on many languages and it is a big plus.
Cortana is quite bad for example, want to bash something? Cortana is there.
You know what's more annoying than "whiners?", people who label anyone who has an opinion or criticism that doesn't align with their own, a "whiner".

Glad you use it that much for work, you must have a job where you have to set a lot of timers :cool:
Score: 17 Votes (Like | Disagree)
Stella Avatar
37 months ago
Impressive number without any personal data mining.
Personal Data mining is irrelevant.

Siri still sucks.
Score: 17 Votes (Like | Disagree)
PickUrPoison Avatar
37 months ago
So Siri’s not nearly the train wreck MacRumors forum posters endlessly and loudly claim it to be.

Never could have guessed...

Most improved, usually beaten by google, usually better than Alexa, and always beats Cortana; a lot of folks here aren’t going to want to hear this. Time to shoot the messenger I suppose.
Score: 15 Votes (Like | Disagree)

Top Stories

apple music change forever

Apple Music Teaser: 'Get Ready – Music is About to Change Forever'

Sunday May 16, 2021 2:39 pm PDT by
The Browse tab in the Music app across Apple's platforms has started displaying a prominent teaser hinting at an upcoming major announcement for Apple Music. Under the heading "Coming soon," the headline says "Get ready – music is about to change forever." An accompanying "Tune-In Video" simply shows an animated Apple Music logo. Rumors have indicated that Apple is preparing to launch a...
m1 ipad pro early customer

M1 iPad Pro Arrives Early for Lucky Customer

Saturday May 15, 2021 11:57 pm PDT by
Days ahead of their expected launch and seemingly before official review embargoes lift, one lucky customer has already gotten their hands on the brand new 12.9-inch M1 iPad Pro. Reddit User PeterDragon50 Posted on Reddit, u/PeterDragon50 has already received their 12.9-inch iPad Pro through retailer Nebraska Furniture Mart. The Reddit user says they placed their order when pre-orders...
2021 mbp hdmi slot 3d

2021 MacBook Pro Leaks Confirm Returning MagSafe and Ports

Friday May 14, 2021 3:06 am PDT by
Apple's upcoming MacBook Pro models are expected to feature a number of major changes such as larger display options and powerful new Apple silicon chips. Among the more surprising updates to this year's MacBook Pro models is the return of three ports that have been missing from the machines for over five years. Expected to come in 14- and 16-inch sizes, the 2021 MacBook Pro models are...
Twitter Feature

Twitter's 'Blue' Subscription Service May Cost $2.99, Will Offer Undo Tweet Option

Saturday May 15, 2021 11:08 am PDT by
Twitter has been working on some kind of subscription service since last summer, and Jane Manchun Wong, who often digs into new features coming in apps, has shared details on just what Twitter is exploring. Twitter's subscription service could be called Twitter Blue, and at the current time, it's priced at $2.99 per month. There will be a "Collections" section that allows users to save and...
AirPods Lineup Not Lossless Feature

AirPods, AirPods Max and AirPods Pro Don't Support Apple Music Lossless Audio

Monday May 17, 2021 10:44 am PDT by
Apple today announced that starting in June, Apple Music songs will be available to stream in Lossless and Hi-Resolution Lossless formats, but lossless audio won't be supported on the AirPods, AirPods Max, or AirPods Pro. Apple's Lossless Audio is encoded as Apple Lossless Audio Codec files, with lossless quality ranging from 16-bit 44.1 kHz playback to 24-bit 48 kHz playback and Hi-Res...
Top Stories 60 Feature

Top Stories: M1 iMac Benchmarks, iPhone Battery Life Tips, Colorful MacBook Air?

Saturday May 15, 2021 6:00 am PDT by
Anticipation is building for the new iMac and iPad Pro models, which have started shipping out ahead of a launch around May 21. Benchmarks for the new machines are starting to come in, and the M1 chips inside of them are registering at around the same figures seen with other M1-based Macs, making for significant upgrades over previous-generation models. This week also saw some rumors about...
apple music spatial audio

Apple Music Launching Spatial Audio With Dolby Atmos and Lossless Audio in June at No Extra Cost

Monday May 17, 2021 6:06 am PDT by
Apple today announced that Apple Music will be gaining support for Spatial Audio with Dolby Atmos at no additional cost starting in June. At launch, Apple Music subscribers will have access to thousands of songs in Spatial Audio from artists like J Balvin, Gustavo Dudamel, Ariana Grande, Maroon 5, Kacey Musgraves, The Weeknd, and many others. Apple says this feature will provide a...
General Music and AirPod 3 Feature

Rumor: Apple to Announce Third-Generation AirPods and HiFi Apple Music Tier on May 18

Thursday May 13, 2021 10:32 pm PDT by
A new rumor suggests that Apple will announce the third-generation AirPods and the recently rumored HiFi, or high-fidelity Apple Music tier, on Tuesday, May 18, via a press release on its website. The new rumor comes from Apple YouTuber Luke Miani who shared the alleged exclusive news with the AppleTrack website. According to the YouTuber, Apple plans to release the next-generation AirPods...
apple music logo

Apple Music Teaser References 'Hi-Res Lossless' and 'Dolby Atmos'

Sunday May 16, 2021 4:04 pm PDT by
Earlier today, Apple Music began teasing a special announcement with the tagline "Get Ready – Music is About to Change Forever." This teaser comes amid a rumor that Apple is preparing to announce the third-generation AirPods alongside a HiFi, or lossless audio streaming tier for Apple Music on Tuesday, May 18. Now, references to "Apple Lossless," "Free Lossless," "Hi-Res Lossless," and...
tile amazon sidewalk integration

Apple Says Tile Trackers Sold Poorly in Apple Stores

Friday May 14, 2021 4:53 am PDT by
Earlier last month, Spotify, Tile, and Match (owner of Tinder), testified at an app store antitrust hearing spearheaded by the U.S. Senate. During the hearing, Spotify called Apple's App Store "an abusive power grab," while Tile said Apple uses its platform to "unfairly limit competition for its products." Now, in response to their testimonies, Apple's vice president and chief compliance...