Siri on iOS Answered 78.5% of Queries Correctly in Latest Test, Trailing Google Assistant at 85.5%

Five months after performing a test that put the smart speakers of multiple companies in the spotlight to determine how well they performed in various categories, Loup Ventures is back today with an IQ test focused entirely on digital AI assistants. To get the necessary results, the researchers asked Siri, Google Assistant, Alexa, and Cortana 800 questions each on a smartphone, and compared their findings to a previous AI test held in April 2017.


For Siri in the new test, Apple's AI helper understood 99 percent of the queries and answered 78.5 percent of them correctly. That's an improvement on a similar AI-focused test from April 2017 (66.1 percent of 800 questions answered correctly). While Loup Ventures looked at similar methodologies when testing smart speakers in February, the researchers explain that it's "not worthwhile to compare" the results across these tests since "the use cases differ greatly between digital assistants and smart speakers."

This is particularly true for Siri on HomePod, which performs well in certain areas but is largely limited to the amount of actions it can perform on the speaker itself. This led Apple's HomePod to become relegated to the "bottom of the totem pole" in an AI assistant performance test during Loup Venture's smart speaker research in February, with Siri answering 52.3 percent of 782 total questions correctly, across the same five categories as the new test.

Loup Ventures grades each digital assistant on two metrics: "Did it understand what was being asked?" and "Did it deliver a correct response?" Questions came from five categories, including Local (example: "Where is the nearest coffee shop?"), Commerce ("Can you order me more paper towels?"), Navigations ("How do I get to uptown on the bus?"), Information ("Who do the Twins play tonight?"), and Command ("Remind me to call Steve at 2pm today").

Charts via Loup Ventures

Questions were asked of Siri on an iPhone running iOS 11.4, Google Assistant on a Pixel XL, Alexa on the iOS app, and Cortana on the iOS app. Siri's best category was Command (90 percent of questions answered correctly), outperforming all rivals when asked to control aspects of the iPhone, smart home products, Apple Music, and more. Following Command, Siri performed well in Local (87 percent), Navigation (83 percent), and began dipping in Information (70 percent) and Commerce (60 percent).

Google Assistant has the edge in every category except Command. Siri’s lead over the Assistant in this category is odd, given they are both baked into the OS of the phone rather than living on a 3rd party app (as Cortana and Alexa do). We found Siri to be slightly more helpful and versatile (responding to more flexible language) in controlling your phone, smart home, music, etc. Our question set also includes a fair amount of music-related queries (the most common action for smart speakers). Apple, true to its roots, has ensured that Siri is capable with music on both mobile devices and smart speakers.

Google Assistant was the top digital assistant in all categories except Command, with Loup Ventures particularly liking Google's "featured snippets" feature that reads off search results of voice queries and is often "exactly what you're looking for." Both Alexa and Cortana were lesser performers in the test due to the iOS app for each limiting what the assistants can do on an iPhone, unlike Siri's ability to perform tasks all over iOS and not just in one app.

In total, Google Assistant answered 85.5 percent of the 800 questions asked correctly and understood all of them, compared to Siri's 78.5 percent answered correctly and 11 misunderstood. Alexa correctly answered 61.4 percent and misunderstood 13, while Cortana was the "laggard" and correctly answered 52.4 percent and misunderstood 19.


Over the 15 month period since April 2017, Siri improved by 13 percentage points, with Loup Ventures pointing out that it was "impressed with the speed at which the technology is advancing" for most of the assistants. The researchers went on to explain that many of the issues they had last year were erased by "improvements to natural language processing and inter-device connectivity."

Loup Ventures also sees more improvements coming down the line, particularly for Siri. This fall, the digital assistant will gain new abilities with Apple's upcoming "Siri Shortcuts" feature in iOS 12, which will let users create automations between apps that can be easily triggered through voice commands. The researches say they are "eager to test that feature" when it launches within iOS 12, likely coming in September.

Top Rated Comments

(View all)
Avatar
28 months ago
Siriously still bad.
Score: 22 Votes (Like | Disagree)
Avatar
28 months ago
Impressive number without any personal data mining.
Score: 20 Votes (Like | Disagree)
Avatar
28 months ago

Siriously still bad.

Absolutely not. Sure whiners will always say that.
Siri was and it is great, few years ago was impossible to dream of speak to your device. I use Siri about every hour for work.
Internet forums users tend to bash everything. Siri also is available on many languages and it is a big plus.
Cortana is quite bad for example, want to bash something? Cortana is there.
Score: 18 Votes (Like | Disagree)
Avatar
28 months ago

Absolutely not. Sure whiners will always say that.
Siri was and it is great, few years ago was impossible to dream of speak to your device. I use Siri about every hour for work.
Internet forums users tend to bash everything. Siri also is available on many languages and it is a big plus.
Cortana is quite bad for example, want to bash something? Cortana is there.

You know what's more annoying than "whiners?", people who label anyone who has an opinion or criticism that doesn't align with their own, a "whiner".

Glad you use it that much for work, you must have a job where you have to set a lot of timers :cool:
Score: 17 Votes (Like | Disagree)
Avatar
28 months ago

Impressive number without any personal data mining.

Personal Data mining is irrelevant.

Siri still sucks.
Score: 17 Votes (Like | Disagree)
Avatar
28 months ago
So Siri’s not nearly the train wreck MacRumors forum posters endlessly and loudly claim it to be.

Never could have guessed...

Most improved, usually beaten by google, usually better than Alexa, and always beats Cortana; a lot of folks here aren’t going to want to hear this. Time to shoot the messenger I suppose.
Score: 15 Votes (Like | Disagree)

Top Stories

When Will the iPhone 12 Launch? Here's What We Know

Wednesday September 16, 2020 6:12 am PDT by
Yesterday's "Time Flies" Apple event saw the release of the Apple Watch Series 6, Apple Watch SE, iPad 8, and iPad Air 4, but no new iPhone models. Rumors before the event strongly alleged that it would not see the unveiling of new iPhones, with many reports pointing to an October launch. The lack of new iPhone models yesterday seems to confirm that the iPhone 12 lineup will not appear...

iOS 14 Picture in Picture No Longer Working With YouTube's Mobile Website in Safari [Without Premium]

Friday September 18, 2020 12:21 pm PDT by
Apple in iOS 14 added Picture in Picture to the iPhone, a feature designed to let you watch a video in a small screen on your device while you continue to do other things on the phone. When Picture in Picture was working with YouTube The YouTube app doesn't support Picture in Picture, but up until yesterday there was a functional workaround that allowed videos from YouTube.com to be watched...

Hands-On With the New Apple Watch Series 6 and Apple Watch SE

Friday September 18, 2020 1:19 pm PDT by
Today's the official launch date for the Apple Watch Series 6 and the Apple Watch SE, both of which Apple announced on Tuesday. We picked up a couple of the new models and thought we'd give them a quick look for MacRumors readers thinking of ordering a new watch. Apple Watch Series 6 & Apple Watch SE Hands-On! When it comes to design, both the $399 Series 6 and the $279 SE look just like...

Here's How You Can Download iOS 14 and iPadOS 14 Around the World [It's Out]

Wednesday September 16, 2020 2:36 am PDT by
Apple's official public release of iOS 14 and iPadOS 14 dropped on Wednesday, September 16, just a day after the company released the Golden Master to third-party developers. Also set to be made available to the general public for the first time are watchOS 7 and tvOS 14. Getting Started With iOS 14 Video Click image to watch iOS 14 Getting Started While that's left a lot of developers...

Apple Releases iOS 14 and iPadOS 14 With Home Screen Redesign, App Library, Compact UI, Translate App, Scribble Support, App Clips, and More

Wednesday September 16, 2020 12:48 pm PDT by
Apple has released iOS 14 and iPadOS 14, the newest operating system updates designed for the iPhone and iPad. As with all of Apple's software updates, iOS 14 and iPadOS 14 can be downloaded for free. iOS 14 is available on the iPhone 6s and later, while iPadOS 14 is available on the iPad Air 2 and later. The updates are available on all eligible devices over-the-air in the Settings app. To ...

iOS 14.2 Beta Adds New Shazam Music Recognition Feature for Control Center

Thursday September 17, 2020 3:36 pm PDT by
Apple today released the first beta of iOS 14.2 to developers for testing purposes, and the new update introduces a Music Recognition control for the Control Center. The new feature lets you discover music playing around you and it recognizes the music playing with in apps, even when you're wearing AirPods. Songs pop up as notifications, and you can tap to listen in Apple Music....

Apple Updates AirPods 2 and AirPods Pro Firmware to Version 3A283

Monday September 14, 2020 11:24 am PDT by
Apple today released new 3A283 firmware updates for the second-generation AirPods and the AirPods Pro. The second-generation AirPods are being updated from the 2D15 firmware they were previously running, while the AirPods Pros are being updated from the 2D27 firmware they had installed previously. Apple does not provide details on what's included in refreshed firmware so we don't know what's ...

Rumor Report Card: Assessing the Accuracy of Leaks After Apple's Event

Friday September 18, 2020 12:57 pm PDT by
Apple hosted its virtual "Time Flies" event this week, where it introduced four new products, including the Apple Watch Series 6, lower-cost Apple Watch SE, a 10.9-inch iPad Air with an all-screen design, and an updated 10.2-inch iPad with a faster A12 Bionic chip. As expected, there were no new iPhones, which are believed to be coming in October instead. Apple also announced that it will be ...

Epic Games Announces 'Fortnite: Save the World' Will No Longer Be Playable on macOS

Friday September 18, 2020 4:50 am PDT by
Epic Games has announced that "Fortnite: Save the World" will no longer be playable on macOS, after Apple terminated Epic Games' developer account. Fortnite has been in violation of the ‌App Store‌ rules since August 13, when it introduced a direct payment option that skirted Apple's in-app purchase system by allowing payments directly to ‌Epic Games‌. Shortly after Epic blatantly...

Deals: Cellular Carriers Introduce First Offers on Apple Watch Series 6 and SE

Friday September 18, 2020 7:43 am PDT by
With the launch of the Apple Watch Series 6 and Apple Watch SE today, cellular carriers have now introduced special offers for these new wearable devices. Note: MacRumors is an affiliate partner with these vendors. When you click a link and make a purchase, we may receive a small payment, which helps us keep the site running. Starting with AT&T, if you buy one Apple Watch Series 3, Series...