Siri on iOS Answered 78.5% of Queries Correctly in Latest Test, Trailing Google Assistant at 85.5%

Five months after performing a test that put the smart speakers of multiple companies in the spotlight to determine how well they performed in various categories, Loup Ventures is back today with an IQ test focused entirely on digital AI assistants. To get the necessary results, the researchers asked Siri, Google Assistant, Alexa, and Cortana 800 questions each on a smartphone, and compared their findings to a previous AI test held in April 2017.

siri iphone x
For Siri in the new test, Apple's AI helper understood 99 percent of the queries and answered 78.5 percent of them correctly. That's an improvement on a similar AI-focused test from April 2017 (66.1 percent of 800 questions answered correctly). While Loup Ventures looked at similar methodologies when testing smart speakers in February, the researchers explain that it's "not worthwhile to compare" the results across these tests since "the use cases differ greatly between digital assistants and smart speakers."

This is particularly true for Siri on HomePod, which performs well in certain areas but is largely limited to the amount of actions it can perform on the speaker itself. This led Apple's HomePod to become relegated to the "bottom of the totem pole" in an AI assistant performance test during Loup Venture's smart speaker research in February, with Siri answering 52.3 percent of 782 total questions correctly, across the same five categories as the new test.

Loup Ventures grades each digital assistant on two metrics: "Did it understand what was being asked?" and "Did it deliver a correct response?" Questions came from five categories, including Local (example: "Where is the nearest coffee shop?"), Commerce ("Can you order me more paper towels?"), Navigations ("How do I get to uptown on the bus?"), Information ("Who do the Twins play tonight?"), and Command ("Remind me to call Steve at 2pm today").

loup ventures july test

Charts via Loup Ventures

Questions were asked of Siri on an iPhone running iOS 11.4, Google Assistant on a Pixel XL, Alexa on the iOS app, and Cortana on the iOS app. Siri's best category was Command (90 percent of questions answered correctly), outperforming all rivals when asked to control aspects of the iPhone, smart home products, Apple Music, and more. Following Command, Siri performed well in Local (87 percent), Navigation (83 percent), and began dipping in Information (70 percent) and Commerce (60 percent).

Google Assistant has the edge in every category except Command. Siri’s lead over the Assistant in this category is odd, given they are both baked into the OS of the phone rather than living on a 3rd party app (as Cortana and Alexa do). We found Siri to be slightly more helpful and versatile (responding to more flexible language) in controlling your phone, smart home, music, etc. Our question set also includes a fair amount of music-related queries (the most common action for smart speakers). Apple, true to its roots, has ensured that Siri is capable with music on both mobile devices and smart speakers.

Google Assistant was the top digital assistant in all categories except Command, with Loup Ventures particularly liking Google's "featured snippets" feature that reads off search results of voice queries and is often "exactly what you're looking for." Both Alexa and Cortana were lesser performers in the test due to the iOS app for each limiting what the assistants can do on an iPhone, unlike Siri's ability to perform tasks all over iOS and not just in one app.

In total, Google Assistant answered 85.5 percent of the 800 questions asked correctly and understood all of them, compared to Siri's 78.5 percent answered correctly and 11 misunderstood. Alexa correctly answered 61.4 percent and misunderstood 13, while Cortana was the "laggard" and correctly answered 52.4 percent and misunderstood 19.

loup ventures july test 2
Over the 15 month period since April 2017, Siri improved by 13 percentage points, with Loup Ventures pointing out that it was "impressed with the speed at which the technology is advancing" for most of the assistants. The researchers went on to explain that many of the issues they had last year were erased by "improvements to natural language processing and inter-device connectivity."

Loup Ventures also sees more improvements coming down the line, particularly for Siri. This fall, the digital assistant will gain new abilities with Apple's upcoming "Siri Shortcuts" feature in iOS 12, which will let users create automations between apps that can be easily triggered through voice commands. The researches say they are "eager to test that feature" when it launches within iOS 12, likely coming in September.

Top Rated Comments

DrJohnnyN Avatar
31 months ago
Siriously still bad.
Score: 22 Votes (Like | Disagree)
Heineken Avatar
31 months ago
Impressive number without any personal data mining.
Score: 20 Votes (Like | Disagree)
ikir Avatar
31 months ago

Siriously still bad.

Absolutely not. Sure whiners will always say that.
Siri was and it is great, few years ago was impossible to dream of speak to your device. I use Siri about every hour for work.
Internet forums users tend to bash everything. Siri also is available on many languages and it is a big plus.
Cortana is quite bad for example, want to bash something? Cortana is there.
Score: 18 Votes (Like | Disagree)
MrGimper Avatar
31 months ago

Absolutely not. Sure whiners will always say that.
Siri was and it is great, few years ago was impossible to dream of speak to your device. I use Siri about every hour for work.
Internet forums users tend to bash everything. Siri also is available on many languages and it is a big plus.
Cortana is quite bad for example, want to bash something? Cortana is there.

You know what's more annoying than "whiners?", people who label anyone who has an opinion or criticism that doesn't align with their own, a "whiner".

Glad you use it that much for work, you must have a job where you have to set a lot of timers :cool:
Score: 17 Votes (Like | Disagree)
Stella Avatar
31 months ago

Impressive number without any personal data mining.

Personal Data mining is irrelevant.

Siri still sucks.
Score: 17 Votes (Like | Disagree)
PickUrPoison Avatar
31 months ago
So Siri’s not nearly the train wreck MacRumors forum posters endlessly and loudly claim it to be.

Never could have guessed...

Most improved, usually beaten by google, usually better than Alexa, and always beats Cortana; a lot of folks here aren’t going to want to hear this. Time to shoot the messenger I suppose.
Score: 15 Votes (Like | Disagree)

Top Stories

apple briefcase

AppleCare Memo Hints at Potential Hardware Announcement Next Tuesday

Thursday December 3, 2020 9:12 am PST by
Following a busy fall season in which Apple hosted three events in as many months, the company may have one more product announcement in store this year. In an internal memo this week, obtained by MacRumors from a reliable source, Apple informed service providers that it has AppleCare-related changes planned for Tuesday, December 8 at approximately 5:30 a.m. Pacific Time. Specifically, Apple ...
iphone11colorswhitebg

Apple Launches iPhone 11 Display Module Replacement Program

Friday December 4, 2020 3:22 pm PST by
Apple today launched a new display replacement program for iPhone 11 devices to address a problem that cases a "small percentage" of iPhone 11 displays to stop responding to touch. Apple says that affected devices were manufactured between November 2019 and May 2020, and if there are touch issues, the iPhone 11 models could have a problem with the display module. Those with iPhone 11...
matte black macbook pro colorware

Apple Researching a Matte Black Finish for MacBooks

Friday December 4, 2020 9:04 am PST by
Apple has filed a patent application for an intense light-absorbant matte black finish for a range of products, including the iPhone, iPad, Apple Watch, and MacBook (via Patently Apple). Image via ColorWare The patent application, filed with the U.S. Patent and Trademark Office, is titled "Anodized Part Having a Matte Black Appearance," and outlines the characteristics of the finish and...
homepod mini amazon echo size

$99 Speaker Showdown: HomePod Mini vs. Amazon Echo and Google Nest Audio

Wednesday December 2, 2020 3:12 pm PST by
Apple recently released the HomePod mini, a new $99 version of the original HomePod that's smaller, cuter, and, most importantly, competitively priced. At $99, the HomePod mini can better compete with affordable smart speakers from companies like Google and Amazon. Subscribe to the MacRumors YouTube channel for more videos. The HomePod mini has been praised for its high-quality sound at its...
wristcam design

$299 'Wristcam' Adds a Pair of Cameras to Your Apple Watch

Thursday December 3, 2020 9:32 am PST by
The Apple Watch has never included a camera, likely due to battery life and space concerns. A new Apple Watch product aims to address that lack by introducing a wrist-worn camera that works with the Apple Watch. The Apple-certified Wristcam attaches to the Apple Watch in the form of a band that adds a rather large camera set to the top of the Apple Watch. It's quite thick and bulky, but can...
iphone 12 5g

Multiple iPhone 12 Users Report Sudden Drops in 5G and LTE Cellular Coverage

Thursday December 3, 2020 1:18 am PST by
Since Apple launched the iPhone 12 in October, an increasing number of users of the new smartphone have been reporting persistent drops in cellular coverage. Multiple reports of dropped 5G and LTE connectivity have appeared on Reddit, on Apple's support forums, and on the MacRumors forums, with many people suffering issues when walking or in transit and some seeing the same problem when...
iOS 14

Apple Releases Third Betas of iOS 14.3 and iPadOS 14.3 to Developers [Update: Public Beta Available]

Wednesday December 2, 2020 10:04 am PST by
Apple today seeded the third betas of upcoming iOS 14.3 and iPadOS 14.3 updates to developers for testing purposes, two weeks after releasing the second betas and a month after the launch of iOS and iPadOS 14.2. iOS and iPadOS 14.3 can be downloaded through the Apple Developer Center or over the air after the proper developer profile has been installed. The iOS 14.3 update brings the...
16 inch MBP Mini Led

Kuo: Two Redesigned MacBook Pros in 2021 and New MacBook Air in 2022, All With Apple Silicon and Mini-LED Displays

Wednesday December 2, 2020 5:46 am PST by
Apple plans to release two redesigned MacBook Pros in 2021 and a new MacBook Air in 2022, all with mini-LED displays and Apple Silicon chips, according to TFI Securities analyst Ming-Chi Kuo. In a research note to investors, seen by MacRumors, Kuo explained that two new MacBook Pro models equipped with an all-new form factor design are expected to launch in 2021, and a new "affordable"...
satechi magnetic charger

Satechi Debuts Magnetic Wireless Charger for iPhone 12 Models

Friday December 4, 2020 8:00 am PST by
Satechi today announced a new USB-C Magnetic Wireless Charging Cable, which is a new wireless charging option for Apple's iPhone 12 models. Designed to work with the magnets built into the iPhone 12 lineup, the Magnetic Wireless Charging Cable adheres to the back of an iPhone much like Apple's MagSafe charger. The Magnetic Wireless Charging Cable works attaches to an iPhone 12 with a...
magsafe duo design

Hands-On With Apple's New MagSafe Duo Charger for iPhone 12 and Apple Watch

Thursday December 3, 2020 2:49 pm PST by
Apple on Tuesday finally released the MagSafe Duo Charger, which is a $129 charger that combines an iPhone 12 MagSafe charger with an Apple Watch charging puck. Subscribe to the MacRumors YouTube channel for more videos. We picked up one of the new MagSafe Duo chargers to see if it's worth Apple's super high asking price. Apple first introduced the MagSafe Duo alongside the new iPhone 12...