Siri on iOS Answered 78.5% of Queries Correctly in Latest Test, Trailing Google Assistant at 85.5%

Five months after performing a test that put the smart speakers of multiple companies in the spotlight to determine how well they performed in various categories, Loup Ventures is back today with an IQ test focused entirely on digital AI assistants. To get the necessary results, the researchers asked Siri, Google Assistant, Alexa, and Cortana 800 questions each on a smartphone, and compared their findings to a previous AI test held in April 2017.


For Siri in the new test, Apple's AI helper understood 99 percent of the queries and answered 78.5 percent of them correctly. That's an improvement on a similar AI-focused test from April 2017 (66.1 percent of 800 questions answered correctly). While Loup Ventures looked at similar methodologies when testing smart speakers in February, the researchers explain that it's "not worthwhile to compare" the results across these tests since "the use cases differ greatly between digital assistants and smart speakers."

This is particularly true for Siri on HomePod, which performs well in certain areas but is largely limited to the amount of actions it can perform on the speaker itself. This led Apple's HomePod to become relegated to the "bottom of the totem pole" in an AI assistant performance test during Loup Venture's smart speaker research in February, with Siri answering 52.3 percent of 782 total questions correctly, across the same five categories as the new test.

Loup Ventures grades each digital assistant on two metrics: "Did it understand what was being asked?" and "Did it deliver a correct response?" Questions came from five categories, including Local (example: "Where is the nearest coffee shop?"), Commerce ("Can you order me more paper towels?"), Navigations ("How do I get to uptown on the bus?"), Information ("Who do the Twins play tonight?"), and Command ("Remind me to call Steve at 2pm today").

Charts via Loup Ventures

Questions were asked of Siri on an iPhone running iOS 11.4, Google Assistant on a Pixel XL, Alexa on the iOS app, and Cortana on the iOS app. Siri's best category was Command (90 percent of questions answered correctly), outperforming all rivals when asked to control aspects of the iPhone, smart home products, Apple Music, and more. Following Command, Siri performed well in Local (87 percent), Navigation (83 percent), and began dipping in Information (70 percent) and Commerce (60 percent).

Google Assistant has the edge in every category except Command. Siri’s lead over the Assistant in this category is odd, given they are both baked into the OS of the phone rather than living on a 3rd party app (as Cortana and Alexa do). We found Siri to be slightly more helpful and versatile (responding to more flexible language) in controlling your phone, smart home, music, etc. Our question set also includes a fair amount of music-related queries (the most common action for smart speakers). Apple, true to its roots, has ensured that Siri is capable with music on both mobile devices and smart speakers.

Google Assistant was the top digital assistant in all categories except Command, with Loup Ventures particularly liking Google's "featured snippets" feature that reads off search results of voice queries and is often "exactly what you're looking for." Both Alexa and Cortana were lesser performers in the test due to the iOS app for each limiting what the assistants can do on an iPhone, unlike Siri's ability to perform tasks all over iOS and not just in one app.

In total, Google Assistant answered 85.5 percent of the 800 questions asked correctly and understood all of them, compared to Siri's 78.5 percent answered correctly and 11 misunderstood. Alexa correctly answered 61.4 percent and misunderstood 13, while Cortana was the "laggard" and correctly answered 52.4 percent and misunderstood 19.


Over the 15 month period since April 2017, Siri improved by 13 percentage points, with Loup Ventures pointing out that it was "impressed with the speed at which the technology is advancing" for most of the assistants. The researchers went on to explain that many of the issues they had last year were erased by "improvements to natural language processing and inter-device connectivity."

Loup Ventures also sees more improvements coming down the line, particularly for Siri. This fall, the digital assistant will gain new abilities with Apple's upcoming "Siri Shortcuts" feature in iOS 12, which will let users create automations between apps that can be easily triggered through voice commands. The researches say they are "eager to test that feature" when it launches within iOS 12, likely coming in September.

Top Rated Comments

(View all)
Avatar
22 months ago
Siriously still bad.
Score: 22 Votes (Like | Disagree)
Avatar
22 months ago
Impressive number without any personal data mining.
Score: 20 Votes (Like | Disagree)
Avatar
22 months ago

Siriously still bad.

Absolutely not. Sure whiners will always say that.
Siri was and it is great, few years ago was impossible to dream of speak to your device. I use Siri about every hour for work.
Internet forums users tend to bash everything. Siri also is available on many languages and it is a big plus.
Cortana is quite bad for example, want to bash something? Cortana is there.
Score: 18 Votes (Like | Disagree)
Avatar
22 months ago

Absolutely not. Sure whiners will always say that.
Siri was and it is great, few years ago was impossible to dream of speak to your device. I use Siri about every hour for work.
Internet forums users tend to bash everything. Siri also is available on many languages and it is a big plus.
Cortana is quite bad for example, want to bash something? Cortana is there.

You know what's more annoying than "whiners?", people who label anyone who has an opinion or criticism that doesn't align with their own, a "whiner".

Glad you use it that much for work, you must have a job where you have to set a lot of timers :cool:
Score: 17 Votes (Like | Disagree)
Avatar
22 months ago

Impressive number without any personal data mining.

Personal Data mining is irrelevant.

Siri still sucks.
Score: 17 Votes (Like | Disagree)
Avatar
22 months ago
So Siri’s not nearly the train wreck MacRumors forum posters endlessly and loudly claim it to be.

Never could have guessed...

Most improved, usually beaten by google, usually better than Alexa, and always beats Cortana; a lot of folks here aren’t going to want to hear this. Time to shoot the messenger I suppose.
Score: 15 Votes (Like | Disagree)

Top Stories

Apple Acquires Weather App Dark Sky

Tuesday March 31, 2020 10:22 am PDT by Juli Clover
Apple has acquired weather app Dark Sky, Dark Sky's developers announced today. Dark Sky is one of the most popular weather apps on the App Store, known for its accuracy and storm warnings. Our goal has always been to provide the world with the best weather information possible, to help as many people as we can stay dry and safe, and to do so in a way that respects your privacy. There is no ...

Seemingly Unreleased Version of Logic Pro X With Live Loops Appears on Apple's Education Site [Updated]

Sunday March 29, 2020 7:23 am PDT by Hartley Charlton
Update: Apple has replaced the Logic Pro X image with an older version. Original story follows. A seemingly unreleased version of Logic Pro X has appeared on Apple's education site, as spotted by a Reddit user. The image from Apple's education products page shows a 16-inch MacBook Pro running Logic Pro X, but with a familiar interface that looks extremely similar to GarageBand's Live Loops ...

Bloomberg: Apple's 5G iPhone Still on Schedule for Fall Launch, But Future Products Could Be Delayed

Monday March 30, 2020 2:40 am PDT by Tim Hardwick
Apple's 5G iPhone is still on track to launch within the company's typical annual fall release schedule, according to a new Bloomberg report on filed on Monday. Signs are that Apple's Chinese-centric manufacturing -- of which Hon Hai is the linchpin -- is slowly getting back on track. The next iPhones with 5G wireless capabilities remain on schedule to launch in the fall, partly because mass...

Case for Upcoming Low-Cost iPhone Shows Up at Best Buy With Alleged April 5 Stock Date

Monday March 30, 2020 4:25 pm PDT by Juli Clover
Apple has a new low-cost iPhone in the works, which is supposed to be launching sometime in the first half of 2020. Given the ongoing situation in the United States and other countries, it's been unclear if the device is going to launch within the planned timeline, but there are signs that it could be coming soon. We started seeing cases for the new low-cost iPhone back in early February,...

Testing Brydge's New Pro+ Keyboard With Trackpad for iPad Pro

Monday March 30, 2020 2:04 pm PDT by Juli Clover
Well ahead of when Apple introduced trackpad support in iOS 13.4, Brydge announced an iPad Pro keyboard with a built-in multi-touch trackpad. We have one of Brydge's new Pro+ keyboards on hand, and thought we'd check it out to see how it works with Apple's new 2020 iPad Pro models. Subscribe to the MacRumors YouTube channel for more videos. The Brydge Pro+ keyboard is similar in design to...

Apple's Work on New Upcoming Products Progressing Normally as Employees Adjust to Telecommuting

Monday March 30, 2020 11:58 am PDT by Juli Clover
Apple's development of upcoming products is progressing as usual despite the fact that Apple employees around the world are working from home, according to a new report today out from Bloomberg. Apple is still working on new versions of the HomePod, Apple TV, MacBook Pro, budget iPads, Apple Watch, iPhone, and iMac, all of which could be released "as early as later this year" and have been...

Apple Releases ProRes RAW Beta for Windows

Monday March 30, 2020 9:33 am PDT by Juli Clover
Apple today released ProRes RAW for Windows in a beta capacity (via Mark Gurman), with the software designed to allow ProRes RAW and ProRes RAW HQ video files to be watched in compatible applications on Windows machines. According to Apple, the software will let the files be played within several Adobe apps: Adobe After Effects (Beta) Adobe Media Encocder (Beta) Adobe Premiere...

2020 iPad Pro Teardown Provides Closer Look at LiDAR Scanner and Confirms Incremental Update

Saturday March 28, 2020 9:56 am PDT by Hartley Charlton
iFixit today shared a video teardown of the new iPad Pro, which Apple unveiled earlier this month. iFixit found that most of the internals of the 2020 iPad Pro are the same as the 2018 model, confirming that the device is a relatively incremental update. The most notable new feature seen inside the new iPad Pro was the LiDAR scanner, which measures the distance to surrounding objects up...

Apple Configurator 2 Updated With New Features, Including Support for Restoring Firmware on 2019 Mac Pro

Tuesday March 31, 2020 5:34 am PDT by Joe Rossignol
Apple Configurator 2 has been updated to version 2.12 with several improvements, including support for restoring firmware on the 2019 Mac Pro. The release notes:• Added support for restoring firmware on the 2019 Mac Pro • Allow access to websites using TLS 1.0 and 1.1 • VPN: Configure Provider Designated Requirement for Custom SSL connection type • VPN: Configure network options for ...

U.S. Government Using Mobile Ad Location Data to Track Compliance With Curbs on Movement

Monday March 30, 2020 4:48 am PDT by Tim Hardwick
The U.S. government is using smartphone location data from the mobile ad industry to track people's movements amid the coronavirus outbreak, according to a Wall Street Journal report. Local governments and the Centers for Disease Control and Prevention have received the anonymized data about people in areas of "geographic interest," with the aim being to create a portal of geolocation...