Siri on iOS Answered 78.5% of Queries Correctly in Latest Test, Trailing Google Assistant at 85.5%

Five months after performing a test that put the smart speakers of multiple companies in the spotlight to determine how well they performed in various categories, Loup Ventures is back today with an IQ test focused entirely on digital AI assistants. To get the necessary results, the researchers asked Siri, Google Assistant, Alexa, and Cortana 800 questions each on a smartphone, and compared their findings to a previous AI test held in April 2017.


For Siri in the new test, Apple's AI helper understood 99 percent of the queries and answered 78.5 percent of them correctly. That's an improvement on a similar AI-focused test from April 2017 (66.1 percent of 800 questions answered correctly). While Loup Ventures looked at similar methodologies when testing smart speakers in February, the researchers explain that it's "not worthwhile to compare" the results across these tests since "the use cases differ greatly between digital assistants and smart speakers."

This is particularly true for Siri on HomePod, which performs well in certain areas but is largely limited to the amount of actions it can perform on the speaker itself. This led Apple's HomePod to become relegated to the "bottom of the totem pole" in an AI assistant performance test during Loup Venture's smart speaker research in February, with Siri answering 52.3 percent of 782 total questions correctly, across the same five categories as the new test.

Loup Ventures grades each digital assistant on two metrics: "Did it understand what was being asked?" and "Did it deliver a correct response?" Questions came from five categories, including Local (example: "Where is the nearest coffee shop?"), Commerce ("Can you order me more paper towels?"), Navigations ("How do I get to uptown on the bus?"), Information ("Who do the Twins play tonight?"), and Command ("Remind me to call Steve at 2pm today").

Charts via Loup Ventures

Questions were asked of Siri on an iPhone running iOS 11.4, Google Assistant on a Pixel XL, Alexa on the iOS app, and Cortana on the iOS app. Siri's best category was Command (90 percent of questions answered correctly), outperforming all rivals when asked to control aspects of the iPhone, smart home products, Apple Music, and more. Following Command, Siri performed well in Local (87 percent), Navigation (83 percent), and began dipping in Information (70 percent) and Commerce (60 percent).

Google Assistant has the edge in every category except Command. Siri’s lead over the Assistant in this category is odd, given they are both baked into the OS of the phone rather than living on a 3rd party app (as Cortana and Alexa do). We found Siri to be slightly more helpful and versatile (responding to more flexible language) in controlling your phone, smart home, music, etc. Our question set also includes a fair amount of music-related queries (the most common action for smart speakers). Apple, true to its roots, has ensured that Siri is capable with music on both mobile devices and smart speakers.

Google Assistant was the top digital assistant in all categories except Command, with Loup Ventures particularly liking Google's "featured snippets" feature that reads off search results of voice queries and is often "exactly what you're looking for." Both Alexa and Cortana were lesser performers in the test due to the iOS app for each limiting what the assistants can do on an iPhone, unlike Siri's ability to perform tasks all over iOS and not just in one app.

In total, Google Assistant answered 85.5 percent of the 800 questions asked correctly and understood all of them, compared to Siri's 78.5 percent answered correctly and 11 misunderstood. Alexa correctly answered 61.4 percent and misunderstood 13, while Cortana was the "laggard" and correctly answered 52.4 percent and misunderstood 19.


Over the 15 month period since April 2017, Siri improved by 13 percentage points, with Loup Ventures pointing out that it was "impressed with the speed at which the technology is advancing" for most of the assistants. The researchers went on to explain that many of the issues they had last year were erased by "improvements to natural language processing and inter-device connectivity."

Loup Ventures also sees more improvements coming down the line, particularly for Siri. This fall, the digital assistant will gain new abilities with Apple's upcoming "Siri Shortcuts" feature in iOS 12, which will let users create automations between apps that can be easily triggered through voice commands. The researches say they are "eager to test that feature" when it launches within iOS 12, likely coming in September.

Top Rated Comments

(View all)
Avatar
26 months ago
Siriously still bad.
Score: 22 Votes (Like | Disagree)
Avatar
26 months ago
Impressive number without any personal data mining.
Score: 20 Votes (Like | Disagree)
Avatar
26 months ago

Siriously still bad.

Absolutely not. Sure whiners will always say that.
Siri was and it is great, few years ago was impossible to dream of speak to your device. I use Siri about every hour for work.
Internet forums users tend to bash everything. Siri also is available on many languages and it is a big plus.
Cortana is quite bad for example, want to bash something? Cortana is there.
Score: 18 Votes (Like | Disagree)
Avatar
26 months ago

Absolutely not. Sure whiners will always say that.
Siri was and it is great, few years ago was impossible to dream of speak to your device. I use Siri about every hour for work.
Internet forums users tend to bash everything. Siri also is available on many languages and it is a big plus.
Cortana is quite bad for example, want to bash something? Cortana is there.

You know what's more annoying than "whiners?", people who label anyone who has an opinion or criticism that doesn't align with their own, a "whiner".

Glad you use it that much for work, you must have a job where you have to set a lot of timers :cool:
Score: 17 Votes (Like | Disagree)
Avatar
26 months ago

Impressive number without any personal data mining.

Personal Data mining is irrelevant.

Siri still sucks.
Score: 17 Votes (Like | Disagree)
Avatar
26 months ago
So Siri’s not nearly the train wreck MacRumors forum posters endlessly and loudly claim it to be.

Never could have guessed...

Most improved, usually beaten by google, usually better than Alexa, and always beats Cortana; a lot of folks here aren’t going to want to hear this. Time to shoot the messenger I suppose.
Score: 15 Votes (Like | Disagree)

Top Stories

Apple Warns Against Closing MacBooks With a Cover Over the Camera

Friday July 10, 2020 11:12 am PDT by
Apple this month published a support document that warns customers against closing their Mac notebooks with a cover over the camera as it can lead to display damage. Image via Reddit Apple says that the clearance between the display and the keyboard is designed to very tight tolerances, which can be problematic. Covering the camera can also cause issues with automatic brightness and True Tone....

iPhone Users Who Experienced 'Batterygate' Can Now File to Receive Around $25 Settlement From Apple

Monday July 13, 2020 6:50 am PDT by
Earlier this year, Apple agreed to settle a U.S. class action lawsuit that accused the company of "secretly throttling" older iPhone models. Now, eligible iPhone owners are beginning to be notified about their legal rights and options. Under the proposed settlement, Apple will provide a cash payment of approximately $25 to each eligible iPhone owner who submits a claim, with its total payout ...

Apple Moving Forward on Semitransparent Lenses for Upcoming AR Headset

Friday July 10, 2020 7:24 am PDT by
Apple and Foxconn have reached a key milestone in the development of Apple's long-rumored augmented reality headset, with the semitransparent lenses for the device moving from prototype to trial production, reports The Information. Apple is developing the lenses on a single production line at a Foxconn factory in Chengdu in southwestern China, where most of Apple’s iPad production is...

Arm-Intel-PowerPC Universal Binaries Are Possible

Saturday July 11, 2020 1:42 pm PDT by
Casual MacRumors visitors may not realize that we have a very active PowerPC forum where users discuss issues related to PowerPC Macs that have not been produced since 2006. Threads range from hardware upgrades and software options to nostalgia: Photo by AphoticD Apple's recently announced transition to Apple Silicon (Arm) based Macs raised some interesting questions about future support...

Possible 'iPhone 12' Battery Certifications Suggest Lower Capacities Than iPhone 11 Series

Monday July 13, 2020 4:22 am PDT by
MySmartPrice has spotted certifications for three new Apple batteries that it believes could be for the upcoming iPhone 12 lineup, despite them being less capacitive than the batteries in the current iPhone 11 series. The batteries are identified with the model numbers A2471, A2431, and A2466, and appear on Safety Korea, China's 3C, and the Danish agency UL Demko. Apple is expected to...

Leaker: 'iPhone 12 Pro' to Come With 6GB of RAM

Friday July 10, 2020 1:59 am PDT by
Later this year, Apple is expected to release four OLED iPhones in three display sizes, including 5.4, 6.7, and two 6.1-inch models. Rumors suggest the 6.7-inch iPhone and one 6.1-inch model will be higher-end devices, and now leaker @L0vetodream has corroborated previous rumors about the internal specs of Apple's upcoming lineup. Rumors suggest Apple will use 5-nanometer A14 chips in its...

Kuo: Apple Silicon Macs to Include 13-inch MacBook Pro and MacBook Air This Year, 14.1-inch and 16-inch MacBook Pro Models Next Year

Friday July 10, 2020 2:58 am PDT by
At last month's WWDC, Apple officially announced that its Mac computers will be transitioned from Intel x86 to homegrown Apple Silicon chips. Apple said it plans to deliver the first Apple Silicon Mac by the end of the year and complete the transition in about two years. According to Apple analyst Ming-Chi Kuo, a 13.3-inch MacBook Pro with a form factor similar to the current 13.3-inch...

Google to 'Dramatically' Improve Chrome Impact on Mac Battery Life

Sunday July 12, 2020 1:56 pm PDT by
Google will address long-standing battery life issues, particularly on Mac devices, reports The Wall Street Journal. Chrome will improve "tab throttling" by better prioritizing active tabs and limiting resource drain from tabs open in the background. This is said to have a "dramatic impact on battery and performance." Google has reportedly been performing early tests on Mac laptops in...

Top Stories: iOS 14 Public Beta, iPhone 12 Size Comparison, 14-Inch MacBook Pro Rumors

Saturday July 11, 2020 6:00 am PDT by
After one round of developer beta testing, Apple unleashed iOS and iPadOS 14 to a wider audience this week, opening it up to members of the public beta program. There are lots of changes and new features to check out, but as with any beta, be careful about installing it on your main devices. Subscribe to the MacRumors YouTube channel for more videos. Other major stories this week included our ...

Apple Shares Humorous 'Working-From-Home Thing' Video

Monday July 13, 2020 9:31 am PDT by
Apple today shared a funny video focused on the problems that people working from home have to deal with, including noisy children, chaotic schedules, communication issues, and more. The video focuses on showing off Apple products and their capabilities that can be useful when working from home, such as the ability to scan a document with an iPhone, mark up a PDF, Siri Reminders, and more.The...