Siri on HomePod Asked 800 Questions and Answered 74% Correctly vs. Just 52% Earlier This Year

Apple analyst Gene Munster of Loup Ventures recently tested the accuracy of digital assistants on four smart speakers by asking Alexa, Siri, Google Assistant, and Cortana a series of 800 questions each on the Amazon Echo, HomePod, Google Home Mini, and Harmon Kardon Invoke respectively.

homepod siri commands
The results indicate that Siri on the HomePod correctly answered 74.6 percent of the questions, a dramatic improvement over the speaker's 52.3 percent success rate when Loup Ventures asked it a similar 782 questions in December 2017.

homepod loup ventures test
Siri on the HomePod remained less accurate than Google Assistant on the Google Home, which correctly answered 87.9 percent of questions in the test. Meanwhile, Alexa on the Echo and Cortana on the Invoke trailed Siri on the HomePod, correctly answering 72.5 percent and 63.4 percent of questions in the test.

loup ventures smart speaker test 1
Munster attributed the HomePod's improved accuracy to "the enabling of more domains in the past year," as a series of software updates in recent months have enabled the speaker to make and receive phone calls, schedule calendar events, set multiple timers, search for songs by lyrics, and more.

Methodology

Loup Ventures says it asked each smart speaker the same 800 questions, and they were graded on two metrics: whether the query was understood and whether a correct response was provided. The question set was designed to "comprehensively test a smart speaker's ability and utility" based on five categories:

  • Local – Where is the nearest coffee shop?

  • Commerce – Can you order me more paper towels?

  • Navigation – How do I get to uptown on the bus?

  • Information – Who do the Twins play tonight?

  • Command – Remind me to call Steve at 2 p.m. today.

The venture capital firm said it continues to modify its question set in order to reflect the changing abilities of digital assistants. "As voice computing becomes more versatile and assistants become more capable, we will continue to alter our test so that it remains exhaustive," said Munster.

Results by Category

loup ventures by category
Google Assistant on the Google Home correctly answered the most questions in four out of the five categories in the test, but fell short of Siri on the HomePod in the "command" category, according to Loup Ventures:

HomePod's lead in this category may come from the fact that the HomePod will pass on full SiriKit requests like those regarding messaging, lists, and basically anything other than music to the iOS device paired to the speaker. Siri on iPhone has deep integration with email, calendar, messaging, and other areas of focus in our Command category. Our question set also contains a fair amount of music-related queries, which HomePod specializes in.

Loup Ventures found the HomePod and Google Home stood "head and shoulders above the others" in both the "local" and "navigation" categories due to propriety data in Apple Maps and Google Maps respectively. "This data is a potential long-term comparative advantage for Siri and Google Assistant," said Munster.

What's Next

Siri's shortcomings were brutally exposed on the HomePod, which relies almost entirely on the assistant to function, so anecdotal evidence of improvement is a good sign, but Apple still has significant ground to make up against its competitors.

Loup Ventures acknowledged that Siri remains limited on the HomePod compared to its expanded capabilities on the iPhone. "This is partially due to Apple's apparent positioning of HomePod not as a 'smart speaker,' but as a home speaker you can interact with using your voice with Siri onboard," said Munster.

The venture capital firm said it will continue to compare the HomePod to other smart speakers over time amid rapid advancements in artificial intelligence.

Related Roundup: HomePod
Buyer's Guide: HomePod (Caution)

Top Rated Comments

BootsWalking Avatar
25 months ago
Siri has indeed gotten much better. Just yesterday I asked the following question and got a richly contextualized answer:

Q: "Hey Siri, is it normal for my new iPad Pro to be bent?"

A: "Yes, it's normal for Apple to bend its customers over"
Score: 34 Votes (Like | Disagree)
AustinIllini Avatar
25 months ago
Still unacceptable for the company that pioneered the virtual assistant.

Siri had a lead and it is now miles behind Alexa in intelligence, which in turn is behind Google Assistant by quite a ways
Score: 12 Votes (Like | Disagree)
ArtOfWarfare Avatar
25 months ago
Why do we even talk about any of these percentages like they're even remotely acceptable?

If your keyboard only produced the correct character 80% of the time, or your touchscreen discarded 20% of touches, or your speaker only streamed 80% of a song, not a soul would be okay with that.

Siri on your iPhone or Mac is a funny novelty. The moment it becomes the only way you can interact with a device? Who on earth would find that acceptable? Maybe you'd be willing to buy such a device if it was priced like a whoopie cushion or silly putty (so, you know, ~$5), but for $300?

And this isn't even getting into the fact that this basic test is EASY! We're not expecting the devices to be productive contributors every conversation - these are tailored queries - softballs - meant to be easily within the realm of what the devices are supposed to be capable of doing!
Score: 11 Votes (Like | Disagree)
Jsameds Avatar
25 months ago

Still unacceptable for the company that pioneered the virtual assistant.

Siri had a lead and it is now miles behind Alexa in intelligence, which in turn is behind Google Assistant by quite a ways

I really don't mind if the one I'm using is slightly behind if that means looking after my privacy in a respectable manner.
Score: 9 Votes (Like | Disagree)
benshive Avatar
25 months ago


Siri had a lead and it is now miles behind Alexa in intelligence, which in turn is behind Google Assistant by quite a ways

According to this test Siri answered more queries correctly than Alexa. Doesn't mean it couldn't still be behind overall but "miles behind" is a pretty big exaggeration.
Score: 9 Votes (Like | Disagree)
Baymowe335 Avatar
25 months ago
I know it's popular to bash Siri here, but I think she works very well and has improved a lot.
Score: 7 Votes (Like | Disagree)

Top Stories

m1 chip macbook air pro

Kuo: Redesigned MacBooks With Apple Silicon to Launch in Second Half of 2021

Tuesday November 24, 2020 7:53 pm PST by
Apple plans to release additional MacBook models with Apple Silicon in the second half of 2021, according to analyst Ming-Chi Kuo, as part of the company's two-year transition away from Intel processors across its Mac lineup. In a research note today, obtained by MacRumors, Kuo said that these MacBook models will feature a new design. Kuo did not specify which models these will be, but he...
2020 apple shopping event

Apple Offering Up to $150 Gift Card With Select Products on Black Friday Through Cyber Monday

Monday November 23, 2020 2:53 am PST by
Apple has announced its annual four-day shopping event, offering customers up to a $150 Apple Store gift card with the purchase of select products between Black Friday and Cyber Monday in the United States. The gift card values in the United States are as follows: $150 for 16-inch MacBook Pro $150 for 21.5-inch iMac $50 for 13-inch MacBook Pro $50 for MacBook Air $50 for iPhone SE,...
app store christmas icon

Apple Shutting Down App Store Connect From December 23 to December 27

Monday November 23, 2020 10:14 am PST by
Apple shuts down App Store Connect for a week around the holidays each year in an effort to give App Store staff time off from work. This year, App Store Connect will be unavailable from December 23 to December 27. With App Store Connect unavailable, Apple will not accept new apps or app updates, so all pricing changes and new app submissions need to be locked in before those dates for...
Target November Deals 1

Black Friday Spotlight: Target Begins Week-Long Sale With Deals on iPhone 12, Powerbeats Pro, and More

Monday November 23, 2020 8:07 am PST by
We've been tracking early Black Friday deals in our dedicated Black Friday Roundup, and in an effort to prepare our readers for the big shopping event we're highlighting sales store-by-store in the lead-up to November 27. Note: MacRumors is an affiliate partner with Target. When you click a link and make a purchase, we may receive a small payment, which helps us keep the site running....
mac mini macbook pro macbook air

Apple M1 Hands-On Comparison: MacBook Air vs. MacBook Pro vs. Mac Mini

Monday November 23, 2020 3:40 pm PST by
Apple's M1 Macs are out in the wild now, but ahead of the holidays, you might still be trying to figure out which one to pick up, either for yourself or as a gift for someone else. We've got all three of the new Macs available, so we thought we'd give MacRumors readers a hands-on overview of each machine in our latest YouTube video. Subscribe to the MacRumors YouTube channel for more videos. ...
max tech xcode benchmark m1 macbook

Video Demos Performance Differences Between 8GB and 16GB Apple M1 MacBook Pro

Monday November 23, 2020 2:54 pm PST by
All of the M1 Mac models use the same M1 chip, so the upgrade options are limited to SSD storage space and RAM. We haven't seen many comparisons that demonstrate the difference between a machine with 8GB RAM and the upgraded 16GB RAM option, but Max Tech today shared a video highlighting the performance between an 8GB MacBook Pro and a 16GB MacBook Pro. The video includes a series of...
new mac mini logicpro screen

M1 Macs Able to Run Up to Six External Displays Using DisplayLink

Tuesday November 24, 2020 6:53 am PST by
It is possible to run up to six external displays from the M1 Mac mini, and five external displays from the M1 MacBook Air and MacBook Pro, with the aid of DisplayPort adapters, according to YouTuber Ruslan Tulupov. This far exceeds Apple's specified limits on external displays with the M1 Macs. Apple's host of new M1 Macs are not capable of supporting as many external displays as their...
iPads black friday 20 sale feature

Apple Black Friday 2020: Best iPad Deals

Tuesday November 24, 2020 8:51 am PST by
Although we're still a few days away from the official launch of Black Friday, we've known for a few weeks that many retailers would be kicking off discounts ahead of November 27. Because of this trend, today we're tracking discounts across Apple's lineup of iPads, including the 10.2-inch iPad and 2020 iPad Pro. More sales are likely to appear as the week progresses, and when they do we'll update ...
13 16 inch macbook pro air trio

Reliable Leaker Suggests Redesigned MacBooks in 2021 Will Include Both Apple Silicon and Intel Models

Wednesday November 25, 2020 9:15 am PST by
Reliable leaker known as "L0vetodream" has today suggested on Twitter that redesigned MacBooks coming in the second half of 2021 will include models with both Apple Silicon chips and Intel processors. The brief Tweet came in response to a MacRumors article from earlier today, which outlined a report from Ming-Chi Kuo claiming that Apple plans to release redesigned MacBook models with Apple ...
ipad pro 2020 display

Black Friday Week Kicks Off With Up to $150 Savings on 2020 iPad Pro

Sunday November 22, 2020 2:37 pm PST by
As we head into Black Friday week, we're seeing some of the best deals of the season so far, with Amazon and Best Buy today discounting the latest iPad Pro models by up to $150 at the lowest prices we've ever tracked on these models. Note: MacRumors is an affiliate partner with some of these vendors. When you click a link and make a purchase, we may receive a small payment, which helps us keep ...