Siri on HomePod Asked 800 Questions and Answered 74% Correctly vs. Just 52% Earlier This Year

Apple analyst Gene Munster of Loup Ventures recently tested the accuracy of digital assistants on four smart speakers by asking Alexa, Siri, Google Assistant, and Cortana a series of 800 questions each on the Amazon Echo, HomePod, Google Home Mini, and Harmon Kardon Invoke respectively.

homepod siri commands
The results indicate that Siri on the HomePod correctly answered 74.6 percent of the questions, a dramatic improvement over the speaker's 52.3 percent success rate when Loup Ventures asked it a similar 782 questions in December 2017.

homepod loup ventures test
Siri on the HomePod remained less accurate than Google Assistant on the Google Home, which correctly answered 87.9 percent of questions in the test. Meanwhile, Alexa on the Echo and Cortana on the Invoke trailed Siri on the HomePod, correctly answering 72.5 percent and 63.4 percent of questions in the test.

loup ventures smart speaker test 1
Munster attributed the HomePod's improved accuracy to "the enabling of more domains in the past year," as a series of software updates in recent months have enabled the speaker to make and receive phone calls, schedule calendar events, set multiple timers, search for songs by lyrics, and more.

Methodology

Loup Ventures says it asked each smart speaker the same 800 questions, and they were graded on two metrics: whether the query was understood and whether a correct response was provided. The question set was designed to "comprehensively test a smart speaker's ability and utility" based on five categories:

  • Local – Where is the nearest coffee shop?

  • Commerce – Can you order me more paper towels?

  • Navigation – How do I get to uptown on the bus?

  • Information – Who do the Twins play tonight?

  • Command – Remind me to call Steve at 2 p.m. today.

The venture capital firm said it continues to modify its question set in order to reflect the changing abilities of digital assistants. "As voice computing becomes more versatile and assistants become more capable, we will continue to alter our test so that it remains exhaustive," said Munster.

Results by Category

loup ventures by category
Google Assistant on the Google Home correctly answered the most questions in four out of the five categories in the test, but fell short of Siri on the HomePod in the "command" category, according to Loup Ventures:

HomePod's lead in this category may come from the fact that the HomePod will pass on full SiriKit requests like those regarding messaging, lists, and basically anything other than music to the iOS device paired to the speaker. Siri on iPhone has deep integration with email, calendar, messaging, and other areas of focus in our Command category. Our question set also contains a fair amount of music-related queries, which HomePod specializes in.

Loup Ventures found the HomePod and Google Home stood "head and shoulders above the others" in both the "local" and "navigation" categories due to propriety data in Apple Maps and Google Maps respectively. "This data is a potential long-term comparative advantage for Siri and Google Assistant," said Munster.

What's Next

Siri's shortcomings were brutally exposed on the HomePod, which relies almost entirely on the assistant to function, so anecdotal evidence of improvement is a good sign, but Apple still has significant ground to make up against its competitors.

Loup Ventures acknowledged that Siri remains limited on the HomePod compared to its expanded capabilities on the iPhone. "This is partially due to Apple's apparent positioning of HomePod not as a 'smart speaker,' but as a home speaker you can interact with using your voice with Siri onboard," said Munster.

The venture capital firm said it will continue to compare the HomePod to other smart speakers over time amid rapid advancements in artificial intelligence.

Related Roundup: HomePod
Buyer's Guide: HomePod (Buy Now)

Top Rated Comments

BootsWalking Avatar
56 months ago
Siri has indeed gotten much better. Just yesterday I asked the following question and got a richly contextualized answer:

Q: "Hey Siri, is it normal for my new iPad Pro to be bent?"

A: "Yes, it's normal for Apple to bend its customers over"
Score: 34 Votes (Like | Disagree)
Dragonmount Avatar
56 months ago
Still unacceptable for the company that pioneered the virtual assistant.

Siri had a lead and it is now miles behind Alexa in intelligence, which in turn is behind Google Assistant by quite a ways
Score: 12 Votes (Like | Disagree)
ArtOfWarfare Avatar
56 months ago
Why do we even talk about any of these percentages like they're even remotely acceptable?

If your keyboard only produced the correct character 80% of the time, or your touchscreen discarded 20% of touches, or your speaker only streamed 80% of a song, not a soul would be okay with that.

Siri on your iPhone or Mac is a funny novelty. The moment it becomes the only way you can interact with a device? Who on earth would find that acceptable? Maybe you'd be willing to buy such a device if it was priced like a whoopie cushion or silly putty (so, you know, ~$5), but for $300?

And this isn't even getting into the fact that this basic test is EASY! We're not expecting the devices to be productive contributors every conversation - these are tailored queries - softballs - meant to be easily within the realm of what the devices are supposed to be capable of doing!
Score: 11 Votes (Like | Disagree)
Jsameds Avatar
56 months ago
Still unacceptable for the company that pioneered the virtual assistant.

Siri had a lead and it is now miles behind Alexa in intelligence, which in turn is behind Google Assistant by quite a ways
I really don't mind if the one I'm using is slightly behind if that means looking after my privacy in a respectable manner.
Score: 9 Votes (Like | Disagree)
benshive Avatar
56 months ago

Siri had a lead and it is now miles behind Alexa in intelligence, which in turn is behind Google Assistant by quite a ways
According to this test Siri answered more queries correctly than Alexa. Doesn't mean it couldn't still be behind overall but "miles behind" is a pretty big exaggeration.
Score: 9 Votes (Like | Disagree)
Baymowe335 Avatar
56 months ago
I know it's popular to bash Siri here, but I think she works very well and has improved a lot.
Score: 7 Votes (Like | Disagree)

Popular Stories

iOS 16

iOS 16.4 Now Available for Your iPhone With These 8 New Features

Friday March 31, 2023 8:55 am PDT by
Following six weeks of beta testing, iOS 16.4 was released to the public this week. The software update includes a handful of new features and changes for the iPhone 8 and newer. To install an iOS update, open the Settings app on the iPhone, tap General → Software Update, and follow the on-screen instructions. Below, we have recapped eight new features and changes added with iOS 16.4,...
CarPlay Phone Call

General Motors to Phase Out Apple CarPlay Starting This Year in EV Transition

Friday March 31, 2023 8:43 am PDT by
General Motors (GM) will phase out Apple CarPlay and Android Auto in its vehicles starting this year, shifting to a built-in infotainment system co-developed with Google (via Reuters). GM owns Buick, Cadillac, Chevrolet, and GMC in the United States. It will stop offering Apple CarPlay and Android Auto starting with the 2024 Chevrolet Blazer, which goes on sale this summer. The company plans ...
iOS 17 on Phone Feature

Three New iOS Features Coming to Your iPhone Following Apple Music Classical

Thursday March 30, 2023 7:13 am PDT by
With the Apple Music Classical app and an Apple Pay Later early access program now available, the list of previously-announced iOS features that have yet to launch is beginning to shrink. However, there are still a few features we are waiting for. Below, we have recapped three more iOS features that are expected to launch in 2023, including an Apple Card savings account for Daily Cash,...
wwdc 2023

Three Products We Might See at WWDC 2023

Friday March 31, 2023 3:37 pm PDT by
Apple this week announced the official dates for the 34th annual Worldwide Developers Conference, with the annual WWDC keynote event set to take place on Monday, June 5. The keynote is where Apple unveils new versions of iOS, macOS, watchOS, and tvOS, and sometimes, we get hardware announcements. Rumors this year suggest there are at least three new devices that are set to be unveiled in the ...
iPhone 15 Pro Multi Purpose button Mute Switch Feature Green 2

iPhone 15 Pro Solid-State Button Sensitivity Can Be Customized to Cater for Cases and Gloves

Thursday March 30, 2023 11:36 pm PDT by
iPhone 15 Pro and iPhone 15 Pro Max users will be able to customize the sensitivity of the solid-state buttons on their device, thanks to a new sensitivity toggle in Settings. That's according to details provided by a hitherto reliable source that shared additional details on the MacRumors forums. Earlier this week, the same anonymous tipster revealed that the iPhone 15 Pro models will use...
apple mixed reality headset concept by david lewis and marcus kane

Kuo: Apple Mixed-Reality Headset May Not Appear at WWDC as Mass Production Pushed Back Yet Again

Thursday March 30, 2023 4:50 am PDT by
Apple has again pushed back mass production of its mixed-reality headset and the device may not appear at this year's Worldwide Developers Conference (WWDC), Apple analyst Ming-Chi Kuo today said. Apple headset concept by David Lewis and Marcus Kane In a tweet, Kuo explained that Apple "isn't very optimistic" about whether the headset will be able to create an "iPhone moment." As a result,...
iPhone 15 Pro Periscope Mock Feature

Periscope Camera Lens Exclusive to iPhone 15 Pro Max Will Be Supplied by Largan

Thursday March 30, 2023 1:18 am PDT by
The periscope camera lens that will be exclusive to the iPhone 15 Pro Max will be solely supplied by Largan, according to the 相機鏡頭中獲利-apple-camera-lens-suppliers-face-two-risks-high-53db8da990b2">latest no by Apple industry analyst Ming-Chi Kuo. Rumors about the iPhone getting a periscope lens have been circulating since early 2020, when Kuo first mentioned the possibility. The analyst...