Siri on HomePod Correctly Answered 52.3% of Queries in New AI Test

In a new test shared today by Loup Ventures, Apple's HomePod was put through its paces in categories including Siri, sound quality, and ease of use. For Siri, Loup Ventures' Gene Munster reported that while the AI assistant understood 99.4 percent of queries asked of it, it answered only 52.3 percent of them correctly. Loup Ventures tested three separate HomePods and asked 782 queries total.

Compared to previous tests of rival speakers, HomePod is "at the bottom of the totem pole" in the AI assistant performance category. Google Home answered 81 percent correctly, Amazon's Alexa answered 64 percent correctly, and Microsoft's Cortana answered 57 percent correctly.

mitchs homepod on shelf
Munster broke this information down further, stating that Siri is good for "local" and "commerce" queries, like asking about nearby coffee shops or assisting in buying new shoes. In this area, Siri beats Alexa and Cortana but still falls behind Google Assistant on Google Home.

Despite the low percentage of correctly answering the 782 total queries asked of it, Munster said Siri's overall performance rose above expectations "given the limited scope of HomePod's music focus."

homepod siri loup ventures

Chart via Loup Ventures

The researchers explained that over time HomePod and Siri should grow to match, or surpass, rival assistants by simply adding query domains like calendar, email, calling, and navigation.

Some domains like navigation, calendar, email, and calling are simply not supported. These questions were met with, “I can’t ___ on HomePod.” Also, in any case that iPhone-based Siri would bring up Google search results, HomePod would reply, “I can’t get the answer to that on HomePod,” which forces you to use your phone or give up on the question altogether. Removing navigation, calling, email, and calendar-related queries from our question set yields a 67% correct response, a jump from overall of 52.3% correct.

This means added support for these domains would bring HomePod performance above that of Alexa (64%) and Cortana (57%), though still shy of Google Home (81%). We know Siri has the ability to correctly answer a whole range of queries that HomePod cannot, evidenced by our note here. Apple’s limiting of HomePod’s domains should change over time, at which point we expect the speaker to be vastly more useful and integrated with your other Apple devices.

As discovered in the research, where HomePod excelled was its "superior" listening skills. The HomePod allows users to speak at a normal volume, even when music from the speaker is particularly loud, and Siri will pick up on the voice and hear the user. "This was HomePod's most stellar feature," according to Munster.

Loup Ventures also favored HomePod's sound quality, saying that "it sounds incredible." Like other reviews and impressions of the device, Munster's enjoyment of the HomePod's audio quality was countered with the speaker's lackluster Siri performance, which he thinks will be changing fairly soon.

Don’t be fooled by HomePod’s sound quality-focused first step into smart speakers; Apple has a grander vision than delivering a better sounding Echo. While not present in the first version of HomePod (i.e. you can’t even make a phone call with HomePod), we believe Apple’s goal is to make Siri a ubiquitous, ambient presence that connects and controls all your connected devices and services – and to make a leap forward in the transition to voice-first computing.

Visit Loup Ventures to read more of the information gathered from the HomePod "smart speaker gauntlet," including the publication's prediction for demand and market share of HomePod through the rest of 2018.

Related Roundup: HomePod
Buyer's Guide: HomePod (Buy Now)

Top Rated Comments

RudySnow Avatar
80 months ago
Me: “Hey Siri, why are you so terrible at answering questions?”

Siri: “I found several articles on the web about crustaceans.”
Score: 132 Votes (Like | Disagree)
Mobster1983 Avatar
80 months ago
That's pretty damn pathetic...
Score: 108 Votes (Like | Disagree)
SeattleMoose Avatar
80 months ago
Siri is Apple's biggest failure. They have had years to make it at least 80% functional and have failed to do so. Just like the ATV, it seems to be just a "hobby" for Apple.
Score: 84 Votes (Like | Disagree)
unobtainium Avatar
80 months ago
Siri sucks. I don’t think I’ve used it in over a year.
Score: 77 Votes (Like | Disagree)
jasvncnt Avatar
80 months ago
What good is "superior" listeng skills if it can't answer many of the questions?
Score: 75 Votes (Like | Disagree)
palmerc2 Avatar
80 months ago
I think Siri is more than adequate for what most need it for the HomePod. Is it better than Google Home? No. Does it need to be better than the competition? Not necessarily. I use Siri all the time and I really don’t have any general complaints, aside from not understanding some of the things I ask it. None of these voice assistants are perfect.
I can only imagine the amount of gloating on these forums if Siri was #1 on the list instead of Google.
Score: 38 Votes (Like | Disagree)

Popular Stories

iPhone 16 Camera Lozenge 2 Perspective Gray

Five Key Upgrades Coming to iPhone 16

Friday March 15, 2024 1:45 pm PDT by
The iPhone is Apple's top-selling product, and it gets an update every year. In 2024, we're expecting the iPhone 16 and iPhone 16 Pro lineup, with an arguably more interesting feature set than we got with the iPhone 15 and iPhone 15 Pro. Subscribe to the MacRumors YouTube channel for more videos. Capture Button All four iPhone 16 models are set to get a whole new button, which will be...
When To Expect New iPads Feature 1

Apple to Announce New iPads on March 26, Rumors Claim

Monday March 18, 2024 4:02 am PDT by
Apple is widely expected to release new iPad Air and OLED iPad Pro models in the next few weeks. According to new rumors coming out of Asia, the company will announce its new iPads on Tuesday, March 26. Chinese leaker Instant Digital on Weibo this morning 日发布%23">claimed that the date will see some sort of announcement from Apple related to new iPads, but stopped short of calling it an...
airpods 3 orange

Two New AirPods 4 Models Expected to Launch in September or October

Sunday March 17, 2024 7:56 am PDT by
Apple suppliers will begin production of two new fourth-generation AirPods models in May, according to Bloomberg's Mark Gurman. Based on this production timeframe, he expects the headphones to be released in September or October. Gurman expects both fourth-generation AirPods models to feature a new design with better fit, improved sound quality, and an updated charging case with a USB-C...
iphone se 4 modified flag edges

iPhone SE 4 Expected to Depreciate Heavily

Tuesday March 12, 2024 9:04 am PDT by
Resale value trends suggest the iPhone SE 4 may not hold its value as well as Apple's flagship models, according to SellCell. According to the report, Apple's iPhone SE models have historically depreciated much more rapidly than the company's more premium offerings. The third-generation iPhone SE, which launched in March 2022, experienced a significant drop in resale value, losing 42.6%...
General iOS 17 Feature Orange Purple

iOS 17.4.1 Update for iPhone is Imminent

Monday March 18, 2024 5:27 am PDT by
iOS 17.4.1 and iPadOS 17.4.1 should be released within the next few days, with a build number of 21E235, according to a source with a proven track record. MacRumors previously reported that Apple was internally testing iOS 17.4.1. As a minor update for the iPhone, it will likely address software bugs and/or security vulnerabilities. It is unclear if the update will include any other changes. ...
M3 iPad Feature 3

New iPads Likely to Begin Shipping in April

Monday March 18, 2024 9:52 am PDT by
Apple's new iPad Pro models with OLED displays will likely begin shipping to customers in April, according to information shared today by Ross Young, CEO of display industry research firm Display Supply Chain Consultants. Bloomberg's Mark Gurman also said the new iPad Pro models might not ship until "deeper" into April in his Power On newsletter on Sunday:I've repeatedly said that new...
iOS 17 Passkey With Apple ACCOUNT Feature

'Apple ID' Expected to Change to 'Apple Account' Starting With iOS 18

Sunday March 17, 2024 7:13 am PDT by
MacRumors was first to report that Apple was planning to rebrand "Apple ID" to "Apple Account" across its software platforms and websites like iCloud.com as early as this year, and now Bloomberg's Mark Gurman has corroborated this change. A mockup of the new Apple Account branding In his Power On newsletter today, Gurman said the new "Apple Account" branding will start to be used later this...