Siri on HomePod Correctly Answered 52.3% of Queries in New AI Test

In a new test shared today by Loup Ventures, Apple's HomePod was put through its paces in categories including Siri, sound quality, and ease of use. For Siri, Loup Ventures' Gene Munster reported that while the AI assistant understood 99.4 percent of queries asked of it, it answered only 52.3 percent of them correctly. Loup Ventures tested three separate HomePods and asked 782 queries total.

Compared to previous tests of rival speakers, HomePod is "at the bottom of the totem pole" in the AI assistant performance category. Google Home answered 81 percent correctly, Amazon's Alexa answered 64 percent correctly, and Microsoft's Cortana answered 57 percent correctly.


Munster broke this information down further, stating that Siri is good for "local" and "commerce" queries, like asking about nearby coffee shops or assisting in buying new shoes. In this area, Siri beats Alexa and Cortana but still falls behind Google Assistant on Google Home.

Despite the low percentage of correctly answering the 782 total queries asked of it, Munster said Siri's overall performance rose above expectations "given the limited scope of HomePod's music focus."

Chart via Loup Ventures

The researchers explained that over time HomePod and Siri should grow to match, or surpass, rival assistants by simply adding query domains like calendar, email, calling, and navigation.

Some domains like navigation, calendar, email, and calling are simply not supported. These questions were met with, “I can’t ___ on HomePod.” Also, in any case that iPhone-based Siri would bring up Google search results, HomePod would reply, “I can’t get the answer to that on HomePod,” which forces you to use your phone or give up on the question altogether. Removing navigation, calling, email, and calendar-related queries from our question set yields a 67% correct response, a jump from overall of 52.3% correct.

This means added support for these domains would bring HomePod performance above that of Alexa (64%) and Cortana (57%), though still shy of Google Home (81%). We know Siri has the ability to correctly answer a whole range of queries that HomePod cannot, evidenced by our note here. Apple’s limiting of HomePod’s domains should change over time, at which point we expect the speaker to be vastly more useful and integrated with your other Apple devices.

As discovered in the research, where HomePod excelled was its "superior" listening skills. The HomePod allows users to speak at a normal volume, even when music from the speaker is particularly loud, and Siri will pick up on the voice and hear the user. "This was HomePod's most stellar feature," according to Munster.

Loup Ventures also favored HomePod's sound quality, saying that "it sounds incredible." Like other reviews and impressions of the device, Munster's enjoyment of the HomePod's audio quality was countered with the speaker's lackluster Siri performance, which he thinks will be changing fairly soon.

Don’t be fooled by HomePod’s sound quality-focused first step into smart speakers; Apple has a grander vision than delivering a better sounding Echo. While not present in the first version of HomePod (i.e. you can’t even make a phone call with HomePod), we believe Apple’s goal is to make Siri a ubiquitous, ambient presence that connects and controls all your connected devices and services – and to make a leap forward in the transition to voice-first computing.

Visit Loup Ventures to read more of the information gathered from the HomePod "smart speaker gauntlet," including the publication's prediction for demand and market share of HomePod through the rest of 2018.

Related Roundup: HomePod
Buyer's Guide: HomePod (Caution)

Top Rated Comments

(View all)
Avatar
34 months ago
Me: “Hey Siri, why are you so terrible at answering questions?”

Siri: “I found several articles on the web about crustaceans.”
Score: 132 Votes (Like | Disagree)
Avatar
34 months ago
That's pretty damn pathetic...
Score: 108 Votes (Like | Disagree)
Avatar
34 months ago
Siri is Apple's biggest failure. They have had years to make it at least 80% functional and have failed to do so. Just like the ATV, it seems to be just a "hobby" for Apple.
Score: 84 Votes (Like | Disagree)
Avatar
34 months ago
Siri sucks. I don’t think I’ve used it in over a year.
Score: 77 Votes (Like | Disagree)
Avatar
34 months ago
What good is "superior" listeng skills if it can't answer many of the questions?
Score: 75 Votes (Like | Disagree)
Avatar
34 months ago

I think Siri is more than adequate for what most need it for the HomePod. Is it better than Google Home? No. Does it need to be better than the competition? Not necessarily. I use Siri all the time and I really don’t have any general complaints, aside from not understanding some of the things I ask it. None of these voice assistants are perfect.

I can only imagine the amount of gloating on these forums if Siri was #1 on the list instead of Google.
Score: 38 Votes (Like | Disagree)

Top Stories

Hands-On With iOS 14 Widgets, Custom Icons, and Home Screen Setup

Tuesday September 22, 2020 3:25 pm PDT by
Apple with iOS 14 introduced widgets on the Home Screen, leading to unprecedented levels of customization for the iPhone. Combined with Shortcuts that let you change an app's icon, iOS 14 lets you create a whole new look for your Home Screen. Subscribe to the MacRumors YouTube channel for more videos. We've been following along with some of the creative alternative Home Screen designs that M...

New Images Leak of iPhone 12 Braided USB-C to Lightning Cable

Thursday September 24, 2020 2:37 am PDT by
Rumors suggest Apple's upcoming iPhone 12 models will ship with a new Lightning to USB-C cable that includes a braided fabric design. Images of the purported cables were leaked in July, and today leaker Mr White has shared new images that give us a closer look at what we might get included in the iPhone 12 box. The photos show a USB-C to Lightning cable with a clearly braided design rather...

Apple's iPhone 12 Event Could Happen on October 13 Based on Rumors From Mobile Operators

Wednesday September 23, 2020 11:51 am PDT by
Apple's upcoming iPhone-centric event could perhaps be held on Tuesday, October 13, according to information shared with MacRumors by an employee at a UK cellular carrier. There's no way for us to confirm the dates at this point in time nor are we sure on the credibility of the source, but even without a rumor, Tuesday, October 13 is a good guess based on Apple's historic launch timelines, ...

New Version of Microsoft Office Coming Next Year That Won't Require a Subscription

Thursday September 24, 2020 1:53 am PDT by
Microsoft will next year offer a new perpetual release of Microsoft Office for Mac and Windows that doesn't require a subscription to use, according to the software giant (via Windows Central). "Microsoft Office will also see a new perpetual release for both Windows and Mac, in the second half of 2021," said Microsoft in a blog post announcing the next version of its Exchange server,...

Interest in iOS 14 Home Screen Ideas Helps Pinterest Break Daily Download Record

Wednesday September 23, 2020 4:37 am PDT by
Apple's introduction of widgets on the Home Screen in iOS 14 has driven a surge in interest among users looking to customize their iPhone, and that has reportedly had a knock-on effect for Pinterest, whose iOS app has seen record downloads as users flock to its content seeking design inspiration. As reported by TechCrunch, App Store intelligence firm Apptopia was first to note the impact of ...

'iPhone 12 mini' Name Reappears in Leaked Apple iPhone 12 Case Stickers

Friday September 25, 2020 1:58 am PDT by
Earlier this week a proven leaker claimed that the iPhone 12 lineup would be named "iPhone 12 mini," "iPhone 12," "iPhone 12 Pro," and "iPhone 12 Pro Max," and today the same nomenclature has appeared again in a photo depicting alleged stickers from unreleased Silicone iPhone cases originating from Apple's international distribution center in Ireland. The photo shows three stickers with the...

Apple Releases iOS 14.0.1 With Fix for Bug That Resets Default Apps After Rebooting

Thursday September 24, 2020 10:12 am PDT by
Apple today released iOS 14.0.1, the first update to the iOS 14 operating system that was released on September 16. Today's update is a bug fix update addressing issues that weren't able to be fixed in the initial iOS 14 launch. The iOS 14.0.1 update is available on all eligible devices over-the-air in the Settings app. To access the new software, go to Settings > General > Software Update. ...

Apple Emphasizes That Solo Loop May Increase in Length Over Time, Updates Sizing Guide With More Specific Instructions

Wednesday September 23, 2020 8:26 am PDT by
Apple on Tuesday updated one of its support documents to emphasize that the new silicone rubber Solo Loop for the Apple Watch may increase in length over time, as mentioned in fine print at the bottom of the Solo Loop product page. Apple has also updated its printable Solo Loop sizing guide with more specific instructions, as noted by 9to5Mac's Michael Steeber. The guide now advises users to ...

iOS 14 Widgets Offer iPhone Users Creative Home Screen Ideas

Sunday September 20, 2020 8:43 pm PDT by
Updated on September 22nd with hands on video. In iOS 14, Apple introduced ‌the concept of Home Screen‌ widgets, which provide information from apps at a glance. Widgets can be pinned to the Home Screen in various spots and sizes, allowing for many different layouts. Despite the relative lack of...

Kuo: Mini-LED Displays to Debut in Apple's Next iPad Pro

Tuesday September 22, 2020 11:45 pm PDT by
Following his report earlier this week indicating that Apple will be accelerating adoption of mini-LED displays in its iPad and Mac notebook lineups thanks to better than expected development from potential secondary supplier Sanan Optoelectronics, analyst Ming-Chi has released a new report today sharing a bit more perspective on the mini-LED display market as it relates to Apple. Kuo says...