Siri on HomePod Correctly Answered 52.3% of Queries in New AI Test

In a new test shared today by Loup Ventures, Apple's HomePod was put through its paces in categories including Siri, sound quality, and ease of use. For Siri, Loup Ventures' Gene Munster reported that while the AI assistant understood 99.4 percent of queries asked of it, it answered only 52.3 percent of them correctly. Loup Ventures tested three separate HomePods and asked 782 queries total.

Compared to previous tests of rival speakers, HomePod is "at the bottom of the totem pole" in the AI assistant performance category. Google Home answered 81 percent correctly, Amazon's Alexa answered 64 percent correctly, and Microsoft's Cortana answered 57 percent correctly.


Munster broke this information down further, stating that Siri is good for "local" and "commerce" queries, like asking about nearby coffee shops or assisting in buying new shoes. In this area, Siri beats Alexa and Cortana but still falls behind Google Assistant on Google Home.

Despite the low percentage of correctly answering the 782 total queries asked of it, Munster said Siri's overall performance rose above expectations "given the limited scope of HomePod's music focus."

Chart via Loup Ventures

The researchers explained that over time HomePod and Siri should grow to match, or surpass, rival assistants by simply adding query domains like calendar, email, calling, and navigation.

Some domains like navigation, calendar, email, and calling are simply not supported. These questions were met with, “I can’t ___ on HomePod.” Also, in any case that iPhone-based Siri would bring up Google search results, HomePod would reply, “I can’t get the answer to that on HomePod,” which forces you to use your phone or give up on the question altogether. Removing navigation, calling, email, and calendar-related queries from our question set yields a 67% correct response, a jump from overall of 52.3% correct.

This means added support for these domains would bring HomePod performance above that of Alexa (64%) and Cortana (57%), though still shy of Google Home (81%). We know Siri has the ability to correctly answer a whole range of queries that HomePod cannot, evidenced by our note here. Apple’s limiting of HomePod’s domains should change over time, at which point we expect the speaker to be vastly more useful and integrated with your other Apple devices.

As discovered in the research, where HomePod excelled was its "superior" listening skills. The HomePod allows users to speak at a normal volume, even when music from the speaker is particularly loud, and Siri will pick up on the voice and hear the user. "This was HomePod's most stellar feature," according to Munster.

Loup Ventures also favored HomePod's sound quality, saying that "it sounds incredible." Like other reviews and impressions of the device, Munster's enjoyment of the HomePod's audio quality was countered with the speaker's lackluster Siri performance, which he thinks will be changing fairly soon.

Don’t be fooled by HomePod’s sound quality-focused first step into smart speakers; Apple has a grander vision than delivering a better sounding Echo. While not present in the first version of HomePod (i.e. you can’t even make a phone call with HomePod), we believe Apple’s goal is to make Siri a ubiquitous, ambient presence that connects and controls all your connected devices and services – and to make a leap forward in the transition to voice-first computing.

Visit Loup Ventures to read more of the information gathered from the HomePod "smart speaker gauntlet," including the publication's prediction for demand and market share of HomePod through the rest of 2018.

Related Roundup: HomePod
Buyer's Guide: HomePod (Neutral)

Top Rated Comments

(View all)
Avatar
30 months ago
Me: “Hey Siri, why are you so terrible at answering questions?”

Siri: “I found several articles on the web about crustaceans.”
Score: 132 Votes (Like | Disagree)
Avatar
30 months ago
That's pretty damn pathetic...
Score: 108 Votes (Like | Disagree)
Avatar
30 months ago
Siri is Apple's biggest failure. They have had years to make it at least 80% functional and have failed to do so. Just like the ATV, it seems to be just a "hobby" for Apple.
Score: 84 Votes (Like | Disagree)
Avatar
30 months ago
Siri sucks. I don’t think I’ve used it in over a year.
Score: 77 Votes (Like | Disagree)
Avatar
30 months ago
What good is "superior" listeng skills if it can't answer many of the questions?
Score: 75 Votes (Like | Disagree)
Avatar
30 months ago

I think Siri is more than adequate for what most need it for the HomePod. Is it better than Google Home? No. Does it need to be better than the competition? Not necessarily. I use Siri all the time and I really don’t have any general complaints, aside from not understanding some of the things I ask it. None of these voice assistants are perfect.

I can only imagine the amount of gloating on these forums if Siri was #1 on the list instead of Google.
Score: 38 Votes (Like | Disagree)

Top Stories

'This App is No Longer Shared' iOS Bug Preventing Some Apps From Opening

Friday May 22, 2020 3:58 pm PDT by
An app bug is causing some iOS users to be unable to open their apps, with affected iPhone and iPad users seeing the message "This app is no longer shared with you" when attempting to access an app. There are multiple complaints about the issue on the MacRumors forums and on Twitter from users who are running into problems. A MacRumors reader describes the issue:Is anyone else experiencing...

Apple Memorial Day Deals: Shop the Best Apple Accessory Sales From Twelve South, eBay, Anker, Mophie, and More

Friday May 22, 2020 6:39 am PDT by
We're now just a few days away from Memorial Day on Monday, May 25, and numerous retailers have opened up discounts in celebration of the holiday. This includes sales on helpful Apple-related accessories like Anker's portable batteries, Beats headphones at eBay, Incase and Incipio's protective iPad and iPhone cases, Mophie's iPhone battery cases, JBL's Bluetooth speakers, and much more. Note:...

Former iOS Chief Scott Forstall Shares Intriguing Story of His Interview With Steve Jobs at NeXT

Friday May 22, 2020 4:01 am PDT by
Former Apple executive and iOS chief Scott Forstall made a rare public appearance this week at Code.org's virtual Code Break event, and in between classes, Forstall shared the intriguing story of how he was hired by Steve Jobs. Forstall revealed that he had been considering working at Microsoft when he went to interview at NexT, the company started by Jobs after he had left Apple. Forstall...

'Apple Glass' Rumored to Start at $499, Support Prescription Lenses, and More

Tuesday May 19, 2020 6:30 am PDT by
Front Page Tech host and leaker Jon Prosser today shared several alleged details about Apple's rumored augmented reality glasses, including an "Apple Glass" marketing name, $499 starting price, prescription lens option, and more. The marketing name will be "Apple Glass" The glasses will start at $499 with the option for prescription lenses at an extra cost There will be displays in both...

Apple's 'Bounce' AirPods Ad Wins 'Best of Advertising' Award

Friday May 22, 2020 10:09 am PDT by
Apple's creative "Bounce" ad designed to highlight the AirPods took top honors in the 99th annual ADC (Art Director's Club) awards for advertising, earning the "Best of Discipline" award along with two Gold Cube awards in the craft in video and branded content categories. Released in June 2019, the ad features a bored man who pulls his AirPods off of their wireless charging pad and then pops ...

Apple's 'AirPods Studio' Over-Ear Headphones Have Reportedly Kicked Off Production

Friday May 22, 2020 7:03 am PDT by
We've been hearing quite a bit recently about Apple's long-rumored over-ear headphones, said to be called "AirPods Studio," and it looks like a launch may be coming in the relatively near future. Artist mockup based on Beats Studio3 Rumors have generally suggested a summer or fall launch for AirPods Studio, with a report earlier this week claiming that suppliers in Vietnam will begin...

Top Stories: Apple Glass and iPhone 12 Rumors, iOS 13.5 Update, and More!

Saturday May 23, 2020 6:00 am PDT by
It was another big week for rumors this week, with a flurry of reports about Apple's augmented reality glasses, the iPhone 12, and Apple's "AirPods Studio" over-ear headphones. This week also saw the release of iOS 13.5, bringing a number of health-related updates to Apple's mobile devices. Subscribe to the MacRumors YouTube channel for more videos. Other topics of interest this week included ...

Apple Releases iPadOS and iOS 13.5 With Exposure Notification API, Face ID Mask Updates, Group FaceTime Changes and More

Wednesday May 20, 2020 10:00 am PDT by
Apple today released iOS and iPadOS 13.5, major updates that come more than a month after the launch of iOS and iPadOS 13.4.1. iOS 13.5 is a major health-related update that brings many features related to the ongoing public health crisis. The iOS and ‌‌iPadOS‌‌ 13.5 updates are available on all eligible devices over-the-air in the Settings app. To access the updates, go to Settings...

Jon Prosser Claims Apple is Working on 'Steve Jobs Heritage Edition' AR Glasses, Gurman Calls Rumor 'Complete Fiction'

Thursday May 21, 2020 4:50 pm PDT by
Apple is working on a limited-edition version of its augmented reality smart glasses that's designed to look like the round, frameless glasses that Steve Jobs was famous for wearing, according to Jon Prosser. Prosser, who runs YouTube show Front Page Tech and who has been sharing a flood of Apple rumors in recent weeks, mentioned the detail in Cult of Mac's latest Cultcast podcast....

T-Mobile and Sprint Offering Free iPhone SE With Trade-In

Thursday May 21, 2020 1:14 pm PDT by
T-Mobile is launching a Memorial Day promotion that will see the company offering a free iPhone SE to customers who trade in an eligible older smartphone in good condition. From Friday to Monday, customers who trade in an existing smartphone can get a free iPhone SE (sales tax still needs to be paid) or up to $500 off a Samsung Galaxy S20. The free iPhone SE will be provided in the form...