Apple is separating the new smartphones into its usual low-cost versus high-cost categories, with big differences between the two models coming down to the camera, display, and battery life.
iOS 13 & watchOS 6 Now Available
Siri on HomePod Correctly Answered 52.3% of Queries in New AI Test
In a new test shared today by Loup Ventures, Apple's HomePod was put through its paces in categories including Siri, sound quality, and ease of use. For Siri, Loup Ventures' Gene Munster reported that while the AI assistant understood 99.4 percent of queries asked of it, it answered only 52.3 percent of them correctly. Loup Ventures tested three separate HomePods and asked 782 queries total.
Compared to previous tests of rival speakers, HomePod is "at the bottom of the totem pole" in the AI assistant performance category. Google Home answered 81 percent correctly, Amazon's Alexa answered 64 percent correctly, and Microsoft's Cortana answered 57 percent correctly.
Munster broke this information down further, stating that Siri is good for "local" and "commerce" queries, like asking about nearby coffee shops or assisting in buying new shoes. In this area, Siri beats Alexa and Cortana but still falls behind Google Assistant on Google Home.
Despite the low percentage of correctly answering the 782 total queries asked of it, Munster said Siri's overall performance rose above expectations "given the limited scope of HomePod's music focus."
Chart via Loup Ventures
The researchers explained that over time HomePod and Siri should grow to match, or surpass, rival assistants by simply adding query domains like calendar, email, calling, and navigation.
Loup Ventures also favored HomePod's sound quality, saying that "it sounds incredible." Like other reviews and impressions of the device, Munster's enjoyment of the HomePod's audio quality was countered with the speaker's lackluster Siri performance, which he thinks will be changing fairly soon.
Compared to previous tests of rival speakers, HomePod is "at the bottom of the totem pole" in the AI assistant performance category. Google Home answered 81 percent correctly, Amazon's Alexa answered 64 percent correctly, and Microsoft's Cortana answered 57 percent correctly.
Munster broke this information down further, stating that Siri is good for "local" and "commerce" queries, like asking about nearby coffee shops or assisting in buying new shoes. In this area, Siri beats Alexa and Cortana but still falls behind Google Assistant on Google Home.
Despite the low percentage of correctly answering the 782 total queries asked of it, Munster said Siri's overall performance rose above expectations "given the limited scope of HomePod's music focus."
The researchers explained that over time HomePod and Siri should grow to match, or surpass, rival assistants by simply adding query domains like calendar, email, calling, and navigation.
Some domains like navigation, calendar, email, and calling are simply not supported. These questions were met with, “I can’t ___ on HomePod.” Also, in any case that iPhone-based Siri would bring up Google search results, HomePod would reply, “I can’t get the answer to that on HomePod,” which forces you to use your phone or give up on the question altogether. Removing navigation, calling, email, and calendar-related queries from our question set yields a 67% correct response, a jump from overall of 52.3% correct.As discovered in the research, where HomePod excelled was its "superior" listening skills. The HomePod allows users to speak at a normal volume, even when music from the speaker is particularly loud, and Siri will pick up on the voice and hear the user. "This was HomePod's most stellar feature," according to Munster.
This means added support for these domains would bring HomePod performance above that of Alexa (64%) and Cortana (57%), though still shy of Google Home (81%). We know Siri has the ability to correctly answer a whole range of queries that HomePod cannot, evidenced by our note here. Apple’s limiting of HomePod’s domains should change over time, at which point we expect the speaker to be vastly more useful and integrated with your other Apple devices.
Loup Ventures also favored HomePod's sound quality, saying that "it sounds incredible." Like other reviews and impressions of the device, Munster's enjoyment of the HomePod's audio quality was countered with the speaker's lackluster Siri performance, which he thinks will be changing fairly soon.
Don’t be fooled by HomePod’s sound quality-focused first step into smart speakers; Apple has a grander vision than delivering a better sounding Echo. While not present in the first version of HomePod (i.e. you can’t even make a phone call with HomePod), we believe Apple’s goal is to make Siri a ubiquitous, ambient presence that connects and controls all your connected devices and services – and to make a leap forward in the transition to voice-first computing.Visit Loup Ventures to read more of the information gathered from the HomePod "smart speaker gauntlet," including the publication's prediction for demand and market share of HomePod through the rest of 2018.
Related Roundup: HomePod
Buyer's Guide: HomePod (Neutral)
[ 544 comments ]
Top Rated Comments
(View all)
21 months ago
Me: “Hey Siri, why are you so terrible at answering questions?”
Siri: “I found several articles on the web about crustaceans.”
Siri: “I found several articles on the web about crustaceans.”
21 months ago
Siri is Apple's biggest failure. They have had years to make it at least 80% functional and have failed to do so. Just like the ATV, it seems to be just a "hobby" for Apple.
21 months ago
What good is "superior" listeng skills if it can't answer many of the questions?
21 months ago
I think Siri is more than adequate for what most need it for the HomePod. Is it better than Google Home? No. Does it need to be better than the competition? Not necessarily. I use Siri all the time and I really don’t have any general complaints, aside from not understanding some of the things I ask it. None of these voice assistants are perfect.
I can only imagine the amount of gloating on these forums if Siri was #1 on the list instead of Google.
21 months ago
So long as it can play/stop/skip music that's the end of the smart technology I would want. I don't care that others answer questions better, I wouldn't use it for that.
21 months ago
That’s what we get for Apple keeping our information private. I feel like it’s a very small price to pay and I am not surprised at all that Google leads the way here. Their sole focus as a company is mining your data for advertising. Ultimately, it doesn’t affect me as I don’t use voice assistants this way. Smart home stuff and music is really it. I’m sure the HomePod works well for these.
21 months ago
So long as it can play/stop/skip music that's the end of the smart technology I would want. I don't care that others answer questions better, I wouldn't use it for that.
You say that, but I guarantee that if Siri was better you would be using it for more than the basics.
[ Read All Comments ]
Burst Mode refers to when the camera on your iPhone captures a series of photos in rapid succession, at a rate of ten frames per second. It's a great way to shoot an action scene or an unexpected...
Apple's Portrait Mode has become a popular way of taking impressive shots using a depth-of-field effect known as bokeh, allowing iPhone users to shoot a photo that keeps the subject sharp with a...
Apple has redesigned the native Camera app on the iPhone 11 and iPhone 11 Pro to make space for the various additional shooting options available in its new flagship phones, and one particularly...
Apple's massive launch week has come to a close, with the iPhone 11, iPhone 11 Pro, iPhone 11 Pro Max, and Apple Watch Series 5 all debuting on Friday. And as if that wasn't enough, Thursday...
With the arrival of the iPhone 11, iPhone 11 Pro and iPhone 11 Pro Max, Apple has introduced an optional new camera feature in iOS 13 that's designed to allow you to correct the alignment of...
For this week's giveaway, we've teamed up with AnyTrans to offer MacRumors readers a chance to win one of Apple's new iPhone 11 Pro Max devices and a lifetime AnyTrans license.
For...
Due to Apple seeding the iOS 13.1 beta prior to the public release of iOS 13, upgrading to a new iPhone may be a bit trickier this year.
In a support document, Apple has provided instructions...
Apple today the golden master version of an upcoming tvOS 13 update to developers, one week after seeding the eleventh beta and more than three months after unveiling the tvOS 13 software at the...
