Siri Shows Increasing Promise, But Accuracy Falls Short of Google Text Searches

In what should not be a surprise given Siri's beta status and its use of new technology for voice interaction with mobile devices, a study from Piper Jaffray analyst Gene Munster finds that Siri's abilities are still eclipsed by the traditional text-entry Google searches that have been in place for years.

The study included presenting Siri with 800 queries in each of two environments, a quiet room and a busy urban street, and then assessing Siri's ability to correctly interpret the queries and provide correct responses. Specifically addressing the outdoor testing, Munster notes:

While Google comprehends 100% of searches, Siri was only able to comprehend 83% of the prompted questions on a busy Minneapolis street. In terms of answer accuracy, Siri correctly answered 62% of queries on the street compared to Google's estimated 86% answer accuracy. Based on these results, we give Google an A+ for comprehension and a B+ for accuracy while we give Siri a B for comprehension and a D for accuracy.

But while Siri has some room for improvement, Munster acknowledges that the technology is only in its infancy and he remains "optimistic about its future" with an eye toward iOS 6 as bringing substantial improvements.

siri examples
Piper Jaffray's testing was conducted using the iPhone's built-in microphone in the quiet setting and Skullcandy microphone headphones for the outdoor setting, with Siri's ability to correctly comprehend queries falling from 89% to 83% when moving outdoors to the noisy urban setting. On the accuracy side, Google's 86% rating is derived from comScore data showing that Google generates 1.14 search result pages per search, suggesting that roughly 86% of time Google presents the data the user is looking for within the first few results.

Piper Jaffray also analyzed the sources for Siri's results, finding that Google provided 60% of the answers, with Yelp and other sources filling in the remainder.

Of queries excluding commands (i.e. call Jay, send text) Google would provide 60% of the answers, Yelp 20%, WolframAlpha 14%, Yahoo 4%, and Wikipedia 2%. Breaking down Siri's reliance further, Google provides 100% of navigation results, 61% of information results, 48% of commerce results and 42% of local results. Among other result aggregators, Yelp provided the most local results (51%) and commerce results (51%), while WolframAlpha provided 34% of information results.

Munster estimates that Google's share of Siri results will drop to 48% with the launch of iOS 6 as navigation, sports, and movie queries shift over to Apple and its new partners.

Top Rated Comments

HarryKeogh Avatar
153 months ago
I'm using Siri to post this message and, as you can see, it's 100% a cure writ and pancakes urethra.
Score: 115 Votes (Like | Disagree)
basesloaded190 Avatar
153 months ago
Siri is not on the same level as Google voice. Just saying.

What Google showed yesterday with offline support was very impressive, I agree.
Score: 18 Votes (Like | Disagree)
Ciclismo Avatar
153 months ago
I'm using Siri to post this message and, as you can see, it's 100% a cure writ and pancakes urethra.

Nominated for "Post of the Month" award.
Score: 15 Votes (Like | Disagree)
Bezetos Avatar
153 months ago

Google is just a dumb engine with no possible applications other than being a directory of the internet. They have never developed beyond coder style searches for specifics and their engineer backgrounds suggest they never will.
I think you've been living under a rock if you seriously believe in this...
Score: 13 Votes (Like | Disagree)
rockland Avatar
153 months ago
Siri is substandard

To me, there's no two ways about it. Uncharacteristically, Apple releases a "beta" product, repeatedly invoking that fact as if it were an excuse.

Which would be fine, if they charged a "beta-worthy" price for iPhone 4S.

Sorry, but it's been well over half a year, and Siri STILL underperforms. Badly.

In my experience and in all kinds of situations and environments, Siri works less than 50% of the time. More like a third of the time. I've pretty much given up on it.

We all love Apple, but to me it's hard not to state the obvious. Siri is (so far) a flop, in terms of living up to its profusely advertised promise. It is not even close to the usual high standards of Apple/SJ. And if he were alive now, I think he would be blowing a gasket about it.
Score: 13 Votes (Like | Disagree)
SleeplessChaos Avatar
153 months ago
Fine, if you want to embarrass yourself...

On the accuracy side, Google's 86% rating is derived from comScore data showing that Google generates 1.14 search result pages per search, suggesting that roughly 86% of time Google presents the data the user is looking for within the first few results.

So this the point. Will you get information quicker using Siri or googling? So you've completely missed the point here.

It's not comparing two things at once. It is comparing how accurately you can get information using both tools. That's just one thing.

That would be biased. Secondly, read again: Google generates 1.14 search result pages per search, suggesting that roughly 86% of time Google presents the data the user is looking for within the first few results. So if you ask Siri "how tall is the empire state building", you'll get a precise answer from Siri (hopefuly) and find that information on the first page of Google results (actually, if you google that you'll now get an answer too). So this is what they're comparing. So again, you're completely missing the point.

And again, just go and read the article.

----------


Another person who missed the whole point of this research...

I'm wondering if we should check your reading comprehension guys...
I don't think he is missing the point of the experiment, I think that he, like me, thinks the point is stupid.

Think about it for more than a millisecond and you'll realize that it's comparing apples and oranges. Google, like every other text based search engine on the web, is simply matching text keywords with a large indexed database built by years of crawling through the web. The search is basically brute force and simple.

Siri is receiving a digital waveform from the microphone, eliminating frequencies from that waveform that are not commonly found in the human voice, translating those remaining frequencies into readable text via a remote server and then interpreting that into a command which it then uses to either search the web using what it assumes to be the correct web search service or perform a function using a local app. That service list includes google search.

The fact of the matter is that the two services are not comparable whatsoever and that for all intents and purposes siri expands upon the abilities of google regardless of the situation.
Score: 8 Votes (Like | Disagree)

Popular Stories

reset password request iphone

Warning: Apple Users Targeted in Phishing Attack Involving Rapid Password Reset Requests

Tuesday March 26, 2024 4:34 pm PDT by
Phishing attacks taking advantage of Apple's password reset feature have become increasingly common, according to a report from KrebsOnSecurity. Multiple Apple users have been targeted in an attack that bombards them with an endless stream of notifications or multi-factor authentication (MFA) messages in an attempt to cause panic so they'll respond favorably to social engineering. An...
iPhone Home Screen Gradient Blank Spaces 1

Sources: iOS 18 Lets Apps Be Placed Anywhere on Home Screen Grid

Sunday March 24, 2024 1:33 pm PDT by
iOS 18 will give iPhone users greater control over Home Screen app icon arrangement, according to sources familiar with the matter. While app icons will likely remain locked to an invisible grid system on the Home Screen, to ensure there is some uniformity, our sources say that users will be able to arrange icons more freely on iOS 18. For example, we expect that the update will introduce...
maxresdefault

Apple Announces WWDC 2024 Event for June 10 to 14

Tuesday March 26, 2024 10:02 am PDT by
Apple today announced that its 35th annual Worldwide Developers Conference is set to take place from Monday, June 10 to Friday, June 14. As with WWDC events since 2020, WWDC 2024 will be an online event that is open to all developers at no cost. Subscribe to the MacRumors YouTube channel for more videos. WWDC 2024 will include online sessions and labs so that developers can learn about new...
iPad Pro 2024 Landscape Camera Feature

New iPad Pro Again Rumored to Feature Landscape Front-Facing Camera

Monday March 25, 2024 5:43 am PDT by
The next-generation iPad Pro will feature a landscape-oriented front-facing camera for the first time, according to the Apple leaker known as "Instant Digital." Instant Digital reiterated the design change earlier today on Weibo with a simple accompanying 2D image. The post reveals that the entire TrueDepth camera array will move to the right side of the device, while the microphone will...
apple maps 3d feature

Apple Maps May Gain Custom Routes With iOS 18

Tuesday March 26, 2024 3:10 pm PDT by
Apple may be planning to add support for "custom routes" in Apple Maps in iOS 18, according to code reviewed by MacRumors. Apple Maps does not currently offer a way to input self-selected routes, with Maps users limited to Apple's pre-selected options, but that may change in iOS 18. Apple has pushed an iOS 18 file to its maps backend labeled "CustomRouteCreation." While not much is revealed...
sonoma desktop wwdc

Apple Releases macOS Sonoma 14.4.1 With Fix for USB Hub Bug

Monday March 25, 2024 10:10 am PDT by
Apple today released macOS Sonoma 14.4.1, a minor update for the macOS Sonoma operating system that launched last September. macOS Sonoma 14.4.1 comes three weeks after macOS Sonoma 14.4. The ‌‌‌‌macOS Sonoma‌‌ 14.4‌.1 update can be downloaded for free on all eligible Macs using the Software Update section of System Settings. There's also a macOS 13.6.6 release for those who...
Generic iOS 18 Feature Purple

iOS 18 Will Finally Bring This Android Feature to iPhone

Monday March 25, 2024 6:42 am PDT by
iOS 18 will allow iPhone users to place app icons anywhere on the Home Screen grid, according to sources familiar with development of the software update. This basic feature has long been available on Android smartphones. While app icons will likely remain locked to an invisible grid system on the Home Screen, our sources said that users will be able to arrange icons more freely on iOS 18....
Generic iOS 18 Feature Purple

iOS 18: What to Expect From 'Biggest' Update in iPhone's History

Wednesday March 27, 2024 11:10 am PDT by
At least some Apple software engineers continue to believe that iOS 18 will be the "biggest" update in the iPhone's history, according to Bloomberg's Mark Gurman. Below, we recap rumored features and changes for the iPhone. "The iOS 18 update is expected to be the most ambitious overhaul of the iPhone's software in its history, according to people working on the upgrade," wrote Gurman, in a r...