Got a tip for us? Share it...

New in OS X: Get MacRumors Push Notifications on your Mac

Resubscribe Now Close

Siri Shows Increasing Promise, But Accuracy Falls Short of Google Text Searches

In what should not be a surprise given Siri's beta status and its use of new technology for voice interaction with mobile devices, a study from Piper Jaffray analyst Gene Munster finds that Siri's abilities are still eclipsed by the traditional text-entry Google searches that have been in place for years.

The study included presenting Siri with 800 queries in each of two environments, a quiet room and a busy urban street, and then assessing Siri's ability to correctly interpret the queries and provide correct responses. Specifically addressing the outdoor testing, Munster notes:
While Google comprehends 100% of searches, Siri was only able to comprehend 83% of the prompted questions on a busy Minneapolis street. In terms of answer accuracy, Siri correctly answered 62% of queries on the street compared to Google's estimated 86% answer accuracy. Based on these results, we give Google an A+ for comprehension and a B+ for accuracy while we give Siri a B for comprehension and a D for accuracy.
But while Siri has some room for improvement, Munster acknowledges that the technology is only in its infancy and he remains "optimistic about its future" with an eye toward iOS 6 as bringing substantial improvements.


Piper Jaffray's testing was conducted using the iPhone's built-in microphone in the quiet setting and Skullcandy microphone headphones for the outdoor setting, with Siri's ability to correctly comprehend queries falling from 89% to 83% when moving outdoors to the noisy urban setting. On the accuracy side, Google's 86% rating is derived from comScore data showing that Google generates 1.14 search result pages per search, suggesting that roughly 86% of time Google presents the data the user is looking for within the first few results.

Piper Jaffray also analyzed the sources for Siri's results, finding that Google provided 60% of the answers, with Yelp and other sources filling in the remainder.
Of queries excluding commands (i.e. call Jay, send text) Google would provide 60% of the answers, Yelp 20%, WolframAlpha 14%, Yahoo 4%, and Wikipedia 2%. Breaking down Siri's reliance further, Google provides 100% of navigation results, 61% of information results, 48% of commerce results and 42% of local results. Among other result aggregators, Yelp provided the most local results (51%) and commerce results (51%), while WolframAlpha provided 34% of information results.
Munster estimates that Google's share of Siri results will drop to 48% with the launch of iOS 6 as navigation, sports, and movie queries shift over to Apple and its new partners.

Top Rated Comments

(View all)

27 months ago
I'm using Siri to post this message and, as you can see, it's 100% a cure writ and pancakes urethra.
Rating: 115 Votes
27 months ago
To me, there's no two ways about it. Uncharacteristically, Apple releases a "beta" product, repeatedly invoking that fact as if it were an excuse.

Which would be fine, if they charged a "beta-worthy" price for iPhone 4S.

Sorry, but it's been well over half a year, and Siri STILL underperforms. Badly.

In my experience and in all kinds of situations and environments, Siri works less than 50% of the time. More like a third of the time. I've pretty much given up on it.

We all love Apple, but to me it's hard not to state the obvious. Siri is (so far) a flop, in terms of living up to its profusely advertised promise. It is not even close to the usual high standards of Apple/SJ. And if he were alive now, I think he would be blowing a gasket about it.
Rating: 27 Votes
27 months ago

Siri is not on the same level as Google voice. Just saying.


What Google showed yesterday with offline support was very impressive, I agree.
Rating: 22 Votes
27 months ago


Google is just a dumb engine with no possible applications other than being a directory of the internet. They have never developed beyond coder style searches for specifics and their engineer backgrounds suggest they never will.

I think you've been living under a rock if you seriously believe in this...
Rating: 19 Votes
27 months ago
Siri is not on the same level as Google voice. Just saying.
Rating: 18 Votes
27 months ago

I'm using Siri to post this message and, as you can see, it's 100% a cure writ and pancakes urethra.


Nominated for "Post of the Month" award.
Rating: 15 Votes
27 months ago
Kids these days are too Ghetto for Siri. They don't realise you may have to, in life, use the "T" at the end of sentences etc.

I'm by no means posh but Siri understands me quite well. Obviously not so much if you're not a slack jawed idiot.
Rating: 12 Votes
27 months ago

This is meaningless without testing multiple accents in all languages and having questions posed in different ways!

Pointless to the extreme is what comes to mind.

Siri is more of an experiment than a real feature, it's the interpretation of the sentance that is the exciting part of siri and it's ability to follow a thread.

Google is just a dumb engine with no possible applications other than being a directory of the internet. They have never developed beyond coder style searches for specifics and their engineer backgrounds suggest they never will.


lol you're such a hater man. I hope Apple is paying you.
Rating: 12 Votes
27 months ago
Many people have bashed siri for not being accurate/ helpful. Honestly when siri came out I never saw myself using it a whole lot. But now, I use it regularly on a daily basis. It has come a long way even though it was just released a couple months ago.
Rating: 11 Votes
27 months ago
Munster's assessment compares two completely different resources, a search engine and a personal assistant.

It compares them only on search (since that is all that Google does).

Siri does search, but of a limited sort (e.g., "what is that plane overhead?" "NCA178 Boeing 747-400 at 18,000 feet."). Google does not do that. Nor does it set alarms or remind you to get gas when you leave work.

If you want Siri to do a web search, you can just ask for it.
"Google what team does Peyton Manning play for?"
"Search the Web for what spices are in Lasagna?"
"Yahoo when is the next Haley's comment?"*
* The comment: 'Haven't we already seen that taily thing?"

There is no learning curve.
Rating: 9 Votes

[ Read All Comments ]