Siri Shows Increasing Promise, But Accuracy Falls Short of Google Text Searches

In what should not be a surprise given Siri's beta status and its use of new technology for voice interaction with mobile devices, a study from Piper Jaffray analyst Gene Munster finds that Siri's abilities are still eclipsed by the traditional text-entry Google searches that have been in place for years.

The study included presenting Siri with 800 queries in each of two environments, a quiet room and a busy urban street, and then assessing Siri's ability to correctly interpret the queries and provide correct responses. Specifically addressing the outdoor testing, Munster notes:

While Google comprehends 100% of searches, Siri was only able to comprehend 83% of the prompted questions on a busy Minneapolis street. In terms of answer accuracy, Siri correctly answered 62% of queries on the street compared to Google's estimated 86% answer accuracy. Based on these results, we give Google an A+ for comprehension and a B+ for accuracy while we give Siri a B for comprehension and a D for accuracy.

But while Siri has some room for improvement, Munster acknowledges that the technology is only in its infancy and he remains "optimistic about its future" with an eye toward iOS 6 as bringing substantial improvements.

siri examples
Piper Jaffray's testing was conducted using the iPhone's built-in microphone in the quiet setting and Skullcandy microphone headphones for the outdoor setting, with Siri's ability to correctly comprehend queries falling from 89% to 83% when moving outdoors to the noisy urban setting. On the accuracy side, Google's 86% rating is derived from comScore data showing that Google generates 1.14 search result pages per search, suggesting that roughly 86% of time Google presents the data the user is looking for within the first few results.

Piper Jaffray also analyzed the sources for Siri's results, finding that Google provided 60% of the answers, with Yelp and other sources filling in the remainder.

Of queries excluding commands (i.e. call Jay, send text) Google would provide 60% of the answers, Yelp 20%, WolframAlpha 14%, Yahoo 4%, and Wikipedia 2%. Breaking down Siri's reliance further, Google provides 100% of navigation results, 61% of information results, 48% of commerce results and 42% of local results. Among other result aggregators, Yelp provided the most local results (51%) and commerce results (51%), while WolframAlpha provided 34% of information results.

Munster estimates that Google's share of Siri results will drop to 48% with the launch of iOS 6 as navigation, sports, and movie queries shift over to Apple and its new partners.

Top Rated Comments

HarryKeogh Avatar
110 months ago
I'm using Siri to post this message and, as you can see, it's 100% a cure writ and pancakes urethra.
Score: 115 Votes (Like | Disagree)
basesloaded190 Avatar
110 months ago

Siri is not on the same level as Google voice. Just saying.


What Google showed yesterday with offline support was very impressive, I agree.
Score: 18 Votes (Like | Disagree)
Ciclismo Avatar
110 months ago

I'm using Siri to post this message and, as you can see, it's 100% a cure writ and pancakes urethra.


Nominated for "Post of the Month" award.
Score: 15 Votes (Like | Disagree)
Bezetos Avatar
110 months ago


Google is just a dumb engine with no possible applications other than being a directory of the internet. They have never developed beyond coder style searches for specifics and their engineer backgrounds suggest they never will.

I think you've been living under a rock if you seriously believe in this...
Score: 13 Votes (Like | Disagree)
rockland Avatar
110 months ago
Siri is substandard

To me, there's no two ways about it. Uncharacteristically, Apple releases a "beta" product, repeatedly invoking that fact as if it were an excuse.

Which would be fine, if they charged a "beta-worthy" price for iPhone 4S.

Sorry, but it's been well over half a year, and Siri STILL underperforms. Badly.

In my experience and in all kinds of situations and environments, Siri works less than 50% of the time. More like a third of the time. I've pretty much given up on it.

We all love Apple, but to me it's hard not to state the obvious. Siri is (so far) a flop, in terms of living up to its profusely advertised promise. It is not even close to the usual high standards of Apple/SJ. And if he were alive now, I think he would be blowing a gasket about it.
Score: 13 Votes (Like | Disagree)
SleeplessChaos Avatar
110 months ago

Fine, if you want to embarrass yourself...

On the accuracy side, Google's 86% rating is derived from comScore data showing that Google generates 1.14 search result pages per search, suggesting that roughly 86% of time Google presents the data the user is looking for within the first few results.

So this the point. Will you get information quicker using Siri or googling? So you've completely missed the point here.

It's not comparing two things at once. It is comparing how accurately you can get information using both tools. That's just one thing.

That would be biased. Secondly, read again: Google generates 1.14 search result pages per search, suggesting that roughly 86% of time Google presents the data the user is looking for within the first few results. So if you ask Siri "how tall is the empire state building", you'll get a precise answer from Siri (hopefuly) and find that information on the first page of Google results (actually, if you google that you'll now get an answer too). So this is what they're comparing. So again, you're completely missing the point.

And again, just go and read the article.

----------


Another person who missed the whole point of this research...

I'm wondering if we should check your reading comprehension guys...

I don't think he is missing the point of the experiment, I think that he, like me, thinks the point is stupid.

Think about it for more than a millisecond and you'll realize that it's comparing apples and oranges. Google, like every other text based search engine on the web, is simply matching text keywords with a large indexed database built by years of crawling through the web. The search is basically brute force and simple.

Siri is receiving a digital waveform from the microphone, eliminating frequencies from that waveform that are not commonly found in the human voice, translating those remaining frequencies into readable text via a remote server and then interpreting that into a command which it then uses to either search the web using what it assumes to be the correct web search service or perform a function using a local app. That service list includes google search.

The fact of the matter is that the two services are not comparable whatsoever and that for all intents and purposes siri expands upon the abilities of google regardless of the situation.
Score: 8 Votes (Like | Disagree)

Top Stories

0 Deals Hero

Black Friday 2020: Best Apple Deals to Plan For

Saturday November 21, 2020 10:00 am PST by
In the lead-up to Black Friday next week, we've been putting a spotlight on the best deals coming from various retailers like Best Buy and Walmart. In an effort to further prepare our readers for the best Black Friday deals, we're breaking down what we think should be on your radar for Black Friday in 2020. Note: MacRumors is an affiliate partner with some of these vendors. When you click a...
m1 mac mini vignette

Apple Lists M1-Based Mac Mini Logic Boards With 10 Gigabit Ethernet in Internal Parts Ordering System

Friday November 20, 2020 9:32 am PST by
While the new Mac mini with the M1 chip is only available with Gigabit Ethernet, Apple has listed multiple M1-based Mac mini logic boards with 10 Gigabit Ethernet in an internal parts list for Apple Authorized Service Providers. For every Mac mini logic board with Gigabit Ethernet in the parts list, obtained by MacRumors, there is a corresponding logic board with 10 Gigabit Ethernet:...
new m1 chip

Craig Federighi: Native Windows on M1 Macs is 'Really up to Microsoft'

Friday November 20, 2020 11:57 am PST by
Following the release of the M1 Macs Apple executives have been doing interviews with a range of publications, and today, Ars Technica published another interview with software engineering chief Craig Federighi, hardware technologies lead Johny Srouji, and marketing VP Greg Joswiak. Much of the interview focuses on topics that the three have already covered in prior discussions, but there is ...
14

Apple Releases iOS 14.2.1 With Fix for Text Message Bug and iPhone 12 Mini Lock Screen Issues

Thursday November 19, 2020 10:16 am PST by
Apple today released iOS 14.2.1, a bug fix update that comes two weeks after the launch of iOS 14.2 and is available for Apple's new iPhone 12 models. The iOS 14.2.1 update can be downloaded for free and it is available on all eligible devices over-the-air in the Settings app. To access the new software, go to Settings > General > Software Update. According to Apple's release notes, iOS...
Walmart November Deals Hero

Black Friday Spotlight: Walmart Will Have AirPods Pro Down to Lowest Price of $169, and More Apple Deals

Thursday November 19, 2020 8:05 am PST by
We've been tracking early Black Friday deals in our dedicated Black Friday Roundup, and in an effort to prepare our readers for the big shopping event we're highlighting sales store-by-store in the lead-up to November 27. Note: MacRumors is an affiliate partner with Walmart. When you click a link and make a purchase, we may receive a small payment, which helps us keep the site running. Next ...
apple leather sleeve

Leather Sleeve for iPhone 12 Models Now Available From Apple

Friday November 20, 2020 12:16 pm PST by
Apple today began selling the Leather Sleeve for the new iPhone 12 models, with the accessory having first been announced alongside the updated iPhones in October. Priced at $129, the Leather Sleeve is not a case and is designed to be removed when the iPhone is in use. It features a cutout at the front that displays the time, and it comes with a matching leather strap. According to Apple, it ...
iOS14AntitrackFacebookSadfeature

Apple Confirms Commitment to App Tracking Transparency in Letter Condemning Facebook's Data Collection [Updated]

Thursday November 19, 2020 11:58 am PST by
Apple in iOS 14 is planning to introduce a new App Tracking Transparency feature that will let users know when companies want to track them across apps and website. Following outcry from developers like Facebook and ad networks unprepared for the change, Apple delayed the implementation of the anti-tracking functionality until early 2021. Eight civil society organizations recently sent a...
maxresdefault

CrossOver Allows x86 Windows Apps to Run on Apple M1 Macs

Wednesday November 18, 2020 6:07 pm PST by
Codeweavers posted a blog post and video tonight showing off CrossOver running on an Apple M1 MacBook Air. This video shows Team Fortress 2 running on a new M1 MacBook Air: CrossOver is software (based on Wine Project) that runs Microsoft Windows apps on the Mac by translating Windows APIs into their Mac equivalents. The Codeweavers team was able to run the current version of CrossOver on...
macbookpro13large

Apple Offers Instructions on What to Do if macOS Big Sur Causes Installation Errors on 2013 and 2014 MacBook Pro

Thursday November 19, 2020 6:12 pm PST by
Following the release of macOS Big Sur last week, a number of 2013 and 2014 MacBook Pro owners found that the update bricked their machines. Affected users saw their Macs get stuck displaying a black screen after attempting to install the new software. Apple has now addressed this issue in a new support document that provides instructions on what to do if macOS Big Sur can't be installed on...
128gb m1 macbook air education cropped

$799 M1 MacBook Air With 128GB Storage for Education Institutions Spotted Online

Friday November 20, 2020 5:15 am PST by
A new configuration of the M1 MacBook Air with 128GB of storage and a lower $799 price has today been spotted on Apple's U.S. Education Institution Hardware and Software Price List. The M1 MacBook Air is only available with 256GB, 512GB, 1TB, or 2TB of storage. There is currently no 128GB configuration on the Apple Store. However, Reddit user "u/dduci97" noticed that Apple has listed...