Siri Shows Increasing Promise, But Accuracy Falls Short of Google Text Searches

In what should not be a surprise given Siri's beta status and its use of new technology for voice interaction with mobile devices, a study from Piper Jaffray analyst Gene Munster finds that Siri's abilities are still eclipsed by the traditional text-entry Google searches that have been in place for years.

The study included presenting Siri with 800 queries in each of two environments, a quiet room and a busy urban street, and then assessing Siri's ability to correctly interpret the queries and provide correct responses. Specifically addressing the outdoor testing, Munster notes:

While Google comprehends 100% of searches, Siri was only able to comprehend 83% of the prompted questions on a busy Minneapolis street. In terms of answer accuracy, Siri correctly answered 62% of queries on the street compared to Google's estimated 86% answer accuracy. Based on these results, we give Google an A+ for comprehension and a B+ for accuracy while we give Siri a B for comprehension and a D for accuracy.

But while Siri has some room for improvement, Munster acknowledges that the technology is only in its infancy and he remains "optimistic about its future" with an eye toward iOS 6 as bringing substantial improvements.

siri examples
Piper Jaffray's testing was conducted using the iPhone's built-in microphone in the quiet setting and Skullcandy microphone headphones for the outdoor setting, with Siri's ability to correctly comprehend queries falling from 89% to 83% when moving outdoors to the noisy urban setting. On the accuracy side, Google's 86% rating is derived from comScore data showing that Google generates 1.14 search result pages per search, suggesting that roughly 86% of time Google presents the data the user is looking for within the first few results.

Piper Jaffray also analyzed the sources for Siri's results, finding that Google provided 60% of the answers, with Yelp and other sources filling in the remainder.

Of queries excluding commands (i.e. call Jay, send text) Google would provide 60% of the answers, Yelp 20%, WolframAlpha 14%, Yahoo 4%, and Wikipedia 2%. Breaking down Siri's reliance further, Google provides 100% of navigation results, 61% of information results, 48% of commerce results and 42% of local results. Among other result aggregators, Yelp provided the most local results (51%) and commerce results (51%), while WolframAlpha provided 34% of information results.

Munster estimates that Google's share of Siri results will drop to 48% with the launch of iOS 6 as navigation, sports, and movie queries shift over to Apple and its new partners.

Top Rated Comments

HarryKeogh Avatar
113 months ago
I'm using Siri to post this message and, as you can see, it's 100% a cure writ and pancakes urethra.
Score: 115 Votes (Like | Disagree)
basesloaded190 Avatar
113 months ago
Siri is not on the same level as Google voice. Just saying.

What Google showed yesterday with offline support was very impressive, I agree.
Score: 18 Votes (Like | Disagree)
Ciclismo Avatar
113 months ago
I'm using Siri to post this message and, as you can see, it's 100% a cure writ and pancakes urethra.

Nominated for "Post of the Month" award.
Score: 15 Votes (Like | Disagree)
Bezetos Avatar
113 months ago

Google is just a dumb engine with no possible applications other than being a directory of the internet. They have never developed beyond coder style searches for specifics and their engineer backgrounds suggest they never will.
I think you've been living under a rock if you seriously believe in this...
Score: 13 Votes (Like | Disagree)
rockland Avatar
113 months ago
Siri is substandard

To me, there's no two ways about it. Uncharacteristically, Apple releases a "beta" product, repeatedly invoking that fact as if it were an excuse.

Which would be fine, if they charged a "beta-worthy" price for iPhone 4S.

Sorry, but it's been well over half a year, and Siri STILL underperforms. Badly.

In my experience and in all kinds of situations and environments, Siri works less than 50% of the time. More like a third of the time. I've pretty much given up on it.

We all love Apple, but to me it's hard not to state the obvious. Siri is (so far) a flop, in terms of living up to its profusely advertised promise. It is not even close to the usual high standards of Apple/SJ. And if he were alive now, I think he would be blowing a gasket about it.
Score: 13 Votes (Like | Disagree)
SleeplessChaos Avatar
113 months ago
Fine, if you want to embarrass yourself...

On the accuracy side, Google's 86% rating is derived from comScore data showing that Google generates 1.14 search result pages per search, suggesting that roughly 86% of time Google presents the data the user is looking for within the first few results.

So this the point. Will you get information quicker using Siri or googling? So you've completely missed the point here.

It's not comparing two things at once. It is comparing how accurately you can get information using both tools. That's just one thing.

That would be biased. Secondly, read again: Google generates 1.14 search result pages per search, suggesting that roughly 86% of time Google presents the data the user is looking for within the first few results. So if you ask Siri "how tall is the empire state building", you'll get a precise answer from Siri (hopefuly) and find that information on the first page of Google results (actually, if you google that you'll now get an answer too). So this is what they're comparing. So again, you're completely missing the point.

And again, just go and read the article.

----------


Another person who missed the whole point of this research...

I'm wondering if we should check your reading comprehension guys...
I don't think he is missing the point of the experiment, I think that he, like me, thinks the point is stupid.

Think about it for more than a millisecond and you'll realize that it's comparing apples and oranges. Google, like every other text based search engine on the web, is simply matching text keywords with a large indexed database built by years of crawling through the web. The search is basically brute force and simple.

Siri is receiving a digital waveform from the microphone, eliminating frequencies from that waveform that are not commonly found in the human voice, translating those remaining frequencies into readable text via a remote server and then interpreting that into a command which it then uses to either search the web using what it assumes to be the correct web search service or perform a function using a local app. That service list includes google search.

The fact of the matter is that the two services are not comparable whatsoever and that for all intents and purposes siri expands upon the abilities of google regardless of the situation.
Score: 8 Votes (Like | Disagree)

Top Stories

iphone 6 in hand

Apple Faces Another iPhone Lawsuit Over 'Programmed Obsolescence'

Monday March 1, 2021 6:44 am PST by
Apple faces a new class-action lawsuit that accuses it of deliberately releasing iOS updates that slowly reduce the performance of an iPhone, forcing customers to upgrade their devices. The lawsuit comes from the Portuguese Consumer Protection Agency, Deco Proteste (via Marketeer), which in a statement says that it will proceed with a case against the Cupertino tech giant because it...
PIA23764 RoverNamePlateonMars web

NASA Mars Perseverance Rover Uses Same PowerPC Chipset Found in 1998 G3 iMac

Tuesday March 2, 2021 3:21 am PST by
NASA's Perseverance rover, which recently made history landing on the surface of Mars, is powered by the same processor used in an iMac more than 23 years old. Image Credit: NASA As reported by NewScientist (via Gizmodo), the rover includes the PowerPC 750 processor, the same chip used in the G3 iMac in 1998. The main chipset is the same; however, there are differences between the...
magsafe charging brick feature

Prosser: iPhone 12 MagSafe Battery Pack to Feature Reverse Charging

Monday March 1, 2021 1:18 pm PST by
Apple is working on MagSafe-compatible battery packs that will be compatible with the iPhone 12 models, and leaker Jon Prosser today said on the Genius Bar podcast that one version of the battery pack will feature "reverse charging." According to Prosser, Apple is working on two versions of the battery packs, one that's a standard version and one that's a premium version with reverse...
rosetta 2

Rosetta May Be Removed From M1 Macs in Some Regions on macOS 11.3

Tuesday March 2, 2021 5:20 pm PST by
Installing the upcoming macOS 11.3 software update on an M1 Mac may result in Rosetta 2 being removed in one or more regions around the world. In the third beta of macOS 11.3 seeded to developers for testing today, MacRumors contributor Steve Moser uncovered new strings in the beta's code indicating that "Rosetta will be removed upon installing this update." Another new string reads "Rosetta ...
iphone 12 pro display video

iPhone 13 to Include 1TB Storage Option and LiDAR Across the Board, Says Wedbush Analyst

Monday March 1, 2021 4:00 am PST by
Apple's forthcoming iPhone 13 could include a 1TB storage option for some models and LiDAR Scanners across the entire lineup, according to a report from Wedbush analysts. In a new note to investors, seen by MacRumors, Wedbush analyst Daniel Ives said that initial Asian supply chain checks gave the firm "increased confidence" that Apple's 5G-driven product cycle would extend well into 2022,...
Apple Prefer Lightning Over USB C Feature

iPhone Sticking With Lightning Port Over USB-C for 'Foreseeable Future'

Tuesday March 2, 2021 9:32 am PST by
Apple will retain the Lightning connector on the iPhone for the "foreseeable future," with no intention of switching to USB-C, according to reliable analyst Ming-Chi Kuo. In spite of much of the industry moving toward USB-C, Apple will not be using it to replace the Lightning connector on the iPhone 13, or indeed on any iPhone model for the time being. In a note seen by MacRumors yesterday,...
iphone 12 120hz thumbnail feature

Kuo: iPhone 13 Lineup to Feature Smaller Notch and Larger Batteries, 120Hz Display for Pro Models, and More

Monday March 1, 2021 7:50 am PST by
iPhone 13 models will all feature a smaller notch, while the two Pro models will be equipped with low-power LTPO display technology for a 120Hz refresh rate, analyst Ming-Chi Kuo said today in a research note obtained by MacRumors. Subscribe to the MacRumors YouTube channel for more videos. Several other sources have previously claimed that some iPhone 13 models will support a 120Hz refresh...
Screen Shot 2021 03 03 at 11

MagSafe Charging Port for iPhone Appears in Apple Patent

Wednesday March 3, 2021 1:30 am PST by
In a newly granted patent, Apple envisions a type of connectivity port using an iteration of its magnetic MagSafe charger to charge an iPhone, potentially paving the way for a future without Lightning. Submitted to the United States Patent and Trademark Office on Tuesday (via Patently Apple), the patent outlines a three-pinned MagSafe charger, similar to the ones found in older MacBook...
apple transfer google photos 1

Apple Launches Service for Transferring iCloud Photos and Videos to Google Photos

Wednesday March 3, 2021 12:04 pm PST by
Apple this week introduced a new service that's designed to make it quick and easy for iCloud users to transfer their stored photos and videos to Google Photos. As outlined in an Apple support document, you can go to Apple's privacy website and sign in to see the "Transfer a copy of your data" option. If you select this and go through all the steps, Apple will transfer your iCloud photos and ...
galaxy s21 iphone 12 pro max front feature2

Kuo: Some 2022 iPhones to Abandon Notch in Favor of 'Punch-Hole' Display

Monday March 1, 2021 8:05 am PST by
At least some 2022 iPhone models will abandon the notch and switch to a "punch-hole display design" instead, similar to Samsung's recent high-end Galaxy smartphones, analyst Ming-Chi Kuo said today in a research note obtained by MacRumors. Kuo said the hole-punch display design will be adopted by high-end 2022 iPhones at a minimum, and if production yields are good enough, all 2022 iPhone...