Siri Shows Increasing Promise, But Accuracy Falls Short of Google Text Searches

In what should not be a surprise given Siri's beta status and its use of new technology for voice interaction with mobile devices, a study from Piper Jaffray analyst Gene Munster finds that Siri's abilities are still eclipsed by the traditional text-entry Google searches that have been in place for years.

The study included presenting Siri with 800 queries in each of two environments, a quiet room and a busy urban street, and then assessing Siri's ability to correctly interpret the queries and provide correct responses. Specifically addressing the outdoor testing, Munster notes:

While Google comprehends 100% of searches, Siri was only able to comprehend 83% of the prompted questions on a busy Minneapolis street. In terms of answer accuracy, Siri correctly answered 62% of queries on the street compared to Google's estimated 86% answer accuracy. Based on these results, we give Google an A+ for comprehension and a B+ for accuracy while we give Siri a B for comprehension and a D for accuracy.

But while Siri has some room for improvement, Munster acknowledges that the technology is only in its infancy and he remains "optimistic about its future" with an eye toward iOS 6 as bringing substantial improvements.

siri examples
Piper Jaffray's testing was conducted using the iPhone's built-in microphone in the quiet setting and Skullcandy microphone headphones for the outdoor setting, with Siri's ability to correctly comprehend queries falling from 89% to 83% when moving outdoors to the noisy urban setting. On the accuracy side, Google's 86% rating is derived from comScore data showing that Google generates 1.14 search result pages per search, suggesting that roughly 86% of time Google presents the data the user is looking for within the first few results.

Piper Jaffray also analyzed the sources for Siri's results, finding that Google provided 60% of the answers, with Yelp and other sources filling in the remainder.

Of queries excluding commands (i.e. call Jay, send text) Google would provide 60% of the answers, Yelp 20%, WolframAlpha 14%, Yahoo 4%, and Wikipedia 2%. Breaking down Siri's reliance further, Google provides 100% of navigation results, 61% of information results, 48% of commerce results and 42% of local results. Among other result aggregators, Yelp provided the most local results (51%) and commerce results (51%), while WolframAlpha provided 34% of information results.

Munster estimates that Google's share of Siri results will drop to 48% with the launch of iOS 6 as navigation, sports, and movie queries shift over to Apple and its new partners.

Top Rated Comments

HarryKeogh Avatar
125 months ago
I'm using Siri to post this message and, as you can see, it's 100% a cure writ and pancakes urethra.
Score: 115 Votes (Like | Disagree)
basesloaded190 Avatar
125 months ago
Siri is not on the same level as Google voice. Just saying.

What Google showed yesterday with offline support was very impressive, I agree.
Score: 18 Votes (Like | Disagree)
Ciclismo Avatar
125 months ago
I'm using Siri to post this message and, as you can see, it's 100% a cure writ and pancakes urethra.

Nominated for "Post of the Month" award.
Score: 15 Votes (Like | Disagree)
Bezetos Avatar
125 months ago

Google is just a dumb engine with no possible applications other than being a directory of the internet. They have never developed beyond coder style searches for specifics and their engineer backgrounds suggest they never will.
I think you've been living under a rock if you seriously believe in this...
Score: 13 Votes (Like | Disagree)
rockland Avatar
125 months ago
Siri is substandard

To me, there's no two ways about it. Uncharacteristically, Apple releases a "beta" product, repeatedly invoking that fact as if it were an excuse.

Which would be fine, if they charged a "beta-worthy" price for iPhone 4S.

Sorry, but it's been well over half a year, and Siri STILL underperforms. Badly.

In my experience and in all kinds of situations and environments, Siri works less than 50% of the time. More like a third of the time. I've pretty much given up on it.

We all love Apple, but to me it's hard not to state the obvious. Siri is (so far) a flop, in terms of living up to its profusely advertised promise. It is not even close to the usual high standards of Apple/SJ. And if he were alive now, I think he would be blowing a gasket about it.
Score: 13 Votes (Like | Disagree)
SleeplessChaos Avatar
125 months ago
Fine, if you want to embarrass yourself...

On the accuracy side, Google's 86% rating is derived from comScore data showing that Google generates 1.14 search result pages per search, suggesting that roughly 86% of time Google presents the data the user is looking for within the first few results.

So this the point. Will you get information quicker using Siri or googling? So you've completely missed the point here.

It's not comparing two things at once. It is comparing how accurately you can get information using both tools. That's just one thing.

That would be biased. Secondly, read again: Google generates 1.14 search result pages per search, suggesting that roughly 86% of time Google presents the data the user is looking for within the first few results. So if you ask Siri "how tall is the empire state building", you'll get a precise answer from Siri (hopefuly) and find that information on the first page of Google results (actually, if you google that you'll now get an answer too). So this is what they're comparing. So again, you're completely missing the point.

And again, just go and read the article.

----------


Another person who missed the whole point of this research...

I'm wondering if we should check your reading comprehension guys...
I don't think he is missing the point of the experiment, I think that he, like me, thinks the point is stupid.

Think about it for more than a millisecond and you'll realize that it's comparing apples and oranges. Google, like every other text based search engine on the web, is simply matching text keywords with a large indexed database built by years of crawling through the web. The search is basically brute force and simple.

Siri is receiving a digital waveform from the microphone, eliminating frequencies from that waveform that are not commonly found in the human voice, translating those remaining frequencies into readable text via a remote server and then interpreting that into a command which it then uses to either search the web using what it assumes to be the correct web search service or perform a function using a local app. That service list includes google search.

The fact of the matter is that the two services are not comparable whatsoever and that for all intents and purposes siri expands upon the abilities of google regardless of the situation.
Score: 8 Votes (Like | Disagree)

Popular Stories

Upcoming Products 2022 Feature

Gurman: Apple Preparing 'Widest Array of New Hardware Products in Its History' for Fall

Sunday January 23, 2022 10:32 am PST by
Apple is working on a number of new products that are set to launch this fall, and Bloomberg's Mark Gurman says that it will be "the widest array" of new devices that Apple has introduced in its history. In his latest "Power On" newsletter, Gurman explains that Apple is working on four new flagship iPhones (iPhone 14, iPhone 14 Max, iPhone 14 Pro, and iPhone 14 Pro Max), an updated low-end Ma...
macbook pro 14 16 2021

Three Months After Launch, Apple Still Struggling to Meet Demand for Redesigned 14-Inch and 16-Inch MacBook Pro

Monday January 24, 2022 7:12 am PST by
Three months after their launch, the 14-inch and 16-inch MacBook Pros continue to experience high demand and seemingly short supply, with shipping dates for both models stretching into multiple weeks in several of Apple's key markets. In the United States, the baseline 14-inch MacBook Pro with the M1 Pro chip is estimated to ship in three to four weeks, promising an arrival by at least...
General Dropbox Feature

macOS 12.3 Will Include Cloud Storage Changes Affecting Dropbox and OneDrive

Tuesday January 25, 2022 3:31 pm PST by
Dropbox today announced that users who update to macOS 12.3 once that software version becomes available may temporarily encounter issues with opening online-only files in some third-party apps on their Mac. In a support document and an email to customers, Dropbox said it is actively working on full support for online-only files on macOS 12.3 and will begin rolling out an updated version of...
Questionable Design Decisions

Apple's Most Questionable Design Decisions in Recent Memory

Sunday January 23, 2022 2:59 am PST by
Apple has always emphasized the depth of thought that goes into the design of its products. In the foreword to Designed by Apple in California, a photo book released by the company in 2016, Jony Ive explains how Apple strives "to define objects that appear effortless" and "so simple, coherent and inevitable that there could be no rational alternative." But every once in a while even Apple...
Apple Watch Red Yellow Green Feature 1

Apple Launches Black Unity Braided Solo Loop With 'Unity Lights' Watch Face

Wednesday January 26, 2022 6:05 am PST by
Apple today announced the Black Unity Braided Solo Loop for the Apple Watch, as well as a new downloadable watch face, to celebrate Black History Month. Following the launch of the limited edition Black Unity Apple Watch Series 6 and Sport Band in 2021, Apple today launched the Black Unity Braided Solo Loop as part of its celebrations for Black History Month this year.Apple is launching a...
att gigabit internet

AT&T Bringing $180/Month 5-Gigabit Internet to 70 Cities

Monday January 24, 2022 9:20 am PST by
AT&T today announced the launch of upgraded AT&T Fiber plans, which support speeds of up to 5 Gigabits for some customers. There are two separate plans, one "2 GIG" plan and one "5 GIG" plan, available to new and existing AT&T Fiber subscribers. According to AT&T, the new plans are available to nearly 5.2 million customers across 70 metro areas including Los Angeles, Atlanta, Chicago, San...
intel vs m1 max chip purple

Benchmarks Confirm Intel's Latest Core i9 Chip Outperforms Apple's M1 Max With Several Caveats

Wednesday January 26, 2022 8:56 am PST by
Benchmark results have started to surface for MSI's new GE76 Raider, one of the first laptops to be powered by Intel's new 12th-generation Core i9 processor. Intel previously said that its new high-end Core i9 processor is faster than Apple's M1 Max chip in the 16-inch MacBook Pro and, as noted by Macworld, early Geekbench 5 results do appear to confirm this claim, but there are several...
ios 15

Apple Releases iOS 15.3 and iPadOS 15.3 With Fix for Safari Bug That Leaks Browsing Activity

Wednesday January 26, 2022 10:00 am PST by
Apple today released iOS 15.3 and iPadOS 15.3, the third major updates to the iOS and iPadOS 15 operating systems that were released in September 2021. iOS and iPadOS 15.3 come almost two weeks after the release of iOS and iPadOS 15.2.1, minor bug fix updates. The iOS 15.3 and iPadOS 15.3 updates can be downloaded for free and the software is available on all eligible devices over-the-air in ...