Siri Shows Increasing Promise, But Accuracy Falls Short of Google Text Searches

In what should not be a surprise given Siri's beta status and its use of new technology for voice interaction with mobile devices, a study from Piper Jaffray analyst Gene Munster finds that Siri's abilities are still eclipsed by the traditional text-entry Google searches that have been in place for years.

The study included presenting Siri with 800 queries in each of two environments, a quiet room and a busy urban street, and then assessing Siri's ability to correctly interpret the queries and provide correct responses. Specifically addressing the outdoor testing, Munster notes:

While Google comprehends 100% of searches, Siri was only able to comprehend 83% of the prompted questions on a busy Minneapolis street. In terms of answer accuracy, Siri correctly answered 62% of queries on the street compared to Google's estimated 86% answer accuracy. Based on these results, we give Google an A+ for comprehension and a B+ for accuracy while we give Siri a B for comprehension and a D for accuracy.

But while Siri has some room for improvement, Munster acknowledges that the technology is only in its infancy and he remains "optimistic about its future" with an eye toward iOS 6 as bringing substantial improvements.


Piper Jaffray's testing was conducted using the iPhone's built-in microphone in the quiet setting and Skullcandy microphone headphones for the outdoor setting, with Siri's ability to correctly comprehend queries falling from 89% to 83% when moving outdoors to the noisy urban setting. On the accuracy side, Google's 86% rating is derived from comScore data showing that Google generates 1.14 search result pages per search, suggesting that roughly 86% of time Google presents the data the user is looking for within the first few results.

Piper Jaffray also analyzed the sources for Siri's results, finding that Google provided 60% of the answers, with Yelp and other sources filling in the remainder.

Of queries excluding commands (i.e. call Jay, send text) Google would provide 60% of the answers, Yelp 20%, WolframAlpha 14%, Yahoo 4%, and Wikipedia 2%. Breaking down Siri's reliance further, Google provides 100% of navigation results, 61% of information results, 48% of commerce results and 42% of local results. Among other result aggregators, Yelp provided the most local results (51%) and commerce results (51%), while WolframAlpha provided 34% of information results.

Munster estimates that Google's share of Siri results will drop to 48% with the launch of iOS 6 as navigation, sports, and movie queries shift over to Apple and its new partners.

Top Rated Comments

(View all)
Avatar
106 months ago
I'm using Siri to post this message and, as you can see, it's 100% a cure writ and pancakes urethra.
Score: 115 Votes (Like | Disagree)
Avatar
106 months ago

Siri is not on the same level as Google voice. Just saying.


What Google showed yesterday with offline support was very impressive, I agree.
Score: 18 Votes (Like | Disagree)
Avatar
106 months ago

I'm using Siri to post this message and, as you can see, it's 100% a cure writ and pancakes urethra.


Nominated for "Post of the Month" award.
Score: 15 Votes (Like | Disagree)
Avatar
106 months ago


Google is just a dumb engine with no possible applications other than being a directory of the internet. They have never developed beyond coder style searches for specifics and their engineer backgrounds suggest they never will.

I think you've been living under a rock if you seriously believe in this...
Score: 13 Votes (Like | Disagree)
Avatar
106 months ago
Siri is substandard

To me, there's no two ways about it. Uncharacteristically, Apple releases a "beta" product, repeatedly invoking that fact as if it were an excuse.

Which would be fine, if they charged a "beta-worthy" price for iPhone 4S.

Sorry, but it's been well over half a year, and Siri STILL underperforms. Badly.

In my experience and in all kinds of situations and environments, Siri works less than 50% of the time. More like a third of the time. I've pretty much given up on it.

We all love Apple, but to me it's hard not to state the obvious. Siri is (so far) a flop, in terms of living up to its profusely advertised promise. It is not even close to the usual high standards of Apple/SJ. And if he were alive now, I think he would be blowing a gasket about it.
Score: 13 Votes (Like | Disagree)
Avatar
106 months ago

Fine, if you want to embarrass yourself...

On the accuracy side, Google's 86% rating is derived from comScore data showing that Google generates 1.14 search result pages per search, suggesting that roughly 86% of time Google presents the data the user is looking for within the first few results.

So this the point. Will you get information quicker using Siri or googling? So you've completely missed the point here.

It's not comparing two things at once. It is comparing how accurately you can get information using both tools. That's just one thing.

That would be biased. Secondly, read again: Google generates 1.14 search result pages per search, suggesting that roughly 86% of time Google presents the data the user is looking for within the first few results. So if you ask Siri "how tall is the empire state building", you'll get a precise answer from Siri (hopefuly) and find that information on the first page of Google results (actually, if you google that you'll now get an answer too). So this is what they're comparing. So again, you're completely missing the point.

And again, just go and read the article.

----------


Another person who missed the whole point of this research...

I'm wondering if we should check your reading comprehension guys...

I don't think he is missing the point of the experiment, I think that he, like me, thinks the point is stupid.

Think about it for more than a millisecond and you'll realize that it's comparing apples and oranges. Google, like every other text based search engine on the web, is simply matching text keywords with a large indexed database built by years of crawling through the web. The search is basically brute force and simple.

Siri is receiving a digital waveform from the microphone, eliminating frequencies from that waveform that are not commonly found in the human voice, translating those remaining frequencies into readable text via a remote server and then interpreting that into a command which it then uses to either search the web using what it assumes to be the correct web search service or perform a function using a local app. That service list includes google search.

The fact of the matter is that the two services are not comparable whatsoever and that for all intents and purposes siri expands upon the abilities of google regardless of the situation.
Score: 8 Votes (Like | Disagree)

Top Stories

iPhone Maker Foxconn Says China's 'Days as the World's Factory Are Done'

Wednesday August 12, 2020 7:55 am PDT by
China will no longer be the world's manufacturing epicenter going forward, according to Apple's largest supply chain partner Foxconn, which has been gradually expanding its operations in other countries amid the U.S.-China trade war. "No matter if it's India, Southeast Asia or the Americas, there will be a manufacturing ecosystem in each," said Foxconn chairman Young Liu, according to Bloombe...

Leaker Jon Prosser: Apple Watch and iPad Launching in September, iPhone 12 Event to Take Place in October

Wednesday August 12, 2020 4:31 pm PDT by
Apple last month confirmed that this year's iPhone 12 models will launch outside of their normal September timeframe and will be "available a few weeks later," which has led to speculation about when an event might be held. Leaker Jon Prosser, who sometimes shares accurate knowledge of Apple's plans, today said that Apple will hold its iPhone 12 event during the week of October 12, with...

Apple Removes Fortnite From App Store [Update: Epic Files Lawsuit Against Apple]

Thursday August 13, 2020 11:58 am PDT by
Just hours after Epic Games introduced a new direct payment option for Fortnite that skirts Apple's in-app purchase rules, Apple has pulled the Fortnite app from the App Store. Fortnite is no longer available for download on the iPhone or the iPad, and Apple provided a statement to MacRumors on Fortnite's removal:Today, Epic Games took the unfortunate step of violating the App Store...

Apple to Launch Bundled Subscription Services Called 'Apple One'

Thursday August 13, 2020 3:41 am PDT by
Apple will launch a new range of subscription service bundles called "Apple One" as soon as October, according to a new report by Bloomberg's Mark Gurman. The series of bundles would allow customers to subscribe to several Apple digital services together. This is expected to result in a lower monthly price than when the services are subscribed to individually. Bloomberg reports that the...

Apple Releases iOS and iPadOS 13.6.1 With Fix for Storage Issue and Green Tinted Displays

Wednesday August 12, 2020 1:31 pm PDT by
Apple today released iOS and iPadOS 13.6.1, minor updates that come a month after the release of the iOS 13.6 update with Car Keys and Audio Apple News+ stories. The iOS and ‌iPadOS‌ 13.6.1 updates are available on all eligible devices over-the-air in the Settings app. To access the updates, go to Settings > General > Software Update. iOS 13.6.1 addresses an issue that could cause...

Apple Begins Selling Refurbished 2020 13-inch MacBook Pro Models

Friday August 14, 2020 4:22 am PDT by
Apple has begun selling discounted refurbished 2020 13-inch MacBook Pro models through its online store in the United States for the first time, with up to $200 savings available on some base configurations compared to brand new models. Currently, Apple's refurbished options for the latest 13-inch MacBook Pro lineup are limited to models with the 8th-generation Intel Core i5 and Core i7...

Apple Releases macOS Catalina 10.15.6 Supplemental Update With Virtualization Bug Fix

Wednesday August 12, 2020 1:20 pm PDT by
Apple today released a supplemental update for macOS Catalina 10.15.6, with the update coming a month after the original launch of macOS Catalina 10.15.6. The ‌‌macOS Catalina‌‌ 10.15.6 Supplemental Update can be downloaded from the Mac App Store using the Update feature in the System Preferences app. According to Apple's release notes, the update fixes a problem that could cause...

Samsung's Galaxy Buds Live vs. Apple's AirPods Pro

Thursday August 13, 2020 1:48 pm PDT by
Samsung last week unveiled new flagship smartphones and the new bean-shaped Galaxy Buds Live, a set of wireless earbuds with Active Noise Cancellation that are designed to rival Apple's AirPods Pro. We got a set of the new Galaxy Buds Live and compared them to the AirPods Pro in our latest YouTube video. Subscribe to the MacRumors YouTube channel for more videos. Priced at $179, the Galaxy...

iPad Pro Keyboard Comparison: Logitech's $160 Folio Touch vs. Apple's $300 Magic Keyboard

Tuesday August 11, 2020 2:11 pm PDT by
Logitech recently debuted the Folio Touch, a keyboard and trackpad case designed for the 11-inch iPad Pro that serves as an alternative to the Magic Keyboard. In our latest YouTube video, we compare the $160 Folio Touch to Apple's $300 Magic Keyboard to see which is better. Subscribe to the MacRumors YouTube channel for more videos. Logitech is selling the Folio Touch for $160, while Apple's...

Spotify Sides With Epic Games in Battle Against Apple's App Store Fees

Thursday August 13, 2020 2:32 pm PDT by
Spotify, which has also faced off with Apple over Apple's App Store policies and fees, today weighed in on Epic Games' fight with Apple over Fortnite's removal from the App Store. Unsurprisingly, Spotify has sided with Epic Games, applauding Epic's decision to "take a stand against Apple." From a statement provided to Recode's Peter Kafka:We applaud Epic Games' decision to take a stand...