Apple's Beats brand in April unveiled the Powerbeats Pro, a redesigned wire-free version of its popular fitness-oriented Powerbeats earbuds.
Siri Shows Increasing Promise, But Accuracy Falls Short of Google Text Searches
In what should not be a surprise given Siri's beta status and its use of new technology for voice interaction with mobile devices, a study from Piper Jaffray analyst Gene Munster finds that Siri's abilities are still eclipsed by the traditional text-entry Google searches that have been in place for years.
The study included presenting Siri with 800 queries in each of two environments, a quiet room and a busy urban street, and then assessing Siri's ability to correctly interpret the queries and provide correct responses. Specifically addressing the outdoor testing, Munster notes:
Piper Jaffray's testing was conducted using the iPhone's built-in microphone in the quiet setting and Skullcandy microphone headphones for the outdoor setting, with Siri's ability to correctly comprehend queries falling from 89% to 83% when moving outdoors to the noisy urban setting. On the accuracy side, Google's 86% rating is derived from comScore data showing that Google generates 1.14 search result pages per search, suggesting that roughly 86% of time Google presents the data the user is looking for within the first few results.
Piper Jaffray also analyzed the sources for Siri's results, finding that Google provided 60% of the answers, with Yelp and other sources filling in the remainder.
The study included presenting Siri with 800 queries in each of two environments, a quiet room and a busy urban street, and then assessing Siri's ability to correctly interpret the queries and provide correct responses. Specifically addressing the outdoor testing, Munster notes:
While Google comprehends 100% of searches, Siri was only able to comprehend 83% of the prompted questions on a busy Minneapolis street. In terms of answer accuracy, Siri correctly answered 62% of queries on the street compared to Google's estimated 86% answer accuracy. Based on these results, we give Google an A+ for comprehension and a B+ for accuracy while we give Siri a B for comprehension and a D for accuracy.But while Siri has some room for improvement, Munster acknowledges that the technology is only in its infancy and he remains "optimistic about its future" with an eye toward iOS 6 as bringing substantial improvements.
Piper Jaffray's testing was conducted using the iPhone's built-in microphone in the quiet setting and Skullcandy microphone headphones for the outdoor setting, with Siri's ability to correctly comprehend queries falling from 89% to 83% when moving outdoors to the noisy urban setting. On the accuracy side, Google's 86% rating is derived from comScore data showing that Google generates 1.14 search result pages per search, suggesting that roughly 86% of time Google presents the data the user is looking for within the first few results.
Piper Jaffray also analyzed the sources for Siri's results, finding that Google provided 60% of the answers, with Yelp and other sources filling in the remainder.
Of queries excluding commands (i.e. call Jay, send text) Google would provide 60% of the answers, Yelp 20%, WolframAlpha 14%, Yahoo 4%, and Wikipedia 2%. Breaking down Siri's reliance further, Google provides 100% of navigation results, 61% of information results, 48% of commerce results and 42% of local results. Among other result aggregators, Yelp provided the most local results (51%) and commerce results (51%), while WolframAlpha provided 34% of information results.Munster estimates that Google's share of Siri results will drop to 48% with the launch of iOS 6 as navigation, sports, and movie queries shift over to Apple and its new partners.
[ 244 comments ]
Top Rated Comments
(View all)
89 months ago
I'm using Siri to post this message and, as you can see, it's 100% a cure writ and pancakes urethra.
89 months ago
Siri is not on the same level as Google voice. Just saying.
What Google showed yesterday with offline support was very impressive, I agree.
89 months ago
I'm using Siri to post this message and, as you can see, it's 100% a cure writ and pancakes urethra.
Nominated for "Post of the Month" award.
89 months ago
Siri is substandard
To me, there's no two ways about it. Uncharacteristically, Apple releases a "beta" product, repeatedly invoking that fact as if it were an excuse.
Which would be fine, if they charged a "beta-worthy" price for iPhone 4S.
Sorry, but it's been well over half a year, and Siri STILL underperforms. Badly.
In my experience and in all kinds of situations and environments, Siri works less than 50% of the time. More like a third of the time. I've pretty much given up on it.
We all love Apple, but to me it's hard not to state the obvious. Siri is (so far) a flop, in terms of living up to its profusely advertised promise. It is not even close to the usual high standards of Apple/SJ. And if he were alive now, I think he would be blowing a gasket about it.
To me, there's no two ways about it. Uncharacteristically, Apple releases a "beta" product, repeatedly invoking that fact as if it were an excuse.
Which would be fine, if they charged a "beta-worthy" price for iPhone 4S.
Sorry, but it's been well over half a year, and Siri STILL underperforms. Badly.
In my experience and in all kinds of situations and environments, Siri works less than 50% of the time. More like a third of the time. I've pretty much given up on it.
We all love Apple, but to me it's hard not to state the obvious. Siri is (so far) a flop, in terms of living up to its profusely advertised promise. It is not even close to the usual high standards of Apple/SJ. And if he were alive now, I think he would be blowing a gasket about it.
89 months ago
Google is just a dumb engine with no possible applications other than being a directory of the internet. They have never developed beyond coder style searches for specifics and their engineer backgrounds suggest they never will.
89 months ago
Fine, if you want to embarrass yourself...
On the accuracy side, Google's 86% rating is derived from comScore data showing that Google generates 1.14 search result pages per search, suggesting that roughly 86% of time Google presents the data the user is looking for within the first few results.
So this the point. Will you get information quicker using Siri or googling? So you've completely missed the point here.
It's not comparing two things at once. It is comparing how accurately you can get information using both tools. That's just one thing.
That would be biased. Secondly, read again: Google generates 1.14 search result pages per search, suggesting that roughly 86% of time Google presents the data the user is looking for within the first few results. So if you ask Siri "how tall is the empire state building", you'll get a precise answer from Siri (hopefuly) and find that information on the first page of Google results (actually, if you google that you'll now get an answer too). So this is what they're comparing. So again, you're completely missing the point.
And again, just go and read the article.
----------
Another person who missed the whole point of this research...
I'm wondering if we should check your reading comprehension guys...
I don't think he is missing the point of the experiment, I think that he, like me, thinks the point is stupid.
Think about it for more than a millisecond and you'll realize that it's comparing apples and oranges. Google, like every other text based search engine on the web, is simply matching text keywords with a large indexed database built by years of crawling through the web. The search is basically brute force and simple.
Siri is receiving a digital waveform from the microphone, eliminating frequencies from that waveform that are not commonly found in the human voice, translating those remaining frequencies into readable text via a remote server and then interpreting that into a command which it then uses to either search the web using what it assumes to be the correct web search service or perform a function using a local app. That service list includes google search.
The fact of the matter is that the two services are not comparable whatsoever and that for all intents and purposes siri expands upon the abilities of google regardless of the situation.
89 months ago
This is meaningless without testing multiple accents in all languages and having questions posed in different ways!
Pointless to the extreme is what comes to mind.
Siri is more of an experiment than a real feature, it's the interpretation of the sentance that is the exciting part of siri and it's ability to follow a thread.
Google is just a dumb engine with no possible applications other than being a directory of the internet. They have never developed beyond coder style searches for specifics and their engineer backgrounds suggest they never will.
lol you're such a hater man. I hope Apple is paying you.
89 months ago
I found Siri performs quite well for me. Not perfect by any means, but I use it regularly and rarely have issues. A lot of my use is in the car with the A/C on so that's a fairly noisy environment. I use it with the built-in mic, not a BT or wired one.
The main difference I see is that the article seemed to focus on searching. While I occasionally use Siri to search, I mostly use it for things like:
- Read an incoming text message to me while I'm driving and reply to it.
- Take a note (in the car or anywhere else). I even add-on to the note using voice.
- Check my calendar.
- Set up appointments in my calendar.
- Initiate calls
- Set up reminders (I'm especially fond of location-based reminders)
As others posted, accent and diction probably make a significant difference so YMMV.
The main difference I see is that the article seemed to focus on searching. While I occasionally use Siri to search, I mostly use it for things like:
- Read an incoming text message to me while I'm driving and reply to it.
- Take a note (in the car or anywhere else). I even add-on to the note using voice.
- Check my calendar.
- Set up appointments in my calendar.
- Initiate calls
- Set up reminders (I'm especially fond of location-based reminders)
As others posted, accent and diction probably make a significant difference so YMMV.
89 months ago
Doesn't using the headphones disable the built in noise cancelling? Seems like results might have been better if they used the built in mic.
89 months ago
Piper Jaffray's testing was conducted using the iPhone's built-in microphone in the quiet setting and Skullcandy microphone headphones for the outdoor setting, with Siri's ability to correctly comprehend queries falling from 89% to 83% when moving outdoors to the noisy urban setting. On the accuracy side, Google's 86% rating is derived from comScore data showing that Google generates 1.14 search result pages per search, suggesting that roughly 86% of time Google presents the data the user is looking for within the first few results.
So, a 1 year old Beta system is within 2-3 percentage points accuracy as compared to a 10 year old, vetted, and vastly more widely used system, and is hands free, voice activated, and can be used in a noisy urban environment?Siri sucks!
At failing. :D
Rocketman
[ Read All Comments ]
Amazon is reportedly readying a high-fidelity music streaming service that's set to launch by the end of the year. According to Music Business Worldwide, Amazon is in discussions with various...
Last year, we shared a list of some of the best HomeKit products you can buy, which was quite popular with MacRumors readers, so we thought we'd follow it up.
In our 2019 HomeKit video,...
Amazon is discounting the 2019 AirPods with Charging Case to $139.99, down from $159.00. Apple just refreshed the AirPods last month with an updated H1 chip and "Hey Siri" support,...
As part of its ongoing effort to rebuild Apple Maps, Apple has added detailed terrain features to the U.S. states of Arizona and New Mexico as well as the southern portion of Nevada, including the...
Tik Tok, which was pulled from the App Store in India last week to comply with a government demand to block downloads over child safety concerns, is allowed to return to the App Store, reports...
Amazon is currently discounting the latest 13-inch MacBook Pro with Touch Bar to a new all-time-low price. Specifically, this is the 2.3 GHz Quad-Core Intel Core i5 model with 8GB RAM and a 512GB...
Over four years after debuting on Android, and following a significant redesign last year, Google Fit is now available on iOS.
The fitness tracking app can track workout sessions completed...
Following a flash sale on Easter, Best Buy has returned this week with a new 4-day sale that offers discounts on the MacBook Pro with Touch Bar, iPhone X, Beats products, iPhone cases, and more. The...
