Apple's Machine Learning Has Cut Siri's Error Rate by a Factor of Two

Steven Levy has published an in-depth article about Apple's artificial intelligence and machine learning efforts, after meeting with senior executives Craig Federighi, Eddy Cue, Phil Schiller, and two Siri scientists at the company's headquarters.

backchannel-apple-machine-learning
Apple provided Levy with a closer look at how machine learning is deeply integrated into Apple software and services, led by Siri, which the article reveals has been powered by a neural-net based system since 2014. Apple said the backend change greatly improved the personal assistant's accuracy.

"This was one of those things where the jump was so significant that you do the test again to make sure that somebody didn’t drop a decimal place," says Eddy Cue, Apple’s senior vice president of internet software and services.

Alex Acero, who leads the Siri speech team at Apple, said Siri's error rate has been lowered by more than a factor of two in many cases.

“The error rate has been cut by a factor of two in all the languages, more than a factor of two in many cases,” says Acero. “That’s mostly due to deep learning and the way we have optimized it — not just the algorithm itself but in the context of the whole end-to-end product.”

Acero told Levy he was able to work directly with Apple's silicon design team and the engineers who write the firmware for iOS devices to maximize performance of the neural network, and Federighi added that Apple building both hardware and software gives it an "incredible advantage" in the space.

"It's not just the silicon," adds Federighi. "It's how many microphones we put on the device, where we place the microphones. How we tune the hardware and those mics and the software stack that does the audio processing. It's all of those pieces in concert. It's an incredible advantage versus those who have to build some software and then just see what happens."

Apple's machine learning efforts extend far beyond Siri, as evidenced by several examples shared by Levy:

You see it when the phone identifies a caller who isn’t in your contact list (but did email you recently). Or when you swipe on your screen to get a shortlist of the apps that you are most likely to open next. Or when you get a reminder of an appointment that you never got around to putting into your calendar. Or when a map location pops up for the hotel you’ve reserved, before you type it in. Or when the phone points you to where you parked your car, even though you never asked it to. These are all techniques either made possible or greatly enhanced by Apple’s adoption of deep learning and neural nets.

Another product born out of machine learning is the Apple Pencil, which can detect the difference between a swipe, a touch, and a pencil input:

In order for Apple to include its version of a high-tech stylus, it had to deal with the fact that when people wrote on the device, the bottom of their hand would invariably brush the touch screen, causing all sorts of digital havoc. Using a machine learning model for “palm rejection” enabled the screen sensor to detect the difference between a swipe, a touch, and a pencil input with a very high degree of accuracy. “If this doesn’t work rock solid, this is not a good piece of paper for me to write on anymore — and Pencil is not a good product,” says Federighi. If you love your Pencil, thank machine learning.

On the iPhone, machine learning is enabled by a localized dynamic cache or "knowledge base" that Apple says is around 200MB in size, depending on how much personal information is stored.

This includes information about app usage, interactions with other people, neural net processing, a speech modeler, and "natural language event modeling." It also has data used for the neural nets that power object recognition, face recognition, and scene classification.

"It's a compact, but quite thorough knowledge base, with hundreds of thousands of locations and entities. We localize it because we know where you are," says Federighi. This knowledge base is tapped by all of Apple's apps, including the Spotlight search app, Maps, and Safari. It helps on auto-correct. "And it's working continuously in the background," he says.

Apple, for example, uses its neural network to capture the words iPhone users type using the standard QuickType keyboard.

Other information Apple stores on devices includes probably the most personal data that Apple captures: the words people type using the standard iPhone QuickType keyboard. By using a neural network-trained system that watches while you type, Apple can detect key events and items like flight information, contacts, and appointments — but information itself stays on your phone.

Apple insists that much of the machine learning occurs entirely local to the device, without personal information being sent back to its servers.

"Some people perceive that we can't do these things with AI because we don't have the data," says Cue. "But we have found ways to get that data we need while still maintaining privacy. That's the bottom line."

"We keep some of the most sensitive things where the ML is occurring entirely local to the device," Federighi says. As an example, he cites app suggestions, the icons that appear when you swipe right.

The full-length article on Backchannel provides several more details about how machine learning and artificial intelligence work at Apple.

Top Rated Comments

Santabean2000 Avatar
66 months ago
Siri stills sucks for me... still doesn't work well enough to be useful.
Score: 32 Votes (Like | Disagree)
GfPQqmcRKUvP Avatar
66 months ago
Not in my experience. Siri is absolute trash compared to Google Now and Cortana. Apple should be embarrassed by how stagnant it has been and the repeated, correctable errors it makes. No amount of public relations whitewashing can fix it.
Score: 29 Votes (Like | Disagree)
Oblivious.Robot Avatar
66 months ago
Yeah, no.
Siri is still pretty much useless for me, although better than before, but still meh with her current capabilities.



Edit - For the screen brightness part, after some retries I found out that it does work, but you have to say it in one sentence as Siri forgets about it in the next. :confused:

Attachment Image
Score: 17 Votes (Like | Disagree)
keysofanxiety Avatar
66 months ago
Apple insists that much of the machine learning occurs entirely local to the device, without personal information being sent back to its servers.
And yet it still has no offline functionality, even if the learning is localised; well from what I can see, anyway. An iPhone 3GS managed offline voice functions like "call Tom". Why the heck can't Siri? Phone, text, open an app -- these are not commands that need to be bounced off a server.

It's fine if you're at Apple HQ, testing Siri on a 10Gb/s Internet connection. But when you're on the road with flaky 3G signal, it's the last thing you need.
Score: 11 Votes (Like | Disagree)
jsclem02 Avatar
66 months ago
The struggle with Siri is still very real.

Attachment Image
Score: 11 Votes (Like | Disagree)
thisisnotmyname Avatar
66 months ago
I like Siri on the iPhone when it works, but at times it totally gets things very wrong that it previously got right.

What really gets me ticked off is how Siri on ATV is worse than useless almost every time I try to make use of it.
I agree. On the phone she works well for me. On the AppleTV it seems like she goes to sleep and the first time I try to interact with her such as "what did he just say?" it lags so long that the ten second rewind is well past whatever I wanted to hear. And then there's the lack of Siri search through home sharing content...
Score: 7 Votes (Like | Disagree)

Top Stories

calculatorapp

iOS 11 Bug: Typing 1+2+3 Quickly in the Calculator App Won't Get You 6

Tuesday October 24, 2017 2:03 pm PDT by
A bug in the built-in Calculator app in iOS 11 is getting some major attention this week, despite the fact that it's been around since iOS 11 was in beta testing. At issue is a calculator animation that causes some symbols to be ignored when calculations are entered in rapid succession. You can try it for yourself: Type 1+2+3 and then the equals sign into the Calculator app quickly. Due to...
gradiente iphone white

Brazilian Electronics Company Revives Long-Running iPhone Trademark Dispute

Tuesday May 19, 2020 1:06 pm PDT by
Apple has been involved in a long-running iPhone trademark dispute in Brazil, which was revived today by IGB Electronica, a Brazilian consumer electronics company that originally registered the "iPhone" name in 2000. IGB Electronica fought a multi-year battle with Apple in an attempt to get exclusive rights to the "iPhone" trademark, but ultimately lost, and now the case has been brought to...
iPhone 13 Dummy Thumbnail 2

Full iPhone 13 Feature Breakdown: Everything Rumors Say We Can Expect

Tuesday August 31, 2021 7:50 am PDT by
With the launch of Apple's iPhone 13 lineup believed to be just a few weeks away, we have compiled all of the coherent rumors from our coverage over the past year to build a full picture of the features and upgrades coming to the company's new smartphones. For clarity, only explicit improvements, upgrades, and new features compared to the iPhone 12 lineup are listed. It is worth noting that...
original iphone

Phil Schiller Says iPhone Was 'Earth-Shattering' Ten Years Ago and Remains 'Unmatched' Today

Monday January 9, 2017 7:15 am PST by
To commemorate the tenth anniversary of the iPhone, Apple marketing chief Phil Schiller sat down with tech journalist Steven Levy for a wide-ranging interview about the smartphone's past, present, and future. The report first reflects upon the iPhone's lack of support for third-party apps in its first year. The argument inside Apple was split between whether the iPhone should be a closed...
iPhone 13 Dummy Thumbnail 2

Kuo: iPhone 13 to Feature LEO Satellite Communications to Make Calls and Texts Without Cellular Coverage

Sunday August 29, 2021 7:39 am PDT by
The iPhone 13 will feature low earth orbit (LEO) satellite communication connectivity to allow users to make calls and send messages in areas without 4G or 5G coverage, according to the reliable analyst Ming-Chi Kuo. In a note to investors, seen by MacRumors, Kuo explained that the iPhone 13 lineup will feature hardware that is able to connect to LEO satellites. If enabled with the relevant...
maxresdefault

New MacBook Pro Models Coming at WWDC, Suggests Leaker

Monday May 24, 2021 1:27 pm PDT by
New MacBook Pro models are coming at WWDC, according to leaker Jon Prosser who has a mixed track record when it comes to predicting Apple's plans. Subscribe to the MacRumors YouTube channel for more videos. Prosser provided no additional information, but there are new 14 and 16-inch MacBook Pro models in the works. The new MacBook Pros will feature the most radical redesign to the MacBook Pro ...
omg lightning cable comparison

Security Researcher Develops Lightning Cable With Hidden Chip to Steal Passwords

Thursday September 2, 2021 6:59 am PDT by
A normal-looking Lightning cable that can used to steal data like passwords and send it to a hacker has been developed, Vice reports. The "OMG Cable" compared to Apple's Lightning to USB cable. The "OMG Cable" works exactly like a normal Lightning to USB cable and can log keystrokes from connected Mac keyboards, iPads, and iPhones, and then send this data to a bad actor who could be over a...
iphone 12 colors 2021

iPhone 12 Colors: Deciding on The Right Color

Thursday November 5, 2020 8:35 am PST by
The iPhone 12 and iPhone 12 Pro arrived last October in a range of color options, with entirely new hues available on both devices, as well as some popular classics. The 12 and 12 Pro have different color choices, so if you have your heart set on a particular shade, you might not be able to get your preferred model in that color. iPhone 12 mini and iPhone 12 The iPhone 12 mini and iPhone 12...
Top Stories 75 Thumbnail

Top Stories: Last-Minute iPhone 13 Rumors, Apple Announces App Store Changes, and More

Saturday September 4, 2021 6:00 am PDT by
The finish line is in sight! Apple's annual iPhone event is likely just a week or so away and all eyes will be on the company as it unveils the next version of its most popular product line. With any luck, we'll also see the next-generation Apple Watch and perhaps even some new AirPods. Other news this week saw Apple making some more changes to its App Store policies in response to a...
M1X MBP Feature

Leaker: Upcoming MacBook Pro to See Price Hike Over Current Model, Equal Performance Across 14 and 16-Inch Sizes

Tuesday August 24, 2021 5:28 am PDT by
The upcoming 14-inch MacBook Pro is set to be more expensive than the current 13-inch MacBook Pro and both the 14 and 16-inch models will offer the same performance, according to the leaker known as "Dylandkt." The leaker shared the information on Twitter, explaining that both of the upcoming MacBook Pro models, expected to come in 14 and 16-inch sizes, will feature the same performance due...