Apple's Machine Learning Has Cut Siri's Error Rate by a Factor of Two

Steven Levy has published an in-depth article about Apple's artificial intelligence and machine learning efforts, after meeting with senior executives Craig Federighi, Eddy Cue, Phil Schiller, and two Siri scientists at the company's headquarters.

backchannel-apple-machine-learning
Apple provided Levy with a closer look at how machine learning is deeply integrated into Apple software and services, led by Siri, which the article reveals has been powered by a neural-net based system since 2014. Apple said the backend change greatly improved the personal assistant's accuracy.

"This was one of those things where the jump was so significant that you do the test again to make sure that somebody didn’t drop a decimal place," says Eddy Cue, Apple’s senior vice president of internet software and services.

Alex Acero, who leads the Siri speech team at Apple, said Siri's error rate has been lowered by more than a factor of two in many cases.

“The error rate has been cut by a factor of two in all the languages, more than a factor of two in many cases,” says Acero. “That’s mostly due to deep learning and the way we have optimized it — not just the algorithm itself but in the context of the whole end-to-end product.”

Acero told Levy he was able to work directly with Apple's silicon design team and the engineers who write the firmware for iOS devices to maximize performance of the neural network, and Federighi added that Apple building both hardware and software gives it an "incredible advantage" in the space.

"It's not just the silicon," adds Federighi. "It's how many microphones we put on the device, where we place the microphones. How we tune the hardware and those mics and the software stack that does the audio processing. It's all of those pieces in concert. It's an incredible advantage versus those who have to build some software and then just see what happens."

Apple's machine learning efforts extend far beyond Siri, as evidenced by several examples shared by Levy:

You see it when the phone identifies a caller who isn’t in your contact list (but did email you recently). Or when you swipe on your screen to get a shortlist of the apps that you are most likely to open next. Or when you get a reminder of an appointment that you never got around to putting into your calendar. Or when a map location pops up for the hotel you’ve reserved, before you type it in. Or when the phone points you to where you parked your car, even though you never asked it to. These are all techniques either made possible or greatly enhanced by Apple’s adoption of deep learning and neural nets.

Another product born out of machine learning is the Apple Pencil, which can detect the difference between a swipe, a touch, and a pencil input:

In order for Apple to include its version of a high-tech stylus, it had to deal with the fact that when people wrote on the device, the bottom of their hand would invariably brush the touch screen, causing all sorts of digital havoc. Using a machine learning model for “palm rejection” enabled the screen sensor to detect the difference between a swipe, a touch, and a pencil input with a very high degree of accuracy. “If this doesn’t work rock solid, this is not a good piece of paper for me to write on anymore — and Pencil is not a good product,” says Federighi. If you love your Pencil, thank machine learning.

On the iPhone, machine learning is enabled by a localized dynamic cache or "knowledge base" that Apple says is around 200MB in size, depending on how much personal information is stored.

This includes information about app usage, interactions with other people, neural net processing, a speech modeler, and "natural language event modeling." It also has data used for the neural nets that power object recognition, face recognition, and scene classification.

"It's a compact, but quite thorough knowledge base, with hundreds of thousands of locations and entities. We localize it because we know where you are," says Federighi. This knowledge base is tapped by all of Apple's apps, including the Spotlight search app, Maps, and Safari. It helps on auto-correct. "And it's working continuously in the background," he says.

Apple, for example, uses its neural network to capture the words iPhone users type using the standard QuickType keyboard.

Other information Apple stores on devices includes probably the most personal data that Apple captures: the words people type using the standard iPhone QuickType keyboard. By using a neural network-trained system that watches while you type, Apple can detect key events and items like flight information, contacts, and appointments — but information itself stays on your phone.

Apple insists that much of the machine learning occurs entirely local to the device, without personal information being sent back to its servers.

"Some people perceive that we can't do these things with AI because we don't have the data," says Cue. "But we have found ways to get that data we need while still maintaining privacy. That's the bottom line."

"We keep some of the most sensitive things where the ML is occurring entirely local to the device," Federighi says. As an example, he cites app suggestions, the icons that appear when you swipe right.

The full-length article on Backchannel provides several more details about how machine learning and artificial intelligence work at Apple.

Top Rated Comments

Santabean2000 Avatar
63 months ago
Siri stills sucks for me... still doesn't work well enough to be useful.
Score: 32 Votes (Like | Disagree)
GfPQqmcRKUvP Avatar
63 months ago
Not in my experience. Siri is absolute trash compared to Google Now and Cortana. Apple should be embarrassed by how stagnant it has been and the repeated, correctable errors it makes. No amount of public relations whitewashing can fix it.
Score: 29 Votes (Like | Disagree)
Oblivious.Robot Avatar
63 months ago
Yeah, no.
Siri is still pretty much useless for me, although better than before, but still meh with her current capabilities.



Edit - For the screen brightness part, after some retries I found out that it does work, but you have to say it in one sentence as Siri forgets about it in the next. :confused:

Attachment Image
Score: 17 Votes (Like | Disagree)
keysofanxiety Avatar
63 months ago
Apple insists that much of the machine learning occurs entirely local to the device, without personal information being sent back to its servers.
And yet it still has no offline functionality, even if the learning is localised; well from what I can see, anyway. An iPhone 3GS managed offline voice functions like "call Tom". Why the heck can't Siri? Phone, text, open an app -- these are not commands that need to be bounced off a server.

It's fine if you're at Apple HQ, testing Siri on a 10Gb/s Internet connection. But when you're on the road with flaky 3G signal, it's the last thing you need.
Score: 11 Votes (Like | Disagree)
jsclem02 Avatar
63 months ago
The struggle with Siri is still very real.

Attachment Image
Score: 11 Votes (Like | Disagree)
thisisnotmyname Avatar
63 months ago
I like Siri on the iPhone when it works, but at times it totally gets things very wrong that it previously got right.

What really gets me ticked off is how Siri on ATV is worse than useless almost every time I try to make use of it.
I agree. On the phone she works well for me. On the AppleTV it seems like she goes to sleep and the first time I try to interact with her such as "what did he just say?" it lags so long that the ten second rewind is well past whatever I wanted to hear. And then there's the lack of Siri search through home sharing content...
Score: 7 Votes (Like | Disagree)

Top Stories

EEC Apple iphone 13

Apple Registers iPhone 13 Models in Eurasia Ahead of September Launch

Friday June 11, 2021 2:16 am PDT by
Nashville Chatter Class has discovered a new Russian-language regulatory filing in the Eurasian Economic Commission (EEC) database pointing towards several unreleased iPhone 13 models that Apple will be launching in the fall. Versions of iPhone running iOS 14 are listed with the model numbers A2628, A2630, A2634, A2635, A2640, A2643, and A2645. None of the numbers correspond to Apple's...
ios 15 home screen icons

iOS 15 Lets You Drag and Drop Images and Text Across Apps

Saturday June 12, 2021 3:17 pm PDT by
Apple this week previewed iOS 15, which is available now in beta for developers ahead of a public release later this year. One smaller but useful new feature added is the ability to drag and drop images, text, files, and more across apps on iPhone. MacStories editor-in-chief Federico Viticci demonstrated the new feature in a tweet: Using cross-app drag and drop on iPhone in iOS 15. Finally 🎉 #WW ...
ipad mini 6

Next iPad Mini Will Allegedly Feature Thinner Bezels, USB-C Port, and Touch ID Power Button

Friday June 11, 2021 1:13 pm PDT by
On his newly launched Front Page Tech website, leaker Jon Prosser has shared renders showing off the alleged design of the next-generation iPad mini, which he says are based on schematics, CAD files, and real images of the device. In line with details shared earlier this month by Bloomberg's Mark Gurman and Debby Wu, Prosser claims that the new iPad mini will feature slimmer bezels around...
iphone12protriplelenscamera

Apple's Orders for Key iPhone 13 Camera Component Expected to Outstrip Entire Android Market

Wednesday June 9, 2021 12:47 am PDT by
Major camera upgrades coming to the iPhone 13 series are putting increased pressure on suppliers to meet Apple's demand for key lens components, according to a new DigiTimes report. Apple has reportedly put Taiwan-based makers of voice coil motor (VCM) components on notice to increase their capacity by 30-40% in order to meet the company's demand, which is expected to outstrip the entire...
apple virtual game controller ios 15

Apple Makes New On-Screen Game Controller Available to Developers on iOS 15 and iPadOS 15

Saturday June 12, 2021 12:36 pm PDT by
During the Platforms State of the Union at WWDC this week, Apple unveiled a new API for iOS 15 and iPadOS 15 that enables developers to implement an on-screen virtual game controller in their iPhone and iPad games with just a few lines of code. While many iPhone and iPad games already offer on-screen controls, Apple's new virtual game controller is available to all developers, easy to add,...
macos monterey tidbits feature copy

macOS Monterey Tidbits: Animated Memoji on Login Screen, Change the Color of the Mouse Pointer, and More

Friday June 11, 2021 10:27 am PDT by
We've highlighted several new features coming in macOS Monterey, such as Low Power Mode and the option to erase a Mac without reinstalling the operating system, but there are some smaller tidbits that we wanted to share. Animated Memoji on Login Screen One small but fun new feature in macOS Monterey is the addition of a personalized Memoji on the login screen, complete with animated facial...
mr white ipod touch 5 protoype3

Unreleased iPod Touch 5 With Chamfered Edges and 30-Pin Dock Connector Shared Online

Thursday June 10, 2021 2:05 am PDT by
Occasional leaker Mr White has today shared interesting images on Twitter of what appears to be an old-school fifth-generation iPod touch prototype with chamfered edges and a brushed aluminum finish. The original iPod touch 5 that Apple released in October 2012 had a unibody anodized aluminum chassis with rounded edges, and was available in several colors, including slate. Another...
maxresdefault

Apple Promotes iPad Pro in New Ad With 'The Little Mermaid' Musical Spin

Saturday June 12, 2021 7:01 am PDT by
In a currently unlisted ad on YouTube, Apple is promoting the versatility, portability, and power of the M1 iPad Pro in a fun musical inspired by The Little Mermaid's "Part of Your World" soundtrack. In the ad, which features the main character using an M1 iPad Pro, Magic Keyboard, and Apple Pencil, multiple users can be seen struggling with their old PCs indoors while hoping that they can...
passwords system preferences

macOS Monterey Features Dedicated Password Section in System Preferences, Built-In Authenticator and More

Friday June 11, 2021 2:32 pm PDT by
macOS Monterey makes several improvements to password management, positioning iCloud Keychain as an ideal password service to replace third-party services like Lastpass and 1Password. In System Preferences, there's a new "Passwords" section that houses all of your iCloud Keychain logins and passwords so they're easier to get to, edit, and manage. There's a similar Passwords section that's...
Dark Sky App Featured

Dark Sky iOS App, Website, and API Now Scheduled to Remain Available Until End of 2022

Thursday June 10, 2021 7:34 am PDT by
Last year, Apple acquired the weather app Dark Sky, and shortly after its purchase, Apple shut down the app for Android. Despite the revamped iOS 15 Weather app taking heavy inspiration from Dark Sky, the weather's app standalone iOS app, web app, and API will remain available until the end of next year, compared to the end of this year, as previously planned. Dark Sky announced in an update ...