Apple's Machine Learning Has Cut Siri's Error Rate by a Factor of Two

Steven Levy has published an in-depth article about Apple's artificial intelligence and machine learning efforts, after meeting with senior executives Craig Federighi, Eddy Cue, Phil Schiller, and two Siri scientists at the company's headquarters.

backchannel-apple-machine-learning
Apple provided Levy with a closer look at how machine learning is deeply integrated into Apple software and services, led by Siri, which the article reveals has been powered by a neural-net based system since 2014. Apple said the backend change greatly improved the personal assistant's accuracy.

"This was one of those things where the jump was so significant that you do the test again to make sure that somebody didn’t drop a decimal place," says Eddy Cue, Apple’s senior vice president of internet software and services.

Alex Acero, who leads the Siri speech team at Apple, said Siri's error rate has been lowered by more than a factor of two in many cases.

“The error rate has been cut by a factor of two in all the languages, more than a factor of two in many cases,” says Acero. “That’s mostly due to deep learning and the way we have optimized it — not just the algorithm itself but in the context of the whole end-to-end product.”

Acero told Levy he was able to work directly with Apple's silicon design team and the engineers who write the firmware for iOS devices to maximize performance of the neural network, and Federighi added that Apple building both hardware and software gives it an "incredible advantage" in the space.

"It's not just the silicon," adds Federighi. "It's how many microphones we put on the device, where we place the microphones. How we tune the hardware and those mics and the software stack that does the audio processing. It's all of those pieces in concert. It's an incredible advantage versus those who have to build some software and then just see what happens."

Apple's machine learning efforts extend far beyond Siri, as evidenced by several examples shared by Levy:

You see it when the phone identifies a caller who isn’t in your contact list (but did email you recently). Or when you swipe on your screen to get a shortlist of the apps that you are most likely to open next. Or when you get a reminder of an appointment that you never got around to putting into your calendar. Or when a map location pops up for the hotel you’ve reserved, before you type it in. Or when the phone points you to where you parked your car, even though you never asked it to. These are all techniques either made possible or greatly enhanced by Apple’s adoption of deep learning and neural nets.

Another product born out of machine learning is the Apple Pencil, which can detect the difference between a swipe, a touch, and a pencil input:

In order for Apple to include its version of a high-tech stylus, it had to deal with the fact that when people wrote on the device, the bottom of their hand would invariably brush the touch screen, causing all sorts of digital havoc. Using a machine learning model for “palm rejection” enabled the screen sensor to detect the difference between a swipe, a touch, and a pencil input with a very high degree of accuracy. “If this doesn’t work rock solid, this is not a good piece of paper for me to write on anymore — and Pencil is not a good product,” says Federighi. If you love your Pencil, thank machine learning.

On the iPhone, machine learning is enabled by a localized dynamic cache or "knowledge base" that Apple says is around 200MB in size, depending on how much personal information is stored.

This includes information about app usage, interactions with other people, neural net processing, a speech modeler, and "natural language event modeling." It also has data used for the neural nets that power object recognition, face recognition, and scene classification.

"It's a compact, but quite thorough knowledge base, with hundreds of thousands of locations and entities. We localize it because we know where you are," says Federighi. This knowledge base is tapped by all of Apple's apps, including the Spotlight search app, Maps, and Safari. It helps on auto-correct. "And it's working continuously in the background," he says.

Apple, for example, uses its neural network to capture the words iPhone users type using the standard QuickType keyboard.

Other information Apple stores on devices includes probably the most personal data that Apple captures: the words people type using the standard iPhone QuickType keyboard. By using a neural network-trained system that watches while you type, Apple can detect key events and items like flight information, contacts, and appointments — but information itself stays on your phone.

Apple insists that much of the machine learning occurs entirely local to the device, without personal information being sent back to its servers.

"Some people perceive that we can't do these things with AI because we don't have the data," says Cue. "But we have found ways to get that data we need while still maintaining privacy. That's the bottom line."

"We keep some of the most sensitive things where the ML is occurring entirely local to the device," Federighi says. As an example, he cites app suggestions, the icons that appear when you swipe right.

The full-length article on Backchannel provides several more details about how machine learning and artificial intelligence work at Apple.

Top Rated Comments

(View all)
Avatar
51 months ago
Siri stills sucks for me... still doesn't work well enough to be useful.
Score: 32 Votes (Like | Disagree)
Avatar
51 months ago
Not in my experience. Siri is absolute trash compared to Google Now and Cortana. Apple should be embarrassed by how stagnant it has been and the repeated, correctable errors it makes. No amount of public relations whitewashing can fix it.
Score: 29 Votes (Like | Disagree)
Avatar
51 months ago
Yeah, no.
Siri is still pretty much useless for me, although better than before, but still meh with her current capabilities.



Edit - For the screen brightness part, after some retries I found out that it does work, but you have to say it in one sentence as Siri forgets about it in the next. :confused:

Score: 17 Votes (Like | Disagree)
Avatar
51 months ago

Apple insists that much of the machine learning occurs entirely local to the device, without personal information being sent back to its servers.

And yet it still has no offline functionality, even if the learning is localised; well from what I can see, anyway. An iPhone 3GS managed offline voice functions like "call Tom". Why the heck can't Siri? Phone, text, open an app -- these are not commands that need to be bounced off a server.

It's fine if you're at Apple HQ, testing Siri on a 10Gb/s Internet connection. But when you're on the road with flaky 3G signal, it's the last thing you need.
Score: 11 Votes (Like | Disagree)
Avatar
51 months ago
The struggle with Siri is still very real.

Score: 11 Votes (Like | Disagree)
Avatar
51 months ago

I like Siri on the iPhone when it works, but at times it totally gets things very wrong that it previously got right.

What really gets me ticked off is how Siri on ATV is worse than useless almost every time I try to make use of it.

I agree. On the phone she works well for me. On the AppleTV it seems like she goes to sleep and the first time I try to interact with her such as "what did he just say?" it lags so long that the ten second rewind is well past whatever I wanted to hear. And then there's the lack of Siri search through home sharing content...
Score: 7 Votes (Like | Disagree)

Top Stories

Hands On With iPhone 12 Models Showing New Sizes and Design

Monday July 6, 2020 2:04 pm PDT by
Ahead of the launch of new iPhones we often see dummy models created based on leaked schematics and specifications, with those models designed to let case makers create cases for the new devices ahead of their release. We got our hands on a set of dummy models that represent the iPhone 12 lineup, giving us our first close look at the iPhone 4-style design and the different size options. Subscri ...

Everything New in iOS 14 Beta 2: New Calendar Icon, Files Widget and More

Tuesday July 7, 2020 11:38 am PDT by
Apple today released the second beta of iOS 14 to developers for testing purposes, tweaking and refining some of the features that are coming in the update. Below, we've rounded up all of the changes that we found in the second beta. - Calendar icon - There's a new Calendar app icon in iOS 14 beta 2, with the day of the week abbreviated rather than spelled out. - Clock icon - The clock...

iPhone 12 Sizes Compared with iPhone SE, 7, 8, SE 2, X, 11, 11 Pro and 11 Pro Max [Update]

Tuesday July 7, 2020 6:49 pm PDT by
Apple is planning on launching the iPhone 12 this fall which is rumored to be coming in 3 different sizes: 5.4", 6.1" and 6.7". The middle size (6.1") matches up with the currently shipping iPhone 11, but the other two sizes will be entirely new. Over the weekend, there was some excitement about how well the new 5.4" iPhone 12 compares to the original iPhone SE. Those who have been hoping...

Tom Hanks Discusses 'Heartbreaking' Shift of WWII Film Greyhound From Theatrical Blockbuster to Apple TV+ Exclusive

Monday July 6, 2020 7:53 am PDT by
Tom Hanks' WWII drama "Greyhound" is set to premiere on Apple TV+ this Friday, July 10, and ahead of that debut the actor gave an interview with The Guardian discussing the film. "Greyhound" was originally planned to see a theatrical release this summer, and was repeatedly delayed in the wake of the ongoing Covid-19 pandemic. Apple won the streaming rights to the film, and in the new...

Developer's Visual Comparison of macOS Catalina and Big Sur Offers Closer Look at Apple's UI Redesign for Macs

Tuesday July 7, 2020 4:00 am PDT by
macOS 11 Big Sur is the next major release of Apple's operating system for Mac, and following its preview at WWDC, one of the biggest discussions has revolved around the all-new user interface redesign. Developers are still learning what the impact the new UI will have on their apps, and with that in mind, app designer Andrew Denty has compiled an extensive visual comparison of the user...

5.4-Inch iPhone 12 Model Size Compared to Original iPhone SE and iPhone 7

Saturday July 4, 2020 9:44 pm PDT by
iPhone 12 dummy models based on leaked schematics have been starting to circulate online and in online marketplaces. Not happy with the circulating size comparisons between the rumored 5.4" iPhone 12 and the original iPhone SE models, MacRumors forum user iZac took matters into his own hands and purchased his own 5.4" dummy model to provide more detailed size comparisons between the original...

Shipping Estimates for 27-Inch iMac Continue to Slip, Now Into September

Monday July 6, 2020 6:55 am PDT by
Amid rumors and hints of a forthcoming update for the iMac, supplies of Apple's current 27-inch iMac continue to dwindle with mid- and high-end stock configurations now seeing shipping estimates pushed back into September. The 27-inch iMac has seen tight supplies and extended shipping estimates for months now, but the situation has been gradually worsening to the point where new buyers can...

Apple Cuts iPhone Trade-In Values as iPhone 12 Launch Nears

Tuesday July 7, 2020 7:46 am PDT by
With just two months to go until the usual timeframe for Apple's iPhone launch events, Apple is cutting back on maximum trade-in values of previous-generation iPhones for those looking to upgrade to a new model. Maximum values on more recent models have dropped by $30–$50, while older models have generally dropped by $5–$20 with a few models seeing no change in value.iPhone XS Max: $500 to...

Analyst Believes iPhone 12 Pricing Will Start $50 Higher Even Without EarPods or Charger in Box

Wednesday July 8, 2020 9:35 am PDT by
Despite multiple reports indicating that Apple will not include EarPods or a wall charger with iPhone 12 models this year, one analyst believes that pricing will still increase slightly compared to the iPhone 11 lineup. In a research note provided to MacRumors, analyst Jeff Pu forecasted that iPhone 12 pricing will start at $749 for a new 5.4-inch model, an increase of $50 over the base...

14-Inch MacBook Pro With Mini-LED Display Expected to Enter Production in 2021

Wednesday July 8, 2020 7:51 am PDT by
Apple suppliers will begin competing to win manufacturing orders for new 14-inch and 16-inch MacBook Pro models with Mini-LED displays in the first quarter of 2021, according to Taiwanese research firm TrendForce. Rumors of a 14-inch MacBook Pro have surfaced since Apple replaced the 15-inch MacBook Pro with a new 16-inch model last year. Apple analyst Ming-Chi Kuo has previously said that...