Apple's Machine Learning Has Cut Siri's Error Rate by a Factor of Two

Steven Levy has published an in-depth article about Apple's artificial intelligence and machine learning efforts, after meeting with senior executives Craig Federighi, Eddy Cue, Phil Schiller, and two Siri scientists at the company's headquarters.

backchannel-apple-machine-learning
Apple provided Levy with a closer look at how machine learning is deeply integrated into Apple software and services, led by Siri, which the article reveals has been powered by a neural-net based system since 2014. Apple said the backend change greatly improved the personal assistant's accuracy.

"This was one of those things where the jump was so significant that you do the test again to make sure that somebody didn’t drop a decimal place," says Eddy Cue, Apple’s senior vice president of internet software and services.

Alex Acero, who leads the Siri speech team at Apple, said Siri's error rate has been lowered by more than a factor of two in many cases.

“The error rate has been cut by a factor of two in all the languages, more than a factor of two in many cases,” says Acero. “That’s mostly due to deep learning and the way we have optimized it — not just the algorithm itself but in the context of the whole end-to-end product.”

Acero told Levy he was able to work directly with Apple's silicon design team and the engineers who write the firmware for iOS devices to maximize performance of the neural network, and Federighi added that Apple building both hardware and software gives it an "incredible advantage" in the space.

"It's not just the silicon," adds Federighi. "It's how many microphones we put on the device, where we place the microphones. How we tune the hardware and those mics and the software stack that does the audio processing. It's all of those pieces in concert. It's an incredible advantage versus those who have to build some software and then just see what happens."

Apple's machine learning efforts extend far beyond Siri, as evidenced by several examples shared by Levy:

You see it when the phone identifies a caller who isn’t in your contact list (but did email you recently). Or when you swipe on your screen to get a shortlist of the apps that you are most likely to open next. Or when you get a reminder of an appointment that you never got around to putting into your calendar. Or when a map location pops up for the hotel you’ve reserved, before you type it in. Or when the phone points you to where you parked your car, even though you never asked it to. These are all techniques either made possible or greatly enhanced by Apple’s adoption of deep learning and neural nets.

Another product born out of machine learning is the Apple Pencil, which can detect the difference between a swipe, a touch, and a pencil input:

In order for Apple to include its version of a high-tech stylus, it had to deal with the fact that when people wrote on the device, the bottom of their hand would invariably brush the touch screen, causing all sorts of digital havoc. Using a machine learning model for “palm rejection” enabled the screen sensor to detect the difference between a swipe, a touch, and a pencil input with a very high degree of accuracy. “If this doesn’t work rock solid, this is not a good piece of paper for me to write on anymore — and Pencil is not a good product,” says Federighi. If you love your Pencil, thank machine learning.

On the iPhone, machine learning is enabled by a localized dynamic cache or "knowledge base" that Apple says is around 200MB in size, depending on how much personal information is stored.

This includes information about app usage, interactions with other people, neural net processing, a speech modeler, and "natural language event modeling." It also has data used for the neural nets that power object recognition, face recognition, and scene classification.

"It's a compact, but quite thorough knowledge base, with hundreds of thousands of locations and entities. We localize it because we know where you are," says Federighi. This knowledge base is tapped by all of Apple's apps, including the Spotlight search app, Maps, and Safari. It helps on auto-correct. "And it's working continuously in the background," he says.

Apple, for example, uses its neural network to capture the words iPhone users type using the standard QuickType keyboard.

Other information Apple stores on devices includes probably the most personal data that Apple captures: the words people type using the standard iPhone QuickType keyboard. By using a neural network-trained system that watches while you type, Apple can detect key events and items like flight information, contacts, and appointments — but information itself stays on your phone.

Apple insists that much of the machine learning occurs entirely local to the device, without personal information being sent back to its servers.

"Some people perceive that we can't do these things with AI because we don't have the data," says Cue. "But we have found ways to get that data we need while still maintaining privacy. That's the bottom line."

"We keep some of the most sensitive things where the ML is occurring entirely local to the device," Federighi says. As an example, he cites app suggestions, the icons that appear when you swipe right.

The full-length article on Backchannel provides several more details about how machine learning and artificial intelligence work at Apple.

Top Rated Comments

Santabean2000 Avatar
60 months ago
Siri stills sucks for me... still doesn't work well enough to be useful.
Score: 32 Votes (Like | Disagree)
GfPQqmcRKUvP Avatar
60 months ago
Not in my experience. Siri is absolute trash compared to Google Now and Cortana. Apple should be embarrassed by how stagnant it has been and the repeated, correctable errors it makes. No amount of public relations whitewashing can fix it.
Score: 29 Votes (Like | Disagree)
Oblivious.Robot Avatar
60 months ago
Yeah, no.
Siri is still pretty much useless for me, although better than before, but still meh with her current capabilities.



Edit - For the screen brightness part, after some retries I found out that it does work, but you have to say it in one sentence as Siri forgets about it in the next. :confused:

Attachment Image
Score: 17 Votes (Like | Disagree)
keysofanxiety Avatar
60 months ago
Apple insists that much of the machine learning occurs entirely local to the device, without personal information being sent back to its servers.
And yet it still has no offline functionality, even if the learning is localised; well from what I can see, anyway. An iPhone 3GS managed offline voice functions like "call Tom". Why the heck can't Siri? Phone, text, open an app -- these are not commands that need to be bounced off a server.

It's fine if you're at Apple HQ, testing Siri on a 10Gb/s Internet connection. But when you're on the road with flaky 3G signal, it's the last thing you need.
Score: 11 Votes (Like | Disagree)
jsclem02 Avatar
60 months ago
The struggle with Siri is still very real.

Attachment Image
Score: 11 Votes (Like | Disagree)
thisisnotmyname Avatar
60 months ago
I like Siri on the iPhone when it works, but at times it totally gets things very wrong that it previously got right.

What really gets me ticked off is how Siri on ATV is worse than useless almost every time I try to make use of it.
I agree. On the phone she works well for me. On the AppleTV it seems like she goes to sleep and the first time I try to interact with her such as "what did he just say?" it lags so long that the ten second rewind is well past whatever I wanted to hear. And then there's the lack of Siri search through home sharing content...
Score: 7 Votes (Like | Disagree)

Top Stories

samsung experience 1

Samsung's 'iTest' Lets You Try a Galaxy Device on Your iPhone

Thursday April 8, 2021 12:42 pm PDT by
Samsung has launched "iTest," an interactive website experience that's designed to allow iPhone users to test out Android on a Galaxy device, or "sample the other side," as Samsung puts it. Subscribe to the MacRumors YouTube channel for more videos. The iTest website is being advertised in New Zealand, according to a MacRumors reader who came across the feature. Visiting the iTest website on...
sonny 2021 ipad mini pro dummies

Leaked Dummy Units Show iPad Mini 6 With Thick Bezels and Home Button, New iPad Pro Models

Thursday April 8, 2021 2:11 am PDT by
Rumors suggest Apple will release refreshed versions of the iPad mini and iPad Pro models in the first half of this year, potentially as soon as this month, and a new leak today has provided us with a possible preview of what to expect in terms of the devices' overall design and camera prospects. Tech leaker and Apple blogger Sonny Dickson this morning shared images on Twitter showing dummy ...
apple music for artists new icon

Apple's Revamped Apple Music for Artists Icon Leads to Speculation About iOS 15 Design Plans

Tuesday April 6, 2021 1:03 pm PDT by
Apple yesterday updated its Apple Music for Artists app with some minor bug fixes and improvements, but also one other notable change -- a new icon. New icon on the right The Apple Music for Artists app now features a simpler, streamlined icon with a pinkish red music logo rather than the multicolored logo that was used before. The icon also has an embossed look that makes it stand out from...
apple find my network

Apple Announces Find My Network With Support for Third-Party Devices

Wednesday April 7, 2021 10:06 am PDT by
Apple today announced the launch of its Find My network accessory program, which is designed to allow third-party Bluetooth devices to be tracked in the Find My app right alongside your Apple devices. According to Apple, the first accessory companies to take advantage of the new Find My integration include Belkin, Chipolo, and VanMoof, with devices set to be available beginning next week. ...
new m1 chip

M1 Mac RAM and SSD Upgrades Found to Be Possible After Purchase

Tuesday April 6, 2021 5:34 am PDT by
Technicians in China have reportedly succeeded in upgrading the memory and storage of the M1 chip, suggesting that Apple's integrated custom silicon for the Mac may be more flexible than previously thought. Reports of maintenance technicians being able to expand the memory and storage of M1 Macs began circulating on Chinese social media over the weekend, but now international reports have...
Intel MBP Is Thin and Lighjt

Intel Ad for 'World's Best Processor' Features a MacBook Pro

Wednesday April 7, 2021 9:51 am PDT by
Intel has been on a relentless marketing drive against Mac computers in recent weeks, positioning them as inferior to Windows laptops powered by Intel processors. In a slight slip-up, however, Intel has accidentally used a MacBook instead of a Windows laptop in one of its newest ads to promote one of its new 11th-generation chips as "the world's best processor." The ad appeared on Reddit and ...
tmobile 5g modem

T-Mobile Launches Unlimited 5G Home Internet for $60/Month

Wednesday April 7, 2021 2:18 pm PDT by
T-Mobile today hosted an Un-carrier event where the company announced the launch of a a new 5G home internet plan, which is priced at $60 per month and offers unlimited data. The service is available to more than 30 million Americans across much of the United States, including 10 million households in rural areas not typically able to access reliable broadband. Connectivity will be either 4G ...
iMessage Android featured

Apple's Rationale for Not Bringing iMessage to Android Revealed in Legal Documents

Friday April 9, 2021 2:22 am PDT by
It's no secret that Apple sees iMessage as a big enough selling point to keep the service exclusive to Apple devices, however new court filings submitted by Epic Games in its ongoing lawsuit with the company reveal just how Apple executives have rationalized their decision not to develop a version of iMessage for Android. Apple clearly recognizes the power that iMessage has to keep users...
14

Apple Seeds Seventh Betas of iOS 14.5 and iPadOS 14.5 to Developers [Update: Public Beta Available]

Wednesday April 7, 2021 10:04 am PDT by
Apple today seeded the seventh betas of upcoming iOS 14.5 and iPadOS 14.5 updates to developers for testing purposes, with the new beta updates coming one week after Apple released the sixth iOS and iPadOS 14.5 betas. iOS and iPadOS 14.5 can be downloaded through the Apple Developer Center or over the air after the proper profile has been installed on an iPhone or iPad. iOS 14.5 is the...
ipad pro and macbook pro

iPad and MacBook Production Reportedly Delayed Due to Global Chip Shortage

Thursday April 8, 2021 2:31 am PDT by
Apple is facing a global shortage of certain components for some of its MacBook and iPad models, causing the Cupertino tech giant and its suppliers to postpone production of the products, according to a new report from Nikkei Asia. According to the report, MacBook production is being hindered due to the shortage of chips mounted onto the circuit board before final assembly, which is a key...