Apple Introducing New Child Safety Features, Including Scanning Users' Photo Libraries for Known Sexual Abuse Material

Apple today previewed new child safety features that will be coming to its platforms with software updates later this year. The company said the features will be available in the U.S. only at launch and will be expanded to other regions over time.

iphone communication safety feature

Communication Safety

First, the Messages app on the iPhone, iPad, and Mac will be getting a new Communication Safety feature to warn children and their parents when receiving or sending sexually explicit photos. Apple said the Messages app will use on-device machine learning to analyze image attachments, and if a photo is determined to be sexually explicit, the photo will be automatically blurred and the child will be warned.

When a child attempts to view a photo flagged as sensitive in the Messages app, they will be alerted that the photo may contain private body parts, and that the photo may be hurtful. Depending on the age of the child, there will also be an option for parents to receive a notification if their child proceeds to view the sensitive photo or if they choose to send a sexually explicit photo to another contact after being warned.

Apple said the new Communication Safety feature will be coming in updates to iOS 15, iPadOS 15 and macOS Monterey later this year for accounts set up as families in iCloud. Apple ensured that iMessage conversations will remain protected with end-to-end encryption, making private communications unreadable by Apple.

Scanning Photos for Child Sexual Abuse Material (CSAM)

Second, starting this year with iOS 15 and iPadOS 15, Apple will be able to detect known Child Sexual Abuse Material (CSAM) images stored in iCloud Photos, enabling Apple to report these instances to the National Center for Missing and Exploited Children (NCMEC), a non-profit organization that works in collaboration with U.S. law enforcement agencies.

Apple said its method of detecting known CSAM is designed with user privacy in mind. Instead of scanning images in the cloud, Apple said the system will perform on-device matching against a database of known CSAM image hashes provided by the NCMEC and other child safety organizations. Apple said it will further transform this database into an unreadable set of hashes that is securely stored on users' devices.

The hashing technology, called NeuralHash, analyzes an image and converts it to a unique number specific to that image, according to Apple.

"The main purpose of the hash is to ensure that identical and visually similar images result in the same hash, while images that are different from one another result in different hashes," said Apple in a new "Expanded Protections for Children" white paper. "For example, an image that has been slightly cropped, resized or converted from color to black and white is treated identical to its original, and has the same hash."

apple csam flow chart
Before an image is stored in iCloud Photos, Apple said an on-device matching process is performed for that image against the unreadable set of known CSAM hashes. If there is a match, the device creates a cryptographic safety voucher. This voucher is uploaded to iCloud Photos along with the image, and once an undisclosed threshold of matches is exceeded, Apple is able to interpret the contents of the vouchers for CSAM matches. Apple then manually reviews each report to confirm there is a match, disables the user's iCloud account, and sends a report to NCMEC. Apple is not sharing what its exact threshold is, but ensures an "extremely high level of accuracy" that accounts are not incorrectly flagged.

Apple said its method of detecting known CSAM provides "significant privacy benefits" over existing techniques:

• This system is an effective way to identify known CSAM stored in iCloud Photos accounts while protecting user privacy.
• As part of the process, users also can't learn anything about the set of known CSAM images that is used for matching. This protects the contents of the database from malicious use.
• The system is very accurate, with an extremely low error rate of less than one in one trillion account per year.
• The system is significantly more privacy-preserving than cloud-based scanning, as it only reports users who have a collection of known CSAM stored in iCloud Photos.

The underlying technology behind Apple's system is quite complex and it has published a technical summary with more details.

"Apple's expanded protection for children is a game changer. With so many people using Apple products, these new safety measures have lifesaving potential for children who are being enticed online and whose horrific images are being circulated in child sexual abuse material," said John Clark, the President and CEO of the National Center for Missing & Exploited Children. "At the National Center for Missing & Exploited Children we know this crime can only be combated if we are steadfast in our dedication to protecting children. We can only do this because technology partners, like Apple, step up and make their dedication known. The reality is that privacy and child protection can co-exist. We applaud Apple and look forward to working together to make this world a safer place for children."

Expanded CSAM Guidance in Siri and Search

iphone csam siri
Third, Apple said it will be expanding guidance in Siri and Spotlight Search across devices by providing additional resources to help children and parents stay safe online and get help with unsafe situations. For example, users who ask Siri how they can report CSAM or child exploitation will be pointed to resources for where and how to file a report.

The updates to Siri and Search are coming later this year in an update to iOS 15, iPadOS 15, watchOS 8, and macOS Monterey, according to Apple.

Note: Due to the political or social nature of the discussion regarding this topic, the discussion thread is located in our Political News forum. All forum members and site visitors are welcome to read and follow the thread, but posting is limited to forum members with at least 100 posts.

Top Rated Comments

levitynyc Avatar
35 months ago
Not gonna lie...that's kinda creepy.
Score: 71 Votes (Like | Disagree)
Exponent Avatar
35 months ago
No, too far, Apple.

What is going to keep you from scanning my library for NeuralHash matches against politics you don’t like? Or criticism of mainland dictatorial China?

if that doesn’t happen in the US, what will keep other countries (read above) from doing just that to their citizens?
Score: 58 Votes (Like | Disagree)
gaximus Avatar
35 months ago
What about photos of "Baby's first bath" will those users get treated as child exploitation?
Score: 56 Votes (Like | Disagree)
arn Avatar
35 months ago

Yeh good luck if say you have small young kids who don't keep their clothes on. Like what,t every baby?

This is also creepy asf sorry. Child predators are bad, obviously, but this isn't the way.
The CSAM thing doesn't detect/determine content of images. It checks photos against a database of specific (actively circulating) child abuse images.

Not to say there aren't legitimate concerns, but worrying that it is going to somehow flag your own kid's photos is not one of them.

(The child safety thing does detect, but seems the worst that does is through up a warning/blurring if you have it on)
Score: 44 Votes (Like | Disagree)
fenderbass146 Avatar
35 months ago
every year we go to 1984 a little more.
Score: 37 Votes (Like | Disagree)
Apple_Robert Avatar
35 months ago
I am against child abuse and child porn. However, I don't agree with what Apple is going to be doing with user phones.
Score: 34 Votes (Like | Disagree)

Popular Stories

iPhone 16 Camera Lozenge 2 Colors

iPhone 16 Plus Rumored to Come in These 7 Colors

Wednesday April 10, 2024 3:52 am PDT by
Apple's iPhone 16 Plus may come in seven colors that either build upon the existing five colors in the standard iPhone 15 lineup or recast them in a new finish, based on a new rumor out of China. According to the Weibo-based leaker Fixed focus digital, Apple's upcoming larger 6.7-inch iPhone 16 Plus model will come in the following colors, compared to the colors currently available for the...
apple tv 4k yellow bg feature

When to Expect a New Apple TV to Launch

Tuesday April 9, 2024 8:30 am PDT by
It has been nearly a year and a half since the current Apple TV was released, so the device is becoming due for a hardware upgrade. Below, we recap rumors about the next Apple TV, including potential features and launch timing. The current model is the third-generation Apple TV 4K, announced in October 2022. Key new features compared to the previous model from 2021 include a faster A15...
apple silicon feature joeblue

Macs to Get AI-Focused M4 Chips Starting in Late 2024

Thursday April 11, 2024 10:10 am PDT by
Apple will begin updating its Mac lineup with M4 chips in late 2024, according to Bloomberg's Mark Gurman. The M4 chip will be focused on improving performance for artificial intelligence capabilities. Last year, Apple introduced the M3, M3 Pro, and M3 Max chips all at once in October, so it's possible we could see the M4 lineup come during the same time frame. Gurman says that the entire...
iOS 18 WWDC 24 Feature 2

iOS 18 May Feature All-New 'Safari Browsing Assistant'

Wednesday April 10, 2024 6:11 am PDT by
iOS 18 will apparently feature a new Safari browsing assistant, according to backend code on Apple's servers discovered by Nicolás Álvarez. MacRumors contributor Aaron Perris confirmed that the code exists, but not many details are known at this time. Álvarez said it seems like the browsing assistant will use iCloud Private Relay's infrastructure to send relevant data to Apple in a...
maxresdefault

Review: Six Months With the iPhone 15 Pro

Wednesday April 10, 2024 10:53 am PDT by
It's been a bit over six months since the iPhone 15 lineup came out in September, and MacRumors videographer Dan Barbera has been using an iPhone 15 Pro Max sans case since launch. Over on our YouTube channel, Dan did a long term review to demo how his phone has held up and his thoughts on the Action button, battery life, and camera features. Subscribe to the MacRumors YouTube channel for more ...
iPhone 16 Pro Sizes Feature

Alleged iPhone 16 Battery Details Show Smaller Capacity for One Model

Tuesday April 9, 2024 3:46 am PDT by
Apple's upcoming iPhone 16 lineup will feature bigger battery capacities compared to previous-generation models with the exception of the iPhone 16 Plus, which will have a smaller battery than its predecessor. That's according to the Chinese Weibo-based leaker OvO Baby Sauce OvO, a relatively new source of supply chain leaks with an as-yet unproven track record for accuracy. The iPhone 16 ...