University Researchers Who Built a CSAM Scanning System Urge Apple to Not Use the 'Dangerous' Technology

Respected university researchers are sounding the alarm bells over the technology behind Apple's plans to scan iPhone users' photo libraries for CSAM, or child sexual abuse material, calling the technology "dangerous."

apple privacy
Jonanath Mayer, an assistant professor of computer science and public affairs at Princeton University, as well as Anunay Kulshrestha, a researcher at Princeton University Center for Information Technology Policy, both penned an op-ed for The Washington Post, outlining their experiences with building image detection technology.

The researchers started a project two years ago to identity CSAM in end-to-end encrypted online services. The researchers note that given their field, they "know the value of end-to-end encryption, which protects data from third-party access." That concern, they say, is what horrifies them over CSAM "proliferating on encrypted platforms."

Mayer and Kulshrestha said they wanted to find a middle ground for the situation: build a system that online platforms could use to find CSAM and protect end-to-end encryption. The researchers note that experts in the field doubted the prospect of such a system, but they did manage to build it and in the process noticed a significant problem.

We sought to explore a possible middle ground, where online services could identify harmful content while otherwise preserving end-to-end encryption. The concept was straightforward: If someone shared material that matched a database of known harmful content, the service would be alerted. If a person shared innocent content, the service would learn nothing. People couldn't read the database or learn whether content matched, since that information could reveal law enforcement methods and help criminals evade detection.

Knowledgeable observers argued a system like ours was far from feasible. After many false starts, we built a working prototype. But we encountered a glaring problem.

Since Apple's announcement of the feature, the company has been bombarded with concerns that the system behind detecting CSAM could be used to detect other forms of photos at the request of oppressive governments. Apple has strongly pushed back against such a possibility, saying it will refuse any such request from governments.

Nonetheless, concerns over the future implications of the technology being used for CSAM detection are widespread. Mayer and Kulshrestha said that their concerns over how governments could use the system to detect content other than CSAM had them "disturbed."

A foreign government could, for example, compel a service to out people sharing disfavored political speech. That's no hypothetical: WeChat, the popular Chinese messaging app, already uses content matching to identify dissident material. India enacted rules this year that could require pre-screening content critical of government policy. Russia recently fined Google, Facebook and Twitter for not removing pro-democracy protest materials.

We spotted other shortcomings. The content-matching process could have false positives, and malicious users could game the system to subject innocent users to scrutiny.

We were so disturbed that we took a step we hadn't seen before in computer science literature: We warned against our own system design, urging further research on how to mitigate the serious downsides....

Apple has continued to address user concerns over its plans, publishing additional documents and an FAQ page. Apple continues to believe that its CSAM detection system, which will occur on a user's device, aligns with its long-standing privacy values.

Popular Stories

App Store vs EU Feature 2

Apple Says It Doesn't Approve of EU Porn App

Monday February 3, 2025 1:15 pm PST by
Apple does not approve of the "Hot Tub" pornography app that was released for the iPhone in the EU using alternative app distribution, Apple said in a statement to MacRumors. Further, Apple is concerned about the potential user safety risks with a pornography app, and says that it undermines consumer trust in the Apple ecosystem. We are deeply concerned about the safety risks that hardcore...
iOS 18

iOS 18.4 Will Include These New Features for Your iPhone

Wednesday February 5, 2025 7:15 am PST by
iOS 18.3 was released last month, so the first iOS 18.4 beta should be coming soon. iOS 18.4 is expected to be a more substantial update for the iPhone, with several new features and changes related to Apple Intelligence and beyond. Apple's website suggests that iOS 18.4 will be released in April, following beta testing. Below, we outline what to expect from the update so far. Apple...
General Apple Invites Feature

Apple Launches New 'Invites' App

Tuesday February 4, 2025 8:00 am PST by
Apple today announced the launch of a new app called "Invites," which is designed to allow users to plan events like birthday parties, graduations, vacations, baby showers, and more. "With Apple Invites, an event comes to life from the moment the invitation is created, and users can share lasting memories even after they get together," said Brent Chiu-Watson, Apple's senior director of...
maxresdefault

An Apple TV Refresh is Coming in 2025 - Here's What You Should Know

Wednesday February 5, 2025 10:17 am PST by
Apple hasn't refreshed the Apple TV since 2022, but rumors suggest that we're finally going to get an update in 2025. We don't have a full picture of what to expect yet, but we have some hints on what's coming. Subscribe to the MacRumors YouTube channel for more videos. Updated A-Series Chip The current Apple TV 4K uses the A15 Bionic chip that was in the iPhone 13 lineup, and it's time for...
applecare apple care banner

Apple Raises Monthly AppleCare+ Subscription Price for All iPhones

Tuesday February 4, 2025 9:35 am PST by
Apple this week increased the prices for its monthly AppleCare+ subscription prices for the iPhone, raising the cost by 50 cents for all models in the United States. Standard AppleCare+ for the iPhone 16 models is now priced at $10.49 per month, for example, up from the prior $9.99 per month price. The 50 cent price increase applies to all available AppleCare+ plans for Apple's current...
iCloud General Feature Redux

'Apple Invites' Leaked on iCloud Website

Tuesday February 4, 2025 7:11 am PST by
Update: The new Apple Invites app has officially been announced. The main iCloud.com page has seemingly confirmed Apple's rumored invites tool, which has yet to be officially announced by the company. The page says "Apple Invites" will be an iCloud+ feature:Upgrade to iCloud+ to get more storage, plan events with Apple Invites, and have peace of mind with privacy features like iCloud...
apple power beats pro 2

Apple Expected to Announce Powerbeats Pro 2 on February 11 With These New Features

Sunday February 2, 2025 6:15 am PST by
Apple previously teased that Powerbeats Pro 2 would be released in 2025, and now an announcement date has leaked. Bloomberg's Mark Gurman today said Apple plans to unveil the wireless earbuds on Tuesday, February 11. Powerbeats Pro 2 will be priced at $250 in the U.S., he said. Powerbeats Pro are a sportier, fitness-focused alternative to AirPods Pro with built-in, adjustable ear hooks...
maxresdefault

Testing Apple's New Invites App for Event Planning

Tuesday February 4, 2025 10:26 am PST by
Apple today surprised us with a new Invites app, which is designed for planning events like birthday parties, vacations, and baby showers. We checked it out in our latest video to see how it works, what you can use it for, and to demonstrate all of the different features in the app. Subscribe to the MacRumors YouTube channel for more videos. Invites is a standalone app that you can download...

Top Rated Comments

Geert76 Avatar
45 months ago
Apple, just cancel this on device surveillance software
Score: 166 Votes (Like | Disagree)
nvmls Avatar
45 months ago
They clearly don't know how the technology works.. oh wait
Score: 125 Votes (Like | Disagree)
nexu Avatar
45 months ago
This is getting more and more disturbing
Score: 117 Votes (Like | Disagree)
TheRealNick Avatar
45 months ago
If they don’t cancel this I’m seriously going to have to look at alternative products, which saddens me.
Score: 108 Votes (Like | Disagree)
dwsolberg Avatar
45 months ago
It's time for Apple to admit it's made a mistake. It's seriously bizarre they ever thought this was a good idea.
Score: 95 Votes (Like | Disagree)
VulchR Avatar
45 months ago
Waiting for the surveillance apologists to argue these two researchers "don't understand" the process...
Score: 93 Votes (Like | Disagree)