University Researchers Who Built a CSAM Scanning System Urge Apple to Not Use the 'Dangerous' Technology

Respected university researchers are sounding the alarm bells over the technology behind Apple's plans to scan iPhone users' photo libraries for CSAM, or child sexual abuse material, calling the technology "dangerous."

apple privacy
Jonanath Mayer, an assistant professor of computer science and public affairs at Princeton University, as well as Anunay Kulshrestha, a researcher at Princeton University Center for Information Technology Policy, both penned an op-ed for The Washington Post, outlining their experiences with building image detection technology.

The researchers started a project two years ago to identity CSAM in end-to-end encrypted online services. The researchers note that given their field, they "know the value of end-to-end encryption, which protects data from third-party access." That concern, they say, is what horrifies them over CSAM "proliferating on encrypted platforms."

Mayer and Kulshrestha said they wanted to find a middle ground for the situation: build a system that online platforms could use to find CSAM and protect end-to-end encryption. The researchers note that experts in the field doubted the prospect of such a system, but they did manage to build it and in the process noticed a significant problem.

We sought to explore a possible middle ground, where online services could identify harmful content while otherwise preserving end-to-end encryption. The concept was straightforward: If someone shared material that matched a database of known harmful content, the service would be alerted. If a person shared innocent content, the service would learn nothing. People couldn't read the database or learn whether content matched, since that information could reveal law enforcement methods and help criminals evade detection.

Knowledgeable observers argued a system like ours was far from feasible. After many false starts, we built a working prototype. But we encountered a glaring problem.

Since Apple's announcement of the feature, the company has been bombarded with concerns that the system behind detecting CSAM could be used to detect other forms of photos at the request of oppressive governments. Apple has strongly pushed back against such a possibility, saying it will refuse any such request from governments.

Nonetheless, concerns over the future implications of the technology being used for CSAM detection are widespread. Mayer and Kulshrestha said that their concerns over how governments could use the system to detect content other than CSAM had them "disturbed."

A foreign government could, for example, compel a service to out people sharing disfavored political speech. That's no hypothetical: WeChat, the popular Chinese messaging app, already uses content matching to identify dissident material. India enacted rules this year that could require pre-screening content critical of government policy. Russia recently fined Google, Facebook and Twitter for not removing pro-democracy protest materials.

We spotted other shortcomings. The content-matching process could have false positives, and malicious users could game the system to subject innocent users to scrutiny.

We were so disturbed that we took a step we hadn't seen before in computer science literature: We warned against our own system design, urging further research on how to mitigate the serious downsides....

Apple has continued to address user concerns over its plans, publishing additional documents and an FAQ page. Apple continues to believe that its CSAM detection system, which will occur on a user's device, aligns with its long-standing privacy values.

Popular Stories

Apple Glass

Apple Smart Glasses: Everything We Know So Far

Wednesday May 21, 2025 8:21 am PDT by
Google recently made waves by showcasing a set of lightweight smart glasses featuring deep Gemini integration and an optional in-lens display. The demo has reignited interest in Apple's own smart glasses project, which has been the subject of rumors for nearly a decade. Here's a recap of where things stand. Current Development Status Apple is actively working on new chips specifically...
Apple Glasses Purple Feature

Apple Smart Glasses Launching in 2026

Thursday May 22, 2025 12:22 pm PDT by
Apple is planning to launch a set of smart glasses by the end of 2026, reports Bloomberg. The glasses will be comparable to the Meta Ray-Bans and the Android XR glasses that Google showed off earlier this week. Apple's smart glasses are expected to include cameras, microphones, and AI capabilities, much like the Meta Ray-Bans. The glasses will be able to take photos, record video, provide...
Apple CarPlay Ultra instrument cluster themes 01

Apple's CarPlay Ultra Is Here – Does Your iPhone Support It?

Thursday May 15, 2025 5:17 am PDT by
Apple's recently announced CarPlay Ultra promises a deeply integrated in-car experience, but not all iPhone users will be able to take advantage of the new feature. According to Apple's press release, CarPlay Ultra requires an iPhone 12 or later running iOS 18.5 or later. This means if you're using an iPhone 11, iPhone XR, or any older model, you'll need to upgrade your device to access...
maxresdefault

OpenAI Buys Jony Ive's AI Startup to 'Completely Reimagine What It Means to Use a Computer'

Wednesday May 21, 2025 10:27 am PDT by
OpenAI is acquiring io, the hardware-based AI startup co-created by Jony Ive, OpenAI announced today. Ive has been working with OpenAI CEO Sam Altman on io for two years, and the duo expects to develop a family of AI devices. In a video shared by OpenAI, Altman and Ive outlined their partnership and what they expect to create as a result of the merger. "I have a growing sense that everything ...
WWDC 2025 Banner

Apple Announces WWDC 2025 Schedule, Including Keynote Time

Tuesday May 20, 2025 8:13 am PDT by
Apple today announced a more detailed schedule for its annual developers conference WWDC, which runs from June 9 through June 13. The schedule confirms that Apple's keynote will begin on Monday, June 9 at 10 a.m. Pacific Time, with a live stream to be available on Apple.com, in the Apple TV app, and on YouTube. During the keynote, Apple is expected to announce iOS 19, iPadOS 19, macOS 16,...
iPod shuffle generations

Kuo: Jony Ive's Futuristic OpenAI Device Like a Neck-Worn iPod Shuffle

Thursday May 22, 2025 8:05 am PDT by
The big news in the technology world this week is that ChatGPT maker OpenAI is working more closely with Apple's former design chief Jony Ive on a futuristic AI device. The company is remaining tight lipped about the device, but Apple supply chain analyst Ming-Chi Kuo has shared some alleged details about its design. In a social media post today, Kuo said the device will be "slightly larger" ...
macOS 16 visionOS Inspired Feature 1

macOS 16: Everything We Know So Far

Tuesday May 20, 2025 7:31 am PDT by
The Worldwide Developers Conference (WWDC), Apple's annual developer and software-oriented event, is less than three weeks away. We haven't heard a great deal about macOS 16 ahead of its announcement this year, so we could be in for some major surprises when June 9 rolls around. Here's what we know so far about the next major update to Apple's Mac operating system. macOS 16 Name? Every year ...

Top Rated Comments

Geert76 Avatar
49 months ago
Apple, just cancel this on device surveillance software
Score: 166 Votes (Like | Disagree)
nvmls Avatar
49 months ago
They clearly don't know how the technology works.. oh wait
Score: 125 Votes (Like | Disagree)
nexu Avatar
49 months ago
This is getting more and more disturbing
Score: 117 Votes (Like | Disagree)
TheRealNick Avatar
49 months ago
If they don’t cancel this I’m seriously going to have to look at alternative products, which saddens me.
Score: 108 Votes (Like | Disagree)
dwsolberg Avatar
49 months ago
It's time for Apple to admit it's made a mistake. It's seriously bizarre they ever thought this was a good idea.
Score: 95 Votes (Like | Disagree)
VulchR Avatar
49 months ago
Waiting for the surveillance apologists to argue these two researchers "don't understand" the process...
Score: 93 Votes (Like | Disagree)