University Researchers Who Built a CSAM Scanning System Urge Apple to Not Use the 'Dangerous' Technology

Respected university researchers are sounding the alarm bells over the technology behind Apple's plans to scan iPhone users' photo libraries for CSAM, or child sexual abuse material, calling the technology "dangerous."

apple privacy
Jonanath Mayer, an assistant professor of computer science and public affairs at Princeton University, as well as Anunay Kulshrestha, a researcher at Princeton University Center for Information Technology Policy, both penned an op-ed for The Washington Post, outlining their experiences with building image detection technology.

The researchers started a project two years ago to identity CSAM in end-to-end encrypted online services. The researchers note that given their field, they "know the value of end-to-end encryption, which protects data from third-party access." That concern, they say, is what horrifies them over CSAM "proliferating on encrypted platforms."

Mayer and Kulshrestha said they wanted to find a middle ground for the situation: build a system that online platforms could use to find CSAM and protect end-to-end encryption. The researchers note that experts in the field doubted the prospect of such a system, but they did manage to build it and in the process noticed a significant problem.

We sought to explore a possible middle ground, where online services could identify harmful content while otherwise preserving end-to-end encryption. The concept was straightforward: If someone shared material that matched a database of known harmful content, the service would be alerted. If a person shared innocent content, the service would learn nothing. People couldn't read the database or learn whether content matched, since that information could reveal law enforcement methods and help criminals evade detection.

Knowledgeable observers argued a system like ours was far from feasible. After many false starts, we built a working prototype. But we encountered a glaring problem.

Since Apple's announcement of the feature, the company has been bombarded with concerns that the system behind detecting CSAM could be used to detect other forms of photos at the request of oppressive governments. Apple has strongly pushed back against such a possibility, saying it will refuse any such request from governments.

Nonetheless, concerns over the future implications of the technology being used for CSAM detection are widespread. Mayer and Kulshrestha said that their concerns over how governments could use the system to detect content other than CSAM had them "disturbed."

A foreign government could, for example, compel a service to out people sharing disfavored political speech. That's no hypothetical: WeChat, the popular Chinese messaging app, already uses content matching to identify dissident material. India enacted rules this year that could require pre-screening content critical of government policy. Russia recently fined Google, Facebook and Twitter for not removing pro-democracy protest materials.

We spotted other shortcomings. The content-matching process could have false positives, and malicious users could game the system to subject innocent users to scrutiny.

We were so disturbed that we took a step we hadn't seen before in computer science literature: We warned against our own system design, urging further research on how to mitigate the serious downsides....

Apple has continued to address user concerns over its plans, publishing additional documents and an FAQ page. Apple continues to believe that its CSAM detection system, which will occur on a user's device, aligns with its long-standing privacy values.

Top Rated Comments

Geert76 Avatar
30 months ago
Apple, just cancel this on device surveillance software
Score: 166 Votes (Like | Disagree)
nvmls Avatar
30 months ago
They clearly don't know how the technology works.. oh wait
Score: 125 Votes (Like | Disagree)
nexu Avatar
30 months ago
This is getting more and more disturbing
Score: 117 Votes (Like | Disagree)
TheRealNick Avatar
30 months ago
If they don’t cancel this I’m seriously going to have to look at alternative products, which saddens me.
Score: 108 Votes (Like | Disagree)
dwsolberg Avatar
30 months ago
It's time for Apple to admit it's made a mistake. It's seriously bizarre they ever thought this was a good idea.
Score: 95 Votes (Like | Disagree)
VulchR Avatar
30 months ago
Waiting for the surveillance apologists to argue these two researchers "don't understand" the process...
Score: 93 Votes (Like | Disagree)

Popular Stories

iOS 17

26 New Things Your iPhone Can Do With Next Month's iOS 17.2 Update

Wednesday November 22, 2023 10:57 pm PST by
Apple made the first beta of iOS 17.2 available to developers in October. Since then we've seen two more betas, and with each iteration Apple continues to add more new features and changes, many of which users have been anticipating for quite a while. Below, we've listed 26 new things that are coming to your iPhone when the finalized version is publicly released in December. 1. Help You...
Apple Watch Series 9

5 Reasons to Wait for Next Year's Apple Watch

Friday November 24, 2023 3:46 am PST by
With all the discounted deals on Apple tech currently available, you might be thinking about upgrading your Apple Watch or buying one for the first time. But if your current smartwatch is doing its job just fine and it's only the idea of a good deal that's piqued your interest, it could be worth holding out until next year when Apple unveils its latest and greatest version. There are already ...
General Black Friday Deals 2022 Green

40+ Apple Black Friday Deals Still Available for AirPods, iPhone, iPad and More

Friday November 24, 2023 5:01 am PST by
Black Friday 2023 has officially ended, but we're still tracking some of the best deals of the year on Apple products like AirPods, iPad, iPhone, MacBook, and many more. Note: MacRumors is an affiliate partner with some of these vendors. When you click a link and make a purchase, we may receive a small payment, which helps us keep the site running. Specifically, in this article we're...
Cyber Monday Deals Feature 2022

40+ Apple Cyber Monday Deals for AirPods, iPad, Apple Watch, and More

Sunday November 26, 2023 9:47 am PST by
Cyber Monday has taken the place of Black Friday, and you'll find many of the same deals still around today. This includes dozens of record low prices on Apple products like AirPods, iPad, Apple Watch, MacBook, iPhone, and more. Note: MacRumors is an affiliate partner with some of these vendors. When you click a link and make a purchase, we may receive a small payment, which helps us keep the...
ipad mini 2021 youtube

What to Expect From the Next iPad Mini Later in 2024

Friday November 24, 2023 1:00 pm PST by
Over 800 days have passed since Apple last updated the iPad mini, as outlined in the MacRumors Buyer's Guide. Fortunately, a new iPad mini is rumored to be released next year, and we have outlined what to expect from the device below. Apple released the current iPad mini in September 2021, with new features at the time including a larger 8.3-inch display, a USB-C port, a Touch ID power...