University Researchers Who Built a CSAM Scanning System Urge Apple to Not Use the 'Dangerous' Technology

Respected university researchers are sounding the alarm bells over the technology behind Apple's plans to scan iPhone users' photo libraries for CSAM, or child sexual abuse material, calling the technology "dangerous."

apple privacy
Jonanath Mayer, an assistant professor of computer science and public affairs at Princeton University, as well as Anunay Kulshrestha, a researcher at Princeton University Center for Information Technology Policy, both penned an op-ed for The Washington Post, outlining their experiences with building image detection technology.

The researchers started a project two years ago to identity CSAM in end-to-end encrypted online services. The researchers note that given their field, they "know the value of end-to-end encryption, which protects data from third-party access." That concern, they say, is what horrifies them over CSAM "proliferating on encrypted platforms."

Mayer and Kulshrestha said they wanted to find a middle ground for the situation: build a system that online platforms could use to find CSAM and protect end-to-end encryption. The researchers note that experts in the field doubted the prospect of such a system, but they did manage to build it and in the process noticed a significant problem.

We sought to explore a possible middle ground, where online services could identify harmful content while otherwise preserving end-to-end encryption. The concept was straightforward: If someone shared material that matched a database of known harmful content, the service would be alerted. If a person shared innocent content, the service would learn nothing. People couldn't read the database or learn whether content matched, since that information could reveal law enforcement methods and help criminals evade detection.

Knowledgeable observers argued a system like ours was far from feasible. After many false starts, we built a working prototype. But we encountered a glaring problem.

Since Apple's announcement of the feature, the company has been bombarded with concerns that the system behind detecting CSAM could be used to detect other forms of photos at the request of oppressive governments. Apple has strongly pushed back against such a possibility, saying it will refuse any such request from governments.

Nonetheless, concerns over the future implications of the technology being used for CSAM detection are widespread. Mayer and Kulshrestha said that their concerns over how governments could use the system to detect content other than CSAM had them "disturbed."

A foreign government could, for example, compel a service to out people sharing disfavored political speech. That's no hypothetical: WeChat, the popular Chinese messaging app, already uses content matching to identify dissident material. India enacted rules this year that could require pre-screening content critical of government policy. Russia recently fined Google, Facebook and Twitter for not removing pro-democracy protest materials.

We spotted other shortcomings. The content-matching process could have false positives, and malicious users could game the system to subject innocent users to scrutiny.

We were so disturbed that we took a step we hadn't seen before in computer science literature: We warned against our own system design, urging further research on how to mitigate the serious downsides....

Apple has continued to address user concerns over its plans, publishing additional documents and an FAQ page. Apple continues to believe that its CSAM detection system, which will occur on a user's device, aligns with its long-standing privacy values.

Top Rated Comments

Geert76 Avatar
27 months ago
Apple, just cancel this on device surveillance software
Score: 166 Votes (Like | Disagree)
nvmls Avatar
27 months ago
They clearly don't know how the technology works.. oh wait
Score: 125 Votes (Like | Disagree)
nexu Avatar
27 months ago
This is getting more and more disturbing
Score: 117 Votes (Like | Disagree)
TheRealNick Avatar
27 months ago
If they don’t cancel this I’m seriously going to have to look at alternative products, which saddens me.
Score: 108 Votes (Like | Disagree)
dwsolberg Avatar
27 months ago
It's time for Apple to admit it's made a mistake. It's seriously bizarre they ever thought this was a good idea.
Score: 95 Votes (Like | Disagree)
VulchR Avatar
27 months ago
Waiting for the surveillance apologists to argue these two researchers "don't understand" the process...
Score: 93 Votes (Like | Disagree)

Popular Stories

iOS 17

Apple Releases iOS 17.0.1 and iPadOS 17.0.1 With Bug Fixes, Plus iOS 17.0.2 for iPhone 15 Models

Thursday September 21, 2023 10:28 am PDT by
Apple today released iOS 17.0.1 and iPadOS 17.0.1 updates for the iPhone and the iPad, adding bug fixes to the new software. The iOS 17.0.1 and iPadOS 17.0.1 updates come just a few days after Apple launched iOS 17 and iPadOS 17. The software, which is build 21A340, can be downloaded on eligible iPhones and iPads over-the-air by going to Settings > General > Software Update. There is a...
iPhone 16 Mock Header With Dynamic Island

Skipping the iPhone 15 Pro? Here's What's Rumored for iPhone 16 Pro

Friday September 22, 2023 9:29 am PDT by
Are you skipping the iPhone 15 Pro and waiting another year to upgrade? If so, we already have some iPhone 16 Pro rumors for you. Below, we recap new features rumored for the iPhone 16 Pro models so far:Larger displays: The iPhone 16 Pro and iPhone 16 Pro Max will be equipped with larger 6.3-inch and 6.9-inch displays, respectively, according to Ross Young, CEO of Display Supply Chain...
Update Your iPhone 15 to iOS 17

Warning: Update Your iPhone 15 to iOS 17.0.2 Before Transferring Data From Another iPhone

Friday September 22, 2023 6:36 am PDT by
If you are unboxing an iPhone 15, iPhone 15 Plus, iPhone 15 Pro, or iPhone 15 Pro Max today, make sure to update the device to iOS 17.0.2 before transferring data to the device from another iPhone, or else you might encounter issues. iOS 17.0.2 is only available for the iPhone 15 lineup. Apple says the update fixes an issue that may prevent transferring data directly from another iPhone...
Apple Watch Ultra 2 double tap gesture 230912

watchOS 10.1 to Enable Apple Watch's New 'Double Tap' Gesture

Thursday September 21, 2023 12:52 pm PDT by
The new Double Tap gesture for the Apple Watch Series 9 and the Apple Watch Ultra 2 will be enabled starting with watchOS 10.1, according to Marques Brownlee, host of the popular tech-focused YouTube channel MKBHD. The first beta of watchOS 10.1 will likely be available by next week, and Apple announced that the software update will be released next month. Brownlee shared his impressions...
emojipedia 15 1 emoji

Emoji Coming to Future iOS 17 Update Include Shaking Head, Brown Mushroom, Lime, Phoenix and More

Tuesday September 19, 2023 12:43 pm PDT by
As Apple was announcing new iPhone models last week, the Unicode Consortium was officially approving new emoji characters that are set to be added to smartphones starting in 2024. Mockup of new emoji from Emojipedia Approved Unicode 15.1 emoji include phoenix, lime, an edible mushroom, shaking head vertically (as in a "yes" nod), shaking head horizontally (a "no" head shake), and broken...
iPhone 15 Pro Fingerprints

Apple Says Fingerprints Can 'Temporarily Alter the Color' of iPhone 15 Pro's Titanium Frame

Thursday September 21, 2023 2:48 pm PDT by
In a support document updated today, Apple indicates that fingerprints might "temporarily alter the color" of the titanium frame on the iPhone 15 Pro and iPhone 15 Pro Max. Fortunately, Apple says this is not a permanent condition, as the fingerprints can be wiped off just like on any other iPhone to restore a clean finish. "For iPhone 15 Pro and iPhone 15 Pro Max, the oil from your skin...