University Researchers Who Built a CSAM Scanning System Urge Apple to Not Use the 'Dangerous' Technology

Respected university researchers are sounding the alarm bells over the technology behind Apple's plans to scan iPhone users' photo libraries for CSAM, or child sexual abuse material, calling the technology "dangerous."

apple privacy
Jonanath Mayer, an assistant professor of computer science and public affairs at Princeton University, as well as Anunay Kulshrestha, a researcher at Princeton University Center for Information Technology Policy, both penned an op-ed for The Washington Post, outlining their experiences with building image detection technology.

The researchers started a project two years ago to identity CSAM in end-to-end encrypted online services. The researchers note that given their field, they "know the value of end-to-end encryption, which protects data from third-party access." That concern, they say, is what horrifies them over CSAM "proliferating on encrypted platforms."

Mayer and Kulshrestha said they wanted to find a middle ground for the situation: build a system that online platforms could use to find CSAM and protect end-to-end encryption. The researchers note that experts in the field doubted the prospect of such a system, but they did manage to build it and in the process noticed a significant problem.

We sought to explore a possible middle ground, where online services could identify harmful content while otherwise preserving end-to-end encryption. The concept was straightforward: If someone shared material that matched a database of known harmful content, the service would be alerted. If a person shared innocent content, the service would learn nothing. People couldn't read the database or learn whether content matched, since that information could reveal law enforcement methods and help criminals evade detection.

Knowledgeable observers argued a system like ours was far from feasible. After many false starts, we built a working prototype. But we encountered a glaring problem.

Since Apple's announcement of the feature, the company has been bombarded with concerns that the system behind detecting CSAM could be used to detect other forms of photos at the request of oppressive governments. Apple has strongly pushed back against such a possibility, saying it will refuse any such request from governments.

Nonetheless, concerns over the future implications of the technology being used for CSAM detection are widespread. Mayer and Kulshrestha said that their concerns over how governments could use the system to detect content other than CSAM had them "disturbed."

A foreign government could, for example, compel a service to out people sharing disfavored political speech. That's no hypothetical: WeChat, the popular Chinese messaging app, already uses content matching to identify dissident material. India enacted rules this year that could require pre-screening content critical of government policy. Russia recently fined Google, Facebook and Twitter for not removing pro-democracy protest materials.

We spotted other shortcomings. The content-matching process could have false positives, and malicious users could game the system to subject innocent users to scrutiny.

We were so disturbed that we took a step we hadn't seen before in computer science literature: We warned against our own system design, urging further research on how to mitigate the serious downsides....

Apple has continued to address user concerns over its plans, publishing additional documents and an FAQ page. Apple continues to believe that its CSAM detection system, which will occur on a user's device, aligns with its long-standing privacy values.

Top Rated Comments

Geert76 Avatar
33 months ago
Apple, just cancel this on device surveillance software
Score: 166 Votes (Like | Disagree)
nvmls Avatar
33 months ago
They clearly don't know how the technology works.. oh wait
Score: 125 Votes (Like | Disagree)
nexu Avatar
33 months ago
This is getting more and more disturbing
Score: 117 Votes (Like | Disagree)
TheRealNick Avatar
33 months ago
If they don’t cancel this I’m seriously going to have to look at alternative products, which saddens me.
Score: 108 Votes (Like | Disagree)
dwsolberg Avatar
33 months ago
It's time for Apple to admit it's made a mistake. It's seriously bizarre they ever thought this was a good idea.
Score: 95 Votes (Like | Disagree)
VulchR Avatar
33 months ago
Waiting for the surveillance apologists to argue these two researchers "don't understand" the process...
Score: 93 Votes (Like | Disagree)

Popular Stories

General Apps Messages

Apple Announces 'Groundbreaking' New Security Protocol for iMessage

Wednesday February 21, 2024 6:00 am PST by
Apple today announced a new post-quantum cryptographic protocol for iMessage called PQ3. Apple says this "groundbreaking" and "state-of-the-art" protocol provides "extensive defenses against even highly sophisticated quantum attacks." Apple believes the PQ3 protocol's protections "surpass those in all other widely deployed messaging apps," according to its blog post:Today we are announcing...
Beyond iPhone 13 Better Blue Face ID Single Camera Hole

Six Reasons to Wait for Next Year's iPhone 17

Thursday February 22, 2024 4:20 am PST by
Apple's iPhone development roadmap runs several years into the future and the company is continually working with suppliers on several successive iPhone models concurrently, which is why we sometimes get rumored feature leaks so far ahead of launch. The iPhone 17 series is no different, and already we have some idea of what to expect from Apple's 2025 smartphone lineup. If you plan to skip...
iOS 17

iOS 17.4 Will Add These New Features to Your iPhone

Monday February 19, 2024 6:52 am PST by
Apple last month confirmed that iOS 17.4 will be released in March, and the update includes several new features and changes for the iPhone. Key new features in iOS 17.4 include major App Store changes in the EU and Apple Podcasts transcripts. The update also adds new emoji and includes preparations for the launch of next-generation CarPlay later this year. More details about the new...
iPhone Notch

Apple Explored These Notch and Dynamic Island Designs for iPhones

Friday February 23, 2024 2:05 pm PST by
With the iPhone 14 Pro models in 2022, Apple introduced the Dynamic Island, which can morph and expand to display system alerts, sports scores, and a variety of other information. The feature makes the space surrounding the front camera and Face ID sensors useful compared to the notch on older iPhone models. Apple explored a variety of ideas for the iPhone's notch area over the years before...
Oled iPads and MackBook Pro Notch

Apple's OLED Roadmap: New iPad Mini, Foldable iPad Pro, and More

Wednesday February 21, 2024 5:29 am PST by
Apple is planning to launch at least nine new devices with OLED displays across the iPad and MacBook product lines, according to an updated forecast from research firm Omdia that sets out the company's plans in detail. As widely rumored, Apple's push to transition to OLED will apparently be jump-started by the release of new 11- and 13-inch iPad Pro models this year. The displays will...
Next Generation CarPlay Porsche 1

Apple Launching Next-Generation CarPlay This Year With These New Features

Wednesday February 21, 2024 2:03 pm PST by
Apple recently updated its website to confirm that the first vehicle models with next-generation CarPlay support will debut "in 2024." This wording is shown on Apple's regional websites for the U.S., Canada, Australia, New Zealand, and many other countries. The iOS 17.4 beta includes code-level references to eight new CarPlay apps:Auto Settings: This app will let you manage paired iPhones...
cracked glass vision pro

Handful of Apple Vision Pro Units Develop Identical Crack in Cover Glass

Friday February 23, 2024 4:46 am PST by
A small number of Apple Vision Pro owners have claimed that their headsets developed a hairline crack down the middle of the front cover glass, despite having never been dropped or mishandled. Photo of hairline crack in Vision Pro front glass (credit: Reddit user Inphenite) The first report was posted in the subreddit /r/VisionPro about 18 days ago, and several more have appeared in the last...
iPad Pro OLED Feature 2

Apple's Upcoming OLED iPad Pro Models Rumored to Be Much Thinner

Tuesday February 20, 2024 1:39 pm PST by
Apple is planning to update the iPad Air and iPad Pro lineups as soon as March, and the new iPad Pro models will be significantly thinner according to dimensions shared by 9to5Mac. Citing sources with knowledge of Apple's plans, the site claims that the larger version will be more than 1mm thinner. The current 12.9-inch iPad Pro measures in at 6.4mm thick, but the new model is said to be 5mm ...