University Researchers Who Built a CSAM Scanning System Urge Apple to Not Use the 'Dangerous' Technology

Respected university researchers are sounding the alarm bells over the technology behind Apple's plans to scan iPhone users' photo libraries for CSAM, or child sexual abuse material, calling the technology "dangerous."

apple privacy
Jonanath Mayer, an assistant professor of computer science and public affairs at Princeton University, as well as Anunay Kulshrestha, a researcher at Princeton University Center for Information Technology Policy, both penned an op-ed for The Washington Post, outlining their experiences with building image detection technology.

The researchers started a project two years ago to identity CSAM in end-to-end encrypted online services. The researchers note that given their field, they "know the value of end-to-end encryption, which protects data from third-party access." That concern, they say, is what horrifies them over CSAM "proliferating on encrypted platforms."

Mayer and Kulshrestha said they wanted to find a middle ground for the situation: build a system that online platforms could use to find CSAM and protect end-to-end encryption. The researchers note that experts in the field doubted the prospect of such a system, but they did manage to build it and in the process noticed a significant problem.

We sought to explore a possible middle ground, where online services could identify harmful content while otherwise preserving end-to-end encryption. The concept was straightforward: If someone shared material that matched a database of known harmful content, the service would be alerted. If a person shared innocent content, the service would learn nothing. People couldn't read the database or learn whether content matched, since that information could reveal law enforcement methods and help criminals evade detection.

Knowledgeable observers argued a system like ours was far from feasible. After many false starts, we built a working prototype. But we encountered a glaring problem.

Since Apple's announcement of the feature, the company has been bombarded with concerns that the system behind detecting CSAM could be used to detect other forms of photos at the request of oppressive governments. Apple has strongly pushed back against such a possibility, saying it will refuse any such request from governments.

Nonetheless, concerns over the future implications of the technology being used for CSAM detection are widespread. Mayer and Kulshrestha said that their concerns over how governments could use the system to detect content other than CSAM had them "disturbed."

A foreign government could, for example, compel a service to out people sharing disfavored political speech. That's no hypothetical: WeChat, the popular Chinese messaging app, already uses content matching to identify dissident material. India enacted rules this year that could require pre-screening content critical of government policy. Russia recently fined Google, Facebook and Twitter for not removing pro-democracy protest materials.

We spotted other shortcomings. The content-matching process could have false positives, and malicious users could game the system to subject innocent users to scrutiny.

We were so disturbed that we took a step we hadn't seen before in computer science literature: We warned against our own system design, urging further research on how to mitigate the serious downsides....

Apple has continued to address user concerns over its plans, publishing additional documents and an FAQ page. Apple continues to believe that its CSAM detection system, which will occur on a user's device, aligns with its long-standing privacy values.

Top Rated Comments

Geert76 Avatar
13 months ago
Apple, just cancel this on device surveillance software
Score: 166 Votes (Like | Disagree)
nvmls Avatar
13 months ago
They clearly don't know how the technology works.. oh wait
Score: 125 Votes (Like | Disagree)
nexu Avatar
13 months ago
This is getting more and more disturbing
Score: 117 Votes (Like | Disagree)
TheRealNick Avatar
13 months ago
If they don’t cancel this I’m seriously going to have to look at alternative products, which saddens me.
Score: 108 Votes (Like | Disagree)
dwsolberg Avatar
13 months ago
It's time for Apple to admit it's made a mistake. It's seriously bizarre they ever thought this was a good idea.
Score: 95 Votes (Like | Disagree)
VulchR Avatar
13 months ago
Waiting for the surveillance apologists to argue these two researchers "don't understand" the process...
Score: 93 Votes (Like | Disagree)

Related Stories

Child Safety Feature Blue

Apple Says NeuralHash Tech Impacted by 'Hash Collisions' Is Not the Version Used for CSAM Detection

Wednesday August 18, 2021 1:13 pm PDT by
Developer Asuhariet Yvgar this morning said that he had reverse-engineered the NeuralHash algorithm that Apple is using to detect Child Sexual Abuse Materials (CSAM) in iCloud Photos, posting evidence on GitHub and details on Reddit. Yvgar said that he reverse-engineered the NeuralHash algorithm from iOS 14.3, where the code was hidden, and he rebuilt it in Python. After he uploaded his...
universal control sidecar

PSA: Universal Control and Sidecar Can Be Used At the Same Time in macOS 12.3

Tuesday March 15, 2022 5:12 am PDT by
Before macOS 12.3 and iPadOS 15.4 were released to the public, many beta testers who tried Universal Control were unable to get it working if they were already using an iPad in Sidecar mode, but it turns out that in the final release you can actually enable both Universal Control and Sidecar simultaneously on different devices, provided your setup allows for this. In the final version of...
universal control 1

macOS Monterey 12.3 Beta 3 Makes Accessing Universal Control Settings Easier

Tuesday February 15, 2022 3:56 pm PST by
In the third beta of macOS Monterey that Apple released to developers today, there is a small Universal Control update that is designed to make it easier to access the various Universal Control settings that you might need when using the feature to control multiple Macs and iPads with a single mouse/trackpad and keyboard. If you open up the Displays section of System Preferences after...
appleprivacyad

Corellium Launching New Initiative to Hold Apple Accountable Over CSAM Detection Security and Privacy Claims

Tuesday August 17, 2021 1:35 am PDT by
Security research firm Corellium this week announced it is launching a new initiative that will "support independent public research into the security and privacy of mobile applications," and one of the initiative's first projects will be Apple's recently announced CSAM detection plans. Since its announcement earlier this month, Apple's plan to scan iPhone users' photo libraries for CSAM or...
Child Safety Feature Purple

Apple's Proposed Phone-Scanning Child Safety Features 'Invasive, Ineffective, and Dangerous,' Say Cybersecurity Researchers in New Study

Friday October 15, 2021 12:23 am PDT by
More than a dozen prominent cybersecurity experts hit out at Apple on Thursday for relying on "dangerous technology" in its controversial plan to detect child sexual abuse images on iPhones (via The New York Times). The damning criticism came in a new 46-page study by researchers that looked at plans by Apple and the European Union to monitor people's phones for illicit material, and called...
european parliament

EU Provisionally Agrees on Law That Would Force Apple to Allow Alternative App Stores, Sideloading, and iMessage Interoperability

Friday March 25, 2022 4:46 am PDT by
European lawmakers have provisionally agreed upon a new law that would force Apple to allow user access to third-party app stores and permit the sideloading of apps on iPhones and iPads, among other sweeping changes designed to make the digital sector fairer and more competitive. The European Council and European Parliament said on Friday they had reached a political agreement on the...
Child Safety Feature Blue

Apple Privacy Chief Explains 'Built-in' Privacy Protections in Child Safety Features Amid User Concerns

Tuesday August 10, 2021 9:07 am PDT by
Apple's Head of Privacy, Erik Neuenschwander, has responded to some of users' concerns around the company's plans for new child safety features that will scan messages and Photos libraries, in an interview with TechCrunch. When asked why Apple is only choosing to implement child safety features that scan for Child Sexual Abuse Material (CSAM), Neuenschwander explained that Apple has "now got ...
Child Safety Feature yellow

Global Coalition of Policy Groups Urges Apple to Abandon 'Plan to Build Surveillance Capabilities into iPhones'

Thursday August 19, 2021 1:23 am PDT by
An international coalition of more than 90 policy and rights groups published an open letter on Thursday urging Apple to abandon its plans to "build surveillance capabilities into iPhones, iPads, and other products" – a reference to the company's intention to scan users' iCloud photo libraries for images of child sex abuse (via Reuters). "Though these capabilities are intended to protect...

Popular Stories

iPhone 14 Pro Purple Rear Flat MacRumors Exclusive

iPhone 14 Pro Predicted to Start With Increased 256GB Storage Alongside Rumored Price Increase

Wednesday August 10, 2022 11:14 am PDT by
Earlier today, analyst Ming-Chi Kuo claimed iPhone 14 Pro models will be more expensive than iPhone 13 Pro models. Kuo did not reveal exact pricing, but he said that the average selling price of all four iPhone 14 models will increase by about 15% overall. While higher prices would be disappointing for customers, it is possible the iPhone 14 Pro and iPhone 14 Pro Max will offer increased...
iPhone 14 Pro Lineup Feature Silver

Kuo: Apple to Increase Prices of iPhone 14 Pro Models

Wednesday August 10, 2022 8:22 am PDT by
Apple plans to increase the prices of iPhone 14 Pro models compared to iPhone 13 Pro models, according to analyst Ming-Chi Kuo. Kuo did not reveal exact pricing for the iPhone 14 Pro models. However, in a tweet today, he estimated that the average selling price of the iPhone 14 lineup as a whole will increase by about 15% compared to the iPhone 13 lineup. In the United States, the iPhone...
iPhone 14 Pro Purple Front and Back MacRumors Exclusive feature

iPhone 14 Is Just a Few Weeks Away: Three Tips to Prepare for the New iPhone

Wednesday August 10, 2022 4:08 am PDT by
The launch of the new iPhone 14 is just a few weeks away, meaning millions of iPhone customers will soon upgrade their existing iPhone or perhaps get an iPhone for the first time. Exclusive MacRumors iPhone 14 Pro renders by graphic designer Ian Zelbo Whether upgrading from an older model or this is your first iPhone, we've rounded up a few tips to help you prepare for the next flagship...
battery percentage ios 16

Here's Why the iPhone Battery Status Icon in iOS 16 Is So Controversial

Wednesday August 10, 2022 4:34 am PDT by
In the latest iOS 16 beta, Apple has updated the status bar battery icon on iPhones with Face ID to display the exact percentage remaining rather than just a visual representation of battery level, and while the change has been largely welcomed, some users are unhappy with the way it has been implemented. In iOS 15 and earlier, battery percent has not been present on iPhones that have...
Apple Watch Body Temperature Finished

'High-Accuracy' Apple Watch Temperature Sensor Revealed by Patent Filing Just Weeks Before Series 8 Unveiling

Wednesday August 10, 2022 5:39 am PDT by
Apple has been granted a patent for a temperature sensor suitable for the Apple Watch, just weeks before the company is expected to unveil the Apple Watch Series 8 with body temperature sensing capabilities. The newly granted patent, spotted by MyHealthyApple, was filed with the United States Patent and Trademark Office and is titled "Temperature gradient sensing in electronic devices."...