University Researchers Who Built a CSAM Scanning System Urge Apple to Not Use the 'Dangerous' Technology

Respected university researchers are sounding the alarm bells over the technology behind Apple's plans to scan iPhone users' photo libraries for CSAM, or child sexual abuse material, calling the technology "dangerous."

apple privacy
Jonanath Mayer, an assistant professor of computer science and public affairs at Princeton University, as well as Anunay Kulshrestha, a researcher at Princeton University Center for Information Technology Policy, both penned an op-ed for The Washington Post, outlining their experiences with building image detection technology.

The researchers started a project two years ago to identity CSAM in end-to-end encrypted online services. The researchers note that given their field, they "know the value of end-to-end encryption, which protects data from third-party access." That concern, they say, is what horrifies them over CSAM "proliferating on encrypted platforms."

Mayer and Kulshrestha said they wanted to find a middle ground for the situation: build a system that online platforms could use to find CSAM and protect end-to-end encryption. The researchers note that experts in the field doubted the prospect of such a system, but they did manage to build it and in the process noticed a significant problem.

We sought to explore a possible middle ground, where online services could identify harmful content while otherwise preserving end-to-end encryption. The concept was straightforward: If someone shared material that matched a database of known harmful content, the service would be alerted. If a person shared innocent content, the service would learn nothing. People couldn't read the database or learn whether content matched, since that information could reveal law enforcement methods and help criminals evade detection.

Knowledgeable observers argued a system like ours was far from feasible. After many false starts, we built a working prototype. But we encountered a glaring problem.

Since Apple's announcement of the feature, the company has been bombarded with concerns that the system behind detecting CSAM could be used to detect other forms of photos at the request of oppressive governments. Apple has strongly pushed back against such a possibility, saying it will refuse any such request from governments.

Nonetheless, concerns over the future implications of the technology being used for CSAM detection are widespread. Mayer and Kulshrestha said that their concerns over how governments could use the system to detect content other than CSAM had them "disturbed."

A foreign government could, for example, compel a service to out people sharing disfavored political speech. That's no hypothetical: WeChat, the popular Chinese messaging app, already uses content matching to identify dissident material. India enacted rules this year that could require pre-screening content critical of government policy. Russia recently fined Google, Facebook and Twitter for not removing pro-democracy protest materials.

We spotted other shortcomings. The content-matching process could have false positives, and malicious users could game the system to subject innocent users to scrutiny.

We were so disturbed that we took a step we hadn't seen before in computer science literature: We warned against our own system design, urging further research on how to mitigate the serious downsides....

Apple has continued to address user concerns over its plans, publishing additional documents and an FAQ page. Apple continues to believe that its CSAM detection system, which will occur on a user's device, aligns with its long-standing privacy values.

Top Rated Comments

Geert76 Avatar
15 weeks ago
Apple, just cancel this on device surveillance software
Score: 166 Votes (Like | Disagree)
nvmls Avatar
15 weeks ago
They clearly don't know how the technology works.. oh wait
Score: 125 Votes (Like | Disagree)
nexu Avatar
15 weeks ago
This is getting more and more disturbing
Score: 117 Votes (Like | Disagree)
TheRealNick Avatar
15 weeks ago
If they don’t cancel this I’m seriously going to have to look at alternative products, which saddens me.
Score: 108 Votes (Like | Disagree)
dwsolberg Avatar
15 weeks ago
It's time for Apple to admit it's made a mistake. It's seriously bizarre they ever thought this was a good idea.
Score: 95 Votes (Like | Disagree)
VulchR Avatar
15 weeks ago
Waiting for the surveillance apologists to argue these two researchers "don't understand" the process...
Score: 93 Votes (Like | Disagree)

Related Stories

Child Safety Feature Purple

Apple's Proposed Phone-Scanning Child Safety Features 'Invasive, Ineffective, and Dangerous,' Say Cybersecurity Researchers in New Study

Friday October 15, 2021 12:23 am PDT by
More than a dozen prominent cybersecurity experts hit out at Apple on Thursday for relying on "dangerous technology" in its controversial plan to detect child sexual abuse images on iPhones (via The New York Times). The damning criticism came in a new 46-page study by researchers that looked at plans by Apple and the European Union to monitor people's phones for illicit material, and called...
General Insta Facebook Messenger Feature Blue

Facebook Delays Plans to Use End-to-End Encryption As Default for Messenger and Instagram Messages Until at Least 2023

Monday November 22, 2021 6:32 am PST by
Facebook, now rebranded to "Meta," is delaying plans to use end-to-end encryption for Messenger and Instagram messages until at least 2023, a year later than the previous deadline promised by the social media behemoth. Writing in The Telegraph, Meta's head of global safety, Antigone Davis, said that the company is pushing back its deadline to roll out end-to-end encryption as the default...
apple park drone june 2018 2

Apple Employees Internally Raising Concerns Over CSAM Detection Plans

Thursday August 12, 2021 11:43 pm PDT by
Apple employees are now joining the choir of individuals raising concerns over Apple's plans to scan iPhone users' photo libraries for CSAM or child sexual abuse material, reportedly speaking out internally about how the technology could be used to scan users' photos for other types of content, according to a report from Reuters. According to Reuters, an unspecified number of Apple employees ...
Child Safety Feature yellow

Global Coalition of Policy Groups Urges Apple to Abandon 'Plan to Build Surveillance Capabilities into iPhones'

Thursday August 19, 2021 1:23 am PDT by
An international coalition of more than 90 policy and rights groups published an open letter on Thursday urging Apple to abandon its plans to "build surveillance capabilities into iPhones, iPads, and other products" – a reference to the company's intention to scan users' iCloud photo libraries for images of child sex abuse (via Reuters). "Though these capabilities are intended to protect...
Child Safety Feature

Security Researchers Express Alarm Over Apple's Plans to Scan iCloud Images, But Practice Already Widespread

Thursday August 5, 2021 1:04 pm PDT by
Apple today announced that with the launch of iOS 15 and iPadOS 15, it will begin scanning iCloud Photos in the U.S. to look for known Child Sexual Abuse Material (CSAM), with plans to report the findings to the National Center for Missing and Exploited Children (NCMEC). Prior to when Apple detailed its plans, news of the CSAM initiative leaked, and security researchers have already begun...
appleprivacyad

Corellium Launching New Initiative to Hold Apple Accountable Over CSAM Detection Security and Privacy Claims

Tuesday August 17, 2021 1:35 am PDT by
Security research firm Corellium this week announced it is launching a new initiative that will "support independent public research into the security and privacy of mobile applications," and one of the initiative's first projects will be Apple's recently announced CSAM detection plans. Since its announcement earlier this month, Apple's plan to scan iPhone users' photo libraries for CSAM or...
apple privacy

Apple Makes iCloud Safari Bookmarks End-To-End Encrypted [Updated]

Monday October 4, 2021 1:28 am PDT by
Apple has toggled end-to-end encryption for Safari bookmarks in iCloud, further expanding the type of user data that the company fully encrypts, offering the highest level of privacy and data protection. Spotted on Reddit, an update to Apple's "iCloud security overview" page has indicated that alongside Safari tabs and history, Safari bookmarks are now end-to-end encrypted, meaning no one,...
eff apple park plane 1

EFF Flew a Banner Over Apple Park During Last Apple Event to Protest CSAM Plans

Friday September 24, 2021 2:06 am PDT by
In protest of the company's now delayed CSAM detection plans, the EFF, which has been vocal about Apple's child safety features plans in the past, flew a banner over Apple Park during the iPhone 13 event earlier this month with a message for the Cupertino tech giant. During Apple's fully-digital "California streaming" event on September 14, which included no physical audience attendance in...

Popular Stories

telsa cyberwhistle

Elon Musk Urges Customers to Buy 'Tesla Cyberwhistle' Instead of Apple Polishing Cloth

Wednesday December 1, 2021 4:01 am PST by
Tesla CEO Elon Musk has encouraged customers to buy the "Cyberwhistle" for $50 instead of Apple's much-discussed Polishing Cloth. The product page, which Musk shared on Twitter on Tuesday evening, offers a limited edition stainless steel whistle with the same distinctive design of the Tesla Cybertruck:Inspired by Cybertruck, the limited-edition Cyberwhistle is a premium collectible made from ...
iPhone SE Cosmopolitan Clean

New iPhone SE Reportedly on Track for Release in First Quarter of 2022

Tuesday November 30, 2021 8:08 am PST by
Apple plans to release a third-generation iPhone SE in the first quarter of 2022, according to Taiwanese research firm TrendForce. If this timeframe proves to be accurate, we can expect the device to be released by the end of March. As previously rumored, TrendForce said the new iPhone SE will remain a mid-range smartphone with added support for 5G:In terms of product development, Apple is...
maxresdefault

Five Features to Look Forward to in the 2022 MacBook Air

Tuesday November 30, 2021 1:51 pm PST by
In 2022, Apple is going to release an updated version of the MacBook Air with some of the biggest design changes that we've seen since 2010, when Apple introduced the 11 and 13-inch size options. In the video below, we highlight five features that you need to know about the new machine. Subscribe to the MacRumors YouTube channel for more videos. No More Wedge Design - Current MacBook...
2017 apple tv

Cyber Monday: Original Apple TV 4K Drops to $99.99 for Amazon Prime Members

Monday November 29, 2021 12:01 pm PST by
We've been tracking Apple product and accessory deals for Cyber Monday 2021 today, and now Woot is offering a solid discount on the previous generation 32GB Apple TV 4K. You can get this device in new condition for just $99.99 if you're an Amazon Prime member. Note that this sale will last for one day only. Note: MacRumors is an affiliate partner with some of these vendors. When you click a...
apple top apps games 2020

Apple Reveals the Most Downloaded iOS Apps and Games of 2021

Thursday December 2, 2021 12:05 am PST by
Along with naming its editorial picks for the top apps and games of 2021, Apple today shared charts for the most downloaded free and paid apps and games in the United States across 2021. The number one most downloaded free iPhone app was TikTok, followed by YouTube, Instagram, Snapchat, and Facebook. The top paid iPhone apps included Procreate Pocket, HotSchedules, The Wonder Weeks, and Touch...
Mac Notebook Upgrade Program

Apple Introduces New MacBook Upgrade Program for Business Partners

Monday November 29, 2021 7:38 am PST by
In association with CIT as the financing partner, Apple has launched a new Mac Upgrade Program for small businesses and Apple business partners that allow companies to easily distribute and upgrade their fleets of MacBooks at an affordable price to all of their workers. As outlined on CIT's website, shared by Max Weinbach, Apple Business Partners can distribute the 13-inch MacBook Pro,...
airpods prototype translucent

Transparent AirPods and 29W Power Adapter Prototypes Surface in Photos

Tuesday November 30, 2021 7:16 am PST by
Images of transparent prototype AirPods and a 29W Apple power adapter have been shared on Twitter by Apple device collector Giulio Zompetti. The prototypes, which appear to be either first-generation or second-generation AirPods, feature clear plastic along the stem and around the outer side of the earbud, with the normal white plastic on the inner side of the earbud. Transparent casings are ...
apple view concept right corner

Apple Planning to Replace the iPhone With AR Headset in 10 Years

Wednesday December 1, 2021 2:29 am PST by
Apple is planning to replace the iPhone with an augmented reality (AR) headset in 10 years, a process that is apparently due to start as soon as next year with the launch of a head-mounted device, according to a recent report. Concept render of Apple's rumored AR headset by Antonio De Rosa In a note to investors seen by MacRumors, eminent analyst Ming-Chi Kuo explained that "Apple's goal is...