University Researchers Who Built a CSAM Scanning System Urge Apple to Not Use the 'Dangerous' Technology

Respected university researchers are sounding the alarm bells over the technology behind Apple's plans to scan iPhone users' photo libraries for CSAM, or child sexual abuse material, calling the technology "dangerous."

apple privacy
Jonanath Mayer, an assistant professor of computer science and public affairs at Princeton University, as well as Anunay Kulshrestha, a researcher at Princeton University Center for Information Technology Policy, both penned an op-ed for The Washington Post, outlining their experiences with building image detection technology.

The researchers started a project two years ago to identity CSAM in end-to-end encrypted online services. The researchers note that given their field, they "know the value of end-to-end encryption, which protects data from third-party access." That concern, they say, is what horrifies them over CSAM "proliferating on encrypted platforms."

Mayer and Kulshrestha said they wanted to find a middle ground for the situation: build a system that online platforms could use to find CSAM and protect end-to-end encryption. The researchers note that experts in the field doubted the prospect of such a system, but they did manage to build it and in the process noticed a significant problem.

We sought to explore a possible middle ground, where online services could identify harmful content while otherwise preserving end-to-end encryption. The concept was straightforward: If someone shared material that matched a database of known harmful content, the service would be alerted. If a person shared innocent content, the service would learn nothing. People couldn't read the database or learn whether content matched, since that information could reveal law enforcement methods and help criminals evade detection.

Knowledgeable observers argued a system like ours was far from feasible. After many false starts, we built a working prototype. But we encountered a glaring problem.

Since Apple's announcement of the feature, the company has been bombarded with concerns that the system behind detecting CSAM could be used to detect other forms of photos at the request of oppressive governments. Apple has strongly pushed back against such a possibility, saying it will refuse any such request from governments.

Nonetheless, concerns over the future implications of the technology being used for CSAM detection are widespread. Mayer and Kulshrestha said that their concerns over how governments could use the system to detect content other than CSAM had them "disturbed."

A foreign government could, for example, compel a service to out people sharing disfavored political speech. That's no hypothetical: WeChat, the popular Chinese messaging app, already uses content matching to identify dissident material. India enacted rules this year that could require pre-screening content critical of government policy. Russia recently fined Google, Facebook and Twitter for not removing pro-democracy protest materials.

We spotted other shortcomings. The content-matching process could have false positives, and malicious users could game the system to subject innocent users to scrutiny.

We were so disturbed that we took a step we hadn't seen before in computer science literature: We warned against our own system design, urging further research on how to mitigate the serious downsides....

Apple has continued to address user concerns over its plans, publishing additional documents and an FAQ page. Apple continues to believe that its CSAM detection system, which will occur on a user's device, aligns with its long-standing privacy values.

Top Rated Comments

Geert76 Avatar
19 months ago
Apple, just cancel this on device surveillance software
Score: 166 Votes (Like | Disagree)
nvmls Avatar
19 months ago
They clearly don't know how the technology works.. oh wait
Score: 125 Votes (Like | Disagree)
nexu Avatar
19 months ago
This is getting more and more disturbing
Score: 117 Votes (Like | Disagree)
TheRealNick Avatar
19 months ago
If they don’t cancel this I’m seriously going to have to look at alternative products, which saddens me.
Score: 108 Votes (Like | Disagree)
dwsolberg Avatar
19 months ago
It's time for Apple to admit it's made a mistake. It's seriously bizarre they ever thought this was a good idea.
Score: 95 Votes (Like | Disagree)
VulchR Avatar
19 months ago
Waiting for the surveillance apologists to argue these two researchers "don't understand" the process...
Score: 93 Votes (Like | Disagree)

Popular Stories

iOS 16

iOS 16.3 Now Available for Your iPhone With These 4 New Features

Friday February 3, 2023 1:13 pm PST by
Apple released iOS 16.3 in late January following nearly six weeks of beta testing. The software update is available for the iPhone 8 and newer, and while it is a relatively minor update, it still includes a handful of new features, changes, and bug fixes. Below, we've recapped new features in iOS 16.3, including support for physical security keys as a two-factor authentication option for...
iPhone 14 Pro Purple Side Perspective Feature Purple

Gurman: Apple Considering New High-End iPhone Alongside Pro and Pro Max

Sunday February 5, 2023 6:07 am PST by
Apple has discussed selling a new top-of-the-line iPhone alongside the Pro and Pro Max models in 2024 at the earliest, according to Bloomberg's Mark Gurman. Based on this timeframe, the device would be part of the iPhone 16 lineup or later. In a September 2022 edition of his weekly "Power On" newsletter, Gurman said there was "potential" for an iPhone 15 Ultra to replace the iPhone 15 Pro...
ipad air purple

Deals: M1 iPad Air Hits Record-Low Prices at TigerDirect, Starting at $313.99 (48% Off) [Updated]

Saturday February 4, 2023 10:05 am PST by
Online retailer TigerDirect has slashed pricing on the M1 iPad Air in several colors, offering the base 64GB configuration for just $313.99 in Purple and Pink. Note: MacRumors is an affiliate partner with TigerDirect. When you click a link and make a purchase, we may receive a small payment, which helps us keep the site running. That's a savings of 48% compared to Apple's normal $599.00...
mac studio pink

Apple May Not Launch Updated Mac Studio With M2 Ultra Chip Due to Similarity With Upcoming Mac Pro

Sunday February 5, 2023 6:06 am PST by
A new version of the Mac Studio with the "M2 Ultra" chip is unlikely to arrive in the near future, according to Bloomberg's Mark Gurman. In the latest edition of his "Power On" newsletter, Gurman explained that since the upcoming Apple silicon Mac Pro is "very similar in functionality to the Mac Studio," Apple may wait until the release of M3- or M4-series chips to update the machine, or...
webkit vs chromium feature

Google Working on Browser for iOS That Would Break Apple's App Store Rules

Saturday February 4, 2023 1:30 am PST by
Google's Chromium developers are working on an experimental web browser for iOS that would break Apple's browser engine restrictions, The Register reports. The experimental browser, which is being actively pursued by developers, uses Google's Blink engine. Yet if Google attempted to release it on the App Store, it would not pass Apple's App Review process. Apple's App Store rules dictate...
Apple Silicon Teal Feature

The Next Big Apple Silicon Device May Not Be a Mac or iPad

Wednesday February 1, 2023 3:57 am PST by
Apple's next device with an Apple silicon chip may not be a Mac or an iPad, but rather an advanced external display, according to recent reports. The display, which is rumored to arrive this year, is expected to sit somewhere between the $1,599 Studio Display and the $4,999 Pro Display XDR – but more exact information about the device's positioning and price point is as yet unknown. While ...
top stories 4feb2023

Top Stories: New HomePod Launches, iOS 16.3.1 Coming Soon, and More

Saturday February 4, 2023 6:00 am PST by
This week saw the launch of the final product from Apple's January hardware announcements, with the second-generation HomePod making its way into customers' hands. Other news and rumors this week included signs of an interim iOS 16.3.1 bug fix update in the works as we await a fresh round of beta testing for a more significant iOS 16.4 update, a report on the potential dangers of using...
maxresdefault

Hands-On With Apple's Second-Generation HomePod

Friday February 3, 2023 1:28 pm PST by
Today is the official launch day for the second-generation HomePod that was introduced in January, and we picked one up to compare it to the original HomePod that Apple discontinued in 2021. Subscribe to the MacRumors YouTube channel for more videos. Design wise, the second-generation HomePod looks a lot like the first-generation model, featuring the same rounded design and acoustic mesh...