University Researchers Who Built a CSAM Scanning System Urge Apple to Not Use the 'Dangerous' Technology

Respected university researchers are sounding the alarm bells over the technology behind Apple's plans to scan iPhone users' photo libraries for CSAM, or child sexual abuse material, calling the technology "dangerous."

apple privacy
Jonanath Mayer, an assistant professor of computer science and public affairs at Princeton University, as well as Anunay Kulshrestha, a researcher at Princeton University Center for Information Technology Policy, both penned an op-ed for The Washington Post, outlining their experiences with building image detection technology.

The researchers started a project two years ago to identity CSAM in end-to-end encrypted online services. The researchers note that given their field, they "know the value of end-to-end encryption, which protects data from third-party access." That concern, they say, is what horrifies them over CSAM "proliferating on encrypted platforms."

Mayer and Kulshrestha said they wanted to find a middle ground for the situation: build a system that online platforms could use to find CSAM and protect end-to-end encryption. The researchers note that experts in the field doubted the prospect of such a system, but they did manage to build it and in the process noticed a significant problem.

We sought to explore a possible middle ground, where online services could identify harmful content while otherwise preserving end-to-end encryption. The concept was straightforward: If someone shared material that matched a database of known harmful content, the service would be alerted. If a person shared innocent content, the service would learn nothing. People couldn't read the database or learn whether content matched, since that information could reveal law enforcement methods and help criminals evade detection.

Knowledgeable observers argued a system like ours was far from feasible. After many false starts, we built a working prototype. But we encountered a glaring problem.

Since Apple's announcement of the feature, the company has been bombarded with concerns that the system behind detecting CSAM could be used to detect other forms of photos at the request of oppressive governments. Apple has strongly pushed back against such a possibility, saying it will refuse any such request from governments.

Nonetheless, concerns over the future implications of the technology being used for CSAM detection are widespread. Mayer and Kulshrestha said that their concerns over how governments could use the system to detect content other than CSAM had them "disturbed."

A foreign government could, for example, compel a service to out people sharing disfavored political speech. That's no hypothetical: WeChat, the popular Chinese messaging app, already uses content matching to identify dissident material. India enacted rules this year that could require pre-screening content critical of government policy. Russia recently fined Google, Facebook and Twitter for not removing pro-democracy protest materials.

We spotted other shortcomings. The content-matching process could have false positives, and malicious users could game the system to subject innocent users to scrutiny.

We were so disturbed that we took a step we hadn't seen before in computer science literature: We warned against our own system design, urging further research on how to mitigate the serious downsides....

Apple has continued to address user concerns over its plans, publishing additional documents and an FAQ page. Apple continues to believe that its CSAM detection system, which will occur on a user's device, aligns with its long-standing privacy values.

Popular Stories

airpods pro 2 pink

Apple Releases New AirPods Pro 2 Firmware

Tuesday May 28, 2024 11:46 am PDT by
Apple today released new firmware update for both the Lightning and USB-C versions of the AirPods Pro 2. The new firmware is version 6F7, up from the 6B34 firmware released in November. Apple does not provide details on what features might be included in the refreshed firmware beyond "bug fixes and other improvements," so it is unclear what's new in the update. Apple does not give...
maxresdefault

Report: These 10 New AI Features Are Coming in iOS 18

Sunday May 26, 2024 12:57 pm PDT by
iOS 18 and macOS 15 will offer an array of new AI features such as auto-generated emojis, suggested replies to emails and messages, and more, Bloomberg's Mark Gurman reports. Subscribe to the MacRumors YouTube channel for more videos. A significant portion of Apple's Worldwide Developers Conference (WWDC) is expected to focus on AI features. Writing his latest "Power On" newsletter, Gurman...
wwdc 2024 main image feature

Apple Confirms Time for June 10 WWDC Keynote, Shares Full Schedule

Tuesday May 28, 2024 10:21 am PDT by
Apple today shared details on the schedule that it has prepared for the 2024 Worldwide Developers Conference, which is set to take place from June 10 to June 14. While WWDC always includes a keynote, Apple has confirmed that it will be held on June 10 at 10:00 a.m. Pacific Time. Apple is expected to announce iOS 18, iPadOS 18, macOS 15, tvOS 18, watchOS 11, and visionOS 2, and at this time,...
Apple iPhone 15 Pro lineup Action button 230912

Apple Green-Lights iPhone 16 Pro Display Production

Tuesday May 28, 2024 5:13 am PDT by
Samsung Display and LG Display have been granted approval for mass production of OLED screens for Apple's upcoming iPhone 16 Pro models, Korea's The Elec reports. Both suppliers apparently received approval earlier this month, paving the way for the commencement of mass production of screens for the iPhone 16 Pro models. While Samsung Display will supply OLED screens for all four iPhone 16...
iPad Pro Landscape Apple Logo Feature

Apple Says Future iPads Could Feature Landscape Apple Logo

Monday May 27, 2024 6:31 am PDT by
French website Numerama interviewed three senior Apple employees about the new iPad Pro models that launched earlier this month. While the discussion did not reveal many new details, it did mention one potential change for future iPads. While the Apple logo on the back of iPads is positioned so that it appears upright in vertical orientation, the devices are often used in landscape...

Top Rated Comments

Geert76 Avatar
36 months ago
Apple, just cancel this on device surveillance software
Score: 166 Votes (Like | Disagree)
nvmls Avatar
36 months ago
They clearly don't know how the technology works.. oh wait
Score: 125 Votes (Like | Disagree)
nexu Avatar
36 months ago
This is getting more and more disturbing
Score: 117 Votes (Like | Disagree)
TheRealNick Avatar
36 months ago
If they don’t cancel this I’m seriously going to have to look at alternative products, which saddens me.
Score: 108 Votes (Like | Disagree)
dwsolberg Avatar
36 months ago
It's time for Apple to admit it's made a mistake. It's seriously bizarre they ever thought this was a good idea.
Score: 95 Votes (Like | Disagree)
VulchR Avatar
36 months ago
Waiting for the surveillance apologists to argue these two researchers "don't understand" the process...
Score: 93 Votes (Like | Disagree)