University Researchers Who Built a CSAM Scanning System Urge Apple to Not Use the 'Dangerous' Technology

Respected university researchers are sounding the alarm bells over the technology behind Apple's plans to scan iPhone users' photo libraries for CSAM, or child sexual abuse material, calling the technology "dangerous."

apple privacy
Jonanath Mayer, an assistant professor of computer science and public affairs at Princeton University, as well as Anunay Kulshrestha, a researcher at Princeton University Center for Information Technology Policy, both penned an op-ed for The Washington Post, outlining their experiences with building image detection technology.

The researchers started a project two years ago to identity CSAM in end-to-end encrypted online services. The researchers note that given their field, they "know the value of end-to-end encryption, which protects data from third-party access." That concern, they say, is what horrifies them over CSAM "proliferating on encrypted platforms."

Mayer and Kulshrestha said they wanted to find a middle ground for the situation: build a system that online platforms could use to find CSAM and protect end-to-end encryption. The researchers note that experts in the field doubted the prospect of such a system, but they did manage to build it and in the process noticed a significant problem.

We sought to explore a possible middle ground, where online services could identify harmful content while otherwise preserving end-to-end encryption. The concept was straightforward: If someone shared material that matched a database of known harmful content, the service would be alerted. If a person shared innocent content, the service would learn nothing. People couldn't read the database or learn whether content matched, since that information could reveal law enforcement methods and help criminals evade detection.

Knowledgeable observers argued a system like ours was far from feasible. After many false starts, we built a working prototype. But we encountered a glaring problem.

Since Apple's announcement of the feature, the company has been bombarded with concerns that the system behind detecting CSAM could be used to detect other forms of photos at the request of oppressive governments. Apple has strongly pushed back against such a possibility, saying it will refuse any such request from governments.

Nonetheless, concerns over the future implications of the technology being used for CSAM detection are widespread. Mayer and Kulshrestha said that their concerns over how governments could use the system to detect content other than CSAM had them "disturbed."

A foreign government could, for example, compel a service to out people sharing disfavored political speech. That's no hypothetical: WeChat, the popular Chinese messaging app, already uses content matching to identify dissident material. India enacted rules this year that could require pre-screening content critical of government policy. Russia recently fined Google, Facebook and Twitter for not removing pro-democracy protest materials.

We spotted other shortcomings. The content-matching process could have false positives, and malicious users could game the system to subject innocent users to scrutiny.

We were so disturbed that we took a step we hadn't seen before in computer science literature: We warned against our own system design, urging further research on how to mitigate the serious downsides....

Apple has continued to address user concerns over its plans, publishing additional documents and an FAQ page. Apple continues to believe that its CSAM detection system, which will occur on a user's device, aligns with its long-standing privacy values.

Top Rated Comments

Geert76 Avatar
35 months ago
Apple, just cancel this on device surveillance software
Score: 166 Votes (Like | Disagree)
nvmls Avatar
35 months ago
They clearly don't know how the technology works.. oh wait
Score: 125 Votes (Like | Disagree)
nexu Avatar
35 months ago
This is getting more and more disturbing
Score: 117 Votes (Like | Disagree)
TheRealNick Avatar
35 months ago
If they don’t cancel this I’m seriously going to have to look at alternative products, which saddens me.
Score: 108 Votes (Like | Disagree)
dwsolberg Avatar
35 months ago
It's time for Apple to admit it's made a mistake. It's seriously bizarre they ever thought this was a good idea.
Score: 95 Votes (Like | Disagree)
VulchR Avatar
35 months ago
Waiting for the surveillance apologists to argue these two researchers "don't understand" the process...
Score: 93 Votes (Like | Disagree)

Popular Stories

iOS 18 Siri Integrated Feature

iOS 18 Will Add These New Features to Your iPhone

Friday April 12, 2024 11:11 am PDT by
iOS 18 is expected to be the "biggest" update in the iPhone's history. Below, we recap rumored features and changes for the iPhone. iOS 18 is rumored to include new generative AI features for Siri and many apps, and Apple plans to add RCS support to the Messages app for an improved texting experience between iPhones and Android devices. The update is also expected to introduce a more...
iPhone 16 Camera Lozenge 2 Colors

iPhone 16 Plus Rumored to Come in These 7 Colors

Wednesday April 10, 2024 3:52 am PDT by
Apple's iPhone 16 Plus may come in seven colors that either build upon the existing five colors in the standard iPhone 15 lineup or recast them in a new finish, based on a new rumor out of China. According to the Weibo-based leaker Fixed focus digital, Apple's upcoming larger 6.7-inch iPhone 16 Plus model will come in the following colors, compared to the colors currently available for the...
apple silicon feature joeblue

Macs to Get AI-Focused M4 Chips Starting in Late 2024

Thursday April 11, 2024 10:10 am PDT by
Apple will begin updating its Mac lineup with M4 chips in late 2024, according to Bloomberg's Mark Gurman. The M4 chip will be focused on improving performance for artificial intelligence capabilities. Last year, Apple introduced the M3, M3 Pro, and M3 Max chips all at once in October, so it's possible we could see the M4 lineup come during the same time frame. Gurman says that the entire...
top stories 13apr2024

Top Stories: M4 Mac Roadmap Leaked, New iPads in Second Week of May, and More

Saturday April 13, 2024 6:00 am PDT by
Apple's hardware roadmap was in the news this week, with things hopefully firming up for a launch of updated iPad Pro and iPad Air models next month while we look ahead to the other iPad models and a full lineup of M4-based Macs arriving starting later this year. We also heard some fresh rumors about iOS 18, due to be unveiled at WWDC in a couple of months, while we took a look at how things ...
new best buy blue

Best Buy Opens Up Sitewide Sale With Record Low Prices on M3 MacBook Air, iPad, and Much More

Saturday April 13, 2024 7:41 am PDT by
Best Buy this weekend has a big sale on Apple MacBooks and iPads, including new all-time low prices on the M3 MacBook Air, alongside the best prices we've ever seen on MacBook Pro, iPad, and more. Some of these deals require a My Best Buy Plus or My Best Buy Total membership, which start at $49.99/year. In addition to exclusive access to select discounts, you'll get free 2-day shipping, an...