Nearly All Mobile Device Makers Cheat on Benchmarks, Except Apple and Motorola

Following Tuesday's report that Samsung artificially inflates its benchmarking scores, well-respected hardware review site AnandTech has published evidence suggesting nearly all mobile manufacturers, with the exception of Apple and Motorola, use CPU/GPU optimizations to game benchmark tests.

Samsung and other OEMs use a variety of methods to enhance device performance when a benchmark is detected. For example, with its Galaxy S 4 Samsung raised its thermal limits (and max GPU frequency) to get an edge on certain benchmarks and also raised its CPU voltage/frequency to its highest state when a benchmark was sensed, a tactic engaged by multiple manufacturers like LG, HTC, and ASUS as well.

In the table below, Anandtech highlights devices that detect benchmarks and immediately respond with max CPU frequency.

anandtechbenchmarks

With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device that runs this silly CPU optimization. It’s possible that older Motorola devices might’ve done the same thing, but none of the newer devices we have on hand exhibited the behavior. It’s a systemic problem that seems to have surfaced over the last two years, and one that extends far beyond Samsung.

AnandTech notes that it’s a continual "cat and mouse" game discovering which devices have optimized for which benchmarks, because targeted benchmarks must be avoided.

The only realistic solution is to continue to evolve the suite ahead of those optimizing for it. The more attention you draw to certain benchmarks, the more likely they are to be gamed. We constantly play this game of cat and mouse on the PC side, it's just more frustrating in mobile since there aren’t many good benchmarks to begin with. […]

There's no single solution here, but rather a multi-faceted approach to make sure we’re ahead of the curve. We need to continue to rev our test suite to stay ahead of any aggressive OEM optimizations, we need to petition the OEMs to stop this madness, we need to work with the benchmark vendors to detect and disable optimizations as they happen and avoid benchmarks that are easily gamed.

Despite all of the effort that OEMs put into benchmark optimizations, the gains are negligible. The impact on CPU tests revealed a 0 to 5 percent performance increase, and a less than 10 percent increase on GPU benchmarks.

Top Rated Comments

NutsNGum Avatar
136 months ago
Guy 1: My phone is faster than your phone!

Guy 2: I have a girlfriend.
Score: 132 Votes (Like | Disagree)
pk7 Avatar
136 months ago
Mad respect to Anandtech for making this chart. I love the title of the table :D

Props to Apple and Motorola for not being so cheap.
Score: 53 Votes (Like | Disagree)
notjustjay Avatar
136 months ago
"I can't believe I have to make this table"

Hah!
Score: 47 Votes (Like | Disagree)
BC2009 Avatar
136 months ago
And this is why Android phones have been able to claim great performance via benchmarks and specs while iPhones have performed just as well or better in practice. I remember folks telling me that the Galaxy S2 performed better than my iPhone 3GS according to benchmarks, but in practice (side-by-side) my iPhone 3GS seemed to be twice as fast with no stutter on tasks that seemed to stutter and lag on the GS2.
Score: 25 Votes (Like | Disagree)
charlitos Avatar
136 months ago
Im Sorry but Samsung got Ys across the board. They are also the only one on that table cheating Geekbench which is one of the most popular benchmarking apps used.
Score: 25 Votes (Like | Disagree)
nezr Avatar
136 months ago
This week's lesson: don't buy Samsung
Score: 21 Votes (Like | Disagree)

Popular Stories

Apple car wheel icon feature yellow

Apple Cancels Electric Car Project

Tuesday February 27, 2024 11:05 am PST by
Apple has canceled all plans to release an autonomous, electric vehicle, reports Bloomberg. Apple has been working on an Apple Car for more than a decade and invested millions of dollars into development before deciding it was not a viable project. Apple's Chief Operating Officer Jeff Williams today told approximately 2,000 employees working on the Apple Car that the project was canceled,...
Beyond iPhone 13 Better Blue Face ID Single Camera Hole

Six Reasons to Wait for Next Year's iPhone 17

Thursday February 22, 2024 4:20 am PST by
Apple's iPhone development roadmap runs several years into the future and the company is continually working with suppliers on several successive iPhone models concurrently, which is why we sometimes get rumored feature leaks so far ahead of launch. The iPhone 17 series is no different, and already we have some idea of what to expect from Apple's 2025 smartphone lineup. If you plan to skip...
General Apps Messages

iOS 17.4 to Add This 'Groundbreaking' New Messaging Feature

Friday February 23, 2024 5:05 am PST by
With iOS 17.4, set to arrive in March 2024, Apple is bringing a new cryptographic security feature to iMessage called PQ3. This "groundbreaking" and "state-of-the-art" protocol provides "extensive defenses against even highly sophisticated quantum attacks," according to Apple. Let's break down what that means. Apple's iMessage service already supports end-to-end encryption, but security...
iOS 18 Mock iPhone 16 Feature Gray

iOS 18 Rumored to Be Compatible With These iPhone Models

Tuesday February 27, 2024 6:31 am PST by
iOS 18 will be compatible with the iPhone XR, and thereby also the iPhone XS and iPhone XS Max models with the same A12 Bionic chip, according to a post on X today from a private account with a proven track record of sharing build numbers for upcoming iOS updates. The post was spotted by MacRumors contributor Aaron Perris, and it has since been deleted. However, this was likely because the...
iOS 17

iOS 17.4 Coming Soon With These New Features for Your iPhone

Monday February 26, 2024 6:08 am PST by
In a press release last month, Apple confirmed that iOS 17.4 will be released in March, and the update includes several new features and changes for the iPhone. Key new features in iOS 17.4 include major App Store changes in the EU, Apple Podcasts transcripts, and an iMessage security upgrade. The update also adds new emoji and includes preparations for the launch of next-generation CarPlay...
applearcade

Game Developers Describe 'Smell of Death' Around Apple Arcade

Monday February 26, 2024 7:24 am PST by
Some game developers are dissatisfied with Apple Arcade amid concerns about the subscription service's future, a new report claims. Sources speaking to mobilegamer.biz described a "smell of death" around Apple's games subscription service and noted the difference between the company's investment in TV and music, and its interest in games. "At the very top of the company there needs to be a ...