Nearly All Mobile Device Makers Cheat on Benchmarks, Except Apple and Motorola

Following Tuesday's report that Samsung artificially inflates its benchmarking scores, well-respected hardware review site AnandTech has published evidence suggesting nearly all mobile manufacturers, with the exception of Apple and Motorola, use CPU/GPU optimizations to game benchmark tests.

Samsung and other OEMs use a variety of methods to enhance device performance when a benchmark is detected. For example, with its Galaxy S 4 Samsung raised its thermal limits (and max GPU frequency) to get an edge on certain benchmarks and also raised its CPU voltage/frequency to its highest state when a benchmark was sensed, a tactic engaged by multiple manufacturers like LG, HTC, and ASUS as well.

In the table below, Anandtech highlights devices that detect benchmarks and immediately respond with max CPU frequency.

anandtechbenchmarks

With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device that runs this silly CPU optimization. It’s possible that older Motorola devices might’ve done the same thing, but none of the newer devices we have on hand exhibited the behavior. It’s a systemic problem that seems to have surfaced over the last two years, and one that extends far beyond Samsung.

AnandTech notes that it’s a continual "cat and mouse" game discovering which devices have optimized for which benchmarks, because targeted benchmarks must be avoided.

The only realistic solution is to continue to evolve the suite ahead of those optimizing for it. The more attention you draw to certain benchmarks, the more likely they are to be gamed. We constantly play this game of cat and mouse on the PC side, it's just more frustrating in mobile since there aren’t many good benchmarks to begin with. […]

There's no single solution here, but rather a multi-faceted approach to make sure we’re ahead of the curve. We need to continue to rev our test suite to stay ahead of any aggressive OEM optimizations, we need to petition the OEMs to stop this madness, we need to work with the benchmark vendors to detect and disable optimizations as they happen and avoid benchmarks that are easily gamed.

Despite all of the effort that OEMs put into benchmark optimizations, the gains are negligible. The impact on CPU tests revealed a 0 to 5 percent performance increase, and a less than 10 percent increase on GPU benchmarks.

Top Rated Comments

NutsNGum Avatar
114 months ago
Guy 1: My phone is faster than your phone!

Guy 2: I have a girlfriend.
Score: 132 Votes (Like | Disagree)
pk7 Avatar
114 months ago
Mad respect to Anandtech for making this chart. I love the title of the table :D

Props to Apple and Motorola for not being so cheap.
Score: 53 Votes (Like | Disagree)
notjustjay Avatar
114 months ago
"I can't believe I have to make this table"

Hah!
Score: 47 Votes (Like | Disagree)
BC2009 Avatar
114 months ago
And this is why Android phones have been able to claim great performance via benchmarks and specs while iPhones have performed just as well or better in practice. I remember folks telling me that the Galaxy S2 performed better than my iPhone 3GS according to benchmarks, but in practice (side-by-side) my iPhone 3GS seemed to be twice as fast with no stutter on tasks that seemed to stutter and lag on the GS2.
Score: 25 Votes (Like | Disagree)
charlitos Avatar
114 months ago
Im Sorry but Samsung got Ys across the board. They are also the only one on that table cheating Geekbench which is one of the most popular benchmarking apps used.
Score: 25 Votes (Like | Disagree)
nezr Avatar
114 months ago
This week's lesson: don't buy Samsung
Score: 21 Votes (Like | Disagree)

Popular Stories

2022 back to school apple

Apple Launches 2022 Back to School Offer: Up to $150 Gift Card With Mac or iPad

Friday June 24, 2022 5:08 am PDT by
Apple today launched its annual "Back to School" promotion for college/university students in the United States and Canada. This year's promotion offers a free Apple gift card with the purchase of an eligible Mac or iPad, rather than free AirPods like last year. Apple is also offering students 20% off AppleCare+ plans during the promotion. Apple is offering a $150 gift card with the purchase ...
apple ar headset concept 2

Apple CEO Tim Cook Hints at AR/VR Headset: 'Stay Tuned and You'll See What We Have to Offer'

Wednesday June 22, 2022 6:34 pm PDT by
CEO Tim Cook this week did an interview with China Daily, where he once again commented on on the future of augmented reality and hinted at Apple's work on an AR/VR headset. Render via designer Ian Zelbo Cook said that Apple is excited about the opportunities available with augmented reality, which is not too far off from prior comments that he's made, but he went on to say that people should ...
airpods pro 2 1

AirPods Pro 2 Said to Feature Upgraded H1 Chip, Find My, Heart Rate Detection, USB-C and More

Friday June 24, 2022 9:48 am PDT by
The next-generation AirPods Pro could come with a long list of new features that include heart rate detection, the ability to function as a hearing aid, and a USB-C port according to a report from 52Audio. The site claims that it has received new information on the AirPods Pro 2, and it has used that information to provide some renders on what the earbuds might look like. Design wise, there...
customize wallpaper setting ios 16

iOS 16 Beta 2 Tidbits: New Wallpaper Colors, Backup Over LTE, SMS Filtering and More

Wednesday June 22, 2022 2:16 pm PDT by
Apple today seeded the second betas of iOS 16 and iPadOS 16 to developers for testing purposes, and the new betas introduce new features and refine some of the changes that Apple made with the first iOS 16 release. Subscribe to the MacRumors YouTube channel for more videos. Lock Screen Photo Wallpaper Customization When customizing a photo on the Lock Screen, there are two new DuoTone and...
home app ios 16

Apple Confirms iPad Will No Longer Be Supported as a Home Hub in iOS 16

Wednesday June 22, 2022 12:38 pm PDT by
The iPad will no longer be able to be used as a home hub following the launch of iOS 16, iPadOS 16, macOS Ventura, and the HomePod 16 software this fall, Apple confirmed today. As discovered in iOS 16 code by MacRumors contributor Steve Moser, Apple says that the iPad will no longer be supported as a home hub. This information will be displayed in the Home app after updating to iOS 16.A home ...
apple ar headset concept 1

Apple Rumored to Announce 'Game-Changer' AR/VR Headset in January 2023

Friday June 24, 2022 2:52 am PDT by
Apple is "likely" to announce its long-rumored mixed-reality headset as soon as January 2023, Apple analyst Ming-Chi Kuo has reiterated. Concept render based on purported leaked information by Ian Zelbo In a detailed post on Medium, Kuo explained that Apple's headset will be a "game-changer" for the augmented-reality and virtual-reality market. Describing some of the headset's...