Nearly All Mobile Device Makers Cheat on Benchmarks, Except Apple and Motorola
Following Tuesday's report that Samsung artificially inflates its benchmarking scores, well-respected hardware review site AnandTech has published evidence suggesting nearly all mobile manufacturers, with the exception of Apple and Motorola, use CPU/GPU optimizations to game benchmark tests.
Samsung and other OEMs use a variety of methods to enhance device performance when a benchmark is detected. For example, with its Galaxy S 4 Samsung raised its thermal limits (and max GPU frequency) to get an edge on certain benchmarks and also raised its CPU voltage/frequency to its highest state when a benchmark was sensed, a tactic engaged by multiple manufacturers like LG, HTC, and ASUS as well.
In the table below, Anandtech highlights devices that detect benchmarks and immediately respond with max CPU frequency.
With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device that runs this silly CPU optimization. It’s possible that older Motorola devices might’ve done the same thing, but none of the newer devices we have on hand exhibited the behavior. It’s a systemic problem that seems to have surfaced over the last two years, and one that extends far beyond Samsung.
AnandTech notes that it’s a continual "cat and mouse" game discovering which devices have optimized for which benchmarks, because targeted benchmarks must be avoided.
The only realistic solution is to continue to evolve the suite ahead of those optimizing for it. The more attention you draw to certain benchmarks, the more likely they are to be gamed. We constantly play this game of cat and mouse on the PC side, it's just more frustrating in mobile since there aren’t many good benchmarks to begin with. […]
There's no single solution here, but rather a multi-faceted approach to make sure we’re ahead of the curve. We need to continue to rev our test suite to stay ahead of any aggressive OEM optimizations, we need to petition the OEMs to stop this madness, we need to work with the benchmark vendors to detect and disable optimizations as they happen and avoid benchmarks that are easily gamed.
Despite all of the effort that OEMs put into benchmark optimizations, the gains are negligible. The impact on CPU tests revealed a 0 to 5 percent performance increase, and a less than 10 percent increase on GPU benchmarks.
Popular Stories
Software from an iPhone prototype running an early build of iOS 26 leaked last week, giving us a glimpse at future Apple devices and iOS features. We recapped device codenames in our prior article, and now we have a list of some of the most notable feature flags that were found in the software code.
In some cases, it's obvious what the feature flags are referring to, while some are more...
Last week, details about unreleased Apple devices and future iOS features were shared by Macworld. This week, we learned where the information came from, plus we have more findings from the leak.
As it turns out, an Apple prototype device running an early build of iOS 26 was sold, and the person who bought it shared the software. The OS has a version number of 23A5234w, and the first...
Next year's iPhone 18 Pro and iPhone 18 Pro Max will be equipped with under-screen Face ID, and the front camera will be moved to the top-left corner of the screen, according to a new report from The Information's Wayne Ma and Qianer Liu.
As a result of these changes, the report said the iPhone 18 Pro models will not have a pill-shaped Dynamic Island cutout at the top of the screen....
Apple is testing iOS 26.3, the next version of iOS 26 that will launch around January. Since iOS 26.3's testing is happening over the holidays, it is a smaller update with fewer features than we've seen in prior betas.
We've rounded up what's new so far, and we'll add to our list with subsequent betas if we come across any other features.
Transfer to Android
Apple is making it simpler...
The first foldable iPhone will feature a series of design and hardware firsts for Apple, according to details shared by the Weibo leaker known as Digital Chat Station.
According to a new post, via machine translation, Apple is developing what the leaker describes as a "wide foldable" device, a term used to refer to a horizontally oriented, book-style foldable with a large internal display....
Macworld's Filipe Espósito today revealed a handful of features that Apple is allegedly planning for iOS 26.4, iOS 27, and even iOS 28.
The report said the features are referenced within the code for a leaked internal build of iOS 26 that is not meant to be seen by the public. However, it appears that Espósito and/or his sources managed to gain access to it, providing us with a sneak peek...
We have been covering iPhone 18 Pro, iPhone 17e, and iPhone Fold details from The Information's report about future iPhone models, and next up is the iPhone Air 2.
The report says that Apple aims to make the iPhone Air 2 more attractive in two ways.
First, Apple is apparently considering adding a second rear camera to the device, which would resolve a key limitation. The current iPhone...
Apple released the AirPods Max on December 15, 2020, meaning the over-ear headphones launched five years ago today. While the AirPods Max were updated with a USB-C port and new color options last year, followed by support for lossless audio and ultra-low latency audio this year, the headphones lack some of the features that have been introduced for newer generations of the regular AirPods and the ...
Apple is developing a high-end iMac featuring the M5 Max chip, according to information from leaked internal software.
Subscribe to the MacRumors YouTube channel for more videos.
The finding comes from leaked kernel debug kit files used by Apple engineers. These kernel debug kit files enumerate unreleased Apple hardware by internal identifiers, such as codenames and platform names, and they...
