Nearly All Mobile Device Makers Cheat on Benchmarks, Except Apple and Motorola

Following Tuesday's report that Samsung artificially inflates its benchmarking scores, well-respected hardware review site AnandTech has published evidence suggesting nearly all mobile manufacturers, with the exception of Apple and Motorola, use CPU/GPU optimizations to game benchmark tests.

Samsung and other OEMs use a variety of methods to enhance device performance when a benchmark is detected. For example, with its Galaxy S 4 Samsung raised its thermal limits (and max GPU frequency) to get an edge on certain benchmarks and also raised its CPU voltage/frequency to its highest state when a benchmark was sensed, a tactic engaged by multiple manufacturers like LG, HTC, and ASUS as well.

In the table below, Anandtech highlights devices that detect benchmarks and immediately respond with max CPU frequency.

anandtechbenchmarks

With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device that runs this silly CPU optimization. It’s possible that older Motorola devices might’ve done the same thing, but none of the newer devices we have on hand exhibited the behavior. It’s a systemic problem that seems to have surfaced over the last two years, and one that extends far beyond Samsung.

AnandTech notes that it’s a continual "cat and mouse" game discovering which devices have optimized for which benchmarks, because targeted benchmarks must be avoided.

The only realistic solution is to continue to evolve the suite ahead of those optimizing for it. The more attention you draw to certain benchmarks, the more likely they are to be gamed. We constantly play this game of cat and mouse on the PC side, it's just more frustrating in mobile since there aren’t many good benchmarks to begin with. […]

There's no single solution here, but rather a multi-faceted approach to make sure we’re ahead of the curve. We need to continue to rev our test suite to stay ahead of any aggressive OEM optimizations, we need to petition the OEMs to stop this madness, we need to work with the benchmark vendors to detect and disable optimizations as they happen and avoid benchmarks that are easily gamed.

Despite all of the effort that OEMs put into benchmark optimizations, the gains are negligible. The impact on CPU tests revealed a 0 to 5 percent performance increase, and a less than 10 percent increase on GPU benchmarks.

Top Rated Comments

NutsNGum Avatar
120 months ago
Guy 1: My phone is faster than your phone!

Guy 2: I have a girlfriend.
Score: 132 Votes (Like | Disagree)
pk7 Avatar
120 months ago
Mad respect to Anandtech for making this chart. I love the title of the table :D

Props to Apple and Motorola for not being so cheap.
Score: 53 Votes (Like | Disagree)
notjustjay Avatar
120 months ago
"I can't believe I have to make this table"

Hah!
Score: 47 Votes (Like | Disagree)
BC2009 Avatar
120 months ago
And this is why Android phones have been able to claim great performance via benchmarks and specs while iPhones have performed just as well or better in practice. I remember folks telling me that the Galaxy S2 performed better than my iPhone 3GS according to benchmarks, but in practice (side-by-side) my iPhone 3GS seemed to be twice as fast with no stutter on tasks that seemed to stutter and lag on the GS2.
Score: 25 Votes (Like | Disagree)
charlitos Avatar
120 months ago
Im Sorry but Samsung got Ys across the board. They are also the only one on that table cheating Geekbench which is one of the most popular benchmarking apps used.
Score: 25 Votes (Like | Disagree)
nezr Avatar
120 months ago
This week's lesson: don't buy Samsung
Score: 21 Votes (Like | Disagree)

Popular Stories

iphone 14 pro hands snowflakes 1

Best Black Friday iPhone Deals Still Available

Wednesday November 23, 2022 1:55 pm PST by
Cellular carriers have always offered big savings on the newest iPhone models during the holidays, and Black Friday 2022 is no different. Right now we're tracking notable offers on the iPhone 14 and iPhone 14 Pro devices from AT&T, Verizon, and T-Mobile. For even more savings, keep an eye on older models like the iPhone 13. Note: MacRumors is an affiliate partner with some of these vendors....
General Black Friday Deals 2022 Green

All the Apple Black Friday Deals You Can Still Get

Friday November 25, 2022 4:40 am PST by
Although Black Friday is now technically over, many Apple products are still seeing major discounts. In this article, you'll find every Apple device with a notable Black Friday sale that's still available. We'll be updating as prices change and new deals arrive, so be sure to keep an eye out if you don't see the sale you're looking for yet. Note: MacRumors is an affiliate partner with some of...
ipad holiday bulbs

Best Black Friday iPad Deals Still Available

Thursday November 24, 2022 12:25 pm PST by
Black Friday deals have been in full swing for the better part of a month, and now that the shopping holiday is officially here we're seeing even more solid discounts on Apple devices. We're highlighting the best sales for all of Apple's product lines, and in this article you'll find the best Black Friday sales on iPad, iPad Pro, iPad Air, and iPad mini. Note: MacRumors is an affiliate partner ...
airpods pro 2

Apple Engineer Addresses Lack of Lossless Support on New AirPods Pro

Friday November 25, 2022 2:58 am PST by
An Apple engineer has addressed the lack of lossless audio support in the second-generation AirPods Pro in a new interview. Current Bluetooth technology in the AirPods lineup means that Apple's audio products do not support Apple Music Lossless audio. Apple has previously hinted that it may develop its own codec and connectivity standard that builds on AirPlay and supports higher quality...
mac imac snowflakes

Best Black Friday iMac and MacBook Deals Still Available

Thursday November 24, 2022 1:07 pm PST by
Our Black Friday coverage continues today with the best deals you can find on MacBook Pro, MacBook Air, and iMac. As with all Black Friday deals, we aren't sure how long any of these will last, and prices are always fluctuating, so if you see something you want, be sure to buy it soon. Note: MacRumors is an affiliate partner with some of these vendors. When you click a link and make a...
maxresdefault

Nothing Phone 1 Displays AirPods Battery Level After Latest OS Update

Friday November 25, 2022 3:33 am PST by
Nothing Phone 1 users today began receiving the Nothing OS 1.1.7 update, which adds support for displaying the battery percentage of connected AirPods, amongst other improvements and bug fixes. If you own a Nothing Phone 1, you can check for the OTA update by going to Settings -> System -> System updates. Bear in mind that as support for displaying AirPods battery level is still an...
apple watch gold ornaments

Best Black Friday Apple Watch Deals Still Available

Wednesday November 23, 2022 9:31 am PST by
We're tracking all of the best Apple product discounts for Black Friday this week, and the Apple Watch always makes a great gift around the holiday season, so you're guaranteed to find solid discounts right now. In this article, you'll discover the best Black Friday sales on Apple Watch Series 8, Apple Watch SE, and Apple Watch Ultra. Note: MacRumors is an affiliate partner with some of these...