Nearly All Mobile Device Makers Cheat on Benchmarks, Except Apple and Motorola

Following Tuesday's report that Samsung artificially inflates its benchmarking scores, well-respected hardware review site AnandTech has published evidence suggesting nearly all mobile manufacturers, with the exception of Apple and Motorola, use CPU/GPU optimizations to game benchmark tests.

Samsung and other OEMs use a variety of methods to enhance device performance when a benchmark is detected. For example, with its Galaxy S 4 Samsung raised its thermal limits (and max GPU frequency) to get an edge on certain benchmarks and also raised its CPU voltage/frequency to its highest state when a benchmark was sensed, a tactic engaged by multiple manufacturers like LG, HTC, and ASUS as well.

In the table below, Anandtech highlights devices that detect benchmarks and immediately respond with max CPU frequency.

anandtechbenchmarks

With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device that runs this silly CPU optimization. It’s possible that older Motorola devices might’ve done the same thing, but none of the newer devices we have on hand exhibited the behavior. It’s a systemic problem that seems to have surfaced over the last two years, and one that extends far beyond Samsung.

AnandTech notes that it’s a continual "cat and mouse" game discovering which devices have optimized for which benchmarks, because targeted benchmarks must be avoided.

The only realistic solution is to continue to evolve the suite ahead of those optimizing for it. The more attention you draw to certain benchmarks, the more likely they are to be gamed. We constantly play this game of cat and mouse on the PC side, it's just more frustrating in mobile since there aren’t many good benchmarks to begin with. […]

There's no single solution here, but rather a multi-faceted approach to make sure we’re ahead of the curve. We need to continue to rev our test suite to stay ahead of any aggressive OEM optimizations, we need to petition the OEMs to stop this madness, we need to work with the benchmark vendors to detect and disable optimizations as they happen and avoid benchmarks that are easily gamed.

Despite all of the effort that OEMs put into benchmark optimizations, the gains are negligible. The impact on CPU tests revealed a 0 to 5 percent performance increase, and a less than 10 percent increase on GPU benchmarks.

Top Rated Comments

NutsNGum Avatar
108 months ago
Guy 1: My phone is faster than your phone!

Guy 2: I have a girlfriend.
Score: 132 Votes (Like | Disagree)
pk7 Avatar
108 months ago
Mad respect to Anandtech for making this chart. I love the title of the table :D

Props to Apple and Motorola for not being so cheap.
Score: 53 Votes (Like | Disagree)
notjustjay Avatar
108 months ago
"I can't believe I have to make this table"

Hah!
Score: 47 Votes (Like | Disagree)
BC2009 Avatar
108 months ago
And this is why Android phones have been able to claim great performance via benchmarks and specs while iPhones have performed just as well or better in practice. I remember folks telling me that the Galaxy S2 performed better than my iPhone 3GS according to benchmarks, but in practice (side-by-side) my iPhone 3GS seemed to be twice as fast with no stutter on tasks that seemed to stutter and lag on the GS2.
Score: 25 Votes (Like | Disagree)
charlitos Avatar
108 months ago
Im Sorry but Samsung got Ys across the board. They are also the only one on that table cheating Geekbench which is one of the most popular benchmarking apps used.
Score: 25 Votes (Like | Disagree)
nezr Avatar
108 months ago
This week's lesson: don't buy Samsung
Score: 21 Votes (Like | Disagree)

Popular Stories

AirPods Pro Gen 3 Mock Feature Red

AirPods Pro 2 Could Start a New Accessory Ecosystem

Friday January 14, 2022 2:34 am PST by
Apple's second-generation AirPods Pro could arrive alongside a new series of accessories, recent leaked images suggest. Alleged leaked photos of the next-generation AirPods Pro obtained by MacRumors showed a charging case with a metal loop on the side for attaching a strap. Apple has not used this design for any of its other AirPod models and it is unclear why it would be added in this...
netflix2

Netflix Again Raises Prices for All Plans, 4K Streaming Now $20 Per Month

Friday January 14, 2022 12:46 pm PST by
Netflix today updated the prices for its streaming plans, and all of its offerings are now more expensive. The Basic plan is now priced at $9.99 per month, the Standard plan is priced at $15.49 per month, and the Premium plan is priced at $19.99 per month. The Basic plan is $1 more expensive, up from $8.99 per month. This plan allows users to watch on just one screen at a time, and it limits ...
iPhone 14 Mock pill and hole 16x9 120hz

Analyst: All iPhone 14 Models to Feature 120Hz Displays, 6GB of RAM, and More

Friday January 14, 2022 7:02 am PST by
Apple is rumored to announce four new iPhone 14 models in September, and ahead of time, analyst Jeff Pu has outlined his expectations for the devices. In a research note with Haitong International Securities, obtained by MacRumors, Pu claimed that all iPhone 14 models will feature ProMotion displays, compared to only Pro models currently. ProMotion enables a variable refresh rate up to 120Hz ...
ios 15

Apple Releases Minor iOS 15.2.1 and iPadOS 15.2.1 Updates

Wednesday January 12, 2022 10:05 am PST by
Apple today released minor 15.2.1 updates for iPhone and iPad users, and the software comes one month after Apple launched iOS 15.2 and iPadOS 15.2 with a slew of improvements. The iOS 15.2.1 and iPadOS 15.2.1 update can be downloaded for free and the software is available on all eligible devices over-the-air in the Settings app. To access the new software, go to Settings > General >...
maxresdefault

Hands-On With LG's 32-Inch UltraFine OLED Pro Display

Wednesday January 12, 2022 1:12 pm PST by
LG in December announced the launch of its new 2022 32-inch UltraFine OLED Pro display, and in our latest YouTube video, we went hands-on with it to see how it compares to Apple's Pro Display XDR and whether it might be worth picking up depending on the price point. Subscribe to the MacRumors YouTube channel for more videos. Officially named the "32BP95E," the display features a resolution of ...
iPad Pro Big Ol Logo Orange

Next iPad Pro Might Feature Large Glass Apple Logo to Allow Wireless Charging

Friday January 14, 2022 10:44 am PST by
Bloomberg's Mark Gurman and Debby Wu last year reported that Apple had tested a new iPad Pro with a glass back for wireless charging capabilities. In a recent edition of his newsletter, Gurman said he expects the new iPad Pro to be released in 2022. While the new iPad Pro is still on track to feature wireless charging, 9to5Mac's Filipe Espósito today reported that Apple may have ultimately...
iPhone 14 Mock pill and hole

iPhone 14 Pro Now Rumored to Feature Both Pill-Shaped and Circular Cutouts

Wednesday January 12, 2022 9:26 am PST by
Apple is widely expected to remove the notch on iPhone 14 Pro models, but there have been conflicting rumors about the new design. Early rumors suggested that Apple would adopt a hole-punch design with Face ID somehow moved completely under the display, and later it was rumored that there would be a pill-shaped cutout instead. Now, display industry consultant Ross Young has claimed that...
fortnite apple logo geforce feature

Fortnite Coming Back to iOS Soon Thanks to Nvidia's GeForce NOW Service

Thursday January 13, 2022 11:19 am PST by
With the Apple vs. Epic Games lawsuit continuing on, there are no signs that Apple has any intention of allowing popular battle royale game Fortnite to return to the App Store on iPhone and iPad. Epic Games has found a workaround though, by partnering up with Nvidia. GeForce NOW, Nvidia's streaming gaming service, will soon add Fortnite support, allowing Fortnite to be played through a...
iMac 27 inch 2020 sale

Deals: Apple's 21.5-Inch iMac Hits Record Low Price of $599.99 ($499 Off) [Update: Out of Stock]

Thursday January 13, 2022 4:05 am PST by
Amazon today has a great deal on the 2017 Intel 21.5-inch iMac (2.3GHz, 8GB RAM, 256GB SSD), priced at $599.99, down from $1,099.00. This is the best price we've ever tracked on this model, and it's only available at Amazon. The sale price will be reflected after an automatic coupon is applied at checkout. Note: MacRumors is an affiliate partner with some of these vendors. When you click a...