Nearly All Mobile Device Makers Cheat on Benchmarks, Except Apple and Motorola

Following Tuesday's report that Samsung artificially inflates its benchmarking scores, well-respected hardware review site AnandTech has published evidence suggesting nearly all mobile manufacturers, with the exception of Apple and Motorola, use CPU/GPU optimizations to game benchmark tests.

Samsung and other OEMs use a variety of methods to enhance device performance when a benchmark is detected. For example, with its Galaxy S 4 Samsung raised its thermal limits (and max GPU frequency) to get an edge on certain benchmarks and also raised its CPU voltage/frequency to its highest state when a benchmark was sensed, a tactic engaged by multiple manufacturers like LG, HTC, and ASUS as well.

In the table below, Anandtech highlights devices that detect benchmarks and immediately respond with max CPU frequency.

anandtechbenchmarks

With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device that runs this silly CPU optimization. It’s possible that older Motorola devices might’ve done the same thing, but none of the newer devices we have on hand exhibited the behavior. It’s a systemic problem that seems to have surfaced over the last two years, and one that extends far beyond Samsung.

AnandTech notes that it’s a continual "cat and mouse" game discovering which devices have optimized for which benchmarks, because targeted benchmarks must be avoided.

The only realistic solution is to continue to evolve the suite ahead of those optimizing for it. The more attention you draw to certain benchmarks, the more likely they are to be gamed. We constantly play this game of cat and mouse on the PC side, it's just more frustrating in mobile since there aren’t many good benchmarks to begin with. […]

There's no single solution here, but rather a multi-faceted approach to make sure we’re ahead of the curve. We need to continue to rev our test suite to stay ahead of any aggressive OEM optimizations, we need to petition the OEMs to stop this madness, we need to work with the benchmark vendors to detect and disable optimizations as they happen and avoid benchmarks that are easily gamed.

Despite all of the effort that OEMs put into benchmark optimizations, the gains are negligible. The impact on CPU tests revealed a 0 to 5 percent performance increase, and a less than 10 percent increase on GPU benchmarks.

Top Rated Comments

(View all)
Avatar
90 months ago
Guy 1: My phone is faster than your phone!

Guy 2: I have a girlfriend.
Score: 132 Votes (Like | Disagree)
Avatar
90 months ago
Mad respect to Anandtech for making this chart. I love the title of the table :D

Props to Apple and Motorola for not being so cheap.
Score: 53 Votes (Like | Disagree)
Avatar
90 months ago
"I can't believe I have to make this table"

Hah!
Score: 47 Votes (Like | Disagree)
Avatar
90 months ago
And this is why Android phones have been able to claim great performance via benchmarks and specs while iPhones have performed just as well or better in practice. I remember folks telling me that the Galaxy S2 performed better than my iPhone 3GS according to benchmarks, but in practice (side-by-side) my iPhone 3GS seemed to be twice as fast with no stutter on tasks that seemed to stutter and lag on the GS2.
Score: 25 Votes (Like | Disagree)
Avatar
90 months ago
Im Sorry but Samsung got Ys across the board. They are also the only one on that table cheating Geekbench which is one of the most popular benchmarking apps used.
Score: 25 Votes (Like | Disagree)
Avatar
90 months ago
This week's lesson: don't buy Samsung
Score: 21 Votes (Like | Disagree)

Top Stories

Apple Takes Legal Action Against Small Company With Pear Logo

Saturday August 8, 2020 11:09 am PDT by
Apple is taking legal action against the developers of the app "Prepear" due to its logo, according to iPhone in Canada. Prepear is an app that helps users discover recipes, plan meals, make lists, and arrange grocery deliveries. The app is a spinoff of "Super Healthy Kids," and the founders claim that they are facing litigation from Apple. Apple reportedly takes issue with Prepear's logo, ...

Apple Seeds iOS 14 and iPadOS 14 Public Beta 4 to Testers

Thursday August 6, 2020 10:05 am PDT by
Apple today seeded new public betas of upcoming iOS 14 and iPadOS 14 updates to its public beta testing group. Today's software releases, which Apple labels as fourth betas to keep them in line with developer betas, are actually the third betas that Apple has provided and they come two weeks after the prior beta releases. Public beta testers who have signed up for Apple's beta testing...

Supposed iPhone 12 Display Unit Leaks

Thursday August 6, 2020 8:13 am PDT by
An image supposedly of an iPhone 12 display unit has been shared online by leaker "Twitter user Mr. White". Compared to images of an iPhone 11 Pro display piece, this new unit has a reoriented display connector, reaching up from the bottom of the display, rather than from the left-hand side on iPhone 11 Pro. This may be due to the logic board moving to the other side of the device. A...

8 Third-Party Home Screen Widgets That You Can Try Out Now on iOS 14

Wednesday August 5, 2020 12:56 pm PDT by
One of the biggest new features of iOS 14 is Home Screen widgets, which provide information from apps at a glance. The widgets can be pinned to the Home Screen in various spots and sizes, allowing for many different layouts. When the iOS 14 beta was first released in June, widgets were limited to Apple's own apps like Calendar and Weather, but several third-party developers have begun to test ...

New 27-Inch iMac's Storage Affixed to Logic Board, 4TB and 8TB Configurations Have Expansion Connector

Friday August 7, 2020 7:46 am PDT by
Following a report from German blog iFun.de that claimed the new 27-inch iMac's flash storage is soldered to the logic board, MacRumors has obtained additional information in an internal document for Apple technicians. In the document, Apple says that the flash storage is indeed affixed to the logic board and cannot be removed. However, for the 4TB and 8TB configurations, Apple says that a...

2020 iMac Teardown Reveals Internal Changes and Similarities

Saturday August 8, 2020 12:44 pm PDT by
A teardown video, shared by OWC, reveals the internal changes in the new 2020 27-inch iMac. The 2020 27-inch iMac was announced earlier this week with 10th-generation Intel Core processors, AMD Radeon Pro 5000 series graphics, up to 128GB of RAM, up to 8TB of storage, a 1080p front-facing FaceTime camera, a True Tone display with a nano-texture glass option, higher fidelity speakers, and...

Kuo: iPhone 12 Camera Lens Supplier Experiencing Quality Issues, But May Not Affect Launch Schedule

Thursday August 6, 2020 9:30 pm PDT by
One of Apple's camera lens suppliers for the upcoming non-Pro iPhone 12 models is having quality issues, according to a new research note from analyst Ming-Chi Kuo seen by MacRumors. The issues are, however, unlikely to have an impact on Apple's shipping schedule, presuming that Apple's other main supplier can make up the difference. Kuo says that Genius Electronic Optical has discovered...

Google's $349 Pixel 4a vs. Apple's $399 iPhone SE

Wednesday August 5, 2020 1:45 pm PDT by
Google this week launched its newest smartphone, the $349 Pixel 4a, a low-cost device that's designed to compete with other affordable devices like Apple's iPhone SE. We picked up one of the new Pixel 4a smartphones and thought we'd check it out to see how it measures up to the iPhone SE, given that the two devices have such similar price points. Subscribe to the MacRumors YouTube channel ...

Sony WH-1000XM4 Noise-Canceling Headphones Now Available for $350

Friday August 7, 2020 2:01 am PDT by
Sony's flagship WH-1000MX3 noise-canceling headphones have been among the best on the market for some time, and this week Sony released its fourth-generation WH-1000XM4 headphones boasting some notable improvements on the previous model. The XM4s feature the same basic design as the XM3s, but are one gram lighter at 254 grams due to slightly larger ear pads and some subtle tweaks to the...

Apple Announces New 27-Inch iMac With 10th-Gen Processors, Up to 128GB RAM, 1080p Webcam, True Tone, and More

Tuesday August 4, 2020 8:07 am PDT by
Apple today announced a new 27-inch iMac with faster 10th-generation Intel Core processor options, next-generation AMD graphics, up to 128GB of RAM, a higher-resolution 1080p front-facing FaceTime camera, a True Tone display with a nano-texture glass option, a T2 chip, higher fidelity speakers, studio-quality microphones, and more. A breakdown of the new 27-inch iMac's features and specs:10th...