Nearly All Mobile Device Makers Cheat on Benchmarks, Except Apple and Motorola

Following Tuesday's report that Samsung artificially inflates its benchmarking scores, well-respected hardware review site AnandTech has published evidence suggesting nearly all mobile manufacturers, with the exception of Apple and Motorola, use CPU/GPU optimizations to game benchmark tests.

Samsung and other OEMs use a variety of methods to enhance device performance when a benchmark is detected. For example, with its Galaxy S 4 Samsung raised its thermal limits (and max GPU frequency) to get an edge on certain benchmarks and also raised its CPU voltage/frequency to its highest state when a benchmark was sensed, a tactic engaged by multiple manufacturers like LG, HTC, and ASUS as well.

In the table below, Anandtech highlights devices that detect benchmarks and immediately respond with max CPU frequency.

anandtechbenchmarks

With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device that runs this silly CPU optimization. It’s possible that older Motorola devices might’ve done the same thing, but none of the newer devices we have on hand exhibited the behavior. It’s a systemic problem that seems to have surfaced over the last two years, and one that extends far beyond Samsung.

AnandTech notes that it’s a continual "cat and mouse" game discovering which devices have optimized for which benchmarks, because targeted benchmarks must be avoided.

The only realistic solution is to continue to evolve the suite ahead of those optimizing for it. The more attention you draw to certain benchmarks, the more likely they are to be gamed. We constantly play this game of cat and mouse on the PC side, it's just more frustrating in mobile since there aren’t many good benchmarks to begin with. […]

There's no single solution here, but rather a multi-faceted approach to make sure we’re ahead of the curve. We need to continue to rev our test suite to stay ahead of any aggressive OEM optimizations, we need to petition the OEMs to stop this madness, we need to work with the benchmark vendors to detect and disable optimizations as they happen and avoid benchmarks that are easily gamed.

Despite all of the effort that OEMs put into benchmark optimizations, the gains are negligible. The impact on CPU tests revealed a 0 to 5 percent performance increase, and a less than 10 percent increase on GPU benchmarks.

Top Rated Comments

NutsNGum Avatar
95 months ago
Guy 1: My phone is faster than your phone!

Guy 2: I have a girlfriend.
Score: 132 Votes (Like | Disagree)
pk7 Avatar
95 months ago
Mad respect to Anandtech for making this chart. I love the title of the table :D

Props to Apple and Motorola for not being so cheap.
Score: 53 Votes (Like | Disagree)
notjustjay Avatar
95 months ago
"I can't believe I have to make this table"

Hah!
Score: 47 Votes (Like | Disagree)
BC2009 Avatar
95 months ago
And this is why Android phones have been able to claim great performance via benchmarks and specs while iPhones have performed just as well or better in practice. I remember folks telling me that the Galaxy S2 performed better than my iPhone 3GS according to benchmarks, but in practice (side-by-side) my iPhone 3GS seemed to be twice as fast with no stutter on tasks that seemed to stutter and lag on the GS2.
Score: 25 Votes (Like | Disagree)
charlitos Avatar
95 months ago
Im Sorry but Samsung got Ys across the board. They are also the only one on that table cheating Geekbench which is one of the most popular benchmarking apps used.
Score: 25 Votes (Like | Disagree)
nezr Avatar
95 months ago
This week's lesson: don't buy Samsung
Score: 21 Votes (Like | Disagree)

Top Stories

cook cbs this morning

CBS This Morning: Apple to Make 'Big Announcement' Tomorrow Morning

Tuesday January 12, 2021 8:46 am PST by
CBS This Morning today shared a short clip of an upcoming interview with Apple CEO Tim Cook in which addressing last week's events at the U.S. Capitol, with Cook saying "it's key that people be held accountable for it." Following the clip, Gayle King of CBS noted that the interview with Cook was not specifically arranged to address the current controversy over Parler and other repercussions, ...
ipad pro 2021 mysmartprice cad

Allegedly Leaked 2021 iPad Pro CAD Images Suggest Few Design Changes

Tuesday January 12, 2021 3:38 am PST by
Tech blogs 91mobile and MySmartPrice on Tuesday posted a series of allegedly leaked factory CAD images of Apple's upcoming fifth-generation 11-inch iPad Pro. Rumors suggest Apple plans to announce two new iPad Pro models in both 11-inch and 12.9-inch versions, and today's images offer perhaps the clearest indication yet that Apple's next-generation iPad Pros will have minimal, if any,...
prototype iphone 12 pro

Prototype iPhone 12 Pro Shown Off in Photos

Wednesday January 13, 2021 3:39 pm PST by
Developer Giulio Zompetti, who often shows off prototype versions of Apple devices, today highlighted a prototype version of the iPhone 12 Pro. The iPhone 12 Pro is running an operating system called SwitchBoard, a nonUI version of the iOS 14 update that Apple uses internally. We've seen SwitchBoard on prototype devices before, as Apple uses it to test new features. Zompetti's prototype...
find my app safari post

Safari Allows Users to Enable Hidden 'Items' Tab in 'Find My' App Ahead of AirTags Launch

Wednesday January 13, 2021 5:45 am PST by
As seen in screenshots obtained by MacRumors in 2019, Apple's long-rumored AirTags items trackers are expected to be managed through the Find My app on iPhone, iPad, and Mac. Now, any user can get an early look at this tab. MacRumors reader David Chu today alerted us that the hidden "Items" tab in the Find My app can be enabled on an iPhone or iPad by typing in the link findmy://items in...
iphone x camera close

iOS 14.4 Will Introduce Warning on iPhones With Non-Genuine Cameras

Thursday January 14, 2021 8:07 am PST by
In the second beta of iOS 14.4 seeded to developers and public testers this week, MacRumors contributor Steve Moser has discovered code indicating that Apple will be introducing a new warning on iPhones that have had their camera repaired or replaced with aftermarket components rather than genuine Apple components. "Unable to verify this iPhone has a genuine Apple camera," the message will...
mac anti reflective coating issue

Apple's Anti-Reflective Coating Repair Program Still in Effect for Some MacBooks With New Mail-In Policy

Tuesday January 12, 2021 10:07 am PST by
In an internal memo obtained by MacRumors, Apple recently informed its network of Apple Authorized Service Providers that mail-in repair is now required for Mac notebooks with anti-reflective coating issues in the United States. The new policy went into effect January 4, 2021 and means that customers who take an eligible 12-inch MacBook or MacBook Pro exhibiting this issue to an Apple...
pioneer carplay wc5700nex

The Best Apple-Related Accessories at CES 2021

Wednesday January 13, 2021 1:16 pm PST by
CES 2021 is taking place digitally this year, and it hasn't been as exciting as in past years because many vendors have opted out. That said, some companies are still showing off some interesting Apple-related accessories that are coming out this year and that will be of interest to Mac, iPad, and iPhone users. Subscribe to the MacRumors YouTube channel for more videos. Pioneer Wireless...
Hue module dimmer switch

Philips Hue Announces New Wall Switch Module, Dimmer Switch, and Outdoor Light Bar

Thursday January 14, 2021 3:11 am PST by
Philips Hue has announced a new wireless dimmer switch module that lets Hue bridge owners directly control the smart lighting from their standard wall switches. The new Philips Hue wall switch module is the ideal addition to any Philips Hue set up. Installed behind existing light switches, it allows users to turn their existing switch into a smart switch and ensures their smart lighting is...
whatsapp wallpapers 1

WhatsApp Affirms User Privacy Following Backlash Over Data Sharing With Facebook

Tuesday January 12, 2021 6:39 am PST by
Following backlash after changing its terms and privacy policy to consolidate a significant amount of data sharing with Facebook, WhatsApp is now assuring users about the privacy measures built into its app. Last week, WhatsApp began informing users of updates to the app's terms of service and privacy policy. The updated agreements, which users must consent to in order to continue using...
apple racial equity justice initiative propel center render 01132021

Apple Launches New Nationwide Racial Equity and Justice Initiative Projects

Wednesday January 13, 2021 4:08 am PST by
Apple today announced a set of new projects as part of its $100 million Racial Equity and Justice Initiative (REJI) to help dismantle systemic barriers to opportunity and combat injustices faced by communities of color. Rendering of the Propel Center The projects include the Propel Center, a global innovation and learning hub for Historically Black Colleges and Universities (HBCUs), an Apple ...