Mobile benchmark dishonest has a protracted story that goes far again for the trade (properly – not less than in smartphone trade years), and has additionally been a controversial protection subject at AnandTech for a number of years now.
I keep in mind again in 2013 the place I had tipped off Brian and Anand about a few of the shenanigans Samsung was doing on the GPU of Exynos chipsets on the Galaxy S4, just for the factor to explode right into a wider evaluation of the follow amongst lots of the cell distributors again then – with all of them being discovered responsible. The Samsung case finally even ended up with a profitable $13.4m class-action lawsuit judgment towards the corporate – with yours actually and AnandTech even being cited within the courtroom submitting.
The naming and shaming did work over the next years, as distributors shortly deserted such strategies out of concern of media backlash – the negatives far outweighed the positives.
In current years nonetheless we noticed an enormous resurgence of such strategies, notably from Chinese distributors. Most predominantly for our extra western viewers this occurred to Huawei simply a few generations in the past with mechanisms that primarily disabled thermal throttling the of telephones – letting extra demanding benchmarks primarily have the SoC burn by to the utmost till thermal shutdowns. The naming and shaming right here once more helped, as the corporate had transitioned from using invisible mechanisms to one thing that was much more sincere and clear, and rather a lot much less problematic for follow-up gadgets.
The drawback is, the Chinese vendor market remains to be big, and we’re not capable of dissect each single gadget and vendor on the market. Cheating in benchmarks right here continued to be a really actual drawback and commonplace follow. Huawei’s rationale again then was that they felt that they wanted to do it as a result of others did it as properly – they usually didn’t wish to lose face to the competitors regarding the advertising and marketing energy of benchmark numbers.
The one large distinction right here nonetheless is that there’s at all times been considerably of a firewall in our protection between what a tool vendor did, and what chip distributors enabled them to do, and that’s the place we come to MediaTek’s habits over the previous couple of years. In most previous circumstances we at all times blamed the gadget distributors for dishonest because it had been their mechanisms and initiative – we hadn’t had proof of enablement by chipset distributors, not less than till now.
Helio P95 outperforming Dimensity 1000L?!
The entire factor bought to my consideration after I had first acquired Oppo’s new Reno3 Pro – the European model with MediaTek’s Helio P95 chipset. The cellphone shocked me fairly a bit at first, as in methods benchmarks comparable to PCMark it was punching fairly above its weight and what I had anticipated out of a Cortex-A75 class SoC. Things bought weirder after I acquired a Chinese Reno3 with the MediaTek Dimensity 1000L – a way more highly effective and up to date chip, however which for some motive carried out worse than its P95 sibling. It’s once you see such odd outcomes that alarm bells go off as there’s one thing that’s fairly amiss.
The entire factor ended up as fairly the journey down the rabbit gap.
Real Performance vs Cheated Performance
(Oppo Reno3 Pro P95)
Naturally, and sadly, my first thought was that there have to be some type of dishonest happening. We had reached out to our pals at UL for a anonymised model of PCMark – the groups there up to now had additionally been an important assist in deterring dishonest behaviour within the trade. To no main shock, the 2 variations of the benchmark did differ of their scores – however I used to be nonetheless aghast on the magnitude of the rating delta: a 30% distinction within the general rating, with as much as a 75% distinction in vital subtests such because the writing workload.
A little bit of…
