Last week we detailed an article protecting MediaTek’s seemingly widespread default inclusion of a benchmark whitelist of their chipset BSP (board help package deal) – a mechanism that permits extra aggressive efficiency tuning of a tool’s energy administration as soon as it detects {that a} benchmark software is working.
Yesterday, UL, the builders of the PCMark and 3DMark benchmarking suites, have adopted up on our investigation and analysed a wider vary of gadgets, and have made the choice to quickly delist all gadgets powered by a variety of MediaTek SoCs, a listing of over 50 gadgets from over 25 totally different distributors.
We had labored with UL early on within the investigation, with them offering us different anonymised variations of the benchmarks which bypass the whitelist detection, thus exposing the dishonest behaviour.
The UL information submit states:
“Using hidden mechanisms to detect benchmarking apps by title and make app-specific efficiency optimizations isn’t an “accepted industry standard.” It is, in truth, the very reverse of the accepted normal.
Likewise, benchmark scores based mostly on hidden app-specific optimizations and settings which can be enabled by default and never out there to the consumer don’t precisely mirror a tool’s true efficiency in on a regular basis use.
Simply put, a tool should run a benchmark as if it was another software. Performance positive factors should come from reacting to the character of the workloads within the take a look at reasonably than the title of the app itself.
As it has with comparable instances previously, we hope this delisting will assist persuade MediaTek to vary its strategy and be part of the remainder of the trade in adopting benchmarking finest practices.”
The assertion is a harsh rebuttal of MediaTek’s public response to our article, once more mentioning that the observe is something however an “industry standard” – particularly damning because it’s coming from one of many main benchmark builders within the trade.
As UL said of their weblog submit, and what we additionally identified in our unique piece, we hope that the adverse reactions to the matter will persuade MediaTek to desert such practices and rethink their view of what the “industry standard” is.
