Geekbench declares all Intel Core Ultra PLUS CPU benchmarks potentially “invalid”
Primate Labs call Geekbench results with Intel’s IBOT tool “invalid”
Primate Labs, the company behind Geekbench, the popular cross-platform benchmarking tool, has responded to the release of Intel’s Core Ultra 200S PLUS series CPUs (see our review here). The company has stated that all Geekbench 6 results using Intel’s new CPU “may be invalid” due to Intel’s new IBOT (Intel Binary Optimisation Tool) technology.
Intel’s IBOT tool restructures and streamlines compatible software to improve performance on Intel’s newest CPUs. Using this tool, software can run with higher IPC (Instructions Per Cycle) and better utilise caches, prefetchers, and CPU pipelines. Basically, it optimises software for Intel’s x86 CPUs. In Primate Labs’ view, Intel has modified its benchmark to boost its performance numbers.
In Geekbench 6, scores for individual workloads can increase by up to 40% using Intel’s IBOT tool, while overall benchmark scores can increase by up to 8%. These are significant performance gains. On the one hand, this showcases clear performance benefits from Intel’s IBOT tool. On the other hand, this can be seen as Intel cheating on workloads that should be fixed. While IBOT does not skip work and delivers the same results, benchmark modding is a grey area. As of now, Primate Labs cannot detect whether Intel users are using IBOT, so it must treat all Core Ultra 200S PLUS results as potentially invalid.
Geekbench 6 and Intel’s Binary Optimization Tool
Intel recently released the Binary Optimization Tool, which modifies instruction sequences in executables in order to improve performance. The techniques used are not publicly documented, and it is unclear how widely applicable these techniques are across different applications. The tool only supports a short list of applications, and Geekbench 6 is one of the few supported applications.
When run under the tool, some Geekbench 6 workload scores increase by up to 40%, and overall scores increase by up to 8%. Since the tool modifies the benchmark, and it is unclear to both Primate Labs and the general public how these changes occur, results generated with the tool are not comparable to results generated without it. In addition, we currently have no way to detect if a Geekbench 6 result was run with or without the Binary Optimization Tool.
As a (hopefully temporary) workaround, the Geekbench Browser will display the following warning on all Geekbench 6 CPU benchmark results from CPUs that support the Binary Optimization Tool: “This benchmark result may be invalid due to binary modification tools that can run on this system.”
While the Binary Optimization Tool only supports a small number of Intel CPUs, this is an important step to ensure scores reported on the Geekbench Browser remain trustworthy. Intel lists the supported CPUs on the Binary Optimization Tool webpage. We expect this list to be dynamic and that it will change over time. Primate Labs’ warnings will be updated accordingly.
– Primate Labs
A controversial technology for “like-for-like” benchmarking
With Intel’s IBOT technology altering workloads, an uneven playing field has emerged in the world of PC benchmarking. Yes, Intel’s tech delivers higher performance without skipping work or changing the quality/end result of the workloads completed. Regardless, Intel has effectively created a tool that creates optimised versions of some games/software for Intel’s hardware. If you want “like-for-like” results when testing various hardware configurations, IBOT should be disabled. However, if you want “real-world” results, IBOT should be active, as it represents what Intel users should do (if they want to maximise performance).
If one thing’s for certain, Intel’s IBOT tool highlights that some parts of Geekbench 6 run in deeply unoptimised ways on modern Intel CPUs. Some areas of the benchmark run 40% faster with IBOT enabled. Those are huge performance gains. An “up to 8%” increase in overall scores is significant. Primate Labs may be right to call results from Intel’s new CPUs “may be invalid”. On the other hand, Intel has just shown that Geekbench isn’t well-optimised for modern Intel hardware.
You can join the discussion on Geekbench’s stance on Intel Binary Optimisation on the OC3D Forums.
