[RETURN HOME]

Benchmarketing 101: Intel SysMark

Posted By Van Smith

Date: August 14, 2001

While Intel's intimate relationship with a major benchmarking organization has been known for some time, the extent of this corroboration is only now beginning to become clear.  It is now evident that BAPCo's SysMark 2000 included Intel processor specific instructions to make the chip giant's CPUs look good, while handicapping the competition.

===================================

BAPCo's Shady History

Over a year ago at Tom's Hardware Guide, we first exposed Intel's relationship with BAPCo, the maker of SysMark 2000 and 2001.  Initially the BAPCo URL was registered in "Whois" under Intel, but, as word of this questionalble connection spread, the entry was altered to remove the chipmaker's name.  However, the new BAPCo listing's address was familiar to those in the industry: 2200 Mission College Blvd., Santa Clara, California -- Intel's domicile.

Soon after we brought this sleight of hand to the public's attention, all connections to the chip giant were obliterated from the Whois entry.  Despite this, BAPCo's own contact information continues to cite 2200 Mission College Blvd. as its official address.

At last winter's Platform Conference, Randall Kennedy of CSA Research stated in his presentation that all of the other member companies in the BAPCo consortium had "fallen by the wayside" leaving only Intel.  In fact, Mr. Kennedy asserted that BAPCo was essentially just a "front" for the chip giant.

Mr. Kennedy's history is interesting.  Several years ago he provided a benchmark for Intel on a contract basis.  An early iteration of the application was supposedly demonstrated publicly by Intel's Pat Gelsinger.  However, the chipmaker terminated the project after it became clear that Intel's own chips fared poorly on Mr. Kennedy's tests.  These benchmarks involved highly demanding concurrent MS Office application loads. 

Mr. Kennedy's CSA Research now produces system testing tools while Mr. Kennedy himself writes occasionally for AnandTech.

===================================

Stacking the deck

Hardware reviewers rely on software tools to make performance analyses.  Most often, the most crucial insights of reviews are dependent upon commercially derived benchmark suites.  With Intel's help distributing their products, the CPU maker's alter ego, BAPCo, has seen its SysMark application level test suite become perhaps the most widely used CPU analysis tool in the industry. 

By controlling the very lens through which all CPUs are viewed, Intel can show their processors in the best possible light while placing its rivals' products in the shadows.

There are many ways to skew benchmarks to make a particular CPU look good.  One cheap and dirty method is to highlight a particular instruction set extension to the exclusion of the competitors' analog.  We mentioned well over a year ago our belief that SysMark 2000 had Intel SSE optimizations to the exclusion of AMD's 3dNow! enhancements.  However, testing this conjecture was difficult.

With the introduction of AMD's Palomino, this situation has changed.  The Palomino is very similar to the Thunderbird version of the Athlon, but boasts full SSE compatibility.  Although Palomino desktop chips are not available yet, we dropped a 1.2 GHz Athlon MP into an Epox 8KHA system and compared performance with a 1.4 GHz Thunderbird Athlon used in the same setup.  The SysMark test we chose to examine was Photoshop since its filters were a prime candidate for SSE optimizations.

===================================

Test Setup

For this test we used SysMark 2000, Patch 5.  The desktop resolution was 1280x1024x32.

===================================

Benchmark Results

Despite its 200 MHz handicap, the 1.2 GHz Palomino easily defeated its faster clocked older brother.  Although the Palomino also supports hardware prefetching, the performance difference is almost entirely due to SSE since the SysMark Photoshop test is relatively bandwidth insensitive as can be seen in this old chipset review.

Our results demonstrate that the 1.2 GHz Palomino has nearly a 30% better score than the 1.4 GHz Thunderbird in the SysMark 2000 Photoshop test. 

Last year, this SSE advantage would belong solely to Intel's Pentium III.

===================================

Conclusion

There is little remaining doubt that Intel's SysMark 2000 played favorites with the chip giant's own processors.  For computer hardware reviewers, the current benchmarking scene is frustrating since few evaluators have at their disposal custom analysis tools.  It is for this reason that we at VHJ continue to push forwards with COSBI, a comprehensive open source benchmark suite.  The goal of COSBI is to provide a broad and free analysis tool box where all source code is openly available to all for inspection.

Certainly the SysMark finagling is understandable from Intel's marketing perspective, especially if the Santa Clara company can get by with this practice unnoticed.  In fact, it might be more accurate to call using Intel's SysMark "benchmarketing" rather than "benchmarking."

Only if independent means of test verification exist can reviewers and readers turn the current benchmarketing condition back into objective and dependable benchmarking.

===================================

Pssst!  Our Donation Page is up.

===================================

[RETURN HOME]