标题: 因为不满缺乏公平性、现实性、公开性，AMD 发表官方博文不支持 Sysmark 2012 [打印本页] 作者: Edison 时间: 2011-6-22 00:57 标题: 因为不满缺乏公平性、现实性、公开性，AMD 发表官方博文不支持 Sysmark 2012 In my last blog I asked whether you would vote for your IT department. I focused on the “democratization of IT” that companies are experiencing as a result of the new applications and devices we are all running. This week I want to focus on the applications, and how we measure and size the devices that run them.
Today most of us run applications that use JavaScript and HTML5. We use applications that have high video content and tools that can be accelerated with GPUs. Off the shelf, Microsoft’s Internet Explorer 9, Office 11 and Movie Maker all use GPU acceleration – they come with it turned on. So does Adobe Flash.
Last week in Seattle we highlighted some 50 applications, some fancy, some everyday, which leverage the GPU to provide a superior experience.
What we need in this brave new world are planning and sizing tools that enable IT to capacity plan, and users to buy the right systems for their needs. And that means we need benchmarks that are relevant for those applications. Having spent some time as a Systems Engineer at IBM, I had the pleasure of working on lots of benchmarks and capacity planning tools, so I understand the importance of which workloads are chosen − and how they are weighted − in determining the relevance of a benchmark’s overall score.
And this brings me to the main point of today’s blog.
The Truth Will Set You Free
AMD has a long history of supporting open standards; if you have any doubt just look at our support for OpenCL. And this support extends to active involvement with open industry consortia that likewise promote open standards. The beauty of open standards is that they are just that – open. Open to analysis, open to improvement and open to criticism.
AMD has for some time been a member of BAPCo, an industry organization that promotes, among other things, a benchmark known as SYSmark. In the past year or so AMD, with openness and transparency, has tried to explain why we believe this benchmark is misleading with respect to today’s commonplace applications − about a year ago I published a blog designed to explore this. If you work for a company that believes in transparency and integrity – and I do – then you have to take a stand and speak up when something is wrong.
BAPCo’s response to this blog was a threat to expel AMD from the consortium.
The heart of our complaint is this: the SYSmark benchmark is not only comprised of unrepresentative workloads (workloads that ignore the importance of heterogeneous computing and, frankly, favor our competitor’s designs), but it actually generates misleading results that can lead to very poor purchasing decisions, causing governments worldwide to historically overspend somewhere in the area of approximately $8B!
Now you’re starting to see why this is relevant to you (presuming you’re a taxpayer).
Good Intentions, Bad Results
AMD decided to do what we believed was the right thing for the industry and our customers, so we continued to work within BAPCo to try to get the next-generation benchmark, SYSmark12 (“SM2012”), right. Our hope was to effect change so that it would be open, transparent and processor-neutral. We got workloads included that represent the things you and I actually do in a day (instead of 35,000 line spreadsheets!).
But the question remained: what weighting would BAPCo ultimately give to the real-world workloads − since it is this weighting that defines the actual benchmark scores.
Unfortunately, our good intentions were met with an outcome that we believe does a disservice to the industry and our customers. We weren’t able to effect positive change within BAPCo, and the resulting benchmark continues to distort workload performance and offers even less transparency to end users. Once again, BAPCo chose to ignore the opportunity to promote openness and transparency.
While SM2012 is marketed as rating performance using 18 applications and 390 measurements, the reality is that only 7 applications and less than 10 percent of the total measurements dominate the overall score. So a small class of operations across the entire benchmark influences the overall score.
In fact, a relatively large proportion of the SM2012 score is based on system performance rated during optical character recognition (OCR) and file compression activities − things an average user will rarely if ever do.
And SM2012 doesn’t represent the evolution of computer processing and how that evolution is influencing average users’ experience. SM2012 focuses only on the serial processing performance of the CPU, and virtually ignores the parallel processing performance of the GPU. In particular, SM2012 scores do not take into account GPU-accelerated applications that are widely used in today’s business environments.
There are more things that AMD objects to in SM2012, like the excessive wall clock time consumed by its installation and execution. But this explanation will hopefully help you understand why, ultimately, we couldn’t look in the other direction.
Moving Forward…to Openness
So how can AMD stay in BAPCo? Simply put, we can’t. We have resigned from BAPCo and asked that our name and logo be removed from marketing materials promoting SM2012.
Now I hear some of you asking, “Isn’t this really just about the long-running antagonism between AMD and your competitor?”
No, it’s not.
It’s about fairness. Fairness to consumers and business users, to governments and other organizations that make purchasing decisions based on benchmarks, and, in the case of SYSmark, needlessly overspend because of it.
It’s about relevance. Because do you want to buy a system based on an outdated approach to measuring performance? Don’t you want your system’s performance measured against relevant measures like HTML5 or GPU acceleration? And shouldn’t a benchmark that measures PC performance be relevant to other devices that are likely in your life (if you’re reading this blog I think it’s safe to presume you use an array of devices – I do). Benchmarks should measure the way people engage with their devices today – not stick to a formula more appropriate for the last millennium.
And it’s about openness. Because you, and IT purchasing managers, should know what a benchmark represents and what the score really means to how the device will be used. That’s being set free.
And this is why we are exploring the options to encourage an alternative consortium, one that will deliver unbiased, representative benchmarks and promote more transparency for our industry. We are committed to working with likeminded companies that want to give consumers and business users an accurate, honest measure of what they can expect from their PCs and mobile devices. And what if ultimately we don’t “win” on these new benchmarks? Well, if the work is done with openness and transparency and results in a useful benchmark, we will make our case and let the market decide.
That’s all we have been asking for from BAPCo.
My hope is you, and the market, will vote for openness.