Search

Benchmarking

Following is an article by Lahey's president, Bob Runyan, on the value of benchmarking and a selection of benchmark codes gathered by Fortran users David Staples and Ken Hamilton.

Do Try This at Home
by Bob Runyan

Which compiler generates the fastest code? LF90? DEC Visual Fortran? The now defunct Microsoft Powerstation? Wouldn't it be wonderful to have a single test, or a single suite of tests that gave a single simple answer?

Unfortunately, benchmarking compilers is not as straightforward as that. Different compilers will perform better on different codes. Different sets of switches will do better on different codes. Different hardware configurations will do better on different codes. Benchmarking that attempts to produce a single index is misguided. To get even a sketch of the true picture, you need to run a lot of different codes with a lot of different switches on a lot of different machines. Even so, is the picture a true representation of how the compilers will perform on your code and on your machine?

The answer is no.

Compiler vendors often will optimize code generation for a particular benchmark or benchmark suite. Published numbers on a benchmark or benchmark suite then might better reflect a compiler vendor's attention to getting good numbers on benchmarks than on the vendor's attention to getting good performance for user code.

Compiler vendors will sometimes generate code that only runs on certain hardware (for example using MMX- or Pentium Pro-specific instructions), then quote benchmark numbers based on the hardware specific code.

We run benchmarks frequently at Lahey and have noted the following:

  1. There is a huge variance in results from test to test. On some tests LF90 beats the competition by a factor of two. On some tests it's the other way around.
  2. There is no commercial compiler in our market that doesn't do very well on some tests.
  3. Usually you will not find a 20% difference between two compilers on the cumulative results of a suite of benchmarks. Unless execution speed is of overriding importance, you should look at other factors like quality of diagnostics, support, and ease of use in making a buying decision.
Should you give up on benchmarks then? I'd recommend giving up on putting any trust in the numbers that compiler vendors publish. If execution speed is very critical for you though, you can do some things on your own. We've made a suite of benchmarks available for download here. Run the benchmarks yourself on your own machine. Note that these are not just the benchmarks that Lahey does well on. We've included some of our bloopers as well. Download the suites available elsewhere and run those too. Most importantly, run your own code! No benchmark is as important as your own.

Benchmark Codes

These benchmarks were gathered by Ken Hamilton and David Staples and consist of a variety of codes, many of which were downloaded from public sites.

Download the benchmarks, bmkh.zip, from Ken Hamilton. The zip file contains other zip files, each of which should be unzipped in its own directory. Ken has provided makefiles for various compilers for each benchmark. Note that the switches that Ken specifies in the makefiles are not necessarily the ones that would be recommended by the various compiler vendors. Check with your compiler vendor for the best set of switches.

Download the benchmarks, bmds.zip, from David Staples. The zip file consists of Fortran sources only. Each source file is a separate benchmark. An internal timer routine is called in each (based on the Fortran 90 standard system_clock routine). These tests are for the most part rather small--some run in under 2 seconds on a fast PC--so averaging multiple runs or beefing up iteration counts and/or the sizes of loops is advised.