BenchmarkingFollowing is an article by Lahey's president, Bob Runyan, on the value of benchmarking and a selection of benchmark codes gathered by Fortran users David Staples and Ken Hamilton.
Do Try This at Home
by Bob Runyan
Which compiler generates the fastest code? LF90? DEC Visual Fortran? The now defunct Microsoft Powerstation? Wouldn't it be wonderful to have a single test, or a single suite of tests that gave a single simple answer?
Unfortunately, benchmarking compilers is not as straightforward as that. Different compilers will perform better on different codes. Different sets of switches will do better on different codes. Different hardware configurations will do better on different codes. Benchmarking that attempts to produce a single index is misguided. To get even a sketch of the true picture, you need to run a lot of different codes with a lot of different switches on a lot of different machines. Even so, is the picture a true representation of how the compilers will perform on your code and on your machine?
The answer is no.
Compiler vendors often will optimize code generation for a particular benchmark or benchmark suite. Published numbers on a benchmark or benchmark suite then might better reflect a compiler vendor's attention to getting good numbers on benchmarks than on the vendor's attention to getting good performance for user code.
Compiler vendors will sometimes generate code that only runs on certain hardware (for example using MMX- or Pentium Pro-specific instructions), then quote benchmark numbers based on the hardware specific code.
We run benchmarks frequently at Lahey and have noted the following:
These benchmarks were gathered by Ken Hamilton and David Staples and consist of a variety of codes, many of which were downloaded from public sites.
Download the benchmarks, bmkh.zip, from Ken Hamilton. The zip file contains other zip files, each of which should be unzipped in its own directory. Ken has provided makefiles for various compilers for each benchmark. Note that the switches that Ken specifies in the makefiles are not necessarily the ones that would be recommended by the various compiler vendors. Check with your compiler vendor for the best set of switches.
Download the benchmarks, bmds.zip, from David Staples. The zip file consists of Fortran sources only. Each source file is a separate benchmark. An internal timer routine is called in each (based on the Fortran 90 standard system_clock routine). These tests are for the most part rather small--some run in under 2 seconds on a fast PC--so averaging multiple runs or beefing up iteration counts and/or the sizes of loops is advised.