A benchmark is a tool used to evaluate or compare the performance of computer software or systems. Typically, this involves the design of a program (or suite of programs) that performs a series of operations that mimic “real world” activities. For example, computer processors (CPUs) can be given calculations in floating-point arithmetic, yielding a result in “flops” (floating point operations per second). Sim-ilarly, several different C-language compilers can be given the same files of source code and rated according to how quickly they produce the executable code, as well as the code’s compactness, speed, or efficiency.
Some examples of computer industry benchmarks include:
• Dhrystone and Whetstone for integer and floating point arithmetic, respectively
• mIPS (millions of instructions per second) and MFLOPS (millions of floating point instructions per second) for microprocessors
• FPS (frames per second) for various types of graphics
• 3DMark for three-dimensional graphics
• test suites using Linpack and LAPACK for super-computers
The devising of appropriate benchmarks is impor-tant because they can help prospective purchasers decide which competing CPU, program development tool, data-base system, or Web server to buy. Often the aspects of systems that are highlighted in advertising are not those that are most relevant to determining their actual utility. For example, CPUs are often compared according to clock speed, but a chip with a superior architecture and algorithm for handling instructions might actually outperform chips with faster clock speeds. By putting chips through their paces using the same arithmetic, data transfer, or graphics instructions, the benchmark pro-vides a more valid comparison.
The most relevant benchmarks tend to focus on re-cre-ating real-world use. Thus database systems can be com-pared in their speed of retrieval or update of data records. Real-world benchmarks also help guard against manufac-turers “tweaking” their systems to create artificially high benchmark results. Nevertheless, benchmarks cannot be used mechanically. While a given industry may have an “industry standard” benchmark, and a given product may be the highest performer using that test, the user must con-sider how well that benchmark reflects the actual work for which the system or program is being purchased. Perfor-mance, however well benchmarked, is usually only one key consideration, with environment (such as network connec-tions), reliability, security, ease of use, and of course cost being other considerations.
Some examples of computer industry benchmarks include:
• Dhrystone and Whetstone for integer and floating point arithmetic, respectively
• mIPS (millions of instructions per second) and MFLOPS (millions of floating point instructions per second) for microprocessors
• FPS (frames per second) for various types of graphics
• 3DMark for three-dimensional graphics
• test suites using Linpack and LAPACK for super-computers
The devising of appropriate benchmarks is impor-tant because they can help prospective purchasers decide which competing CPU, program development tool, data-base system, or Web server to buy. Often the aspects of systems that are highlighted in advertising are not those that are most relevant to determining their actual utility. For example, CPUs are often compared according to clock speed, but a chip with a superior architecture and algorithm for handling instructions might actually outperform chips with faster clock speeds. By putting chips through their paces using the same arithmetic, data transfer, or graphics instructions, the benchmark pro-vides a more valid comparison.
The most relevant benchmarks tend to focus on re-cre-ating real-world use. Thus database systems can be com-pared in their speed of retrieval or update of data records. Real-world benchmarks also help guard against manufac-turers “tweaking” their systems to create artificially high benchmark results. Nevertheless, benchmarks cannot be used mechanically. While a given industry may have an “industry standard” benchmark, and a given product may be the highest performer using that test, the user must con-sider how well that benchmark reflects the actual work for which the system or program is being purchased. Perfor-mance, however well benchmarked, is usually only one key consideration, with environment (such as network connec-tions), reliability, security, ease of use, and of course cost being other considerations.
No comments:
Post a Comment