TopWare 18: Liquid/Image.iso/liquid/top2062/bnchmark.txt

Text File | 1993-02-17 | 17KB | 381 lines

   Windows Graphics Benchmarks -- Problems and Solutions
                     February 17, 1993

     This paper is intended to shed light on a problem that
has arisen in the graphics board market:  The reliability of
Winbench, an important test used to judge the performance of
Windows accelerators, has been compromised due to
manufacturers' ability to engineer inflated test results.
The developers of the WinBench test, Ziff-Davis, have
recognized the problem, and are currently working to revise
WinBench.  However, many consumers continue to use WinBench
as their exclusive measure of Windows graphics performance.
This paper suggests alternative benchmarks that should be
part of a complete test suite, even after a revised version
of WinBench is available.

     The problem with Winbench arose when a graphics board
manufacturer discovered a way to design its Windows driver
software to inflate the test results without, however, a
commensurate increase in real-world performance.  In its
current versions, WinBench is susceptible to such
manipulation because it is largely a repetitive, non-random
test.  Given these characteristics, developers have been
able to produce maximum scores by programming the graphics
driver to ignore repeated calls to execute the same
operation.  Also, bitmaps, patterns, and text strings can be
cached to dramatically improve Winbench results without
increasing performance in real applications.

     In a highly competitive environment, manufacturers
constantly refine their products to gain a competitive
advantage.  As a result of the industry's emphasis on this
easy-to-use test, the graphics WINMARK has become an
important weapon in the graphics board marketing war.
Predictably, "benchmark aware drivers" with artificially
inflated WINMARK scores have become a common phenomenon.
The problem is that users can no longer reliably estimate
real-world performance on the basis of these WINMARK scores.

     Hercules recommends that consumers evaluate Windows
graphics performance using both their intended applications
and two other simple benchmark tests, WinTach and Speedy,
discussed below.  After careful study of the methods used to
make display drivers "benchmark aware", Ziff-Davis is
revising Winbench.  The new version, dubbed v3.11, is
intended to thwart the techniques that have been employed to
inflate currently available versions of WinBench (v2.5 and
v3.1).  Hercules looks forward to the general availability
of this new test, and hopes to be able to recommend it as a
reliable measure of graphics performance.


Alternatives to WinBench

     WinTach and Speedy differ from WinBench in that they do
not concentrate on tight-loop iterative testing.  Unlike
current versions of WinBench that, for example, draw a line
multiple times in the same place on the screen in order to
measure line drawing speed, WinTach and Speedy execute a
wide variety of different commands in a long sequence. This
makes it much more difficult for driver developers to
"special-case" the test and artificially inflate its
results.

     It is in the consumer's interest to employ as many
tests as are available when judging graphics performance.
Every test has a particular scope and embodies a point of
view reflected in the decisions of its author.  To form a
representative picture, the consumer should use as many
thoughtfully designed tools as are available.  Both WinTach
and Speedy can be distributed freely, allowing users to
evaluate performance for themselves in their own systems.


Using Benchmark Tests

     In order to get an accurate measure of performance, it
is important to test different devices in the same or
parallel configurations.  For example, the results of a test
delivered when running in 1204 x 768 by 256 color resolution
should not be compared with the results of a test run at 640
x 480 by 256 colors.  Neither should tests taken using
different CPU platforms (e.g. a 386/25 and a 486/33) be
compared unless relative performance of similarly configured
computers is the object of the test.  Color depth must not
be overlooked, comparing 1024 x 768 by 16.7 million color
operation to 1024 x 768 by 256 color operation is only
meaningful if the relative difference between the
performance of the two color depths is the object of the
test.  Be careful to compare apples to apples or know that
the apple to orange comparison is your objective.

     Refresh rates are also a consideration in graphics
performance.  Although qualitative factors like the
intensity and stability of the display are not reflected in
benchmark scores, they do have a significant impact on a
user's experience.  In many cases, increasing the refresh
rate brings down graphics benchmark numbers, especially on
DRAM-based boards.  When comparing two devices, one
operating at 80Hz refresh and the other operating at 60Hz
refresh, be sure to cite this difference.

     In describing a test result, be diligent about
including all the information that makes it possible to
compare your results with others:  "Graphics accelerator X
delivered a benchmark result of Y when run in 1024 x 768,
256 color mode (large fonts) at 72Hz refresh rate on a '486-
33 ISA-bus computer with 8MB of system memory."  There are
often other variables that affect performance such as disk
or memory cache, the presence of ATM, whether the system is
running from a network, the degree of disk fragmentation,
and numerous other factors.  Whenever possible, use the same
system to compare multiple devices in order to reduce the
number of variables that may not be consistent between
different systems.


A Description of the Tests

WinTach

     WinTach was produced in 1992 by Texas Instruments. The
developers of WinTach profiled four typical Windows
application types to characterize the API (Application
Programming Interface) calls that typify these programs.
WinTach combines these API calls with real-world program
logic in order to simulate the performance of the Windows
system under test running these applications:

Word Processing
     Applications centered around the creation of text
documents that use a variety of fonts, styles, and
formatting options. Although the largest percentage of time
in a word processor is spent on text entry, graphics
performance is needed when large amounts of text are
repeatedly redrawn, such as when a page is scrolled.

CAD/Drawing
     Applications centered around the creation of technical
drawings or illustrations where the computer's power is used
to store and manipulate a large number of drawing objects.
While the largest percentage of time in a drawing
application is spent entering drawing elements, graphics
performance is needed when the drawing must be re-generated,
re-sized, or scrolled.

Spreadsheet
     Applications centered around the management of tabular
data.  While a large amount of time is spent entering data,
the primary graphics performance requirement is in scrolling
to the various parts of the spreadsheet.  Graphing the
numerical data contained in the spreadsheet is another
operation that increases graphics performance demands.

Paint
     Applications centered around the creation and
manipulation of bitmapped images.  Color depth is the most
important determinant of image quality, but a variety of
graphics primitives are employed to modify images.  Common
operations include lines, rectangles, floodfills, and
freehand drawing with different brush styles.

     When executed, WinTach evaluates the graphics system
and displays the results as the "RPM," or Relative
Performance Measurement.  This RPM index is calculated for
each individual application, indicating the system's
performance executing that class of application, as well as
for overall performance (a simple average of the four
individual application indices).  The RPM index is scaled to
regard both the resolution and color depth of the test
environment.  The program itself contains a complete
explanation of the RPM calculation.  This explanation is
found in the WinTach Overview available from the program's
Help menu under the topic entitled Calculation of the
WinTach RPM Index.  The RPM index enables the user to easily
compare the graphics performance of different Windows
workstations or graphics adapters.

     WinTach contains a very thorough on-line explanation
that includes a breakdown of the specific Windows GDI calls
made by the four application benchmarks, ranked by call
frequency.

     Because Texas Instruments, the developer of the TMS340
family of graphics chips, created WinTach it has been
criticized as being inherently biased towards T.I.'s own
graphics processors.  At Hercules, we have not found this to
be the case.  Hercules' product line includes both TMS34020-
based boards and IIT AGX-based boards.  We have found the
relative strengths of both designs to be accurately
reflected in their WinTach results.  The test measures the
efficiency of a graphics board and its driver executing
standard API calls regardless of the board's underlying
architecture.  If, however, you are concerned that WinTach
unfairly favors T.I.-based products, simply do not use it to
compare a T.I.-based board with a non-T.I.-based board.

     Another criticism of WinTach is that the weighting of
API calls does not accurately reflect real-world
applications.  As a component test, (a test which focuses on
graphics operations as opposed to system operations as a
whole) WinTach is an excellent relative measure.  By running
WinTach on two different boards or drivers at the same
resolution and color depth in the same machine, you'll
collect reliable data descriptive of the relative
performance of the two devices and/or drivers.  How closely
WinTach mirrors the behavior of actual applications is a
separate question from the comparative analysis of graphics
sub-system performance.  In practice, we have found the
relative performance ranking of boards tested with WinTach
to remain unchanged when the same boards are tested using
much more involved application scripts.


Speedy

     Speedy can be used in a variety of ways to compare the
relative performance of different graphics hardware and
display driver combinations.  Speedy tests a wide range of
Windows Graphics Device Interface (GDI) calls and delivers a
single SPEEDMARK number for comparison with other
systems/devices running Speedy.  Speedy exercises all of the
raster operations defined in the GDI and so is also a good
test of the completeness and compatibility of a display
driver.

     Speedy was developed by Hercules to illustrate the
strengths of more advanced graphics boards.  There are two
main areas in which Speedy differs from other graphics
performance benchmarks.  First, in contrast to WinBench and
WinTach whose individual tests run sequentially, Speedy
tests multiple applications executing in parallel.  This
parallelism, together with the use of random coordinates
inside each individual test, makes it very difficult to
enhance a display driver specifically to improve the Speedy
test result.

     Additionally, significant stress is added to the test
when a display driver is required to juggle multiple device
contexts.  The availability of multiple applications running
simultaneously is one of the defining characteristics of the
Windows operating system.  The degree to which the CPU can
off-load graphics tasks and go about other chores, such as
calculating pixels-per-second, is part of the performance
advantage of a Windows graphics accelerator.

     The second difference is that Speedy ramps through
colors using Windows' RGB model regardless of the actual
pixel depth of the display adapter.  This means that a 256
color driver will need to dither in order to simulate colors
that are not available in its palette.  Other benchmark
tests do not concern themselves with color depth so that
executing on a 16.7 million color device is not visually
different or more demanding of a display driver than
executing on a 16 color device.

     The most straightforward use of the Speedy program is
to choose Auto Run from the Control menu (or press the F2
key).  Speedy will fill the screen with eight graphics
applications running in parallel and deliver a number in the
center of the screen.  After 70 seconds of operation, a
SPEEDMARK number will also appear in the center window.  To
terminate Speedy press the F10 key or choose Kill...Kill All
Instances from the Control menu of any of the running
applications.

     When Speedy is launched and Auto Run selected, the
number in the center of the screen represents the total
number of pixels-per-second written by the eight separate
applications, divided by 100,000.  This number is computed
once every second.  After Speedy has been running for 70
seconds a mean is calculated and presented as the SPEEDMARK
rating.  The mean calculation does not include the first ten
seconds of operation which typically will involve
initialization and font caching.  This initialization period
will be noticed as the test appearing to stutter before
proceeding steadily.  Although the test may continue to run
indefinitely, the SPEEDMARK rating will not change after it
is presented.

The following applications make up Speedy's Auto Run test:

MemToScr is a memory to screen Bit Block Transfer (BitBlt)
test which measures the efficiency with which the display
hardware/driver combination transfers bitmapped data from
host memory to screen memory.

PolyHatch measures the efficiency of the display
hardware/driver combination when drawing polygons that are
filled with a pattern.

ScrToScr is a screen to screen BitBlt test measuring the
efficiency with which the display hardware/driver
combination can relocate bitmapped data from one part of the
screen to another.

VecSolid tests the efficiency of the display hardware/driver
combination when drawing solid lines of varying colors.

Rops is a test that exercises all 256 raster operations as
defined in the Windows GDI.  A raster operation is a
definition of how the bits in a source bitmap are logically
combined with the bits in a destination bitmap.

Fonts tests the efficiency of the display hardware/driver
combination when "realizing" or writing several True Type
character sizes and colors.

VecStyle tests the efficiency of the display hardware/driver
combination when drawing lines of varying colors and
patterns.

PolySolid measures the efficiency of the display
hardware/driver combination when drawing polygons that are
filled with solid colors.  Since the specified color may not
be available in the current color palette, this call may
require the driver to present a dithered color to most
closely simulate the specified color.

In addition to the tests included in Speedy's Auto Run the
following test is also available.

DIBToScreen tests the efficiency of the display
hardware/driver combination when managing BitBlts where the
source bitmap is not stored in the same format as the screen
bitmap.  This call tests the decoding ability of the driver.


Obtaining the Tests

     If you do not already have the tests, Both WinTach and
Speedy are available on the Hercules Computer Technology
BBS:
                 [phone] (V.32bis)
                   [phone] (V.32)
                [phone] (1200/2400)

     From the main menu of the Hercules BBS, simply press
"D" to initiate a download and specify the file name, either
WINTACH.ZIP or SPEEDY.ZIP.

     At Hercules, we are interested in your graphics
performance.  Run WinTach and Speedy on your computer, print
out this section of the document, fill out the information
below and fax it to us at [phone].

Hercules Graphite Card (model HG210) Benchmark Test Results

Testing 1024 x 768 x 256 color resolution at 72Hz refresh
rate on a Compaq Deskpro/i with 8MB of memory.  Using
AGX.DRV with the large fonts option selected.

    Processor            WinTach           Speedy
     386DX/25             16.78             16.57
     386DX/33             20.06             19.48
     486DX/33             28.46             25.67
     486DX2/66            34.16             28.62

==========================================================

My Name -_______________________

My System -
Manufacturer:___________________        Model:____________

Processor:__________  Speed:__________  Memory:___________

My Graphics -
Manufacturer:________________   Model:____________________

Resolution:_________  # of Colors:_______  Refresh:_______

Driver (Named in WinTach):_______________________

WinTach RPM: ___________           SpeedMark: __________

Please provide me with more information about Hercules
graphics accelerators by (circle your preference) calling,
faxing, writing, e-mailing me at: