Current Status of HLSW Benchmarks and Guidelines for Benchmark Submission
--------------------------------------------------------------------------

Nik Dutt, U.C. Irvine
HLSW92 Benchmark Coordinator

Last Updated: September 18, 1992.
First Created: April 9, 1992.

Please send comments, corrections, suggestions and criticisms
to "dutt@ics.uci.edu"

-----------------------------------------------------------------------------
Contents
--------

1. Introduction

2. Comments on the old set of benchmarks (before 1992)

3. Guidelines for new benchmark submission

4. Status of new HLSW92 benchmarks

5. Brief history of HLSW benchmarking

-----------------------------------------------------------------------------

1. Introduction
---------------

Although the benchmarking effort for high-level synthesis has been alive
and active for several years, the current (i.e., older than 1991) set of
HLSW benchmarks are not very robust and lack the rigor associated with most
benchmarking efforts. This is certainly not intentional, and perhaps arose
from our (i.e., the HLS community's) eagerness in trying to get as many
examples into the HLSW benchmark set as possible.

However, we have reached a point of maturity in HLS where several researchers
use (or attempt to use) the benchmarks for comparative evaluation of their
results. These comparisons are often confusing and sometimes incorrect,
due to the inherent ambiguity in the existing set of benchmarks.

As part of HLSW92, we are attempting to rectify this situation by providing
a set of guidelines for benchmark submission and use, together with some
sample benchmarks that follow these guidelines.
This document attempts to describe the status of the benchmarking effort
prior to HLSW92 (Sec. 2), suggests some new guidelines for introducing more
rigor into benchmarking for HLS (Sec. 3), and points to a new set of benchmark
examples that follow these guidelines (Sec. 4).
The document concludes with a brief history of the benchmarking effort
for the past few High-Level Synthesis Workshops (Sec. 5).

-----------------------------------------------------------------------------

2. (Old) Benchmarks for HLSW 1991
----------------------------------

The benchmarks for HLSW 1991 are currently available from MCNC under the
directory labeled "HLSynth91". The design descriptions vary in complexity,
application, language, description style and scope.

The following design descriptions comprise the benchmark set for HLSW 1991:

o Processors

mc6502 (ISPS)
mc68000 (ISPS)
frisc (Hdw-C)
mark-1 (ISPS)

o Peripheral/Glue Chips

i8251 (ISPS, Hdw-C)
Amiga BLIT (C)

o DSP Chips

fft (paper description)
pitch extraction (paper description)
Kalman filter (ISPS)
fifth-order elliptic filter (VHDL, Hdw-C)

o FSM Controllers and Small Examples

gcd (Hdw-C)
ecc (Hdw-C)
TLC (Hdw-C)
tseng (Hdw-C)
diffeq (Hdw-C)
daior (Hdw-C)
daiop (Hdw-C)

The current benchmark set has several shortcomings which are described
below.

2.1 Lack of Sufficient Documentation
---------------------------------------

Most of the designs have little or no documentation. In addition to the
HDL descriptions we need to have English descriptions of the functionality,
comments in the HDL descriptions and pointers to the original source of
information (e.g., data sheet).

2.2 Lack of Simulation Vectors
---------------------------------

Except for some Hardware-C benchmarks, none of the HLSW benchmarks have any
simulation vectors for inputs and expected outputs used to test typical
behavioral sequences. Some of the ISPS benchmarks may have been simulated
before they were submitted, but unfortunately, the test vectors are not
part of the benchmark set. For the other descriptions, one can surmise that
most of them have never been simulated or checked for behavioral correctness!

As a result, the very behavior these models is in question (i.e., does the
HDL description *really* describe the design?). Furthermore, the lack of
input and output simulation vectors for the behavioral descriptions
implies that the outputs of synthesis tools on these examples may never
have been simulated (or verified) for correctness.

This is a serious deficiency in the current set of benchmarks, since most
researchers take existing descriptions and adapt them (e.g., translate
into another HDL or modify the description style) to suit specific tool
requirements. In the absence of a set of "sanity-check" test vectors, there
is no mechanism to ensure the preservation of behavior for modified (or
translated) design descriptions. Furthermore, outputs of synthesis tools
that have not been simulated for correct operational behavior are of
dubious value.

2.3 Variety of HDLs and Formats
----------------------------------

The old set of benchmarks are written using a variety of HDLs. In some
instances, the benchmark is simply a "paper description" of a design.

While the variance in HDLs cannot be avoided, it is preferable for the
benchmarks to be written in "well-known", robust HDLs (e.g., VHDL or Verilog)
which have publicly available reference manuals and simulators. This makes
the benchmarks available to a larger community of HLS researchers.

2.4 Ambiguity in Assumptions and Simplifications
---------------------------------------------------

In describing standard parts, the HDL description may have several
assumptions built-in (e.g., timing behavior), both due to the lack of
sufficient information and/or due to the limitations of the synthesis
tools used.

Similarly, several simplifications (e.g., ignoring tristating) or omissions
(e.g., omitting several instructions for a complex processor) may be made
in developing the input HDL description.

The existing benchmark examples for standard parts lack a clear description
of the assumptions and simplifications made.

2.5 Design Level vs. Description Level
-----------------------------------------

The benchmarks vary in their level of design (e.g. simple FSMs to complex
microprocessors), but are all described using behavioral constructs.
In some instances, the design is already partitioned into functional blocks,
with behavioral descriptions for each block (e.g., the I8251).
Furthermore, some designs can easily be described at two levels: abstract
behavior and partitioned functional blocks, resulting in different
synthesis issues for each HDL model.

Hence some indication of the level of the design and its partitioning is
useful when comparing the results of synthesis.

2.6 TextBook Examples
------------------------

Several benchmark examples are of academic interest or are quite trivial
in their scope and range. Unfortunately, this results from the difficulty
in obtaining industrial-strength examples. Moreover, in instances where
such designs are somehow acquired, they often have insufficient information
or documentation to make them useful for use in HLS.

It behooves our industrial participants/researchers/observers to make
"real" examples available to the HLS research community so that
we can begin realistic comparative analyses of HLS tools and systems.

This is an appeal for more "real" design examples to augment the current
set of HLSW benchmarks.

-----------------------------------------------------------------------------

3. Guidelines for Benchmark Submission (HLSW 92)
-------------------------------------------------

Based on comments from the previous section, the following guidelines
are suggested as a first step towards introducing more rigor in the
benchmarking process, and towards the creation of a robust set of
benchmark examples for testing HLS tools and systems.

3.1 "Well-Known" HDL Description
---------------------------------

The design must be described using a "well-known" HDL which has a publicly
available LRM, and which has a publicly available simulator. Sample
HDLs that fit this criterion include VHDL, Verilog and Hardware-C.

The HDL description must be liberally commented to allow readability.

3.2. Design Documentation, Assumptions, Simplifications
--------------------------------------------------------

The source of the design information should be specified (e.g., data sheet,
initial design spec., etc.).

A description of the design's functionality (using English, flowcharts,
block diagrams, etc.) must accompany the HDL description.

All assumptions and simplifications made in writing the HDL model must be
clearly stated.

3.3 Simulation Vectors
-----------------------

A set of input and expected output functional test vectors must accompany the
HDL description for simulating typical operational behaviors of the design.
These test vectors are not designed to exhaustively test the design.
Instead, they give some level of confidence in the behavioral HDL model,
and allow translation and validation of the model into another HDL or
description style.

The test vectors must also be accompanied by a (English) description of
what functionality is being tested.

The input and expected output vectors should be described in a generic format
that allows ease of use in different simulation environments. A brief
description of the test vector format must accompany the test vector set.

3.4 Simulator Details
----------------------

Each benchmark design must indicate the name, version, and availability
(where appropriate) of the simulator used to test the design.

3.5 Synthesis Outputs
---------------------

The outputs of synthesis tools must be simulated using the same simulator
and test vectors used to check the behavior of the input description.

-----------------------------------------------------------------------------

4. New Benchmarks for HLSW 1992 (status as of Sep. 18 1992)
-------------------------------------------------------------

The directory "HLSynth92" in the MCNC benchmark repository has the
following design examples that conform to the new guidelines:

o Processors and Sequencers

Am2901 Microprocessor slice
Am2910 Microprogram address sequencer

o Peripheral/Glue Chips

I8251 USART

o DSP Chips

Kalman filter
fifth-order elliptic filter

o FSM Controllers and Small Examples

gcd
TLC
diffeq
controlled counter

-----------------------------------------------------------------------------

5. Brief History
----------------

This sections gives a brief history of the benchmarking effort for HLS.
Please inform "dutt@ics.uci.edu" is there are any errors, omissions, or
corrections.

5.1 1987-1988
--------------

The benchmarking effort began during the summer of 1987 when an informal
benchmarking discussion was organized by Gaetano Borriello and Ewald Detjens
during DAC 1987. The urgent need for a set of benchmarks led to the HLSW 1988
Call for Participation stating: "The objective of the workshop is to begin the
development of a set of ``high-level synthesis benchmarks'' that can provide
a means of comparing different synthesis systems and guide future work to
include a complete range of digital circuits". A mailing list of parties
interested in the benchmarking effort was also established.

Four classes of circuits and representative examples were proposed as a
starting point for the benchmark set:

o Simple controller (I8251 UART)
o Simple microprocessor (MC6502)
o Signal processing application (fifth order elliptic wave filter)
o Complex microprocessor (MC68000)

Except for the wave filter example (which was a textual description), all the
other examples were written in ISPS.

5.2 1988-1989
--------------

Robert Mayo took over as the benchmark coordinator after 1988.
In an effort to expand the benchmark set, design examples continued to be
solicited from the HLS community. Since the benchmark set had few design
examples, design descriptions were added to the benchmark set on an "as-is"
basis, with the intention of broadening the range and number of design examples.

Listed below is the set of benchmarks that were used for HLSW 1989 (these
files are available from MCNC under the directory labeled "HLSynth89"):

o Processors:
mc6502 (ISP description)
mc68000 (ISP description)

o Peripheral/glue chips:
i8251 (ISP description, Hardware C description)
Amiga BLIT chip (C description)

o DSP chips:
fft (paper description only)
pitch extraction (paper description only)
kalman filter (ISP description)

o Small examples:
Traffic light controller (V & VHDL descriptions)
GCD (V & VHDL descriptions)
Counter (V & VHDL descriptions)
Prefetch (V & VHDL descriptions)

5.3 1990-1991
--------------

Robert Mayo moved the benchmark set to the MCNC file distribution center
during October of 1990. In 1991, some additional examples were provided by
Giovanni De Micheli (Stanford). These examples were written in HardwareC,
and included sample input and output test patterns for simulation of the
behavior. The complete set of benchmarks used for HLSW 1991 are available
from MCNC under the directory labeled "HLSynth91". This set augmented the
"HLSynth89" benchmark set with the following HardwareC design descriptions:

daio_receiver DAIO Receiver
daio_phase_decoder DAIO Phase Decoder
diffeq Differential Equation Solver
ecc Error Correction System
elliptic Fifth Order Elliptic Filter
frisc RISC Microprocessor
gcd Greatest Common Divisor Alg.
parker1986 Example from Alice Parker 1986
traffic Traffic Light Controller
tseng Example from Tseng

----------------------------------------------------------------------------