Benchmark procedures:

Instructions for users and developpers: on installing a new version of Chromalgema it is advised to run the benchmarks. It consists in a serie of tests to run the program on a large scope of options and parameters values, and different datasets comprising one, two or three components.

The benchmark directory tree-structure is detailed below. It can be as deep as 6 levels and  classifies the tests according to the following order of the options/parameters:
Each test directory contains a  parameter file named param.dat and possibly data files, unless generic data files located in the parent directory (One/Two/ThreeComp) are used. It also contains a subdirectory Ref which contains the reference output files: colonne#.out and output.

When the script chtest.sh is run, the benchmark directory is searched for all sub directories containing param.dat file and the corresponding test is run, the output files
colonne#.out and output which are then created are compared with their homonyms in the Ref directory with a diff whose result is saved in a Diff file. The number of lines exhibiting differences is summarized for each test in the report file in the benchmark directory.

For users: some differences between the current tests and the reference ones are normal: for instance the cpu time evaluations can be different from on computer to the other. Also in the case of EO optimization, elapsed time for each generation is output, which is also machine dependent. These differences will appear in the output file. Differences in the colonne#.out files should be investigated.

For developpers only: whenever changes are made to the program, the benchmark procedure chtest.sh should be run and differences should be explained and justified. When  the new version  is accepted, the new reference files should be created and stored in the Ref directories. This is automatized using the script chtestgen.sh in the benchmark directory (remember to sent the new Ref contents with CVS)

The scripts
chtestgen.sh and chtest.shin the benchmark directory make use of other scripts: chtestgenfile.sh, verifrep.sh, and  chtestfile.sh, which are located in the bin directory.

benchmark directory tree structure

OneComp  One component data

TwoComp Two components data  
ThreeComp  Three components data

                                                                                                                                                                                                  MAJ: MP 05/10/04