Automatic Testing

Quick Test

To run a "quick" test run this command from $RSFSRC:

admin/quick_test.csh

This script currently (June 19, 2010) runs a set of 147 examples in $RSFSRC/book that make 4GB of total data. They run in about 2 minutes total on a 2.4 GHz Intel Core 2 Duo MacBook when nothing needs to be updated, and 15 minutes when everything is rebuilt from scratch. Your mileage will vary.

Some of the 147 examples do not pass all the tests on my system. Your mileage will probably vary on this as well. Indeed, I will be curious to see how the tests work on your system (hint, hint :-) Post your testing report on rsf-devel.

Explanation of different steps

The following commands use sfbooklist, sffiglist, and sftestlist. Run them from $RSFSRC.

1. Get an inventory

sfbooklist size=1024 public=y list=filter skipfile=admin/skiplist.txt book

This step is not really necessary, but I do it to get a quick list of all the examples and a list of the subset that will be tested. I usually capture the output in a file and do a diff with the previous run. That way I will notice when new examples have been added to book since the last time I ran the tests.

The filters size= public= skipfile= limit the collection of examples to be executed. The size= filter skips any example that makes more than 1 GB of data. public=y means it will run examples that download public data from the server (the default is public=n). The default options for local= and private= are both n, so examples that use that kind of external data will be skipped. skipfile= specifies a list of examples by name that will be skipped. An example skipfile is $RSFSRC/admin/skiplist.txt, it specifies examples I can't run because of stuff I haven't installed and others that take a long time to run on my laptop. There is also a default filter to skip any example that does not yet have an .rsfproj file.

Running this step will tell you which examples pass these set of filters. The following options will run all examples:

sfbooklist rsfproj=both public=y private=y local=y book

2. Run the examples

sfbooklist size=1024 list=filter skipfile=admin/skiplist.txt timer=file command="scons >scons.log 2>&1" book

Step 2 is the same as Step 1 except I also specify a command string to run in each directory. The command string runs scons to update the examples and capture the console output in log files. There is also an option timer=file which appends a single line execution-time summary to a .rsftimer file in each directory. The other options for timer are none and log which sends the summary to the console instead of a file.

The output at the end of this step will include a count of how many examples failed. I usually capture the output of this step into a file and grep it for "error" to see which examples crashed. Then I can look at the corresponding scons.log file to see what happened, or just cd to the directory and run scons manually to see it happen again.

3. Compare the results

rm book/*/*/*/.rsftest
sfbooklist size=1024 list=filter skipfile=admin/skiplist.txt command="sffiglist rsftest=y list=none > list.log" book

Step 3 is like Step 2 except for a different command string. This command string runs the script sffiglist in each directory to compare the *.vpl files in the Fig directory with those in the lock directory. A summary of the comparison is put into an .rsftest file in each directory.

4. Summarize tests

sftestlist outfile=testlist.txt book

Step 4 runs a different script called sftestlist. This script just collects data from all the .rsftest files and makes a summary.

The examples will be listed by name. The examples without an .rsftest file are just those that did not pass the filters. There should not be any examples with .rsftest file errors. This kind of error probably indicates a bug in one of the testing scripts.

Examples with non matching figs is where the manual labor comes in. Some of those are probably just roundoff differences. But some are examples that don't reproduce on your system. The only way to tell is to actually look at the figs. The script sffiglist can help with that manual work.

Useful Scripts

1. List all the examples in book that use <program>

   sfbooklist uses=<program> list=filter book

2. rebuild all the examples in book that use <program>

   sfbooklist uses=<program> list=filter command=scons book

3. Compare the newly generated figs with the corresponding reference figs stored in $RSFFIGS (defaulted to $RSFROOT/share/figs)

   sfbooklist uses=<program> list=filter command="sffiglist rsftest=y list=none" book

4. Display any figs that are different from their $RSFFIGS counterparts

   sfbooklist uses=<program> list=filter command="sffiglist rsftest=y list=none show=diff" book