Editing Automatic Testing

==Quick Test==

To run a "quick" test run this command from <tt>$RSFSRC</tt>:

<syntaxhighlight lang="bash">admin/quick_test.csh</syntaxhighlight>

This script currently (June 19, 2010) runs a set of 147 examples in <tt>$RSFSRC/book</tt> that make 4GB of total data.  They run in about 2 minutes total on a 2.4 GHz Intel Core 2 Duo MacBook when nothing needs to be updated, and 15 minutes when everything is rebuilt from scratch.  Your mileage will vary.

Some of the 147 examples do not pass all the tests on my system. Your mileage will probably vary on this as well.  Indeed, I will be curious to see how the tests work on your system (hint, hint :-)  Post your testing report on [https://lists.sourceforge.net/lists/listinfo/rsf-devel rsf-devel].

==Explanation of different steps==

The following commands use [http://www.ahay.org/RSF/sfbooklist.html sfbooklist], [http://www.ahay.org/RSF/sffiglist.html sffiglist], and [http://www.ahay.org/RSF/sftestlist.html sftestlist]. Run them from <tt>$RSFSRC</tt>.

===1. Get an inventory===
<syntaxhighlight lang="bash">sfbooklist size=1024 public=y list=filter skipfile=admin/skiplist.txt book</syntaxhighlight>

This step is not really necessary, but I do it to get a quick list of all the examples and a list of the subset that will be tested.  I usually capture the output in a file and do a diff with the previous run.  That way I will notice when new examples have been added to book since the last time I ran the tests.

The filters <tt>size= public= skipfile=</tt> limit the collection of examples to be executed.  The '''size=''' filter skips any example that makes more than 1 GB of data.  '''public=y''' means it will run examples that download public data from the server (the default is <tt>public=n</tt>).  The default options for '''local=''' and '''private=''' are both <tt>n</tt>, so examples that use that kind of external data will be skipped.  '''skipfile=''' specifies a list of examples by name that will be skipped.  An example skipfile is [http://sourceforge.net/p/rsf/code/HEAD/tree/trunk/admin/skiplist.txt $RSFSRC/admin/skiplist.txt], it specifies examples I can't run because of stuff I haven't installed and others that take a long time to run on my laptop.  There is also a default filter to skip any example that does not yet have an .rsfproj file.

Running this step will tell you which examples pass these set of filters. The following options will run all examples:
<syntaxhighlight lang="bash">sfbooklist rsfproj=both public=y private=y local=y book</syntaxhighlight>

===2. Run the examples===
<syntaxhighlight lang="bash">sfbooklist size=1024 list=filter skipfile=admin/skiplist.txt timer=file command="scons >scons.log 2>&1" book</syntaxhighlight>

Step 2 is the same as Step 1 except I also specify a command string to run in each directory.  The command string runs <tt>scons</tt> to update the examples and capture the console output in log files.  There is also an option '''timer=file''' which appends a single line execution-time summary to a .rsftimer file in each directory.  The other options for timer are <tt>none</tt> and <tt>log</tt> which sends the summary to the console instead of a file.

The output at the end of this step will include a count of how many examples failed.  I usually capture the output of this step into a file and grep it for "error" to see which examples crashed.  Then I can look at the corresponding scons.log file to see what happened, or just cd to the directory and run scons manually to see it happen again.

===3. Compare the results===
<syntaxhighlight lang="bash">rm book/*/*/*/.rsftest
sfbooklist size=1024 list=filter skipfile=admin/skiplist.txt command="sffiglist rsftest=y list=none > list.log" book</syntaxhighlight>

Step 3 is like Step 2 except for a different command string.  This command string runs the script '''sffiglist''' in each directory to compare the <tt>*.vpl</tt> files in the <tt>Fig</tt> directory with those in the lock directory.  A summary of the comparison is put into an <tt>.rsftest</tt> file in each directory.

===4. Summarize tests===
<syntaxhighlight lang="bash">sftestlist outfile=testlist.txt book</syntaxhighlight>

Step 4 runs a different script called '''sftestlist'''.  This script just collects data from all the <tt>.rsftest</tt> files and makes a summary.  

The examples will be listed by name.  The examples without an <tt>.rsftest</tt> file are just those that did not pass the filters.  There should not be any examples with <tt>.rsftest</tt> file errors.  This kind of error probably indicates a bug in one of the testing scripts.  

Examples with non matching figs is where the manual labor comes in.  Some of those are probably just roundoff differences.   But some are examples that don't reproduce on your system.  The only way to tell is to actually look at the figs.  The script <tt>sffiglist</tt> can help with that manual work.

==Useful Scripts==

# List all the examples in <tt>book</tt> that use <tt><program></tt><pre>sfbooklist uses=<program> list=filter book</pre>
# rebuild all the examples in <tt>book</tt> that use <tt><program></tt><pre>sfbooklist uses=<program> list=filter command=scons book</pre>
# Compare the newly generated figs with the corresponding reference figs stored in <tt>$RSFFIGS</tt> (defaulted to <tt>$RSFROOT/share/figs</tt>)<pre>sfbooklist uses=<program> list=filter command="sffiglist rsftest=y list=none" book</pre>
# Display any figs that are different from their <tt>$RSFFIGS</tt> counterparts<pre>sfbooklist uses=<program> list=filter command="sffiglist rsftest=y list=none show=diff" book</pre>