Automatic Testing: Difference between revisions
| Line 30: | Line 30: | ||
Running this step will tell you which examples pass these set of filters. | Running this step will tell you which examples pass these set of filters. | ||
== | ==Useful Scripts== | ||
# List all the examples in <tt>book</tt> that use <tt><program></tt><pre>sfbooklist uses=<program> list=filter book</pre> | # List all the examples in <tt>book</tt> that use <tt><program></tt><pre>sfbooklist uses=<program> list=filter book</pre> | ||
Revision as of 15:06, 22 June 2010
Quick Test
To run a "quick" test run this command from $RSFSRC:
<bash>admin/quick_test.csh</bash>
This script currently (June 19, 2010) runs a set of 147 examples in $RSFSRC/book that make 4GB of total data. They run in about 2 minutes total on a 2.4 GHz Intel Core 2 Duo MacBook when nothing needs to be updated, and 15 minutes when everything is rebuilt from scratch. Your mileage will vary.
Seven of the 147 examples do not pass all the tests on my system, perhaps because of bugs not yet found :-( These examples are:
- book/gee/mda/levint
- book/jsg/lpf/lpf
- book/sep/bspl/plane3
- book/sep/pwd/alias
- book/sep/pwd/hole
- book/sep/pwd/signoi
- book/sep/stack/miginv
Your mileage will probably vary on this as well. Indeed, I will be curious to see how the tests work on your system (hint, hint :-) Post your testing report on rsf-devel.
Explanation of different steps
Get an inventory
<bash>sfbooklist size=1024 public=y list=filter skipfile=admin/skiplist.txt book</bash>
This step is not really necessary, but I do it to get a quick list of all the examples and a list of the subset that will be tested. I usually capture the output in a file and do a diff with the previous run. That way I will notice when new examples have been added to book since the last time I ran the tests.
The filters size= public= skipfile= limit the collection of examples to be executed. The size= filter skips any example that makes more than 1 GB of data. public=y means it will run examples that download public data from the server (the default is public=n). The default options for local= and private= are both n, so examples that use that kind of external data will be skipped. skipfile= specifies a list of examples by name that will be skipped. An example skipfile is RSFSRC/admin/skiplist.txt, it specifies examples I can't run because of stuff I haven't installed and others that take a long time to run on my laptop. There is also a default filter to skip any example that does not yet have an .rsfproj file.
Running this step will tell you which examples pass these set of filters.
Useful Scripts
- List all the examples in book that use <program>
sfbooklist uses=<program> list=filter book
- rebuild all the examples in book that use <program>
sfbooklist uses=<program> list=filter command=scons book
- Compare the newly generated figs with the corresponding reference figs stored in $RSFFIGS (defaulted to $RSFROOT/share/figs)
sfbooklist uses=<program> list=filter command="sffiglist rsftest=y list=none" book
- Display any figs that are different from their $RSFFIGS counterparts
sfbooklist uses=<program> list=filter command="sffiglist rsftest=y list=none show=diff" book