{"id":302,"date":"2012-09-22T13:54:44","date_gmt":"2012-09-22T13:54:44","guid":{"rendered":"http:\/\/ahay.org\/blog\/?p=302"},"modified":"2015-08-04T23:51:33","modified_gmt":"2015-08-04T23:51:33","slug":"how-is-regression-testing-done-in-madagascar","status":"publish","type":"post","link":"https:\/\/ahay.org\/blog\/2012\/09\/22\/how-is-regression-testing-done-in-madagascar\/","title":{"rendered":"How is regression testing done in Madagascar?"},"content":{"rendered":"

Testing for reproducibility is an important principle behind Madagascar’s design. It works on several levels.<\/p>\n

    \n
  1. Inside a project directory (with SConstruct<\/tt> file that contains from rsf.proj import *<\/strong>), run\n
    <\/i>
    <\/i><\/a><\/div><\/div>
    scons lock<\/pre><\/div>\n
    to create Result<\/em> files (figures) and to copy them to a different location (specified by RSFFIGS<\/strong> or $RSFROOT\/share\/madagascar\/figs<\/tt> by default). Papers included with Madagascar (under $RSFSRC\/book<\/tt>) have their result figures saved in a repository<\/a>.
    \nTo come back and test if the results are still reproducible, run <\/p>\n
    <\/i> 
    <\/i><\/a><\/div><\/div>
    scons test<\/pre><\/div>\n
     or <\/p>\n
    <\/i> 
    <\/i><\/a><\/div><\/div>
    scons figurename.test<\/pre><\/div>\n
    This command performs an intelligent comparison of figures using Joe Dellinger’s sfvplotdiff<\/a> and reports an error if the figures are different. In the case of an error, you can run<\/p>\n
    <\/i> 
    <\/i><\/a><\/div><\/div>
    scons figurename.flip<\/pre><\/div>\n
     to flip between the new version of the figure and the old version and on the screen and to compare them visually. Based on that comparison, you can either “lock” the new version with <\/p>\n
    <\/i> 
    <\/i><\/a><\/div><\/div>
    scons figurename.lock<\/pre><\/div>\n
     or debug the error that caused the difference and try to fix it.<\/li>\n
  2. To test all projects where a particular program, say sfspike<\/tt>, is used, run\n
    <\/i> 
    <\/i><\/a><\/div><\/div>
    cd $RSFSRC\/book; scons sfspike.test<\/pre><\/div>\n
     This is useful for regression testing for changes in programs that may cause reproducibility failures. You can also run <\/p>\n
    <\/i> 
    <\/i><\/a><\/div><\/div>
    cd $RSFSRC\/book; scons test<\/pre><\/div>\n
     to test all projects and all Madagascar programs. By default, testing is limited to projects that use only publicly available data and less that 1 Mb of disk space. This behavior can be changes by giving all=y<\/tt> or size=<\/tt> parameters to scons test<\/tt>.<\/li>\n
  3. A collection of scripts developed by Jim Jennings and explained on the Automatic Testing<\/a> page performs fine-grain testing with extended diagnostics.<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"
    Testing for reproducibility is an important principle behind Madagascar’s design. It works on several levels. Inside a project directory (with SConstruct file that contains from rsf.proj import *), run scons lock to create Result files (figures) and to copy them to a different location (specified by RSFFIGS or $RSFROOT\/share\/madagascar\/figs by default). Papers included with Madagascar […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_import_markdown_pro_load_document_selector":0,"_import_markdown_pro_submit_text_textarea":"","activitypub_content_warning":"","activitypub_content_visibility":"","activitypub_max_image_attachments":4,"activitypub_interaction_policy_quote":"anyone","activitypub_status":"","footnotes":""},"categories":[4],"tags":[],"class_list":["post-302","post","type-post","status-publish","format-standard","hentry","category-faq"],"_links":{"self":[{"href":"https:\/\/ahay.org\/blog\/wp-json\/wp\/v2\/posts\/302","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ahay.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ahay.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ahay.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ahay.org\/blog\/wp-json\/wp\/v2\/comments?post=302"}],"version-history":[{"count":1,"href":"https:\/\/ahay.org\/blog\/wp-json\/wp\/v2\/posts\/302\/revisions"}],"predecessor-version":[{"id":613,"href":"https:\/\/ahay.org\/blog\/wp-json\/wp\/v2\/posts\/302\/revisions\/613"}],"wp:attachment":[{"href":"https:\/\/ahay.org\/blog\/wp-json\/wp\/v2\/media?parent=302"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ahay.org\/blog\/wp-json\/wp\/v2\/categories?post=302"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ahay.org\/blog\/wp-json\/wp\/v2\/tags?post=302"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}