Editing Reproducibility (section)

=Why "do" reproducibility?=

==Knowledge management==
* The complexity of modern numerical experiments makes it necessary for the author himself to keep track meticulously of what he has done: "In the mid 1980's, we noticed that a few months after completing a project, the researchers at our laboratory were usually unable to reproduce their own computational work without considerable agony." [1]
"It takes some effort to organize your research to be reproducible. We found that although the effort seems to be directed to helping other people stand thisup on your shoulders, the principal beneficiary is generally the author herself. This is because time turns each one of us into another person, and by making effort to communicate with strangers, we help ourselves to communicate with our future selves." [2]
* Collaboration with other researchers, bringing a new employee/student up to speed or continuing to build on the work of a former team member: "Research cooperation can happen effortlessly if you use a uniform system for filing your research." [2]
* Speeding up the progress of science, and helping readers: "In a traditional article the author merely outlines the relevant computations: the limitations of a paper medium prohibit a complete documentation including experimental data, parameter values, and the author's programs. Consequently, the reader has painfully to re-implement the author's work before verifying and utilizing it. Even if the reader receives the author's source files (a feasible assumption considering the recent progress in electronic publishing), the results can be recomputed only if the various programs are invoked exactly as in the original publication. The reader must spend valuable time merely rediscovering minutiae, which the author was unable to communicate conveniently." [1]
* A reproducible experiment is the ultimate level of documentation. Code that works shows what actually happened in the experiment, regardless of the quality of theoretical explanations elsewhere
* A reproducible experiment is the first step towards an industrial implementation

==Error catching==
* Makes total peer review possible. "Many eyes make all bugs shallow" (Raymond, The Cathedral and the Bazaar)
* Discourages false claims, is an essential requirement for scientific integrity [4]
* Helps avoid errors caused by subtle differences in the implementation of current experiment and reference experiment [4]
* A reproducible experiment also serves as a regression test for the underlying software, helping avoind unintentional side effects of future modifications.

==Large benefit for small marginal cost==
"Our experience shows that it is only slightly more difficult to give birth to a "living" document than a "dead" one. The major hurdles in preparing a doctoral dissertation, research monograph or textbook are these: (1) mastering the subject matter itself, (2) writing the ancillary technical programs, (3) using a text editor and word processing system, and (4) writing the command scripts that run the programs to make the illustrations. The difference between preparing a "live" document and a "dead" one, lies in the command scripts. Will they be run once and then forgotten, or will they be attached to the figure-caption pushbutton?" [3]

==The last resort when returns start to diminish==
Scientific progress is [http://en.wikipedia.org/wiki/The_Structure_of_Scientific_Revolutions not uniform]. In the first stages after a new idea is discovered to be valuable, scientists proceed to exploit the "lowest-hanging fruit"/"stake their territory" as soon as possible. In this stage experimental methodology tends to not be followed in a pedantic fashion, as second-order inaccuracies do not have a large influence on the results. However, when returns start to diminish, improvements can drown in these second-order errors, and experiments need to be carefully set up (and made truly reproducible) in order to be able to perform accurate comparisons. More commentary on this topic, with application to the seismic processing industry, is provided in [5].