Editing Reproducibility (section)

=Pitfalls of reproducibility=

A few problems arise whenever reproducibility is put in practice. Fortunately, Madagascar has the potential to avoid all of them:

==The "out-of-date" pitfall==

Let us presume a reproducible experiment was set up and verified, then archived to disk... is that all?

No. Software dependencies and platforms change in time. Each change is incremental, but overall they add up. An archived experiment will not run a few years later, when libraries are different, versions of compilers/interpreters are different, etc. The only way to archive an experiment statically, "maintenance-free" is to conserve the actual physical machine it was run on. Nothing less will do: even archiving an image of the whole disk does not help, as old OSs will lack drivers for new hardware. The solution is to re-run the experiment every time something changed in the system, from a bug fix in the source code of the experiment's tools to a new version of a library called by a dependency of a dependency. The experiments function as regression tests.

==The "legacy maintenance" pitfall==

Perpetual regression testing works well, but somebody thinking two steps ahead may wonder how it scales with growth in the number of experiments. By necessity, the entire community will have to participate in the testing of the experiments. Since every participant in the community scales her own experiments to maximize use of own resources, nobody will have enough resources to run all experiments for everybody, or even significantly more than her own share! 

True problems arise when people stop participating in the community, by graduating, changing employers, shirking their testing duties, not being able to perform them, etc, but the experiments they created remain in the system and need to be tested. To see the magnitude of the problem, try to imagine the Stanford Exploration Project continously testing and debugging all numerical experiments done since the inception of the group in 1973. (Moore's Law would help, but unfortunately it is already slowing down dramatically) 

The first solution is finding "sterile" community members, who are willing to run tests and perform maintenance, but do not generate their own experiments. Possible candidates are corporate sponsors, foundations set up by professional associations, and libraries (yes, 21st century archiving involves keeping documents "alive"!). Then there is the sad, last-resort solution of dropping support for the oldest, least-referenced experiments. This should be avoided as much as possible, because bringing an experiment up to date after it was left to become obsolete can involve practically rewriting it, which is much more expensive than continous maintenance. Those who have any doubts should download one of the SEP CDs from the early 90s and try to reproduce the papers.