Editing Reproducibility (section)

==The "legacy maintenance" pitfall==

Perpetual regression testing works well, but somebody thinking two steps ahead may wonder how it scales with growth in the number of experiments. By necessity, the entire community will have to participate in the testing of the experiments. Since every participant in the community scales her own experiments to maximize use of own resources, nobody will have enough resources to run all experiments for everybody, or even significantly more than her own share! 

True problems arise when people stop participating in the community, by graduating, changing employers, shirking their testing duties, not being able to perform them, etc, but the experiments they created remain in the system and need to be tested. To see the magnitude of the problem, try to imagine the Stanford Exploration Project continously testing and debugging all numerical experiments done since the inception of the group in 1973. (Moore's Law would help, but unfortunately it is already slowing down dramatically) 

The first solution is finding "sterile" community members, who are willing to run tests and perform maintenance, but do not generate their own experiments. Possible candidates are corporate sponsors, foundations set up by professional associations, and libraries (yes, 21st century archiving involves keeping documents "alive"!). Then there is the sad, last-resort solution of dropping support for the oldest, least-referenced experiments. This should be avoided as much as possible, because bringing an experiment up to date after it was left to become obsolete can involve practically rewriting it, which is much more expensive than continous maintenance. Those who have any doubts should download one of the SEP CDs from the early 90s and try to reproduce the papers.