Reproducibility - Revision history

Sfomel at 19:28, 15 June 2011

2011-06-15T19:28:23Z

← Older revision		Revision as of 19:28, 15 June 2011
Line 1:		Line 1:
	[[Image:Reproducibility.jpg\|right\|Reproducibility is paramount in numerically-intensive sciences]]		[[Image:Reproducibility.jpg\|right\|Reproducibility is paramount in numerically-intensive sciences]]

			'''(written by I. Vlad)'''

	''"At the end of the day, reproducibility is what separates science from superstition" (S. Fomel)''		''"At the end of the day, reproducibility is what separates science from superstition" (S. Fomel)''

Sfomel: /* The "legacy maintenance" pitfall */

2011-06-15T19:26:11Z

The "legacy maintenance" pitfall

← Older revision		Revision as of 19:26, 15 June 2011
Line 38:		Line 38:
	==The "legacy maintenance" pitfall==		==The "legacy maintenance" pitfall==

	Perpetual regression testing works well, but somebody thinking two steps ahead may wonder how it scales with growth in the number of experiments. By necessity, the entire community will have to participate in the testing of the experiments. Since every participant in the community scales ~~his~~ own experiments to maximize use of own resources, nobody will have enough resources to run all experiments for everybody, or even significantly more than ~~his~~ own share!		Perpetual regression testing works well, but somebody thinking two steps ahead may wonder how it scales with growth in the number of experiments. By necessity, the entire community will have to participate in the testing of the experiments. Since every participant in the community scales her own experiments to maximize use of own resources, nobody will have enough resources to run all experiments for everybody, or even significantly more than her own share!

	True problems arise when people stop participating in the community, by graduating, changing employers, shirking their testing duties, not being able to perform them, etc, but the experiments they created remain in the system and need to be tested. To see the magnitude of the problem, try to imagine the Stanford Exploration Project continously testing and debugging all numerical experiments done since the inception of the group in 1973. (Moore's Law would help, but unfortunately it is already slowing down dramatically)		True problems arise when people stop participating in the community, by graduating, changing employers, shirking their testing duties, not being able to perform them, etc, but the experiments they created remain in the system and need to be tested. To see the magnitude of the problem, try to imagine the Stanford Exploration Project continously testing and debugging all numerical experiments done since the inception of the group in 1973. (Moore's Law would help, but unfortunately it is already slowing down dramatically)

Sfomel: /* The "out-of-date" pitfall */

2011-06-15T19:25:31Z

The "out-of-date" pitfall

← Older revision		Revision as of 19:25, 15 June 2011
Line 34:		Line 34:
	Let us presume a reproducible experiment was set up and verified, then archived to disk... is that all?		Let us presume a reproducible experiment was set up and verified, then archived to disk... is that all?

	No. Software dependencies and platforms change in time. Each change is incremental, but overall they add up. An archived experiment will not run a few years later, when ~~all~~ libraries are different, versions of compilers/interpreters are different, etc. The only way to archive an experiment statically, "maintenance-free" is to conserve the actual physical machine it was run on. Nothing less will do: even archiving an image of the whole disk does not help, as old OSs will lack drivers for new hardware. The solution is to re-run the experiment every time something changed in the system, from a bug fix in the source code of the experiment's tools to a new version of a library called by a dependency of a dependency. The experiments function as regression tests.		No. Software dependencies and platforms change in time. Each change is incremental, but overall they add up. An archived experiment will not run a few years later, when libraries are different, versions of compilers/interpreters are different, etc. The only way to archive an experiment statically, "maintenance-free" is to conserve the actual physical machine it was run on. Nothing less will do: even archiving an image of the whole disk does not help, as old OSs will lack drivers for new hardware. The solution is to re-run the experiment every time something changed in the system, from a bug fix in the source code of the experiment's tools to a new version of a library called by a dependency of a dependency. The experiments function as regression tests.

	==The "legacy maintenance" pitfall==		==The "legacy maintenance" pitfall==

Sfomel: /* The last resort when returns start to diminish */

2011-06-15T19:24:52Z

The last resort when returns start to diminish

← Older revision		Revision as of 19:24, 15 June 2011
Line 24:		Line 24:

	==The last resort when returns start to diminish==		==The last resort when returns start to diminish==
	Scientific progress is [http://en.wikipedia.org/wiki/The_Structure_of_Scientific_Revolutions not uniform]. In the first stages after a new idea is discovered to be valuable, scientists proceed to exploit the "lowest-hanging fruit"/"stake their territory" as soon as possible. In this stage experimental methodology tends to not be followed in a pedantic fashion, as second-order inaccuracies do not have a large influence on the results. ~~When~~ returns start to diminish ~~however~~, improvements can drown in these second-order errors, and experiments need to be carefully set up (and made truly reproducible) in order to be able to perform accurate comparisons. More commentary on this topic, with application to the seismic processing industry, is provided in [5].		Scientific progress is [http://en.wikipedia.org/wiki/The_Structure_of_Scientific_Revolutions not uniform]. In the first stages after a new idea is discovered to be valuable, scientists proceed to exploit the "lowest-hanging fruit"/"stake their territory" as soon as possible. In this stage experimental methodology tends to not be followed in a pedantic fashion, as second-order inaccuracies do not have a large influence on the results. However, when returns start to diminish, improvements can drown in these second-order errors, and experiments need to be carefully set up (and made truly reproducible) in order to be able to perform accurate comparisons. More commentary on this topic, with application to the seismic processing industry, is provided in [5].

	=Pitfalls of reproducibility=		=Pitfalls of reproducibility=

Sfomel at 19:23, 15 June 2011

2011-06-15T19:23:42Z

← Older revision		Revision as of 19:23, 15 June 2011
Line 2:		Line 2:
	''"At the end of the day, reproducibility is what separates science from superstition" (S. Fomel)''		''"At the end of the day, reproducibility is what separates science from superstition" (S. Fomel)''

	Making a numerical experiment reproducible means filing and documenting all data, source code and scripts used to produce the experimental results (figures, tables, graphs, etc). The filing and documentation is done such that the same results can be ~~produced automatically~~ by an independent reviewer. In numerically-intensive sciences, it is possible to not only ''describe'' the experimental setup, but to actually reproduce the entire experiment~~, completely~~. ~~However, paradoxically~~, the increase in computational complexity of the last decades meant that "today, few published results are reproducible in any practical sense. To verify them requires almost as much effort as it took to create them originally." [3]		Making a numerical experiment reproducible means filing and documenting all data, source code and scripts used to produce the experimental results (figures, tables, graphs, etc). The filing and documentation is done such that the same results can be reproduced by an independent reviewer. In numerically-intensive sciences, it is possible to not only ''describe'' the experimental setup, but to actually reproduce the entire experiment. Paradoxically, the increase in computational complexity of the last decades meant that "today, few published results are reproducible in any practical sense. To verify them requires almost as much effort as it took to create them originally." [3]

	=Why "do" reproducibility?=		=Why "do" reproducibility?=

Nick: leading quote

2011-02-23T03:30:07Z

leading quote

← Older revision		Revision as of 03:30, 23 February 2011
Line 1:		Line 1:
	[[Image:Reproducibility.jpg\|right\|Reproducibility is paramount in numerically-intensive sciences]]		[[Image:Reproducibility.jpg\|right\|Reproducibility is paramount in numerically-intensive sciences]]
			''"At the end of the day, reproducibility is what separates science from superstition" (S. Fomel)''

	Making a numerical experiment reproducible means filing and documenting all data, source code and scripts used to produce the experimental results (figures, tables, graphs, etc). The filing and documentation is done such that the same results can be produced automatically by an independent reviewer. In numerically-intensive sciences, it is possible to not only ''describe'' the experimental setup, but to actually reproduce the entire experiment, completely. However, paradoxically, the increase in computational complexity of the last decades meant that "today, few published results are reproducible in any practical sense. To verify them requires almost as much effort as it took to create them originally." [3]		Making a numerical experiment reproducible means filing and documenting all data, source code and scripts used to produce the experimental results (figures, tables, graphs, etc). The filing and documentation is done such that the same results can be produced automatically by an independent reviewer. In numerically-intensive sciences, it is possible to not only ''describe'' the experimental setup, but to actually reproduce the entire experiment, completely. However, paradoxically, the increase in computational complexity of the last decades meant that "today, few published results are reproducible in any practical sense. To verify them requires almost as much effort as it took to create them originally." [3]

Nick: /* Reproducible science in practice */ updated link

2009-06-23T21:31:27Z

Reproducible science in practice: updated link

← Older revision		Revision as of 21:31, 23 June 2009
Line 86:		Line 86:
	* [http://lcavwww.epfl.ch/reproducible_research/ Reproducible Neurophysiological Data Analysis]: a setup from U. Paris 5 for reproducible research in neurophysiology with Sweave. Sweave itself is advertised as a [http://www.stat.umn.edu/~charlie/Sweave/ package for reproducible research]		* [http://lcavwww.epfl.ch/reproducible_research/ Reproducible Neurophysiological Data Analysis]: a setup from U. Paris 5 for reproducible research in neurophysiology with Sweave. Sweave itself is advertised as a [http://www.stat.umn.edu/~charlie/Sweave/ package for reproducible research]
	* [http://drexel-coas-elearning.blogspot.com/2006/09/open-notebook-science.html Open Notebook Science]		* [http://drexel-coas-elearning.blogspot.com/2006/09/open-notebook-science.html Open Notebook Science]
	* [http://www.reproducibleresearch.~~org~~/ reproducibleresearch.~~org~~]		* [http://www.reproducibleresearch.net/ reproducibleresearch.net]

	The links above point to explicit attempts to bundle together whole scientific experiments, as described in scientific papers. It is worth noting that:		The links above point to explicit attempts to bundle together whole scientific experiments, as described in scientific papers. It is worth noting that:

Nick: /* About reproducibility */

2009-03-20T11:49:19Z

About reproducibility

← Older revision		Revision as of 11:49, 20 March 2009
Line 64:		Line 64:
	* [2] [http://sepwww.stanford.edu/research/redoc/IRIS.html Making Research Reproducible], introduction to the first SEP reproducibility system distributed through the WWW. 1996		* [2] [http://sepwww.stanford.edu/research/redoc/IRIS.html Making Research Reproducible], introduction to the first SEP reproducibility system distributed through the WWW. 1996
	* [http://www.ad-astra.ro/journal/2/vlad_reproducibility.pdf Reproducibility in computer-intensive sciences], Ad Astra short note (2000)		* [http://www.ad-astra.ro/journal/2/vlad_reproducibility.pdf Reproducibility in computer-intensive sciences], Ad Astra short note (2000)
	* [1] "Making Scientific computations reproducible (2000)" Schwab, M.; Karrenbach, N.; Claerbout, J., , Computing in Science & Engineering, Vol. 2, Issue6, Nov.-Dec. 2000, p.61-67		* [1] [http://sep.stanford.edu/lib/exe/fetch.php?id=sep%3Aresearch%3Areproducible&cache=cache&media=sep:research:reproducible:cip.pdf "Making Scientific computations reproducible (2000)" Schwab, M.; Karrenbach, N.; Claerbout, J., , Computing in Science & Engineering, Vol. 2, Issue6, Nov.-Dec. 2000, p.61-67]
	* [http://www.agu.org/eos_elec/000381e.html Complete PostScript: An Archival and Exchange Format for the Sciences?] (2003) EOS 84(36), P. Wessel, U. of Hawaii at Manoa, Honolulu		* [http://www.agu.org/eos_elec/000381e.html Complete PostScript: An Archival and Exchange Format for the Sciences?] (2003) EOS 84(36), P. Wessel, U. of Hawaii at Manoa, Honolulu
	* [http://www.dlib.org/dlib/september04/vandesompel/09vandesompel.html Rethinking Scholarly Communication], Sept. 2004 article in the D-Lib Magazine		* [http://www.dlib.org/dlib/september04/vandesompel/09vandesompel.html Rethinking Scholarly Communication], Sept. 2004 article in the D-Lib Magazine

Nick: /* Reproducible science in practice */ reproducibleresearch.org

2009-02-01T11:51:18Z

Reproducible science in practice: reproducibleresearch.org

← Older revision		Revision as of 11:51, 1 February 2009
Line 86:		Line 86:
	* [http://lcavwww.epfl.ch/reproducible_research/ Reproducible Neurophysiological Data Analysis]: a setup from U. Paris 5 for reproducible research in neurophysiology with Sweave. Sweave itself is advertised as a [http://www.stat.umn.edu/~charlie/Sweave/ package for reproducible research]		* [http://lcavwww.epfl.ch/reproducible_research/ Reproducible Neurophysiological Data Analysis]: a setup from U. Paris 5 for reproducible research in neurophysiology with Sweave. Sweave itself is advertised as a [http://www.stat.umn.edu/~charlie/Sweave/ package for reproducible research]
	* [http://drexel-coas-elearning.blogspot.com/2006/09/open-notebook-science.html Open Notebook Science]		* [http://drexel-coas-elearning.blogspot.com/2006/09/open-notebook-science.html Open Notebook Science]
			* [http://www.reproducibleresearch.org/ reproducibleresearch.org]

	The links above point to explicit attempts to bundle together whole scientific experiments, as described in scientific papers. It is worth noting that:		The links above point to explicit attempts to bundle together whole scientific experiments, as described in scientific papers. It is worth noting that:

Nick: /* Reproducible science in practice */

2008-10-26T15:25:49Z

Reproducible science in practice

← Older revision		Revision as of 15:25, 26 October 2008
Line 78:		Line 78:
	=Reproducible science in practice=		=Reproducible science in practice=

	* Some [http://sepwww.stanford.edu/doku.php?id=sep:research:reports Stanford Exploration Project reports]		* Some [http://sepwww.stanford.edu/doku.php?id=sep:research:reports Stanford Exploration Project reports] between 1990 and 2007
	* [http://software.seg.org/ software.seg.org]: Even if the algorithms are available, the input data is also needed for a completely reproducible numerical experiment. Besides publishing code for papers in Geophysics, this SEG site also makes some datasets publicly available		* [http://software.seg.org/ software.seg.org]: Even if the algorithms are available, the input data is also needed for a completely reproducible numerical experiment. Besides publishing code for papers in Geophysics, this SEG site also makes some datasets publicly available
	* [http://www.agu.org/pubs/esupp_about.html AGU Electronic Data Supplements]: Since 1994, software and data were accepted, peer reviewed and made available through FTP. More information on the [http://www.agu.org/pubs/agu_elec.html AGU Electronic Services and Publications] page		* [http://www.agu.org/pubs/esupp_about.html AGU Electronic Data Supplements]: Since 1994, software and data were accepted, peer reviewed and made available through FTP. More information on the [http://www.agu.org/pubs/agu_elec.html AGU Electronic Services and Publications] page