Editing Package overview (section)

==Madagascar components==

===Standalone programs===
The list of all standalone programs is available [https://ahay.org/RSF/ online]. Most programs act as filters on input data and can be chained through Unix pipes, i.e.:
<bash>
< data.rsf sfwindow n1=100 | sfbandpass fhi=60 > data2.rsf
</bash>
This approach follows the Unix philosophy, as formulated by Doug McIlroy, the inventor of Unix pipes (Salus, 1994<ref>Salus, P. H.,  1994, A quarter-century of Unix: Addison-Wesley.</ref>):   
#Write programs that do one thing and do it well.  
#Write programs to work together.  
#Write programs to handle text streams because that is a universal interface. 

Following the Unix convention, programs have brief <tt>man</tt> pages explaining the program's purpose and parameters. You can access this documentation by running a program without parameters. To search for a program by a keyword, use <tt>sfdoc -k <keyword></tt>. 

The [[guide to Madagascar programs]] provides more detailed documentation for selected programs, while the [[task-centric program list]] attempts to categorize them. The programs in actual use can be found in [[Reproducible Documents]].

===Data format===

For data, Madagascar uses the [[Guide to RSF file format| Regularly Sampled Format]] (RSF), which is based on the concept of hypercubes (n-D arrays, or regularly sampled functions of several variables), much like the SEPlib (its closest relative), DDS, or the regularly-sampled version of the Javaseis format (SVF). Up to 9 dimensions are supported. For 1D, it is conceptually analogous to a time series, 2D to a raster image, and 3D to a [http://en.wikipedia.org/wiki/Voxel voxel volume]. The format (actually a [http://en.wikipedia.org/wiki/Meta meta]format) makes use of an ASCII file with metadata (information about the data), including a pointer (<tt>in=</tt> parameter) to the location of the file with the actual data values. Irregularly sampled data are currently handled as a pair of datasets, one containing data and the second containing the corresponding irregular geometry information. Programs for conversion to and from other formats, such as SEG-Y and SU, are provided. 

Madagascar currently uses the Vplot vector graphics format for graphics. Converters to other graphics formats (Postscript, PNG, GIF, JPEG) are also provided.

===Reproducible documents===

A reproducible document consists of LaTeX source combined with SCons rules required to fully build the document. These rules are expressed in terms of SCons extensions provided as part of Madagascar. 

This is the key to Madagascar's reproducibility. An introduction to reproducible Madagascar documents is at [[Reproducible_computational_experiments_using_SCons]].

===Vplot graphics===

In contrast to most other Madagascar Components, graphics components produce Vplot data as output.

Vplot is a device-independent graphics format that allows both vector and raster elements (as such, 
it is comparable to Postscript). Several output devices can interpret Vplot files. The typical usage is for a visual display in X-windows. A list of them is [[Guide to Madagascar programs#Plotting programs | provided on the wiki]].

Here is an example of a Madagascar pipe. In this case, it takes a subsection of a file, low-pass 
filters it, and saves the result

<syntaxhighlight lang="bash">
< data.rsf sfwindow n1=100 | sfbandpass fhi=60 > data2.rsf 
</syntaxhighlight>

In this more elaborate case, the final output is passed to a graphics program and plotted.

<syntaxhighlight lang="bash">
< data.rsf sfwindow n1=100 | sfbandpass fhi=60 | sfcontour | xtpen
</syntaxhighlight>

More extensive examples are in [[Guide to Madagascar programs]]. The novice reader should read the material below before proceeding to that page.

===Reproducibility and Project Management===

Madagascar uses and extends [http://www.scons.org/ SCons], an open-source 
software construction package to document and maintain data processing flows. Documented projects become 
computational recipes that can be easily exchanged among Madagascar users. 

SCons is a rule-based package in Python typically used as a build system analogous to <tt>make</tt>. Familiarity with any build system will
help understand SCons. SCons statements, as Python statements, are invoked in the sequence in which they are 
written, but as such, they only define rules. The rules are invoked by a dependency graph, which
SCons builds based on those rules. Components regarded as "up-to-date" are not rebuilt. 

SCons allows for user-contributed Builders (meta-rule categories), and Madagascar uses this capability extensively. 
The idea is that building an output file based on a workflow chain is analogous to building a 
software package based on a software toolchain. The calculation is seen simply as a build with dependencies. 
This setup greatly benefits developing alternative workflows using a given dataset. The system 
maintains an awareness of already completed calculations. Without user intervention, redundant calculations 
are avoided.

Madagascar calculations are thus expressed as SCons scripts (<tt>SConstruct</tt> files). SCons extensions follow SCons conventions, beginning 
with an uppercase letter. The most common Madagascar extensions are <tt>Flow()</tt>, <tt>Result()</tt>, and <tt>End()</tt>. A <tt>Flow()</tt> invocation wraps Madagascar computational components. <tt>Result()</tt> is a version of <tt>Flow()</tt> with a graphical output. Finally an
End() invokes the default rules for multiple results. 

Finally, Madagascar enables a collection of reproducible documents to be organized into living books. Each 
reproducible book contains a collection of Madagascar recipes (<tt>SConstruct</tt> files) used to generate book figures. The recipes 
cover a variety of data processing and imaging tasks described in the books. Figures and recipes serve 
a dual purpose concerning Madagascar maintenance. They provide demos to introduce new users to the 
functionality of the package and, at the same time, [[Automatic Testing|regression tests]] to assure the system's stability 
under change.