Editing Tutorial (section)

==Introduction to Madagascar==
To begin, let's talk about the core principles of Madagascar and the RSF file format.  
[[Image:scheme.png|frame|center|The hierarchy of Madagascar.  Fundamentally, everything builds off of the RSF file format.  As you go up the chain the complexity level may increase, but the capabilities of the processing package increase as well.]]
===Madagascar's design===
There are a few layers to Madagascar.  At the bottom-most layer is the RSF file format, a common exchange format used by all Madagascar programs.  Non-Madagascar programs can also read/write to and from RSF because it is an open exchange format.  The next level of Madagascar contains the actual Madagascar programs that manipulate RSF files to process data.  Concurrent to this level is the VPLOT graphics library which allows users to plot and visualize RSF files.  The scripting utilities in Python and SCons are up another level from the core programs.  These scripting utilities enable users to make powerful scripts that can perform even the most advanced data processing tasks.  The last level includes support for LaTeX, which allows you to create documents combining the features of Madagascar with LaTeX's powerful typesetting.  Throughout the course of these tutorials, we will examine all of these components and demonstrate how they can be used individually and together.  When combined, the individual components of Madagascar allow us to: conduct experiments, process data, visualize our results, make reproducible scripts that can be shared with others, and write papers to document our experiments.  Thus, Madagascar provides a fully integrated research environment.

===RSF file format===
As previously mentioned, the lowest level of Madagascar is the RSF file format, which is the format used to exchange information between Madagascar programs.  Conceptually, the RSF file format is one of the easiest to understand, as RSF files are simply regularly sampled hypercubes of information.   For reference, a hypercube is a hyperdimensional cube (or array) that can best be visualized as an <math>N</math>-dimensional array, where <math>N</math> is between 1 and 9 in Madagascar. 

RSF hypercubes are defined by two files, the header file and the binary file.  The header file contains information about the dimensionality of the hypercube as well as the data contained within the hypercube.  Information contained in the header file may include the following: 
      
*number of elements on all axes,     
*the origin of the axes,     
*the sampling interval of elements on the axes,     
*the type of elements in the axes (i.e., float, integer),     
*the size of the elements (e.g. single or double precision),     
*and the location of the actual binary file. 
Since we often want to view this information about files without deciphering it, we store the header file as an ASCII text file in the local directory, usually with the suffix '''.rsf''' .  At any time, you can view or edit the contents of the header files using a text editor such as gedit, VIM, or Emacs.

The binary file is stored remotely (i.e., in a separate directory) containing the actual hypercube data.  Because the hypercube data can be very large (<math>10s</math> of GB or TB), we usually store the binary files in a remote directory with the suffix '''.rsf@''' .  The user can specify the remote directory using the '''DATAPATH'''  environmental variable.  The advantage of doing this is that we can store large binary data files on a fast remote filesystem if we want, and we can avoid working in local directories.  
[[Image:format.png|frame|center|Cartoon of the RSF file format.  The header file points to the binary file, which can be separate from each other.  The header file, which is text, is tiny compared to the binary file.]]

Because the header and binary are separated from one another, it is possible that we can lose either the header or binary for a particular RSF file.  If the header is lost, then we can simply reconstruct the header using our previous knowledge of the data and a text editor.  However, if we lose the binary file, then we cannot reconstruct the data regardless of what we do.  Therefore, you should try and avoid losing either the header or binary data.  The best way to prevent data loss is to make your research reproducible so that your results can be replicated later.
Sometimes, we need to store RSF files for archiving or transfer to other machines.  Fortunately, we can avoid transferring the header and binary separately by using the combined header/binary format for RSF files.  Files can be constructed using the combined header/binary format by specifying additional parameters on the command line, in particular '''--out=stdout''' , for any Madagascar program.  The output file will then be header/binary combined, which allows you to transfer the file without fear of losing either the header or binary.  Be careful, though: header/binary combined files can be very large and might slow down your local filesystem.  A best practice is only to use combined header/binary files when absolutely necessary for file transfers.  Note: header/binary combined files are usually automatically converted to header/binary separate files when processed by a Madagascar program.

===Additional documentation===

For more complete documentation on the RSF file format see the following links:

A [[Guide_to_RSF_file_format|gentle guide to the RSF file format]]

A [[RSF_Comprehensive_Description|detailed guide to the RSF file format]]