Editing
Tutorial
(section)
Jump to navigation
Jump to search
Warning:
You are not logged in. Your IP address will be publicly visible if you make any edits. If you
log in
or
create an account
, your edits will be attributed to your username, along with other benefits.
Anti-spam check. Do
not
fill this in!
===RSF file format=== As previously mentioned, the lowest level of Madagascar is the RSF file format, which is the format used to exchange information between Madagascar programs. Conceptually, the RSF file format is one of the easiest to understand, as RSF files are simply regularly sampled hypercubes of information. For reference, a hypercube is a hyperdimensional cube (or array) that can best be visualized as an <math>N</math>-dimensional array, where <math>N</math> is between 1 and 9 in Madagascar. RSF hypercubes are defined by two files, the header file and the binary file. The header file contains information about the dimensionality of the hypercube as well as the data contained within the hypercube. Information contained in the header file may include the following: *number of elements on all axes, *the origin of the axes, *the sampling interval of elements on the axes, *the type of elements in the axes (i.e., float, integer), *the size of the elements (e.g. single or double precision), *and the location of the actual binary file. Since we often want to view this information about files without deciphering it, we store the header file as an ASCII text file in the local directory, usually with the suffix '''.rsf''' . At any time, you can view or edit the contents of the header files using a text editor such as gedit, VIM, or Emacs. The binary file is stored remotely (i.e., in a separate directory) containing the actual hypercube data. Because the hypercube data can be very large (<math>10s</math> of GB or TB), we usually store the binary files in a remote directory with the suffix '''.rsf@''' . The user can specify the remote directory using the '''DATAPATH''' environmental variable. The advantage of doing this is that we can store large binary data files on a fast remote filesystem if we want, and we can avoid working in local directories. [[Image:format.png|frame|center|Cartoon of the RSF file format. The header file points to the binary file, which can be separate from each other. The header file, which is text, is tiny compared to the binary file.]] Because the header and binary are separated from one another, it is possible that we can lose either the header or binary for a particular RSF file. If the header is lost, then we can simply reconstruct the header using our previous knowledge of the data and a text editor. However, if we lose the binary file, then we cannot reconstruct the data regardless of what we do. Therefore, you should try and avoid losing either the header or binary data. The best way to prevent data loss is to make your research reproducible so that your results can be replicated later. Sometimes, we need to store RSF files for archiving or transfer to other machines. Fortunately, we can avoid transferring the header and binary separately by using the combined header/binary format for RSF files. Files can be constructed using the combined header/binary format by specifying additional parameters on the command line, in particular '''--out=stdout''' , for any Madagascar program. The output file will then be header/binary combined, which allows you to transfer the file without fear of losing either the header or binary. Be careful, though: header/binary combined files can be very large and might slow down your local filesystem. A best practice is only to use combined header/binary files when absolutely necessary for file transfers. Note: header/binary combined files are usually automatically converted to header/binary separate files when processed by a Madagascar program.
Summary:
Please note that all contributions to Madagascar are considered to be released under the GNU Free Documentation License 1.3 or later (see
My wiki:Copyrights
for details). If you do not want your writing to be edited mercilessly and redistributed at will, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource.
Do not submit copyrighted work without permission!
Cancel
Editing help
(opens in new window)
Navigation menu
Personal tools
English
Not logged in
Talk
Contributions
Create account
Log in
Namespaces
Page
Discussion
English
Views
Read
Edit
View history
More
Search
Getting Madagascar
download
Installation
GitHub repository
SEGTeX
Introduction
Package overview
Tutorial
Hands-on tour
Reproducible documents
Hall of Fame
User Documentation
List of programs
Common programs
Popular programs
The RSF file format
Reproducibility with SCons
Developer documentation
Adding programs
Contributing programs
API demo: clipping data
API demo: explicit finite differences
Community
Conferences
User mailing list
Developer mailing list
GitHub organization
LinkedIn group
Development blog
Twitter
Slack
Tools
What links here
Related changes
Special pages
Page information