RSF Comprehensive Description

Introduction

The Regularly Sampled Format (RSF) is a specific arrangement of information defined by the behavior of the implementation of the C API of Madagascar on a reference machine architecture with a reference version of a Linux distribution and associated dependencies. RSF is the way in which most Madagascar programs expect their input to be and in which they structure their output. Due to portability of code, an exact reference setup has not been specified, and it is expected that Madagascar programs will read and write in the same way on any machines on which the package was compiled successfully. While a intuitive introduction to RSF already exists, this document attempts to describe RSF in an exhaustive fashion, for the use of programmers attempting to interface Madagascar with other packages.

The RSF originated as a representation of a discrete set of values of a single-valued function defined on a n-dimensional space. A real-world example of this is acoustic wavefield propagation in a space-time volume. Each dimension of the space is either discrete and regular, or continuous, but sampled discretely and regularly. In this context, regularity is defined as the property of a set of reals or integers of being consecutive integer multiples of a finite quantity of the same kind of themselves. RSF can be visualized as "matrices with physical dimensions".

The physicality of dimensions, while useful for jargon, for explanations and for the definition of parameters below, is not compulsory. Ultimately RSF is just a sane way of storing n-d arrays on disk. Just like programming languages use intrinsic methods and user-defined procedures to work with arrays held in memory, Madagascar uses both programs in its main distribution and user-written programs to work with data stored in RSF. RSF datasets are just out-of-core arrays.

Encoded information

A RSF dataset consists of: (1) the sequence of numerical values in the array and (2) of information about this "out-of-core array" (metainformation -- data about data).

Numerical data

Like arrays stored in computer memory, the sequence of numerical values is ordered in progressive order along consecutive dimensions. For example, the array

a11 a12 a13
a21 a22 a23
a31 a32 a33

will be written as: a11 a21 a31 a12 a22 a32 a13 a23 a33 A array with three dimensions will be ordered as follows: a111 a211 a311 a121 a221 a321 a131 a231 a331 a112 a212 a312 a122 a222 a322 a132 a232 a332

Metainformation

The distinct pieces of metainformation will be assigned names in italics, to help define the format later.

To the extent that is technically practical, each program that acted on the array will record:

• Name of program (prog)
• Directory in which the program was run (dir)
• User that ran the program (user)
• Short hostname of machine on which the program was run (host). For example, this would be machine instead machine.university.edu
• Date and time (up to seconds) at which the program was started (datetime)
• A pointer to the binary data (pointer)
• Data type, i.e. integer, real or complex (type)
• Data encoding, i.e. name of protocol for representing the data (format)
• Size of each data sample in bytes (esize)
• For each dimension # in the dataset (1 <= # <= 9), specify:
  • Number of elements in that dimension: n#, i.e.: n1, n2, n3... (nelem_axis#)
  • Origin on that axis: o#, i.e.: o1, o2, o3... (orig_axis#)
  • Sampling interval on that axis: d#, i.e.: d1, d2, d3... (sampl_axis#)
  • Label for that axis: label#, i.e.: label1, label2, label3... (label_axis#)
  • Physical unit for that axis: unit#, i.e.: unit1, unit2, unit3... (unit_axis#)

RSF components

The information to be encoded in a RSF dataset, described in the previous section, consists of two distinct parts: the metainformation and the data sequence. Depending on the context, the two parts can be in a single file, in a stream following each other, or in two separate files. So, depending the context, RSF can be a file format, a protocol or a metaformat – hence the preference for the weaselly term "dataset". This section defines the context-independent content of each of the two parts, and the sections that follow shows how they come together in various contexts.

ASCII header

The previous section mentioned that each program will record metainformation "to the extent that is technically practical". The current interpretation of this statement is that in the case of a linear workflow (i.e. prog1 | prog2 | prog3...) all of them will record the metainformation. However, in the case of a merged workflow (i.e. prog1 > file1.rsf; prog2 > file2.rsf; prog3 file1 file2 >file3.rsf ) only the metainformation entries from one file (usually the one going into stdin) is kept.

Using the nomenclature described in the previous section, a metainformation entry made by a program is formatted as follows:

prog dir: user@host datetime

        in="pointer"
        data_format="format_type"
        esize=esize
        n1=nelem_axis1
        o1=orig_axis1
        d1=sampl_axis1
        unit1="unit_axis1"
        label1="label_axis1"

The header must contain only ASCII printable characters.

All entries must be of the form: key=val, with no spaces adjacent to the equal sign. Lines that

The mandatory fields are those in the first line, in, data_format, esize and n1. These are the minimum necessary for a sequence of data values to be read. The first line of the recording (prog dir: user@host datetime) is written by all Madagascar programs, but is currently not used in any way (i.e. it is not mandatory). Indentation of the fields after the first line and the presence of the blank lines are not mandatory either.

The order of the entries after the first line is not mandatory.

The pointer can be either a path to a separate file holding the data and only the data, or the string stdin (when the RSF dataset is contained in a single file or transmitted over a stream).

The valid values of format are native, xdr and ascii, and the valid values of type can be short, int, float, double, complex, uchar, byte. These correspond to the similarly-named C language data types. The data_format with the best support and highest amount of optimization throughout Madagascar is native_float.

Given the encodings currently implemented in madagascar, esize can be 1, 2, 4 or 8.

Programs write to the ASCII header only those non-compulsory fields whose values have been modified. For example, a program performing summation of a 3-dimensional array over axis 3 will write to the header only n3=1, but not the values for n1 and n2, which have not been changed. Madagascar programs read the ASCII headers from the beginning to the end, and overwrite existing fields if new values appear.

Data

(data sequence)

Implementation of context detection

Madagascar I/O is an overloaded operation completely transparent for the user..

RSF files in streams

Input

When a Madagascar program writes a RSF file to disk (i.e.: sfprog >file.rsf), it will create a header file and a binary file as described above.

If the output is to a stream, or if the parameter --out=stdout is passed to the program, then the program will write to the stdout stream the ASCII header, followed by the sequence of three special characters: EOL EOL EOT (\014\014\004), followed by the binary.

Output

When a Madagascar program reads from the stdin stream, it expects either a EOF character indicating the end of the ASCII header (after which it transfers the stdin to reading from the binary cube), or a EOL EOL EOT sequence indicating that the data follows immediately on stdin.