RSF Comprehensive Description - Revision history

Nick: Clarified issue raised by Francesco

2010-02-25T17:07:51Z

Clarified issue raised by Francesco

← Older revision		Revision as of 17:07, 25 February 2010
Line 68:		Line 68:

	Given the encodings currently implemented in madagascar, <tt>esize</tt> can be 1, 2, 4 or 8.		Given the encodings currently implemented in madagascar, <tt>esize</tt> can be 1, 2, 4 or 8.

			Axis length values (n1, n2, ... n9) must be greater than zero. It is allowed to not specify axis length values of higher dimensions than the intrinsic dimensionality of the data. For example, for a 1000x20x500 data cube, it is allowed to not specify any values for n4, n5, ... n9. It is also allowed to set the higher-dimension axis length values to 1. Gaps in the axis length sequence are not allowed (example: if n4 is given a value greater than one, then n1, n2 and n3 must be specified, even if their values are set to one.

	Programs write to the ASCII header only those non-compulsory fields whose values have been modified. For example, a program performing summation of a 3-dimensional array over axis 3 will write to the header only <tt>n3=1</tt>, but not the values for <tt>n1</tt> and <tt>n2</tt>, which have not been changed. Madagascar programs read the ASCII headers from the beginning to the end, and overwrite existing fields if new values appear.		Programs write to the ASCII header only those non-compulsory fields whose values have been modified. For example, a program performing summation of a 3-dimensional array over axis 3 will write to the header only <tt>n3=1</tt>, but not the values for <tt>n1</tt> and <tt>n2</tt>, which have not been changed. Madagascar programs read the ASCII headers from the beginning to the end, and overwrite existing fields if new values appear.

Nick: /* Encoded information */

2009-12-23T15:45:12Z

Encoded information

← Older revision		Revision as of 15:45, 23 December 2009
Line 9:		Line 9:

	==Encoded information==		==Encoded information==
	A RSF dataset consists of: (1) the sequence of numerical values in the array and (2) of information about this "out-of-core array" (metainformation -- data about data).		A RSF dataset consists of: (1) the sequence of numerical values in the array and (2) of information about this "out-of-core array" (metainformation -- data about data). This section attempts to ''define'' the information contained in a RSF dataset is, without concerning itself with the way this information is encoded, which will be done in the section that follows.

	~~===Metainformation===~~		A discrete sequence of numbers is quite simple and needs no further explanation. The metainformation requires a more detailed definition. Its distinct components will be assigned names in italics, to help define the encoding later.
	The distinct ~~pieces of metainformation~~ will be assigned names in italics, to help define the ~~format~~ later.

	To the extent that is technically practical, each program that acted on the array will record:		To the extent that is technically practical, each program that acted on the array will record:

Nick: /* Introduction */

2009-12-23T15:26:36Z

Introduction

← Older revision		Revision as of 15:26, 23 December 2009
Line 2:		Line 2:
	==Introduction==		==Introduction==

	The Regularly Sampled Format (RSF) is a specific arrangement of information defined by the behavior of the implementation of the [http://rsf.svn.sourceforge.net/viewvc/rsf/trunk/api/c/ C API of Madagascar] ~~on a reference machine architecture with a reference version of a Linux distribution and associated dependencies~~. RSF is the way in which most Madagascar programs expect their input to be and in which they structure their output. Due to portability of code, an exact reference setup has not been specified, and it is expected that Madagascar programs will read and write in the same way on any machines on which the package was compiled successfully. While a [http://reproducibility.org/wiki/Guide_to_RSF_file_format intuitive introduction to RSF] already exists, this document attempts to describe RSF in an exhaustive fashion, for the use of programmers attempting to interface Madagascar with other packages.		The Regularly Sampled Format (RSF) is a specific arrangement of information defined by the behavior of the implementation of the [http://rsf.svn.sourceforge.net/viewvc/rsf/trunk/api/c/ C API of Madagascar]. RSF is the way in which most Madagascar programs expect their input to be and in which they structure their output. While a [http://reproducibility.org/wiki/Guide_to_RSF_file_format intuitive introduction to RSF] already exists, this document attempts to describe RSF in an exhaustive fashion, for the use of programmers attempting to interface Madagascar with other packages.

	The RSF originated as a representation of a discrete set of values of a single-valued function defined on a ''n''-dimensional [http://en.wikipedia.org/wiki/Space_%28mathematics%29 space]. A real-world example of this is acoustic wavefield propagation in a space-time volume. Each dimension of the space is either discrete and regular, or continuous, but sampled discretely and regularly. In this context, regularity is defined as the property of a set of reals or integers of being consecutive integer multiples of a finite quantity of the same kind of themselves. RSF can be visualized as "matrices with physical dimensions".		The RSF originated as a representation of a discrete set of values of a single-valued function defined on a ''n''-dimensional [http://en.wikipedia.org/wiki/Space_%28mathematics%29 space]. A real-world example of this is acoustic wavefield propagation in a space-time volume. Each dimension of the space is either discrete and regular, or continuous, but sampled discretely and regularly. In this context, regularity is defined as the property of a set of reals or integers of being consecutive integer multiples of a finite quantity of the same kind of themselves. RSF can be visualized as "matrices with physical dimensions".

Nick: /* Single-stream RSF */

2009-12-23T15:25:19Z

Single-stream RSF

← Older revision		Revision as of 15:25, 23 December 2009
Line 95:		Line 95:

	==Single-stream RSF==		==Single-stream RSF==
	A single-stream RSF dataset consists of the ASCII header in which the last value of <tt>in</tt> is <tt>"stdin"</tt>. The ASCII header is followed by the sequence of three special characters: EOL EOL EOT (<tt>\014\014\004</tt>), followed by the data sequence. This stream can be written to a single rsf file with the standard ".rsf" extension if a special option passed to the madagascar program ~~is used~~. It can also be piped in the standard Unix manner.		A single-stream RSF dataset consists of the ASCII header in which the last value of <tt>in</tt> is <tt>"stdin"</tt>. The ASCII header is followed by the sequence of three special characters: EOL EOL EOT (<tt>\014\014\004</tt>), followed by the data sequence. This stream can be written to a single rsf file with the standard ".rsf" extension if a special option is passed to the madagascar program. It can also be piped in the standard Unix manner.

Nick: /* Single-stream RSF */

2009-12-23T15:24:48Z

Single-stream RSF

← Older revision		Revision as of 15:24, 23 December 2009
Line 95:		Line 95:

	==Single-stream RSF==		==Single-stream RSF==
	A single-stream RSF dataset consists of the ASCII header in which the last value of <tt>in</tt> is <tt>"stdin"</tt>. The ASCII header is followed by the sequence of three special characters: EOL EOL EOT (<tt>\014\014\004</tt>), followed by the data sequence. This stream can be written to a single rsf file with the standard ".rsf" extension~~, or~~ can be piped in the standard Unix manner.		A single-stream RSF dataset consists of the ASCII header in which the last value of <tt>in</tt> is <tt>"stdin"</tt>. The ASCII header is followed by the sequence of three special characters: EOL EOL EOT (<tt>\014\014\004</tt>), followed by the data sequence. This stream can be written to a single rsf file with the standard ".rsf" extension if a special option passed to the madagascar program is used. It can also be piped in the standard Unix manner.

Nick: /* Multi-file RSF */

2009-12-23T15:23:49Z

Multi-file RSF

← Older revision		Revision as of 15:23, 23 December 2009
Line 90:		Line 90:

	==Multi-file RSF==		==Multi-file RSF==
	When a Madagascar program writes a RSF file to disk (i.e.: <tt>sfprog >file.rsf</tt>), it will create:		When a Madagascar program writes a RSF file to disk (i.e.: <tt>sfprog >file.rsf</tt>), it will create by default:
	* a file with the ASCII header and the extension ".rsf"		* a file with the ASCII header and the extension ".rsf"
	* a file with the data sequence, in the encoding specified by the header, and the extension ".rsf@". Temporary files created during piping operations are allowed have no extension. The data file is located usually in the directory specified by the <tt>$DATAPATH</tt> environmental variable. The <tt>in</tt> field in the ASCII header specifies the absolute path of the data file.		* a file with the data sequence, in the encoding specified by the header, and the extension ".rsf@". Temporary files created during piping operations are allowed have no extension. The data file is located usually in the directory specified by the <tt>$DATAPATH</tt> environmental variable. The <tt>in</tt> field in the ASCII header specifies the absolute path of the data file.

Nick: /* ASCII header */

2009-12-23T15:22:31Z

ASCII header

← Older revision		Revision as of 15:22, 23 December 2009
Line 53:		Line 53:

	</pre>		</pre>

			Datasets of dimensions higher than 1 will have values for parameters describing the other axes such as n2, n3, etc.

	The header must contain only [http://en.wikipedia.org/wiki/ASCII#ASCII_printable_characters ASCII printable characters].		The header must contain only [http://en.wikipedia.org/wiki/ASCII#ASCII_printable_characters ASCII printable characters].

Nick: added illustration

2009-12-23T15:20:35Z

added illustration

← Older revision		Revision as of 15:20, 23 December 2009
Line 1:		Line 1:
			[[Image:Fotolia_2600038_XS.jpg\|right\|]]
	==Introduction==		==Introduction==

Nick at 13:18, 23 December 2009

2009-12-23T13:18:40Z

← Older revision		Revision as of 13:18, 23 December 2009
Line 93:		Line 93:
	==Single-stream RSF==		==Single-stream RSF==
	A single-stream RSF dataset consists of the ASCII header in which the last value of <tt>in</tt> is <tt>"stdin"</tt>. The ASCII header is followed by the sequence of three special characters: EOL EOL EOT (<tt>\014\014\004</tt>), followed by the data sequence. This stream can be written to a single rsf file with the standard ".rsf" extension, or can be piped in the standard Unix manner.		A single-stream RSF dataset consists of the ASCII header in which the last value of <tt>in</tt> is <tt>"stdin"</tt>. The ASCII header is followed by the sequence of three special characters: EOL EOL EOT (<tt>\014\014\004</tt>), followed by the data sequence. This stream can be written to a single rsf file with the standard ".rsf" extension, or can be piped in the standard Unix manner.

	~~==Data stream RSF==~~

	~~==Implementation of context detection==~~
	~~Madagascar I/O is an overloaded operation completely transparent for the user..~~

	~~==RSF files in streams==~~
	~~===Input===~~
	~~When a Madagascar program writes a RSF file to disk (i.e.: <tt>sfprog >file.rsf</tt>), it will create a header file and a binary file as described above.~~

	If the output is to a stream, or if the parameter <tt>--out=stdout</tt> is passed to the program, then the program will write to the stdout stream the ASCII header, followed by the sequence of three special characters: EOL EOL EOT (<tt>\014\014\004</tt>), followed by the binary.

	~~===Output===~~
	When a Madagascar program reads from the stdin stream, it expects either a EOF character indicating the end of the ASCII header (after which it transfers the stdin to reading from the binary cube), or a EOL EOL EOT sequence indicating that the data follows immediately on stdin.

Nick: /* Single-file RSF */

2009-12-23T13:15:27Z

Single-file RSF

← Older revision		Revision as of 13:15, 23 December 2009
Line 86:		Line 86:
	In this example dimension 1 is defined as the column direction, dimension 2 as the line direction, and dimension 3 as the "out of page" direction. This example follows the linear algebra / Fortran convention. It must be noticed that of only the reality of ordering of numerical values in the file is of substance; the imaginary representation with "vertical" and "horizontal" directions is not (the C language actually uses an inverse order for the dimensions). Thus, dimension 1 in a RSF dataset is in general defined as the dimension along which the first two elements in the data file are organized, then dimension 2 as the dimension adjacent to it, and so on.		In this example dimension 1 is defined as the column direction, dimension 2 as the line direction, and dimension 3 as the "out of page" direction. This example follows the linear algebra / Fortran convention. It must be noticed that of only the reality of ordering of numerical values in the file is of substance; the imaginary representation with "vertical" and "horizontal" directions is not (the C language actually uses an inverse order for the dimensions). Thus, dimension 1 in a RSF dataset is in general defined as the dimension along which the first two elements in the data file are organized, then dimension 2 as the dimension adjacent to it, and so on.

	==Single-~~file~~ RSF==		==Multi-file RSF==
			When a Madagascar program writes a RSF file to disk (i.e.: <tt>sfprog >file.rsf</tt>), it will create:
			* a file with the ASCII header and the extension ".rsf"
			* a file with the data sequence, in the encoding specified by the header, and the extension ".rsf@". Temporary files created during piping operations are allowed have no extension. The data file is located usually in the directory specified by the <tt>$DATAPATH</tt> environmental variable. The <tt>in</tt> field in the ASCII header specifies the absolute path of the data file.

			==Single-stream RSF==
			A single-stream RSF dataset consists of the ASCII header in which the last value of <tt>in</tt> is <tt>"stdin"</tt>. The ASCII header is followed by the sequence of three special characters: EOL EOL EOT (<tt>\014\014\004</tt>), followed by the data sequence. This stream can be written to a single rsf file with the standard ".rsf" extension, or can be piped in the standard Unix manner.

	==Data stream RSF==		==Data stream RSF==