Tutorial
Introduction
Welcome to a brief introduction to Madagascar. The purpose of this document is to teach new users how to use the powerful tools in Madagascar to: process data, produce documents and build your own Madagascar programs. To gradually introduce you to Madagascar, we have created a series of tutorials that are targeted to distinct audiences and designed to make you an experienced Madagascar user in a short-time period. The tutorials are divided by interest into three main categories:
- Users learn about Madagascar, how to use the processing programs, and build scripts.
- Authors learn how to build reproducible documents using Madagascar.
- Developers build new Madagascar programs that add additional functionality to Madagascar.
Each tutorial is designed to be completed in a short period of time. Additionally, each tutorial has hands-on examples that you should be able to reproduce on your computer as you go along with the tutorials. Most tutorials will use scripts that you can edit, modify, or play with to further gain experience and understanding. By the end of the tutorial series, you should be able to use all of the tools inside of Madagascar. Please note that this tutorial series does not explicitly show you how to process certain types of data, or how to perform common data processing operations (e.g. CMP semblance picking, time migration,etc.). Additional tutorials on those specific subjects will be added over time. The purpose of this document is simply to familiarize you with the Madagascar framework in a general sense. Before you go on, here are some notes on notation:
- important names, or program names are usually bold in the text. For example: sfwindow
- code snippets are always in the following formatting:
sfwindow
Users
The Users tutorials demonstrate how to use the Madagascar framework to create, process and visualize data, and to create reproducible scripts for processing data. The main goals of the Users tutorials are to learn about:
- the Madagascar framework,
- the RSF file format,
- the command line interface,
- how to interact with files on the command line,
- commonly used programs,
- how to make plots in Madagascar,
- how to make reproducible scripts,
- how to use SCons and Python,
- how to visualize your data.
By the end of this tutorial group you should be able to fully use all of Madagascar's built-in tools for data processing and scripting. By using these tools, you'll be able to process data ranging from tens of megabytes to tens of terabytes in a reproducible fashion.
Introduction to Madagascar
To begin, let's talk about the core principles of Madagascar, and the RSF file format.
Madagascar's design
There are a few layers to Madagascar. At the bottom-most layer, is the RSF file format, which is a common exchange format that all Madagascar programs use. Non-Madagascar programs can also read/write to and from RSF because it is an open exchange format. The next level of Madagascar contains the actual Madagascar programs that manipulate RSF files to process data. Concurrent to this level is the VPLOT graphics library which allows users to plot and visualize RSF files. The scripting utilities in Python and SCons are up another level from the core programs. These scripting utilities allow users to make powerful scripts that can perform even the most advanced data processing tasks. The last level includes support for LaTeX which allows you to make documents combining the features of Madagascar with the powerful typesetting of LaTeX. Throughout the course of these tutorials, we will examine all of these components, and demonstrate how they can be used individually, as well as together. When combined, the individual components of Madagascar allow us to: conduct experiments, process data, visualize our results, make reproducible scripts that can be shared with others, and write papers to document our experiments. Thus, Madagascar provides a fully integrated research environment.
RSF file format
As previously mentioned, the lowest level of Madagascar is the RSF file format, which is the format used to exchange information between Madagascar programs. Conceptually, the RSF file format is one of the easiest to understand, as RSF files are simply regularly sampled hypercubes of information. For reference, a hypercube is a hyper dimensional cube (or array) that can best be visualized as an -dimensional array, where is between 1 and 9 in Madagascar.
RSF hypercubes are defined by two files, the header file and the binary file. The header file contains information about the dimensionality of the hypercube as well as the data contained within the hypercube. Information contained in the header file may include the following:
- number of elements on all axes,
- the origin of the axes,
- the sampling interval of elements on the axes,
- the type of elements in the axes (i.e. float, integer),
- the size of the elements (e.g. single or double precision),
- and the location of the actual binary file.
Since we often want to view this information about files without deciphering it, we store the header file as an ASCII text file in the local directory, usually with the suffix .rsf . At any time, you can view or edit the contents of the header files using a text editor such as gedit, VIM, or Emacs.
The binary file is a file stored remotely (i.e. in a separate directory) that contains the actual hypercube data. Because the hypercube data can be very large ( of GB or TB) we usually store the binary files in a remote directory with the suffix .rsf@ . The remote directory is specified by the user using the DATAPATH environmental variable. The advantage to doing this, is that we can store the large binary data file on a fast remote filesystem if we want, and we can avoid working in local directories.
Because the header and binary are separated from one another, it is possible that we can lose either the header or binary for a particular RSF file. If the header is lost, then we can simply reconstruct the header using our previous knowledge of the data and a text editor. However, if we lose the binary file, then we cannot reconstruct the data regardless of what we do. Therefore, you should try and avoid losing either the header or binary data. The best way to avoid data loss is to make your research reproducible so that your results can be replicated later. Sometimes though we need to store RSF files for archiving or to transfer to other machines. Fortunately, we can avoid transferring the header and binary separately by using the combined header/binary format for RSF files. Files can be constructed using the combined header/binary format by specifying additional parameters on the command line, in particular --out=stdout , for any Madagascar program. The output file will then be header/binary combined, which allows you to transfer the file without fear for losing either the header or binary. Be careful though: header/binary combined files can be very large, and might slow down your local filesystem. A best practice is to only use combined header/binary files when absolutely necessary for file transfers. Note: header/binary combined files are usually automatically converted to header/binary separate files when processed by a Madagascar program.
Additional documentation
For more complete documentation on the RSF file format see the following links:
A gentle guide to the RSF file format
A detailed guide to the RSF file format
Command line interface
Madagascar was designed initially to be used from the command line. Programmers create Madagascar programs (prefixed with the sf designation) that read and write Madagascar files. These programs are designed to be as general as possible, so that each one could operate on any dataset in RSF format, provided you also supply the correct parameters and have the right type of data for the program.
Using programs
Madagascar programs follow the standard UNIX conventions for reading and writing RSF files to and from stdin and stdout . This is also the convention that Seismic Unix uses, so it should be familiar to SU users. For example: the program sfattr allows us to get attributes about an RSF file (mean, standard deviation, min, max, etc.) To get the attributes for a pre-computed file, we might use
sfattr < file.rsf
rms = 1.41316
mean = 0.999667
2-norm = 357.503
variance = 0.997693
std dev = 0.998846
max = 5.05567 at 36 8 27
min = -3.59936 at 18 9 6
nonzero samples = 64000
total samples = 64000
We demonstrate how to read and write RSF files using sfwindow which is a program that allows us to select portions of an RSF file. When sfwindow is used without any additional parameters, we are able to make a copy of a file with a different filename. For example:
sfwindow < file.rsf > file2.rsf
gives us two files, file.rsf and file2.rsf which are identical but not the same file. If your intention is simply to copy a file, you can also use sfcp . In addition to specifying files to read in and out on the command line we can specify the parameters for each program that are necessary for it to run, or to produce the desired result. The general format for specifying parameters on the command line is key=val , where key is the name of the parameter that you want to set, and val is the value of the parameter. There are four (4) types of values that are acceptable: integers, floating point numbers, booleans, or strings. Going back to the window program, we can specify the number of traces or pieces of the file that we want to keep like:
sfwindow < file.rsf n1=10 > file-win.rsf
Self-documentation
Of course, we can specify as many parameters as we'd like on the command line. To figure out which parameters are needed for a specific program, just type the name of the program with no input files our output files on the command line to bring up a program's self-documentation. For example, sfwindow 's self documentation is: �egin{verbatimtab}[4] NAME sfwindow DESCRIPTION Window a portion of a dataset. SYNOPSIS sfwindow < in.rsf > out.rsf verb=n squeeze=y j#=(1,...) d#=(d1,d2,...) f#=(0,...) min#=(o1,o2,,...) n#=(0,...) max#=(o1+(n1-1)*d1,o2+(n1-1)*d2,,...) PARAMETERS float d#=(d1,d2,...) sampling in #-th dimension largeint f#=(0,...) window start in #-th dimension int j#=(1,...) jump in #-th dimension float max#=(o1+(n1-1)*d1,o2+(n1-1)*d2,,...) maximum in #-th dimension float min#=(o1,o2,,...) minimum in #-th dimension largeint n#=(0,...) window size in #-th dimension bool squeeze=y [y/n] if y, squeeze dimensions equal to 1 to the end bool verb=n [y/n] Verbosity flag USED IN bei/dpmv/krchdmo bei/dpmv/matt bei/dwnc/sigmoid bei/fdm/kjartjac bei/fld/cube bei/fld/shotmovie bei/fld/synmarine bei/fld/yc bei/ft1/autocor \end{verbatimtab} The self-documentation tells us the function of the program, as well as the parameters that are available to be specified. The parameter format is type - name=default value [options] and then a short description of the parameter. File parameters request a name of a file. For example:
file=junk.rsf
Note: strings with spaces must be enclosed in quotation marks (e.g. "value").
Piping
Sometimes we want to chain multiple commands together without writing intermediate Madagascar RSF files in the process. We can avoid making intermediate files by using another standard UNIX construct, pipes. Pipes allow us to connect the standard output from one Madagascar program to the standard input to another program without first writing to a file. For example we could do the following without pipes:
sfwindow < junk.rsf > junk-win.rsf sftransp < junk-win.rsf > junk2.rsf
Or we could do the equivalent using pipes on one line:
sfwindow < junk.rsf | sftransp > junk2.rsf
Pipes simply make these statements more compact, and allow us to reduce the number of files that we need to save to disk. Piping is very powerful, because there is no limit to the number of statements that you can pipe together. For example:
sfwindow < junk.rsf | sftransp plane=12 | sftransp plane=23 | sfnoise > junk2.rsf
If you're using multiple programs, and do not want to save the intermediate files, then pipes will greatly reduce the number of files that you have to keep track of.
Interacting with files from the command line
Ultimately though, 95\
- sfin , used to get header information,
- sfattr , used to get file attributes,
- sfwindow , used to select portions of RSF files,
- and sftransp , used to reorder files.
Here are detailed usage examples and explanations of what the above programs do: sfin is one of the most used program on the command line, because most often we simply need to check the dimensionality of our files to make sure that we have them in the correct order.
sfin junk.rsf
junk.rsf:
in="/var/tmp/junk.rsf@"
esize=4 type=float form=native
n1=100 d1=0.004 o1=0 label1="Time" unit1="s"
n2=34 d2=0.1 o2=0 label2="Distance" unit2="km"
3400 elements 13600 bytes
sfattr is also commonly used from the command line to check files for correct values. Most often, we use sfattr to ensure that files are not filled with zeros, or with NaN's after a long computation, or to make sure that the values are reasonable. sfattr can be used to obtain basic statistics about the files as well.
sfattr < junk.rsf
*******************************************
rms = 1
mean = 1
2-norm = 58.3095
variance = 0
std dev = 0
max = 1 at 1 1
min = 1 at 1 1
nonzero samples = 3400
total samples = 3400
*******************************************
sfwindow is used to select subsets of the data contained in an RSF file for computation elsewhere. Typically, you specify the data subset you want to keep using, the n, j, and f parameters which specify the number of indices in the arrays to keep, the jump in indices, and the first index to keep from the file in the respective dimension. For example if we want to keep the 15th-30th time samples from the first axis in junk.rsf, we might use the following command:
sfwindow < junk.rsf f1=15 n1=15 j1=1 > junk-win.rsf
sftransp is used to reorder RSF files for other programs to be used. For example:
sftransp < junk.rsf plane=12 > junk-transposed.rsf
swaps the first and second axes, so that now the first axis is distance and the second axis is time. For more information about commonly used Madagascar programs please see the guide to Madagascar programs: http://reproducibility.org/wiki/Guide_to_madagascar_programs.
\documentclass[12pt]{geophysics}
\usepackage{hyperref}
\usepackage{comment}
\usepackage{verbatim}
\usepackage{moreverb}
\usepackage{tabularx}
\setfigdir{Fig}
Plotting
VPLOT provides a method for making plots that are small in size, aesthetically pleasing, and easily compatible with Latex for rapid creation of production quality images in Madagascar.
VPLOT
The VPLOT file format (.vpl suffix) is a self-contained binary data format that describes in vector format how to draw a plot on the screen using an interpreter. Since VPLOT is not a standard imaging format, VPLOT files must be viewed with third-party interpreters which we call pens. Each pen interfaces VPLOT with a third-party graphing library such as X11, plplot, opengl, and others. This flexibility makes VPLOT files almost as portable as standard image formats such as: jpeg, png, and gif. Unlike rasterized formats, VPLOT files can be scaled to any size without losing image quality.
Creating plots
To generate VPLOT files, we need to pass our computed RSF files through VPLOT filters, that convert RSF data files to VPLOT files. The VPLOT filters are named by the type of plot that they produce. Table~(table:plotting) lists all of the available VPLOT filters. �egin{table} �egin{tabularx}{ extwidth}{|l|X| } \hline sfgraph & create line plots, or scatter plots
sfgrey & create raster plots or 2D image plots
sfgrey3 & create 3D image plots of panels (or slices) of a 3D cube
sfbox & make box-line plots
sfcontour & make contour plots
sfcontour3 & make contour plots of 3D surfaces
sfplotrays & make plots of rays
\hline \end{tabularx} \caption{List of available plotting programs in Madagascar}
(table:plotting)
\end{table} To actually create a plot, we can use the plotting programs on the command line in the same fashion that we would use a Madagascar program:
sfspike n1=100 | sfnoise > junk.rsf sfgraph < junk.rsf title="noise" > junk.vpl
Visualizing plots
In this example, we create a single trace full of noise and then send it to sfgraph to produce a single VPLOT file, junk.vpl. As you may have noticed, this only creates the file which is useful for saving the plot,does not allow us to visualize the data. To visualize the data we need to use a pen , which tells your machine how to actually draw the plot. A typical Madagascar installation will have multiple pens available for you to use. By default, you should use sfpen which will pick the best pen available for you. You can use sfpen to visualize your plots in the following manner:
sfpen < junk.vpl
This will pop up a screen on your window with the plot shown. Depending on which pen you are using you may be able to interact with the pen interface to control various parameters of the plot as shown by the buttons at the top of the screen. Depending on the pen that you are using, there may be keyboard shortcuts to many of the buttons. NOTE: oglpen uses a mouse interface that can be accessed by right-clicking on the plot.
Converting VPLOT to other formats
If you want to build reports or documents using other programs, or want to send your images to someone who does not have Madagascar you will need to convert your VPLOT files to other image formats for transfer. To convert a VPLOT plot to another format use the tool vpconvert . vpconvert allows you to convert VPLOT files to any of the following formats, provided that you have the appropriate third-party libraries installed:
- avi,
- eps,
- gif,
- jpeg/jpg,
- mpeg/mpg (movie format),
- pdf,
- png,
- ppm,
- ps,
- svg,
- and tif.
Here's an example of how to use vpconvert :
vpconvert junk.vpl format=jpeg
NOTE: details on how to install these third-party libraries are not included with the Madagascar library, and we provide no support on installing them. Most users will be able to install them using either package management software (on Linux and Mac) or pre-compiled binaries.
\documentclass[12pt]{geophysics}
\usepackage{hyperref}
\usepackage{comment}
\usepackage{verbatim}
\usepackage{moreverb}
\usepackage{tabularx}
\setfigdir{Fig}
Scripting Madagascar
Madagascar's command line interface allows us to create, manipulate and plot files. Often though, we want to create scripts that will perform these operations automatically for us. Scripting is especially important when processing large amounts of data, or performing a complex chain of file manipulations in order to ensure that operations are completed in the right order or with proper parameters. In Madagascar, there are two main ways of creating scripts: shell scripts, and Python scripts using SCons.
Shell scripting
Shell scripting is the first option for creating scripts. In shell scripting we simply copy the command lines that we would use and paste them into a file that is recognized by either BASH, C-shell or another shell of your choosing. Shell scripting may be familiar to users of other packages as SU, because it is the primary method of scripting in other packages. An example of a Madagascar shell script is shown below:
#!/bin/bash sfspike n1=100 > junk.rsf < junk.rsf sfgraph > junk.vpl
For simple processing flows, shell scripting is a quick and easy approach to saving your processing flow. However, shell scripts quickly become unmanageable when more complicated processing flows are used. Additionally, shell scripts have no way to avoid repeating commands that have already been successfully run, which causes shell scripts to spend a significant amount of time duplicating already completed work. Because of these issues, Madagascar's main scripting option is to use a build manager called SCons for scripting instead of shell scripts.
SCons
SCons is a build manager written in 100\
SConstructs and commands
SCons scripts are referred to as SConstructs. In order to use SCons, you must create an SConstruct in the local directory where you want to work. Since SCons is written in Python, an SConstruct is simply a text file written using Python syntax. If you don't know Python, you can still use SCons, because the syntax is simple. First, a primer on Python syntax. In SConstructs we are going to deal with Python functions and strings. Python functions are simple, and should be familiar to anyone who has used a programming language. For example, calling a Python function, foo, looks like:
One argument - foo(1). Many arguments - foo(1,2,3,4,5,123)
Python functions can take many arguments, and arguments of different types, but the syntax is the same. Python strings are also very similar to other programming languages. In Python anything inside double quotes is automatically considered to be a string, and not a variable declaration. However, Python also supports a few notations to indicate strings (or long strings) as shown below:
"this is a string" 'this is a string' """this is a string""" "'this is a string"'
Somtimes in Python you will need to nest a string within a string. To do use one of the string representations for the outer string, and use a different one for the inner string. For example:
"""sfgraph title="junk plot" """ OR "' sfgraph title="junk plot" "' OR ' sfgraph title="junk plot" '
Fundamentally, SConstructs are composed of only three commands: Flow , Plot and Result . The first command is Flow . A Flow creates a relationship between the input file, the output file, and the processing command used to create the output file from the input file. The syntax for a Flow is:
Flow(output file,input file,command)
where, target and source are file names (strings), and command is a string that contains the Madagascar program to be used, along with the command line parameters needed for that program to run. For example:
Flow("spike1","spike","sfadd scale=4.0") ,
creates a dependency relationship between the output file 'spike1' and the input file 'spike'. The dependency indicates that 'spike1' cannot be created without 'spike' and that if 'spike' changes then 'spike1' also changes. The relationship in this case is that 'spike1' should be 'spike' scaled by four times. The equivalent command on the command line would be:
< spike.rsf sfadd scale=4.0 > spike1.rsf
Note: the file names of the input and output files do not need to include '.rsf' on the end of the files as SCons automatically adds the suffix to all of our file names. Now that we can create relationships between files using Flow s, we can create an SConstruct to do all of our data processing using SCons. However, we often want to visualize the results of our processing in order to quality control the results. In order to create Plot s (or other visualizations) in Madagascar we have two additional SCons commands: Plot and Result . Plot and Result tell SCons how to use Madagascar's plotting programs to create visualizations of files on the fly after they have been computed. The syntax for both Plot and Result is as follows:
Plot(input file, command) OR Result(input file, command)
In both cases, the Plot or Result command tells SCons to build a VPLOT file with same file name as the input file (with the .vpl suffix instead) from the input file using the command provided. For example, if we want to make a graph of a file we could use:
Plot("spike1","sfgraph pclip=100")
Behind the scenes, SCons establishes that we want to use "spike1.rsf" to create a Plot called "spike1.vpl" using sfgraph. The equivalent command line operation is:
< spike1.rsf sfgraph pclip=100 > spike1.vpl
Result can be used in the same way as Plot , illustrated above. At this point, you might be asking yourself, what's the difference between Plot and Result ? The answer is that Plot creates all VPLOT files in the local directory, whereas Result creates its VPLOT files in a subdirectory called Fig. Fig is a location used to store Plot s that we want to later use when creating papers using Latex. By default, you should use Plot when creating visualizations. Only use Result when you want to save something to be used in a paper. Note: since the VPLOTs from Plot and Result are placed in different locations you can use both Plot and Result for a single RSF file, but create two different plots for the same file.
Creating an SConstruct
Now that we have the three SConstruct commands, we can write our first SConstruct. To do so, open the SConstruct file in your favorite text editor. Before we can create any Flow , Plot or Result statements we have to add a first statement to the SConstruct. Enter the following statement (verbatim) into your new SConstruct:
from rsf.proj import *
This statement tells SCons where to find the Flow , Plot and Result commands, and must be included in every SConstruct. After that statement has been entered, you can enter as many Flow , Plot and Result commands as you wish making sure to use proper syntax. It's helpful to use a text editor that has Python syntax highlighting, as that will help you find and remedy strings that are not closed. You can also create Python comments using the mark to indicate the beginning of a comment to help document your SConstructs.
Lastly, you must include the following statement at the end of your SConstruct:
End()
This statement tells SCons that the script is done and that it should not look for anything else in the script. Make sure to include this statement as the very last item in every SConstruct, otherwise you will get interesting error statements during execution. Here's a sample SConstruct:
from rsf.proj import * # Remember, this statement comes first... ALWAYS
'''Flow''' ("spike",None,"sfspike n1=100 k1=50")
# None is a trick, see Advanced SCons for more information
'''Flow''' ("spike1","spike","sfadd scale=4.0")
'''Flow''' ("noise","spike1","sfnoise")
'''Plot''' ("spike",'sfgraph title="spike" ' ) # Note string nesting
'''Plot''' ("spike1",'sfgraph title="spike1" ')
'''Plot''' ("noise",'sfgraph title="noisy" ')
'''Result''' ("noise",'sfgraph title="noisy" pclip=75 ')
End() # Remember, this always ends the script.
Note: you do not have to order your Flow , Plot and Result commands as shown above. You can mix Flow , Plot and Result in any order. SCons automatically establishes the relationships between related files and commands.
Executing SCons
Now that you have an SConstruct, you can start processing data the Madagascar way. To do so open a terminal and navigate to the local directory where your SConstruct is located. To execute your SConstruct simply run: scons . When you run SCons, it will check to make sure that all necessary dependencies are found and that all of your commands inside the SConstruct are valid. If not, SCons will return an error message showing you where you made a mistake. If everything is OK, then SCons will begin creating your files in the local directory and will output its progress as it executes the Madagascar programs on the command line. For example, running the sample SConstruct from above should show something similar to the following output:
scons: Reading SConscript files ... scons: done reading SConscript files. scons: Building targets ... /opt/rsf/bin/sfspike n1=100 k1=50 > spike.rsf < spike.rsf /opt/rsf/bin/sfadd scale=4.0 > spike1.rsf < spike1.rsf /opt/rsf/bin/sfnoise > noise.rsf < spike.rsf /opt/rsf/bin/sfgraph title="spike" > spike.vpl < spike1.rsf /opt/rsf/bin/sfgraph title="spike1" > spike1.vpl < noise.rsf /opt/rsf/bin/sfgraph title="noisy" > noise.vpl < noise.rsf /opt/rsf/bin/sfgraph title="noisy" pclip=75 > Fig/noise.vpl scons: done building targets.
If the execution of scons ends with, "scons: done building targets" then the script completed successfully. Check your local directory for your output files.
Common errors in SConstructs
There are two common errors that most users will experience when executing SConstructs: missing dependency errors, and misconfiguration errors. We'll demonstrate both of these errors to help new users troubleshoot them below. The first error is caused by missing a dependency in the SConstruct. To introduce this error into your SConstruct modify the sample SConstruct from above to the following:
from rsf.proj import * # Remember, this statement comes first... ALWAYS
'''Flow''' ("spike1","spike","sfadd scale=4.0")
'''Flow''' ("noise","spike1","sfnoise")
'''Plot''' ("spike",'sfgraph title="spike" ' ) # Note string nesting
'''Plot''' ("spike1",'sfgraph title="spike1" ')
'''Plot''' ("noise",'sfgraph title="noisy" ')
'''Result''' ("noise",'sfgraph title="noisy" pclip=75 ')
End() # Remember, this always ends the script.
Now when you run scons , you should get an error message:
scons: Reading SConscript files ... scons: done reading SConscript files. scons: Building targets ... scons: *** [spike1.rsf] Source `spike.rsf' not found, needed by target `spike1.rsf'. scons: building terminated because of errors.
In this case, SCons tells you exactly which file is missing and which output file is missing one of its dependencies. To solve this problem, add the Flow to create 'spike' to the SConstruct. If your SConstruct uses a file that is not created inside the SConstruct, and is complaining about a missing dependency, then make sure the file you are looking for is in a location that the SConstruct can access. The second error, is caused by having a misconfigured command. To demonstrate this type of error change your SConstruct to:
from rsf.proj import * # Remember, this statement comes first... ALWAYS
'''Flow''' ("spike",None,"sfspike n1=100 k1=50")
# None is a trick, see Users 4 for more information
'''Flow''' ("spike1","spike","sfasd scale=4.0")
# HERE IS THE ERROR, NOTE sfasd instead of sfadd
'''Flow''' ("noise","spike1","sfnoise")
'''Plot''' ("spike",'sfgraph title="spike" ' ) # Note string nesting
'''Plot''' ("spike1",'sfgraph title="spike1" ')
'''Plot''' ("noise",'sfgraph title="noisy" ')
'''Result''' ("noise",'sfgraph title="noisy" pclip=75 ')
End() # Remember, this always ends the script.
In this case, we've introduced a typo into one of our commands. When running scons, the result is:
scons: Reading SConscript files ... scons: done reading SConscript files. scons: Building targets ... /opt/rsf/bin/sfspike n1=100 k1=50 > spike.rsf < spike.rsf sfasd scale=4.0 > spike1.rsf sh: sfasd: command not found scons: *** [spike1.rsf] Error 127 scons: building terminated because of errors.
Again, the error message is pretty descriptive and could help you track down the error relatively quickly. It's important to note that in this case, the error was caught after SCons tried to execute the command and failed to do so, whereas the dependency error was caught before any commands were executed. This means that this typo error would kill a script that's been running a long time without any issues up till that point. Fortunately, SCons would restart the script at the point of failure thereby saving you all the additional time to recompute everything before this point in the script. These are only two of the most common errors that new users will see. For additional information about debugging SConstructs, or for exceptionally strange errors please consult the RSF users mailing list.
Additional SCons commands
Here are some additional SCons commands that are useful to know.
Viewing Plots
If you're running an SConstruct and want to view the plots that it creates as it creates them, then you can use: scons view to force SCons to show the plots interactively. Doing so allows you to view your output at runtime and stop the SConstruct as it's running if you don't like what you see.
Viewing only one plot
If you want to view the plot associated with only a single file you can tell SCons to view that plot by using the command: scons file.view where file is the filename of the item that you want to view. For example: scons spike.view would show the plot of the spike.rsf file. The advantage to using this command is that SCons will only build the dependencies needed to view that specific plot, which can save you a lot of time in longer processing scripts.
Building specific targets
If you want to build a specific RSF file, or a few specific RSF files you can simply specify them on the command line when you execute SCons as follows: scons file.rsf file2.rsf file3.rsf ... . The advantage to doing this, is that SCons will only build those files and their dependencies instead of running through the entire script.
Cleaning up
After we've executed our SConstruct and viewed the results of the processings flows, we might decide that we want to clean up our project and go work on something else. Normally, one would have to delete all of the files and then move on, but SCons automates this process using the command: scons -c . Simply execute it in the local directory and SCons will remove all of the built RSF files, and VPLOT files. SCons will not remove files that it does not have rules for.
Dry-runs
Sometimes we want to test an SConstruct through the end of the script without waiting for it to run completely. To do so, we can use scons -n which tells SCons to simulate an actual execution. During a dry-run SCons will check all of the commands, and dependencies to ensure that they exist and are properly configured. If not, then SCons will tell you which commands are misconfigured. This command can save you hours of debugging! Use it liberally.
Parallel executions
For long scripts we can use parallel execution to run multiple processing commands at the same time. Parallel execution requires that the commands can function independently of one another (i.e. there are commands that do not depend on each other's output). To use parallel execution use: pscons which launches the maximum number of parallel jobs that your computer can handle at once. Parallel execution can greatly speed up your processing flows if SCons can parallelize it. In the worst case, SCons won't be able to parallelize your script and it will run at the same speed as if you just used scons .
Final thoughts
At this point you should have a solid understanding of how to use SCons and create SConstructs to script your Madagascar processing flows. Admittedly, SCons is somewhat complicated and difficult to understand at first, but don't give up. By using SCons, you are able to create powerful Madagascar scripts that are completely reproducible and enjoy the benefits of using a powerful build management system. However, we have not shown you the most useful aspects of SCons, which are demonstrated in the next tutorial, where we show how to integrate Python with SCons, thereby making processing flows that are very powerful.
\documentclass[12pt]{geophysics}
\usepackage{hyperref}
\usepackage{comment}
\usepackage{verbatim}
\usepackage{moreverb}
\usepackage{tabularx}
\setfigdir{Fig}
Advanced SCons
Now that you have a grasp of how to use SCons to put together simple processing flows, we're going to show you how to abuse SCons to make more advanced processing flows that can handle multiple input and output files properly. Additionally, we're going to demonstrate some SCons tricks that make your life easier, and allow you to work faster, and smarter.
Multiple input files
Many Madagascar programs require multiple input files and/or output multiple files. In order for SCons to properly recognize that these additional files are dependencies for a specific output file we have to change the syntax that we use for Flow , Plot and Result statements. To do so, we'll need to use Python lists to help us keep everything together when using our SCons commands. We first discuss the case where we need multiple input files. An example of a Madagascar program that requires multiple input files is sfcat . For reference, sfcat is used to concatenate multiple files together, essentially a file manipulation program. For example, we might use sfcat on the command line in the following fashion: �egin{verbatimtab}[4] < file.rsf sfcat axis=2 file1.rsf file2.rsf file3.rsf file4.rsf > catted.rsf \end{verbatimtab} To replicate this behavior using SCons we need to tell our Flow statements about the presence of multiple input files. Important: if we do not indicate to SCons that we have multiple input files then the dependency chain will not be correct and we cannot guarantee our results are correct . We can easily tell SCons of the presence of multiple input files by using a Python list as our input file, instead of a string: �egin{verbatimtab}[4] Flow('catted',['file','file1','file2','file3','file4'], 'sfcat axis=2 Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle {SOURCES[1:-1]}') \end{verbatimtab} As you may have noticed, there are two new items in this '''Flow''' statement, but let's start by discussing only the list of file names: '''['file','file1','file2','file3','file4']''' . The list of file names is simply a Python list of strings that contains each of the names of the files that we want to use in this Flow command. As usual, we don't have to append the '.rsf' suffix to the end of these names because SCons adds it for us. The second new part to the '''Flow''' command is: '''${SOURCES[1:-1]''' }, referred to as the SCons source list, which tells SCons about the presence of additional input files in the command, and to substitute the names into the command automatically. Without this command, SCons would not include the files in the final command. As an example of what the SCons source list does, compare the two SConstructs below against one another. The top is correct, the bottom is incorrectly configured: �egin{verbatimtab}[4] # Correct from rsf.proj import * Flow('file',None,'spike n1=100') Flow('file1',None,'spike n1=100') Flow('file2',None,'spike n1=100') Flow('file3',None,'spike n1=100') Flow('file4',None,'spike n1=100') Flow('catted',['file','file1','file2','file3','file4'], 'sfcat axis=2 } {SOURCES[1:-1]}') End() \end{verbatimtab}
# Wrong
from rsf.proj import *
Flow('file',None,'spike n1=100')
Flow('file1',None,'spike n1=100')
Flow('file2',None,'spike n1=100')
Flow('file3',None,'spike n1=100')
Flow('file4',None,'spike n1=100')
Flow('catted',['file','file1','file2','file3','file4'],'sfcat axis=2')
End()
If you noticed the command line output from SCons, you would find that for the incorrect SConstruct, SCons ran the following command:
< file.rsf sfcat axis=2 > catted.rsf
which is not correct. This is because SCons was not informed that the additional sources actually are used inside the command and did not substitute them in. The SCons source list contains a reference to all of the file names that we passed in our Python list earlier. In order to access those names we have to use a specific notation, but it is essentially a Python list enclosed in curly brackets that begins with a $. Since the source list is a Python list, we can get the file names in a few ways if we follow standard Python list conventions. Standard Python list conventions are:
- List indexing starts with index 0,
- Lists may be negatively indexed, which returns the items from the end (e.g. LIST[-1]),
- Lists may be sliced using the LIST[start:end] notation, where start and end are indices,
- List slicing indices are inclusive for the starting index, and exclusive for the ending index (e.g. LIST[0:4] returns LIST[0],LIST[1],LIST[2],LIST[3] but NOT LIST[4],
- Open slicing indices may be used (e.g. LIST[2:] gets everything from index 2 to the end, and LIST[:4] returns everything from 0 to but not including 4).
- Negative and positive indices may be used together (e.g. LIST[1:-1] returns all elements but the first and last).
These are the most useful conventions to remember, and the ones you will most frequently see. Please see the Python documentation (freely available online) for more information about dealing with Lists. Using the above conventions the following Flow statements are all equivalent for lettings SCons know about the presence of multiple input files: �egin{verbatimtab}[4] Flow('catted',['file','file1','file2','file3','file4'], "' sfcat axis=2 {SOURCES[2]}
![{\displaystyle {SOURCES[1]}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/b207dd37cc8a09318a92d16866d0ed276b0f39c4)
{SOURCES[4]}
![{\displaystyle {SOURCES[3]}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1de1f005a29deef089a614ead29d4e4f0c11d547)
"') Flow('catted',['file','file1','file2','file3','file4'], "' sfcat axis=2 Failed to parse (syntax error): {\displaystyle {SOURCES[1:5]} "') Flow('catted',['file','file1','file2','file3','file4'], "' sfcat axis=2 } {SOURCES[1:-1]} "') \end{verbatimtab} Note: never use SOURCES[0] because SOURCES[0] corresponds to 'file' which is already used by SCons for standard input. Also, never use open slicing on the SOURCES list, because at the end of the SOURCES list are extra items added by SCons for safe keeping that will break the command if accidentally used.
Multiple outputs
For multiple outputs, we can use the same conventions as before, except we specify a list of output files instead of input files, and we use the TARGETS SCons list, instead of SOURCES. For example:
Flow(['pef','lag'], 'dat', 'sflpef lag=${TARGETS[1]}').
None inputs
Sometimes, Flow s are created that don't have an input file. For example, files created using sfspike do not require input files. To get around the need for an input file, we can use the Python keyword None , equivalent to NULL in C or Java, to indicate to SCons that no input file is needed. For example:
Flow('spike',None,'sfspike n1=100')
Note: Plot and Result may not be created with None inputs.
Toggling standard input and standard output
When None inputs are used, then standard input is no longer needed and can be disabled. To turn off standard input on a Flow , add another argument to the Flow statement:
Flow('spike',None,'sfspike n1=100',stdin=0)
When SCons runs this Flow , the output command line will be:
sfspike n1=100 > spike.rsf
Likewise, we can toggle output to standard output as well. Standard output has two options, redirect to null or completely off. For some programs we need to redirect standard output to null, and others will require standard output to be completely off. To toggle standard output off use the following syntax:
Flow('spike',None,'sfspike n1=100',stdout=-1)
OR to redirect to null:
Flow('spike',None,'sfspike n1=100',stdout=0)
Plots with a different output name
Occasionally, you might want to create a plot with a different name than the input file. For example, a file might have multiple axes, and you could window along one of the axes, to create multiple graphs from a single input file. To distinguish between the different plots, you can rename the output files from Plot and Result commands using a syntax similar to Flow:
Plot('output','input','sfgraph')
This Plot command will produce output.vpl instead of input.vpl. In this way, you can create multiple visualizations of the same file. This applies to Result commands as well.
\documentclass[12pt]{geophysics} \usepackage{hyperref} \usepackage{comment} \usepackage{verbatim} \usepackage{moreverb} \usepackage{tabularx} \setfigdir{Fig}
Integrating Python with SCons
Because SCons is written in Python, we can take advantage of all the features of the Python programming language. By doing so, we are able to make our life a lot easier by: compartmentalizing build commands, creating functions for commonly used scripts, organizing our scripts to make them easier to understand. Instead of giving an exhaustive tutorial of the combination of SCons and Python, we're going to demonstrate a few of the more commonly features of Python inside of SConstructs. We leave it to the end user to develop additional techniques that further exploit the power of Python.
A forewarning
While Python and SCons are compatible with one another they are not completely interchangeable. To understand why that is the case, you need to understand the underlying design of SCons. SCons is a declarative language, which means that you tell SCons what to do through the: Flow, Plot, and Result commands, and then SCons decides how and when to execute those commands. Python on the other hand is imperative , which means that as Python reads a Python script, it executes the commands immediately. Python does not take time to decide how or when to execute your commands. This point causes a bit of confusion to users who start to combine Python and SCons together because they expect SCons commands to execute in the order that they place them, which is not the case. Because of this, you may not be able to use certain Python features in their native Python way. For example, loops which require repeated computation, and whose results depend on the result of the last iteration are not possible using SCons. It is also not usually possible to use Python variables that change during execution with SCons because the variable value that SCons will use is always the last value of the variable. Typically, you cannot use conditional statements in SCons, where the choice depends on a file that SCons will build. To be safe, you should always assume that whatever Python does happens before SCons starts running commands.
Variables
The most effective way to combine SCons and Python is to use Python to manage important variables for your scripts. For example, if you want to test a variety of values for a certain processing Flow, then you might save the value to a variable and then let Python format it correctly for you, instead of changing the string each time you want to run a new test. You can also save all your variables in a convenient location and then easily change them to test different parameters as well. For example: �egin{verbatimtab}[4] nx = 100 nz = 100 Flow('model',None, "'sfspike n1="'+str(nx)+"'n2="'+str(nz)) \end{verbatimtab} Note: variables must be converted to strings in order to be combined into the command statements. Because these variables are converted using string concatenation, there is a possibility that a user could give a value of a wrong type.
String substitution
While using variables is convenient, formatting them in the fashion shown above is not convenient. An easier way to format variables for strings is to use string substitution. String substitution works in the same way as the C - printf function works, i.e. we place markers that indicate where values should be substituted and what format they should be in. The above example using string substitution is: �egin{verbatimtab}[4] nx = 100 nz = 100 Flow('model',None, "' sfspike n1= "' \end{verbatimtab} In this statement, both nx, and nz are formatted as integers, and contained inside a tuple. All variables to be used for string substitution must be contained within the tuple at the end of the string statement. For reference, all C-printf like formatting choices are available in Python as well. Note: treat booleans as integers in Python.
Dictionaries
When scripts have a large number of variables, it is often easier to contain them within a Python dictionary instead of letting them float around the script. Dictionaries are declared in key=value format in Python using the dict keyword. For example:
parameters = dict(nx=100,nz=100,verb=True,final_file='output123')
To access variables from within the dictionary, we use list-like indexing where the index given is the name of the variable that we want to access:
nx = parameters['nx'] # Returns 100
We can also set variables within the dictionary, or modify their values after the initial declaration:
parameters['nx'] = 200 # Sets nx to 200 parameters['ny'] = 150 # Adds ny, and sets it to 150
To use the dictionary for string substitution, we only need to modify our formatters to include the key names of the variables that we wish to access from the dictionary. For example: �egin{verbatimtab}[4] Flow('model',None, "' sfspike n1= "' \end{verbatimtab} Notice that the formatters now have the name of the variable inside parentheses: \ By using dictionaries with string substitution, we can add flexibility to our scripts, and improve their readability, which ultimately improves the ability of others to reproduce our work. Thus, you should strive to use dictionaries wherever possible in your SConstructs.
Loops
Perhaps the most useful construct from Python that can be added to SConstructs are loops. By using loops, we can automate many redundant processes. To use a Python loop we just use the standard Python syntax in the following fashion: �egin{verbatimtab}[4] from rsf.proj import * for i in range(10): count = str(i) # Convert integer to string Flow('spike-'+count,None,'sfspike n1=100') Plot('spike-'+count,'sfgraph') End() \end{verbatimtab} In order for the loop to work, you need to make sure that all the files that are created have a unique file name. The easiest way to do so is to use the loop index in the filename, the count variable in this case. The count variable must be a string, because only strings can be concatenated together in Python. If you want to make more complicated file names (from nested loops) then examine the printf like syntax for Python strings. All the usual Python rules apply to these loops. Typically, for loops are easier to understand than while loops in Python, and so we recommend using for loops for most purposes.
Functions
Python functions are incredibly useful because you can compartmentalize repeated uses of commonly used Flow s, and then use them in multiple SConstructs. For the best use of Python functions you should use the following conventions:
- always use keyword=value arguments to help document your code,
- list file names for input and output first, then other arguments,
- use default values for all arguments, even if they are None,
- use the locals() dictionary to get the values of the arguments,
- always perform an action inside the function (i.e. create a Flow, Plot or Result),
- do not return anything from the function,
- and always accept **kwargs as the last argument to allow for dictionary substitution.
Here's an example of a well-defined function to create a Ricker wavelet with a peak frequency specified by the user: �egin{verbatimtab}[4] def ricker(out='wave',freq=20.0,kt=100,nt=1001,dt=0.01,ot=0.0,**kwargs): Flow(out,None, "' sfspike n1= nsp=1 mag=1.0 k1= sfricker1 frequency= "' \end{verbatimtab} Notice that we use the python function locals() for string substitution. This function returns a dictionary that contains only the names and values of the named arguments for the function. To call this function, you can use it as a normal Python function. Since there are default arguments, not all arguments need to be passed as long as you are OK with the default value.
ricker('wave',freq=30,kt=50)
If you are using a dictionary that has all of your variables in it, then you can call the function using explicit parameter fetching:
ricker('wave',parameters['freq'],parameters['kt'],...)
where you have to explicitly grab certain variables from the parameter dictionary. Conversely, you can use automatic parameter fetching:
ricker('wave',**parameters)
which will look for all the named arguments to the ricker function in the parameter dictionary, and send their values to the function. When there are many parameters, and you have already set them in the dictionary, then automatic parameter fetching is the best way to go.
Modules
Of course, functions can be compiled into groups and then placed into Python modules for widespread re-use throughout your SConstructs. Python modules are currently located in $RSFROOT/book/Recipes, which is where you should place your modules as well. As usual, you must use correct Python syntax to access functions contained within modules. For example, if you create a module called myutil.py, then you can access your functions in the following manner: �egin{verbatimtab}[4] import myutil myutil.ricker(...) \end{verbatimtab}
Classes
The least used, but most powerful part of Python that you can bring into your SConstructs are Python classes. For example, if you are writing a script to process multiple models in the exact same way, but that have different parameters you would have to write separate Flow statements to process each of them, OR you could write a Python class that takes the model parameters and uses those parameters to generate Flow statements automatically, similar to functions. However, a class can allow you to group functions together into a single coherent body and allow you to drastically reduce the amount of code that must be reused. We refer the reader to the Python documentation for more information on creating and using classes.
Final thoughts
Congratulations on completing the Madagascar User tutorial series. Now, you should have all the tools to: use Madagascar programs, write SConstructs to script your processing flows, and combine Python with SCons to make powerful scripts that can process data in ways not previously possible. From here, you can continue learning about how to write your own Madagascar programs, or learn about how to make reproducible documents using the Madagascar framework.