Editing Tutorial (section)

==Integrating Python with SCons==
Because SCons is written in Python, we can take advantage of all the features of the Python programming language.  By doing so, we are able to make our life a lot easier by: compartmentalizing build commands, creating functions for commonly used scripts, organizing our scripts to make them easier to understand.  Instead of giving an exhaustive tutorial of the combination of SCons and Python, we're going to demonstrate a few of the more commonly features of Python inside of SConstructs.  We leave it to the end user to develop additional techniques that further exploit the power of Python.
===A forewarning===
While Python and SCons are compatible with one another they are not completely interchangeable.  To understand why that is the case, you need to understand the underlying design of SCons. SCons is a ''declarative''  language, which means that you tell SCons what to do through the: Flow, Plot, and Result commands, and then SCons decides how and when to execute those commands.  Python on the other hand is ''imperative'' , which means that as Python reads a Python script, it executes the commands immediately.  Python does not take time to decide how or when to execute your commands.  This point causes a bit of confusion to users who start to combine Python and SCons together because they expect SCons commands to execute in the order that they place them, which is not the case.  Because of this, you may not be able to use certain Python features in their native Python way.  For example, loops which require repeated computation, and whose results depend on the result of the last iteration are not possible using SCons.  It is also not usually possible to use Python variables that change during execution with SCons because the variable value that SCons will use is ''always''  the last value of the variable.  Typically, you cannot use conditional statements in SCons, where the choice depends on a file that SCons will build.  
To be safe, you should always assume that whatever Python does happens ''before''  SCons starts running commands.
===Variables===
The most effective way to combine SCons and Python is to use Python to manage important variables for your scripts.  For example, if you want to test a variety of values for a certain processing Flow, then you might save the value to a variable and then let Python format it correctly for you, instead of changing the string each time you want to run a new test.  You can also save all your variables in a convenient location and then easily change them to test different parameters as well.  For example:
<syntaxhighlight lang="python">
nx = 100
nz = 100
Flow('model',None,'sfspike n1='+str(nx)+'n2='+str(nz))
</syntaxhighlight>
Note: variables must be converted to strings in order to be combined into the command statements.  Because these variables are converted using string concatenation, there is a possibility that a user could give a value of a wrong type.

===String substitution===
While using variables is convenient, formatting them in the fashion shown above is not convenient.  An easier way to format variables for strings is to use string substitution.   String substitution works in the same way as the C - printf function works, i.e. we place markers that indicate where values should be substituted and what format they should be in.  The above example using string substitution is:
<syntaxhighlight lang="python">
nx = 100
nz = 100
Flow('model',None,
'''
sfspike n1=%d n2=%d 
''' % (nx,nz) ) 
</syntaxhighlight>
In this statement, both nx, and nz are formatted as integers, and contained inside a tuple.  All variables to be used for string substitution must be contained within the tuple at the end of the string statement.  For reference, all C-printf like formatting choices are available in Python as well.  Note: treat booleans as integers in Python.

===Dictionaries===
When scripts have a large number of variables, it is often easier to contain them within a Python dictionary instead of letting them float around the script.  Dictionaries are declared in key=value format in Python using the '''dict'''  keyword.  For example:
<syntaxhighlight lang="python">
parameters = dict(nx=100,nz=100,verb=True,final_file='output123')
</syntaxhighlight>
To access variables from within the dictionary, we use list-like indexing where the index given is the name of the variable that we want to access:
<syntaxhighlight lang="python">
nx = parameters['nx'] # Returns 100
</syntaxhighlight>
We can also set variables within the dictionary, or modify their values after the initial declaration:
<syntaxhighlight lang="python">
parameters['nx'] = 200 # Sets nx to 200
parameters['ny'] = 150 # Adds ny, and sets it to 150
</syntaxhighlight>
To use the dictionary for string substitution, we only need to modify our formatters to include the key names of the variables that we wish to access from the dictionary.  For example:
<syntaxhighlight lang="python">
Flow('model',None,
	'''
	sfspike n1=%(nx)d n2=%(nz)d 
	''' % parameters )
</syntaxhighlight>
Notice that the formatters now have the name of the variable inside parentheses: %(nx)d before the formatting expression.  Then, the entire dictionary is passed to the string for substitution.  At runtime, Python places the values for the keys from the dictionary into the string.  If the values are the wrong type, or the key does not exist in the dictionary, then Python will throw an error at runtime, and prevent you from running with a bad value.

By using dictionaries with string substitution, we can add flexibility to our scripts, and improve their readability, which ultimately improves the ability of others to reproduce our work.  Thus, you should strive to use dictionaries wherever possible in your SConstructs.

===Loops===
Perhaps the most useful construct from Python that can be added to SConstructs are loops.  By using loops, we can automate many redundant processes.  To use a Python loop we just use the standard Python syntax in the following fashion:
<syntaxhighlight lang="python">
from rsf.proj import *
for i in range(10):
    count = str(i) # Convert integer to string
    Flow('spike-'+count,None,'sfspike n1=100 mag='+count)
    Plot('spike-'+count,'sfgraph')
End()
</syntaxhighlight>
In order for the loop to work, you need to make sure that all the files that are created have a unique file name.  The easiest way to do so is to use the loop index in the filename, the count variable in this case.  The count variable must be a string, because only strings can be concatenated together in Python.   If you want to make more complicated file names (from nested loops) then examine the printf like syntax for Python strings. 
All the usual Python rules apply to these loops.  Typically, for loops are easier to understand than while loops in Python, and so we recommend using for loops for most purposes.

===Functions===
Python functions are incredibly useful because you can compartmentalize repeated uses of commonly used '''Flow''' s, and then use them in multiple SConstructs.  For the best use of Python functions you should use the following conventions:
*always use keyword=value arguments to help document your code, 
*list file names for input and output first, then other arguments, 
*use default values for all arguments, even if they are None, 
*use the locals() dictionary to get the values of the arguments, 
*always perform an action inside the function (i.e. create a Flow, Plot or Result), 
*do not return anything from the function, 
*and always accept **kwargs as the last argument to allow for dictionary substitution. 
Here's an example of a well-defined function to create a Ricker wavelet with a peak frequency specified by the user:
<syntaxhighlight lang="python">
def ricker(out='wave',freq=20.0,kt=100,nt=1001,dt=0.01,ot=0.0,**kwargs):
    Flow(out,None,
    	 '''
	 spike n1=%(nt)d o1=%(ot)f d1=%(d1)f 
         nsp=1 mag=1.0 k1=%(kt)d l1=%(kt)d | 
	 ricker1 frequency=%(freq)f 
	 ''' % locals())
</syntaxhighlight>
Notice that we use the python function '''locals()'''  for string substitution.  This function returns a dictionary that contains only the names and values of the named arguments for the function.  
To call this function, you can use it as a normal Python function.  Since there are default arguments, not all arguments need to be passed as long as you are OK with the default value.  
<syntaxhighlight lang="python">
ricker('wave',freq=30,kt=50)
</syntaxhighlight>
If you are using a dictionary that has all of your variables in it, then you can call the function using explicit parameter fetching:
<syntaxhighlight lang="python">
ricker('wave',parameters['freq'],parameters['kt'],...)
</syntaxhighlight>
where you have to explicitly grab certain variables from the parameter dictionary.  Conversely, you can use automatic parameter fetching: 
<syntaxhighlight lang="python">
ricker('wave',**parameters)
</syntaxhighlight>
which will look for all the named arguments to the ricker function in the parameter dictionary, and send their values to the function.  When there are many parameters, and you have already set them in the dictionary, then automatic parameter fetching is the best way to go.

===Modules===
Of course, functions can be compiled into groups and then placed into Python modules for widespread re-use throughout your SConstructs.  Commonly used Python modules are currently located in <tt>&#36;RSFROOT/book/Recipes</tt>, which is where you should place your modules as well.  As usual, you must use correct Python syntax to access functions contained within modules.  For example, if you create a module called myutil.py, then you can access your functions in the following manner:
<syntaxhighlight lang="python">
import myutil
myutil.ricker(...)
</syntaxhighlight>

===Classes===
The least used, but most powerful part of Python that you can bring into your SConstructs are Python classes.   For example, if you are writing a script to process multiple models in the exact same way, but that have different parameters you would have to write separate Flow statements to process each of them, OR you could write a Python class that takes the model parameters and uses those parameters to generate Flow statements automatically, similar to functions.  However, a class can allow you to group functions together into a single coherent body and allow you to drastically reduce the amount of code that must be reused.  
We refer the reader to the Python documentation for more information on creating and using classes.

===Final thoughts===
Congratulations on completing the Madagascar User tutorial series.  Now, you should have all the tools to: use Madagascar programs, write SConstructs to script your processing flows, and combine Python with SCons to make powerful scripts that can process data in ways not previously possible.  From here, you can continue learning about how to write your own Madagascar programs, or learn about how to make reproducible documents using the Madagascar framework.