Advanced Installation: Difference between revisions

From Madagascar
Jump to navigation Jump to search
← Older editNewer edit →
Karl (talk | contribs)
313 edits
Karl (talk | contribs)
313 edits
Line 888: Line 888:
# (Optionally) <b>Open vplots using double click</b>: [[machacks|Instructions]].
# (Optionally) <b>Open vplots using double click</b>: [[machacks|Instructions]].
# Install python with libraries including jupyter and  ipython.  I recommend the Anaconda distribution.  which is available at https://www.anaconda.com/download
# Install python with libraries including jupyter and  ipython.  I recommend the Anaconda distribution.  which is available at https://www.anaconda.com/download
# (Optionally) <b>SWIG</b> is required for some of the options of the Python api (used if you are coding in Python).  run  <pre>conda install swig</pre> . I had to run<pre>cp `which siwg` /usr/local/bin</pre> to get ./configure to find swig.  
# (Optionally) <b>SWIG</b> is required for some of the options of the Python api (used if you are coding in Python).  run  <pre>conda install swig</pre> . I had to run<pre>cp `which swig` /usr/local/bin</pre> to get ./configure to find swig.  
#Proceed with configuration and installation following the normal procedure.  You may need to use Apple's compiler (clang) instead of gcc.  Use one of these commands in the $RSFSRC directory <pre>./configure CC=clang CXX=clang++</pre> or <pre>./configure CC=clang CXX=clang++ API=python --prefix=`pwd`</pre>
#Proceed with configuration and installation following the normal procedure.  You may need to use Apple's compiler (clang) instead of gcc.  Use one of these commands in the $RSFSRC directory <pre>./configure CC=clang CXX=clang++</pre> or <pre>./configure CC=clang CXX=clang++ API=python --prefix=`pwd`</pre>
#build the system with: <pre>make install</pre>
#build the system with: <pre>make install</pre>

Revision as of 21:55, 14 December 2018

Before reading this document, please familiarize yourself with the short Installation guide.

What the installation process does

The term "installation" in the title is used for brevity, and it actually covers all three steps: configuration, build and install.

  1. Configure: determine what tools are available on the system and how they should be used to built the software. Creates a layer of abstraction so that the build is platform-independent. Should ideally either solve or flag all problems, so that the build either works, or does not proceed at all.
  2. Build: compiles the software and documentation using RSFSRC/build as a "workplace"
  3. Install: moves the compiled executables and the documentation to the final locations in $RSFROOT, sometimes changing filenames. Kept separate from build so that it can be done by root, and to avoid build failures leaving junk files all over the system.

A successful installation will have created in $RSFROOT the following directories:

  • bin/: executable programs
  • doc/: auto-generated HTML documentation
  • include/: header files with info on library procedures; fonts
  • lib/: libraries and Python modules

Old (1.0 and 1.1) version installation

Use $RSFROOT/etc/madagascar/ as the location of env scripts instead of $RSFROOT/share/madagascar/etc/.

Old (0.9.9) version installation

Environment variables

Example configuration for bash: 

export RSFROOT=/usr/local/rsf # directory where Madagascar will be installed. 
if [ -n "$PYTHONPATH" ]; then
export PYTHONPATH=${PYTHONPATH}:$RSFROOT/lib
else
export PYTHONPATH=$RSFROOT/lib
fi
export PATH=$RSFROOT/bin:$PATH
export DATAPATH=/var/tmp/
export MANPATH=$RSFROOT/share/man:$(manpath)
export LD_LIBRARY_PATH=$RSFROOT/lib:$LD_LIBRARY_PATH

Notice the slash at the end of the DATAPATH variable.

Example configuration for csh and tcsh: 

setenv RSFROOT /usr/local/rsf
if ($?PYTHONPATH) then
setenv PYTHONPATH ${PYTHONPATH}:$RSFROOT/lib
else
setenv PYTHONPATH $RSFROOT/lib
endif
set path = ($RSFROOT/bin $path)
setenv DATAPATH /var/tmp/
setenv MANPATH $RSFROOT/share/man:`manpath`
setenv LD_LIBRARY_PATH $RSFROOT/lib:$LD_LIBRARY_PATH

Notice the backticks surrounding the call to manpath.

Be aware that on some systems /var/tmp gets automatically cleaned at some intervals, so if you want to keep your data binaries for a long time, set DATAPATH to another location where you have write access and that allows large files.

Software construction

  1. Configuration. Change to the top source directory and run 
    ./configure
    You can examine the config.py file that this command generates. Additional options are available. You can obtain a full list of customizable variables by running scons -h. For example, to install C++ and Fortran-90 API bindings in addition to the basic package, run 
    ./configure API=c++,fortran-90
  2. Building and installing the package. Run scons install or the following two commands in succession: 
    make; make install
    or 
    scons; scons install
    If you need "root" privileges for installing under $RSFROOT, you may need to run 
    su; scons install
    or 
    sudo scons install
  3. Cleaning. To clean all intermediate files generated by SCons, run 
    make clean
    or 
    scons -c

Prerequisites

Basic prerequisites are described in the short Installation guide. Here are some additional details.

Compilers

Madagascar has been built successfully with the following compilers, and possibly with others:

  • gcc
  • Intel (icc/ifort)
  • open64
  • clang
  • cc (Solaris)

Python and SCons

As described below under Platform-specific installation advice, Madagascar supports the oldest non-deprecated Python version currently supported by the latest stable version of SCons. If your version of Python is older and you experience problems you should probably upgrade.

Madagascar includes the latest stable version of SCons and the configure scripts will try to install it for you in RSFROOT if you don't have it already. However, if you have an older version of SCons the configure scripts will not try to install the newer version. Your older version might work fine, but Madagascar attempts to support only the latest stable version of SCons, so if you have problems you should upgrade.

To install the SCons bundled with Madagascar go to RSFSRC/scons, unpack the tar file, and type 

python setup.py install

This will install SCons in the standard location. You might need root privileges. If you don't have root privileges, or you don't want to interfere with the system SCons you can install it somewhere else with a --prefix option. A logical choice is to put it in RSFROOT like this: 

python setup.py install --prefix=$RSFROOT

Location

As long as you set the environment variables and directory permissions correctly, it does not matter in what part of your filesystem you place the install. If you have the luxury of installing anywhere, it is good practice to follow the Filesystem Hierarchy Standard and either:

  1. Install everything (including figs if you do testing) under /usr/local/rsf, with the source tree in /usr/local/rsf/src, OR
  2. Put the source tree in /usr/local/src/rsf, and specify RSFROOT=/usr/local, so that header files and binaries go in /usr/local/bin and /usr/local/include. To follow the standard, before installing set RSFDOC=/usr/local/share/rsf/doc and create the appropriate directories. The auto-generated HTML documentation will get put there. Also, if installed, the figs directory for testing should be /usr/local/share/rsf/figs/.
  3. Package Madagascar (i.e. build a RPM, etc) and install it in the default locations. For RPMs, those are as like the ones from the previous option, just directly in the /usr/ hierarchy, instead of in the /usr/local/ one.

Disk space

At present (Feb 2007, r2530), the source directory containing the build tree from the development version was approx. 200Mb, the full installation (bin/, doc/, include/ and lib) is 31Mb, and figs/ (the optional directory if you want to do testing) is about 10 Gb. The stable version is significantly smaller.

The only Madagascar-related directory where disk space can be an issue is $DATAPATH. Real 3-D seismic datasets can be measured in Terabytes. Buggy programs/processing flows can fill up $DATAPATH. A real problem are "disk memory leaks" -- removing header files with anything else than sfrm will leave the binaries intact. Crashed jobs which start to write to binary but never get to write the header also produce "leaks". Experience has shown that over time $DATAPATH inexorably fills up. You may need to

  1. keep irreplaceable data and expensive results in a separate place;
  2. remove the oldest files in $DATAPATH whenever the amount of free space declines under a preset threshold.

Dependencies

Some platforms feature complete lists of dependencies. See Platform-specific installation advice for details.

C++ API

A C++ compiler. SCons is smart and will try to find it for you. If it does not work specify the path to your compiler in the CXX environment variable (can be passed as an option to the configuration script, like the API one).

F77 API

A Fortran 77 compiler. If SCons does not find one, then you can either specify its path through the F77 variable, or if the executable is in your path, add its name to the list of F77 compilers in RSFSRC/configure.py .

F90 API

Same as for Fortran 77 – just substitute F90. If using the gfortran compiler, make sure to get the latest version. If you have more than one compiler installed on your system, specify the desired one at configuration time: <bash> ./configure API=f90 F90=/path/to/preferred/compiler </bash>

Java API

There are two styles of API, old (1.0 release and previous) and new. The two APIs are not compatible with each other.

The "old" interface requires the Mines Java Toolkit for fast Java file IO. The Mines JTK, is an open-source Java package that can be downloaded from Mines JTK. Currently, the "old" interface will also build alongside the new one if the MINESJTK environment variable exists, and can be used as was previously.

To install the new API, you need the (Oracle) JDK. Set your JAVA_HOME environment variable to the location of the jdk (on Ubuntu 10.04 this is: /usr/lib/jvm/java-6-openjdk), then reconfigure (./configure API=java ...) and reinstall.

Ignore the SWIG warnings (there are lots).

Make sure to set your LD_LIBRARY_PATH to $RSFROOT/lib .

If you want to include additional Java packages, you can set them using your shell's CLASSPATH variable. This environment variable is now automatically passed onto all Java classes in SCons.

The installation can be tested using the example demonstrating the new API in api/java/test .

Matlab API

Besides Matlab itself, you need Mex, which compiles C code into regular Matlab functions. Use the MATLAB and MEX environment variables to specify their paths if they are installed, but not found.

Octave API

The Octave function compiler (mkoctfile) is sometimes bundled in a separate package, so it may be missing from the Octave installation.

Python API

This API requires SWIG, numpy and the Python development kit. Numpy requires Python 2.4 or newer (i.e. RHEL 5 or newer). However, these dependencies are unnecessary for the common case when Python is just used as glue to create chains of programs, and it only needs to read the RSF header, and not the binary. To allow Python metaprograms in madagascar to function, and programming in this style to be done, a fallback development kit implementing only the header-related functionality will be installed in the lack of these dependencies.

Python modules in user space

Python is an evolving language. Many large systems have old versions for stability reasons, and administrators of such large systems tend to not install all software users may wish, and to not allow access to rpm either. To install a module in your user space, download the tarball, unzip it, cd into the directory and run: 

python setup.py install --prefix=/path/to/your/place

The installer will create a subdirectory named lib, or lib64 under the directory above. These lib* dirs will have a directory named python, or python2.3 for example, and those will have a subdirectory named site-packages. Add all paths to these site-packages subdirectories in your PYTHONPATH environment variable. Some (numpy) may create a bin directory that needs to be added to PATH.

Environment variables

Besides the variables defined in env.sh or env.csh (see the short Installation guide), Madagascar programs may read the variables below. They usually have reasonable defaults and were introduced just to provide more power to the advanced user.

For future documentation writers: the environment variables read by Madagascar that have not been documented below can be found by running the script $RSFSRC/admin/find_env_var.py. If the script does not exist or does not work, a summary of all environment variable calls can be obtained by going to $RSFSRC, temporarily moving the directory build/ outside RSFSRC, and typing 

grep environ.get *.py */*.py */*/*.py */*/*/*.py
grep getenv           */*.c  */*/*.c  */*/*/*.c

Used by the Madagascar core

Variables introduced by Madagascar's non-graphic programs
Name Default Meaning
RSF_DATASERVER ftp://egl.beg.utexas.edu/ Data server for benchmark datasets
RSFDOC $RSFROOT/doc Directory for the HTML self-doc
RSFFIGS $RSFROOT/figs Directory with figures for testing examples in $RSFSRC/book
RSFALTFIGS $RSFFIGS Alternate directory with figures for testing examples not in $RSFSRC/book
RSFMEMSIZE 100 Maximum RAM (Mb) to be used by some programs
RSFSRC undefined Root of the Madagascar source tree
TMPDATAPATH $DATAPATH Datapath for temporary files on local disk.
LATEX2HTML undefined LateX2HTML customization directory


Variables introduced by Madagascar graphics programs
Name Default Meaning
DEFAULT_PAPER_SIZE "letter" For pspen. Other options: legal, a3, a4, a5.
FATMULT ? Fatness multiplication factor.
GIFBORDER 0.25 For vplot2gif (spacing)
GIFDELAY 100 For vplot2gif (for animations)
IMAGE_TYPE 'png' Icon type for LateX2HTML
PATTERNMULT None Pattern multiplication factor
PLOTSTYLE None Used in vplot
PPI 75 For vplot2gif (screen resolution)
PPMSCALE 1 For vplot2gif
PSBORDER 0.05 For vplot2eps (border around the plot)
PSPRINTER postscript or colorps For pspen
PSTEXPENOPTS color=n fat=1 fatmult=1.5 invras=y Other vplot2eps options
VPLOTFONTDIR $RSFROOT/include Dir with backup fonts in case the runtime-loaded vplot fonts are not found
VPLOTSPOOLDIR /tmp Where to put vplot tmp files
WSTYPE "default" Workstation type.


Variables set by OS/other apps, read-only to Madagascar
Name Primarily used/set by
CWPROOT Seismic Unix
DISPLAY Operating System (OS)
HOME OS
LD_LIBRARY_PATH linker
MATLABPATH Matlab
XAUTHORITY X-Windows

Used by the Madagascar build process and parallelization utilities

Type scons -h in RSFSRC to get a list of environment variables that affect the build process, with explanations, defaults and actual values. Below are more detailed explanations for some of them:

  • RSF_THREADS: used by pscons to determine on how many threads to run on the local node, overriding the number of threads detected by Madagascar
  • RSF_CLUSTER: used by pscons to determine on which cluster nodes to run, and on how many CPUs

Used by the Matlab API

To use the Matlab API, you need to add $RSFROOT/lib to MATLABPATH

Used by the Octave API

To use the Octave API, you need to add $RSFROOT/lib to Octave's path. Determine Octave's version with 

octave -v | head -1

If your version is lower than 2.9.6, type at a Unix command line: 

echo 'LOADPATH = "::$RSFROOT/lib/octave"' >> ~/.octaverc

For later versions, use: 

echo 'addpath([getenv("RSFROOT") "/lib/octave"])' >> ~/.octaverc

Used by the Java API

New-style (post-1.0) API: Needs JAVA_HOME

Old-style API (1.0 and before): The path to the downloaded Mines JTK must be specified in the MINESJTK environment variable in order to install the Java API. For example: 

export MINESJTK=/home/user/edu_mines_jtk.jar

RSFROOT for NFS-shared user home directories

Heterogeneous networks with user home directories shared through NFS are quite common in many institutions. In addition, even when the architecture is the same (i.e. 64-bit) and the operating system is the same (i.e. RHEL), the difference between operating system versions may be very significant because clusters may run legacy versions, while desktop workstations may run the latest-and-greatest (even beta), and entirely different Madagascar versions may be needed to support both.

One possible solution of detecting the distribution version and architecture and setting RSFROOT appropriately is shown below. In the example network, all RHEL4 machines have the same architecture, but there are RHEL 3 machines with several architectures: 

REDHAT_RELEASE=`awk -F'release' '{ print $2 }' /etc/redhat-release | awk -F' ' '{ print $1 }'`

RSFROOT=/usr/local/rsf/rhel$REDHAT_RELEASE

if [ $REDHAT_RELEASE == '4' ] ; then
    export RSFROOT
elif [ $REDHAT_RELEASE == '3' ] ; then
    export RSFROOT=$RSFROOT/$ARCH
fi

Of course, the Madagascar administrator will have to download appropriate versions of Madagascar to each $RSFROOT, and compile them on the appropriate system.

If you have many kinds of systems to maintain, with multiple versions of Madagascar, and users have more than one shell, you may find it easy to outsource the complex logic to the easy-to-debug Python, i.e.: 

export RSFROOT=`$M8R_SETUP/get_rsfroot.py`
export PYTHONPATH=`$M8R_SETUP/edit_pythonpath.py`
export PATH=`$M8R_SETUP/edit_path.py`

and similarly for (t)csh. The Python scripts determine the operating system and its version, determine the machine name, and simply print to stdout the desired string.

Eclipse + Pydev

If you use Eclipse with Pydev, configure the interpreter by adding $RSFROOT/lib to the PYTHONPATH for your chosen interpreter.

Platform-specific installation advice

Supported platforms

Madagascar attempts to support any POSIX-compliant operating system demanded by users. For systems that bundle Python (i.e. Linux distributions, BSDs), backwards compatibility will attempt to cover those systems that were bundled with the oldest non-deprecated Python version currently supported by the latest stable version of SCons. For example, in early 2009 the stable SCons release (1.2) supported Python 2.2 or newer. Python 2.2 was bundled by RHEL3, so RHEL 3 and newer are supported.

Attempts for backward compatibility with a given operating system are also stopped if the operating system itself becomes unsupported. For example, Python 2.2 was bundled by Fedora 1 and newer, but in January 2010 only Fedora 11 and 12 are actively maintained. Thus, in January 2010 Madagascar was not attempting to support Fedora 1, even though it included Python 2.2.

Please keep in mind that the above statements constitute only general guidelines for what will be attempted, and do not constitute in any way a warranty of support. An application of the above guidelines to some Linux distributions follows:

Support info

Distribution Life Cycle Supported versions
RHEL/CentOS/Scientific Linux 7 years
  • 6 until 2017-11-30
  • 5 until 2014-03-31
Fedora Release X maintained until one month after the release of X+2
  • 15 until 2012-06-24
  • 14 until 2011-12-02
Ubuntu Releases every 6 mo, maintained for 1.5 yrs; LTS versions every 2 yrs, maintained for 5 yrs
  • 15.10 until July 2016
  • 14.04 LTS until end of April 2019
  • 12.04.4 LTS until end of April 2017
Debian Usually: stable releases every 1.5-3 yrs, release X maintained 1 yr after release X+1
  • 6 until its TBD end of life (approx. 2014)
  • 5 until 2012-04
openSUSE openSUSE releases Lifetime of 1.5-2.5 years
  • 11.4 until 2012-09-10
  • 11.3 until 2012-01-15

Ubuntu

In Ubuntu 18.04, you can install the essential Madagascar depenency by running 

sudo apt-get install libxaw7-dev freeglut3-dev libnetpbm10-dev libgd-dev libplplot-dev \
libavcodec-dev libcairo2-dev libjpeg-dev swig python-dev python-numpy g++ gfortran \
libopenmpi-dev libfftw3-dev libsuitesparse-dev python-epydoc scons git emacs25

In Ubuntu 15.10 Wily Werewolf (as well as some previous versions), you can install the essential Madagascar depenency by running 

sudo apt-get install libxaw7-dev

and (optionally) all of Madagascar's dependencies by running 

sudo apt-get install libxaw7-dev freeglut3-dev libnetpbm10-dev libgd-dev libplplot-dev \
libavcodec-dev libcairo2-dev libjpeg-dev swig python-dev python-numpy g++ gfortran \
libopenmpi-dev libfftw3-dev libsuitesparse-dev python-epydoc

In Ubuntu 13.04 Raring Ringtail, you can install all of Madagascar's dependencies by running 

sudo apt-get install freeglut3-dev g++ gfortran libgd2-xpm-dev libx11-dev libxaw7-dev \
libnetpbm10-dev swig python-dev python-numpy libblas-dev libcairo2-dev liblapack-dev \
libavcodec-dev python-epydoc scons

In Ubuntu 12.10 Quantal Quetzal, you can install all of Madagascar's dependencies by running 

sudo apt-get install freeglut3-dev g++ gfortran libgd2-xpm-dev libx11-dev libxaw7-dev \
libnetpbm10-dev swig python-dev python-numpy libblas-dev libcairo2-dev liblapack-dev \
libavcodec-dev python-epydoc

In Ubuntu 12.04 Precise Pangolin, you can install all of Madagascar's dependencies by running 

sudo apt-get install scons openmpi-bin libopenmpi-dev freeglut3-dev g++ gfortran libgd2-xpm-dev \
libglew1.6-dev libx11-dev libxaw7-dev libnetpbm10-dev swig python-dev python-scipy python-numpy \
libtiff4-dev scons units libblas-dev libcairo2-dev liblapack-dev libavcodec-dev python-epydoc

In Ubuntu 10.10 Maverick Meerkat, you can install all of Madagascar's dependencies by running 

sudo apt-get install freeglut3-dev g++ gfortran libgd2-xpm-dev libglew1.5-dev libjpeg62-dev \
libx11-dev libxaw7-dev libnetpbm10-dev swig python-dev python-scipy python-numpy libtiff4-dev \
scons units libblas-dev libcairo2-dev libavcodec-dev libplplot-dev

In Ubuntu 9.04 Jaunty Jackalope, the corresponding command is 

sudo apt-get install freeglut3-dev g++ gfortran libc6-dev libgd2-xpm-dev libglew1.5-dev libjpeg62-dev \
libx11-dev libxaw7-dev libnetpbm10-dev swig python-dev python-scipy python-numpy libtiff4-dev scons units

Earlier versions may work with 

sudo apt-get install mesa-libGL-devel g++ g77 libc6-dev libgd2-xpm-dev libglew-dev libjpeg62-dev \
libx11-dev libxaw7-dev libnetpbm10-dev swig python-dev python-scipy python-numpy libtiff4-dev scons units

If working with the development version, you will also need git or subversion.

Fedora, RedHat, CentOS, Scientific Linux, openSUSE

Install essential Madagascar dependencies with 

sudo yum install gcc libXaw-devel

Dependency package names, sorted by Linux distribution and Madagascar feature they provide. Packages that are not included in the standard distro repositories are hyperlinked to their providers. The tables below cover build dependencies.

Names of packages that are runtime dependencies are highlighted in the tables below (task under construction).

Note: In the future, it should be possible for the configuration scripts to output the dependency tables below, so that they are guaranteed to be in synch with a given Madagascar version

Minimal install ("Core"), publishing and development

Core LaTeX Development version C++ API F77 API, F90 API Python API Java API Octave API Matlab API
Fedora 15 binutils, gcc, glibc-headers, python texlive-latex subversion gcc-c++ gcc-gfortran numpy, swig, python-devel Java (Sun's? IcedTea?) octave, octave-devel Matlab with Mex
Fedora 14 binutils, gcc, glibc-headers, python texlive-latex subversion gcc-c++ gcc-gfortran numpy, swig Java (Sun's? IcedTea?) octave, octave-devel Matlab with Mex
Fedora 13 binutils, gcc, glibc-headers, python texlive-latex subversion gcc-c++ gcc-gfortran numpy, swig Java (Sun's? IcedTea?) octave, octave-devel Matlab
CentOS 5 binutils, gcc, glibc-headers; python (needs RPMforge's RHEL5 repository) ? subversion gcc-c++ gcc-gfortran numpy, swig Java (Sun's? IcedTea?), Mines JTK Octave, octave-devel Matlab
openSUSE 11.0 gcc, python texlive-latex subversion gcc-c++ gcc-fortran NumPy, swig Java (Sun's? IcedTea?), Mines JTK Octave Matlab


Graphics and visualization

The minimum requirement is X11 graphics for running xtpen.

X11 graphics vplot2gif vplot2avi Some sort of movies? TIFF output JPEG output PLplot graphics OpenGL graphics ppm (?) unknown
Fedora 13 libXaw-devel gifsicle ffmpeg (needs the RPM Fusion repository enabled) ffmpeg-devel (needs the RPM Fusion repository enabled) libtiff-devel libjpeg-devel plplot-devel mesa-libGL-devel, freeglut, freeglut-devel netpbm-devel ?
openSUSE 11.0 xorg-x11-devel ? ? ? libtiff-devel libjpeg-devel plplot-devel mesa-libGL-devel, freeglut, freeglut-devel netpbm-devel cairo-devel, gd-devel, glew-devel

Numerical and file manipulation utilities

OpenMP MPI BLAS/ATLAS Interface to the Fast Discrete Curvelet Transform sfunits
Fedora 13 libgomp openmpi, openmpi-devel; openmpi-libs (?) blas, blas-devel, atlas, atlas-devel pyct units
CentOS 5 ? ? ? pyct Gnu Units

Other

Inclusions from Seismic Unix
Fedora 14, 15 Seismic Unix
CentOS 5 Seismic Unix

Command to install all dependencies present in the public repositories

Usually package management software will not install again a package that is already installed, so it should be safe to copy and paste the command below to a command line:

Fedora 15 yum -y install binutils gcc glibc-headers scons texlive-latex subversion gcc-c++ gcc-gfortran numpy python python-devel swig octave octave-devel libgomp openmpi openmpi-devel blas blas-devel atlas atlas-devel units gifsicle ffmpeg ffmpeg-devel libtiff-devel libjpeg-devel plplot-devel mesa-libGL-devel freeglut freeglut-devel libXaw-devel netpbm-devel
Fedora 13 yum -y install binutils gcc glibc-headers scons texlive-latex subversion gcc-c++ gcc-gfortran numpy python swig octave octave-devel libgomp openmpi openmpi-devel blas blas-devel atlas atlas-devel units gifsicle ffmpeg ffmpeg-devel libtiff-devel libjpeg-devel plplot-devel mesa-libGL-devel freeglut freeglut-devel libXaw-devel netpbm-devel
CentOS 5 yum -y install binutils freeglut freeglut-devel gcc gcc-c++ gcc-gfortran glibc-headers libjpeg-devel libXaw-devel netpbm-devel
openSUSE 11.0 zypper install cairo-devel gcc gcc-c++ gcc-fortran gd-devel glew-devel libjpeg-devel libtiff-devel octave scons subversion texlive-latex xorg-x11-devel

List of runtime dependencies only (needed by packagers of Madagascar in order to properly list dependencies): UNDER CONSTRUCTION

Fedora 25

The following explains how to install madagascar on Fedora 25.

We use the following compiler: gcc, g++ and gfortran.

  • Install some packages

sudo dnf install scons lapack-static lapack64-static atlas-static fftw-static gd gd-devel libtool-ltdl-devel \ libXaw-devel mesa-libGL-devel freeglut-devel netpbm-devel plplot-devel ffmpeg-devel swig openmpi openmpi-devel suitesparse-devel This installed the following packages: 

scons-2.5.1-1.fc25.noarch
lapack-static-3.6.1-2.fc25.x86_64
lapack64-static-3.6.1-2.fc25.x86_64
atlas-static-3.10.2-14.fc25.x86_64
fftw-static-3.3.5-3.fc25.x86_64
gd-2.2.4-1.fc25.x86_64
gd-devel-2.2.4-1.fc25.x86_64
libtool-ltdl-devel-2.4.6-13.fc25.x86_64
libXaw-devel-1.0.13-4.fc25.x86_64
mesa-libGL-devel-17.0.5-3.fc25.x86_64
freeglut-devel-3.0.0-3.fc24.x86_64
netpbm-devel-10.78.00-1.fc25.x86_64
plplot-devel-5.11.1-13.fc25.x86_64
ffmpeg-devel-3.1.9-1.fc25.x86_64
swig-3.0.11-1.fc25.x86_64
openmpi-1.10.5-1.fc25.x86_64
openmpi-devel-1.10.5-1.fc25.x86_64
suitesparse-devel-4.4.6-7.fc25.x86_64

and many other dependencies.

  • Get the sources from the git repository
git clone https://github.com/ahay/src

If you run now the configure script, you will get: 

checking for Python ... /usr/bin/python
checking Python version ... 2.7.13
checking for RSFROOT ... /usr/local/rsf
checking for SCons ... /usr/bin/scons
checking SCons version ... v2.5.1.rel_2.5.1:3735:9dc6cee5c168[MODIFIED]
Running RSFROOT=/usr/local/rsf /usr/bin/scons  config ...
------------------------
scons: Reading SConscript files ...
checking platform ... (cached) linux [fedora]
checking for C compiler ... (cached) gcc
checking if gcc works ... yes
checking if gcc accepts '-x c -std=gnu99 -Wall -pedantic' ... yes
checking for ar ... (cached) ar
checking for libraries ... ['m']
checking complex support ... yes
checking for X11 headers ... /usr/include
checking for X11 libraries ... /usr/lib
checking for OpenGL ... yes
checking for sfpen ... (cached) xtpen
checking for ppm ... yes
checking for tiff ... yes
checking for GD (PNG) ... yes
checking for GD (GIF) ... yes
checking for plplot ... no

  Optional package: plplot-devel
checking for ffmpeg ... yes
checking for cairo (PNG) ... yes
checking for cairo (SVG) ... yes
checking for cairo (PDF) ... yes
checking for jpeg ... yes
checking for BLAS ... yes
checking for LAPACK ... no

  Optional package: blas + blas-devel + atlas + atlas-devel
checking for SWIG ... (cached) /usr/bin/swig
checking for numpy ... (cached) yes
checking API options ... (cached) []
checking for C++ compiler ... (cached) g++
checking if g++ works ... yes
checking if g++ accepts '-std=c++11 -U__STRICT_ANSI__ -Wall -pedantic' ... yes
checking for MPICC ... (cached) /usr/lib64/openmpi/bin/mpicc
checking if /usr/lib64/openmpi/bin/mpicc works ... yes
checking for MPICXX ... (cached) /usr/lib64/openmpi/bin/mpicxx
checking if /usr/lib64/openmpi/bin/mpicxx works ... yes
checking for MPIRUN ... (cached) /usr/lib64/openmpi/bin/mpirun
checking for Posix threads ... yes
checking for OpenMP ... yes
checking for CUDA ... (cached) no
checking for FFTW ... yes
checking if FFTW supports threads ... yes
checking for SuiteSparse ... yes
checking for pfft ... no
scons: done reading SConscript files.
scons: Building targets ...
shell_script(["env.sh"], [])
shell_script(["env.csh"], [])
scons: done building targets.
------------------------
Done with configuration.

Libraries plplot and lapack are not found. File config.log says 

/bin/ld: cannot find -lplplotd

and 

gcc -o .sconf_temp/sfconftest_21 .sconf_temp/conftest_21.o -L/usr/lib64/atlas -lm -lf77blas -lcblas -latlas -llapack -lcblas
/usr/lib64/atlas/liblapack.a(dgesv.o): In function `dgesv_':
(.text+0x69): undefined reference to `xerbla_'
/usr/lib64/atlas/liblapack.a(ATL_f77wrap_dgesv.o): In function `atl_f77wrap_dgesv_':
(.text+0x10d): undefined reference to `ATL_xerbla'
/usr/lib64/atlas/liblapack.a(ATL_dgetf2.o): In function `ATL_dgetf2':
(.text+0x184): undefined reference to `ATL_dger'
...

So library plplotd doesn't exist on Fedora 25 and the configure script is looking for the wrong lapack library.

If compiling now, 2 other libraries won't be found: gfortran and quadmath

  • Regarding the plplotd library, edit file framework/configure.py and change the line 724:
plplot = context.env.get('PLPLOT','plplotd')

to 

plplot = context.env.get('PLPLOT','plplot')
  • As for the libraries
    • lapack
    • gfortran
    • quadmath

We must use the lapack library installed in /usr/lib64, not the one installed in /usr/lib64/atlas. And libraries gfortran and quadmath are installed in /usr/lib/gcc/x86_64-redhat-linux/6.3.1/. Edit file framework/configure.py and add the following 4 lines: 

['/usr/lib/gcc/x86_64-redhat-linux/6.3.1/',
 '/usr/lib64/',

and 

LIBS.append('gfortran')
LIBS.append('quadmath')

so that we end up with the following blas function: 

def blas(context):
    context.Message("checking for BLAS ... ")
    text = '''
    #ifdef __APPLE__
    #include <Accelerate/Accelerate.h>
    #else
    #ifdef HAVE_MKL
    #include <mkl.h>
    #else
    #include <cblas.h>
    #endif
    #endif
    int main(int argc,char* argv[]) {
    float d, x[]={1.,2.,3.}, y[]={3.,2.,1.};
    d = cblas_sdot(3,x,1,y,1);
    return 0;
    }\n'''

    if plat['OS'] == 'cygwin':
        context.env['ENV']['PATH'] = context.env['ENV']['PATH'] + \
                                         ':/lib/lapack'

    res = context.TryLink(text,'.c')
    if res: 
        context.Result(res)
        context.env['BLAS'] = True 
    else:
        # first try blas
        LIBS = path_get(context,'LIBS')
        blas = context.env.get('BLAS','blas')
        LIBS.append(blas)
        res = context.TryLink(text,'.c')
        if res: 
            context.Result(res)
            context.env['LIBS'] = LIBS 
            context.env['BLAS'] = blas 
        else:
            # some systems require cblas and atlas
            for atlas_dir in filter(os.path.isdir,
                                    ['/usr/lib/gcc/x86_64-redhat-linux/6.3.1/',  # <--- add this line
                                     '/usr/lib64/',                              # <--- add this line
                                     '/usr/lib64/atlas/',
                                     '/usr/lib/atlas/']):
                context.env['LIBPATH'].append(atlas_dir)
            LIBS.pop()
            LIBS.append('f77blas')
            LIBS.append('cblas')
            LIBS.append('atlas')
            LIBS.append('gfortran')  # <----------------------------------------------- add this line
            LIBS.append('quadmath')  # <----------------------------------------------- add this line
            res = context.TryLink(text,'.c')
            if res: 
                context.Result(res)
                context.env['LIBS'] = LIBS 
                context.env['BLAS'] = 'cblas'
            else:
                context.Result(context_failure)
                context.env['CPPDEFINES'] = \
                    path_get(context,'CPPDEFINES','NO_BLAS')
                LIBS.pop()
                LIBS.pop()
                LIBS.pop()
                context.env['BLAS'] = None 
                need_pkg('blas', fatal=False)
  • Run the configure script, for example:
./configure --prefix=/usr/local/rsf

Output should give: 

checking for Python ... /usr/bin/python
checking Python version ... 2.7.13
checking for RSFROOT ... /usr/local/rsf
checking for SCons ... /usr/bin/scons
checking SCons version ... v2.5.1.rel_2.5.1:3735:9dc6cee5c168[MODIFIED]
Running RSFROOT=/usr/local/rsf /usr/bin/scons  config ...
------------------------
scons: Reading SConscript files ...
checking platform ... (cached) linux [fedora]
checking for C compiler ... (cached) gcc
checking if gcc works ... yes
checking if gcc accepts '-x c -std=gnu99 -Wall -pedantic' ... yes
checking for ar ... (cached) ar
checking for libraries ... ['m']
checking complex support ... yes
checking for X11 headers ... /usr/include
checking for X11 libraries ... /usr/lib
checking for OpenGL ... yes
checking for sfpen ... (cached) xtpen
checking for ppm ... yes
checking for tiff ... yes
checking for GD (PNG) ... yes
checking for GD (GIF) ... yes
checking for plplot ... yes
checking for ffmpeg ... yes
checking for cairo (PNG) ... yes
checking for cairo (SVG) ... yes
checking for cairo (PDF) ... yes
checking for jpeg ... yes
checking for BLAS ... yes
checking for LAPACK ... yes
checking for SWIG ... (cached) /usr/bin/swig
checking for numpy ... (cached) yes
checking API options ... (cached) []
checking for C++ compiler ... (cached) g++
checking if g++ works ... yes
checking if g++ accepts '-std=c++11 -U__STRICT_ANSI__ -Wall -pedantic' ... yes
checking for MPICC ... (cached) /usr/lib64/openmpi/bin/mpicc
checking if /usr/lib64/openmpi/bin/mpicc works ... yes
checking for MPICXX ... (cached) /usr/lib64/openmpi/bin/mpicxx
checking if /usr/lib64/openmpi/bin/mpicxx works ... yes
checking for MPIRUN ... (cached) /usr/lib64/openmpi/bin/mpirun
checking for Posix threads ... yes
checking for OpenMP ... yes
checking for CUDA ... (cached) no
checking for FFTW ... yes
checking if FFTW supports threads ... yes
checking for SuiteSparse ... yes
checking for pfft ... no
scons: done reading SConscript files.
scons: Building targets ...
shell_script(["env.sh"], [])
shell_script(["env.csh"], [])
scons: done building targets.
------------------------
Done with configuration.
  • Then run the compilation:
make
  • and finally the installation
sudo make install

Debian 5

Specific dependencies:

  • Debian 5.0 ("Lenny"): Please make sure you have the libc6-dev package before trying to compile from source. The libXaw7-dev package might be a dependency for xtpen (was in Debian 4.0)

Yellow Dog Linux 6.1 on Sony PlayStation 3

See Will Burnett's guide (PowerPoint)

Mac OS X

Install necessary Mac OS X applications using:

  • MacPorts, an easy-to-use system for compiling, installing, and upgrading open-source software on Mac OS X.
  • Fink, a tool that brings the full world of Unix Open Source software to Mac OS X.

Installation on Mac OS X Mountain Lion requires the following:

  1. Xcode: Download and install XCode from Apple using App Store application. In Xcode, make sure to enable and install the command-line tools in Xcode/Preferences/Downloads
  2. (Optionally) X11: Install X11 libraries from Xquartz.
  3. (Optionally) gcc: Install the Gnu C compiler from HPC Mac OS X.
  4. (Optionally) SEGTeX: To use SEGTeX, you may need TeX Live. MacPorts and Fink provide an easy way to install it with commands 
    sudo port install texlive
    or 
    sudo fink install texlive
  5. (Optionally) FFTW: Using MacPorts, run 
    sudo port install fftw-3-single
    sudo port install fftw-3 +gfortran
  6. (Optionally) MPI: Using MacPorts, run 
    sudo port install openmpi
    sudo port select --set mpi openmpi-mp-fortran
  7. (Optionally) ppmpen: Using MacPorts, run 
    sudo port install libnetpbm
  8. (Optionally) Open vplots using double click: Instructions.
  9. Install python with libraries including jupyter and ipython. I recommend the Anaconda distribution. which is available at https://www.anaconda.com/download
  10. (Optionally) SWIG is required for some of the options of the Python api (used if you are coding in Python). run 
    conda install swig
    . I had to run
    cp `which swig` /usr/local/bin
    to get ./configure to find swig.
  11. Proceed with configuration and installation following the normal procedure. You may need to use Apple's compiler (clang) instead of gcc. Use one of these commands in the $RSFSRC directory 
    ./configure CC=clang CXX=clang++
    or 
    ./configure CC=clang CXX=clang++ API=python --prefix=`pwd`
  12. build the system with: 
    make install
  13. After installing a new version of python you must run:
./configure 
make install

In summer of 2018 on a new MacBook Pro running Mac OS 10.14.1 (Mojave) I encountered an "abort trap: 6" error message when running sfpen from command line. I do not have a problem running sfpen inside scons. I changed to use xtpen from the command line. I continue to use this work around.

OpenSolaris

Use pkg to install missing components such as X11 headers. 

pfexec pkg install SUNWxorg-headers

MS Windows

Due to its size, this topic has been assigned its own Wiki page.

How to adapt Madagascar to a new platform

The most laborious part of adapting madagascar to a new platform is finding the proper dependency names. This usually proceeds as follows: dependency X fails with a "missing file" error either as a header file in config.log, or a missing library during the build step. Possible package names are found through an internet search for the missing file name and the distribution name or by using specific rpm search tools. Packages are installed and the configure (and, if necessary) build processes are repeated until the error goes away.

Multi-user installs

Some organizations may find it desirable to deny write access of some users to all RSFSRC/RSFROOT except their own user directory. Fortunately, this can be easily done by placing the restricted user dirs outside RSFSRC/RSFROOT, i.e. in their home dirs, say /home/joe/rsfsrc. In order to move a user's directory out of RSFSRC, you must:

  • "tell" the SConstruct in the user's dir where to find RSFSRC so that when the user builds in his directory, it can import configure.py and config.py You do that by setting the environment variable RSFSRC to the absolute path of the Madagascar source root, and by making sure that lines 2 and 3 in the users' SConstruct files are
srcroot = os.environ.get('RSFSRC', '../..')
sys.path.append(srcroot)

and then replace ../.. throughout the SConstruct using os.path.join and the srcroot variable.

  • "tell" the build scripts about the user's dir, so that it is included in the builds launched from RSFSRC. You do that with a symbolic link:
ln -s /home/joe/rsfsrc $RSFSRC/user/joe

When the link exists, those of Joe's programs that are mentioned in the "prog" string in SConstruct get included in the distribution, complete with self-doc. If Joe is just learning how to code and his stuff breaks the build, just remove the symbolic link. Even if build+installs are done after the link is removed, his stable programs and self-doc will continue to remain installed system-wide as long as the admin does not type scons -c install (not likely).

  • point the user's RSFDOC environment variable to a location where the user has write access
  • edit the users' SConstruct so that it uses the RSF library and headers already installed in $RSFROOT/lib and $RSFROOT/include , instead of building again the whole librsf with user-specific flags in RSFSRC/filt/lib/. To do that, replace in the user's SConstruct the env.Prepend statement with
rsfroot = os.environ.get('RSFROOT','/usr/local/rsf')

env.Prepend(CPPPATH=[os.path.join(rsfroot,'include')],
            LIBPATH=[os.path.join(rsfroot,'lib')],
            LIBS=['rsf'])
  • If the link from RSFSRC to Joe's directory was not made, add Joe's directory to his own path so that he can execute his own binaries.

To understand how $DATAPATH disk space issues may become an issue in a multi-user environment, refer to the Disk Space subsection at the beginning of this document.

Keeping your stuff separate

A user may add his own programs and recipes to the Madagascar system. He may also create his own computational examples, data, and locked figures for testing. All of these components can be placed in their default locations, but it is not necessary to make them public. To keep these items private simply do not add them to the repository.

However, it might be desirable to keep these components in separate places. For example, if you keep your private programs in RSFSRC/user you will have to remember to make a copy somewhere else if you ever want to delete the Madagascar installation to perform a fresh install. Yup, I deleted all my programs that way once. Good thing I had a back up! Fortunately, it is easy to keep each of these components in a separate place if desired.

Keeping programs separate

User programs are ordinarily kept in a subdirectory of RSFSRC/user. However, if you want to keep your programs separate all you have to do is put your subdirectory somewhere else and make a link to it in RSFSRC/user: <bash> ln -s path_to_my_programs $RSFSRC/user/my_programs </bash> The additional instructions above for "multi-user installs" are for the case where the other users do not have write access to RSFSRC. However, if you have full write access and only want to keep the programs in a separate place the link is the only thing you need.

Problems when symbolic linking and using SVN

If you decide to use a symbolic link e.g. "ln -s ..." as above, you will have difficulties checking in your code into the SVN repository. SVN typically does not follow the symbolic link but will only the save the symbolic link.

See this answer from Stackoverflow (http://stackoverflow.com/questions/24850844/follow-symlinks-in-svn)



If I understand your problem, you have:

project1/ project1/link1 -> ../../some/where/else

If you do a simple svn add link1 it adds a symlink entry to the subversion repository, but what you're trying to accomplish is getting the stuff under somewhere else into the tree.

If this is the case, then you're fighting in the wrong direction, you should make the real files and directories under link1, and make the target locations symlinks into the link1 directory. That would be a simple solution to the problem.

Another solution would be to make the location ../../some/where/else an svn location in it's own right, and make link1 an externals definition to that location. When you commit in the main directory, the externals would be committed at the same time which would achieve storing the information; however to ensure that the other location was kept in sync, you would have to make sure to update it to the same version as the stored data.

In my case, on my desktop, I have a settings directory:

$HOME/settings

This is a checked out copy of a directory containing .bashrc, .profile, .vimrc, .vim, etc. files and folders from an svn repo. All my profile files and directories were symlinks into this folder, so for example .profile -> settings/.profile, etc. When I make changes, they are reflected in the svn tree, which I can commit back to in order to ensure that I don't lose my config settings when I go to another system.

If you really want svn to follow symlinks as hardlinks (You can't make directory hardlinks because it would be a bad thing™), then you'll have to hack the svn client source so that it did what you wanted; but this would be a maintenance nightmare.

You could get away with making the symlink into a bind mount point directed at the target, but that has it's own issues - you would need to be root to accomplish this, and it ends up leaving funny entries in your /proc/mounts to accomplish this:

mount --bind /absolute/path/to/some/where/else project1/link1

Keeping recipes separate

Computational recipes written in Python and imported by the SConstruct file of a workflow are normally stored in RSFSRC/book/Recipes. The install process copies these recipes to a directory like $RSFROOT/lib/python2.5/site-packages/rsf/recipes and adds this directory to your PYTHONPATH so that Python can find them.

However, you can put you own recipes anywhere you want. You only have to add that place to your PYTHONPATH like this (bash): 

export PYTHONPATH=${PYTHONPATH}:path_to_my_recipes

Or like this (csh): 

setenv PYTHONPATH ${PYTHONPATH}:path_to_my_recipes

Keeping examples separate

Madagascar's public collection of example workflows are stored in RSFSRC/book, but you can put your private workflows anywhere you want. No special instructions are required.

However, Madagascar assumes that the workflows are organized into a three-level book/chapter/section directory hierarchy when it creates a directory tree for the data and locked figures associated with your workflow. It is not required, but it might be easier to find the data and locked figures if you put your workflows in a three-level directory tree something like this: path_to_my_book/chapter/section/SConstruct.

Keeping data separate

The location of the data portion of your *.rsf files is controlled by your DATAPATH environment variable. However, you may want to keep the data for your private workflows in a different place, or several different places, than the data created by the public examples in RSFSRC/book. The way to do that is to temporarily change the DATAPATH variable in the SConstruct before importing rsf.proj like this: 

import os
os.environ['DATAPATH'] = 'path_to_my_private_data'

from rsf.proj import *

Keeping locked figures separate

The command "scons lock" in the directory of a workflow will store a "locked" copy of your figures for regression testing. Normally these figures are stored in the location pointed to by your RSFFIGS variable, and that is where the figures from the figures repository should be stored for testing in RSFSRC/book.

However, you may want to keep your private figures in a different place. The way to do that is to temporarily change the RSFFIGS variable in the SConstruct before importing rsf.proj like this: 

import os
os.environ['RSFFIGS'] = 'path_to_my_private_figures'

from rsf.proj import *

If you also create a RSFALTFIGS environment variable pointing to path_to_my_private_figures, then the testing script sffiglist will automatically test your figures against those in RSFALTFIGS when the sffiglist command is executed from a location outside of RSFSRC/book.

Capturing error and warning messages

The messages during configuration are few and their importance quite high, so they should be watched "in person". A complete log of the configuration process is recorded in RSFSRC/configure.log

Console messages generated during the build step can be captured to a log file and observed at the same time with a command like this (tcsh): 

nice +10 nohup /usr/bin/time -p scons -k |& tee log_build.asc

The log file can be of course named otherwise than log_build.asc. The file can be later grepped for error and warnings with commands such as: 

grep -c error log_build.asc
grep error log_build.asc | awk '/error.c/ {next}; /error.h/ {next}; /error.o/ {next}; {print}'
grep -c warning log_build.asc
grep warning log_build.asc | awk '/imaginary constants are a GCC extension/ {next}; {print}'

Advanced troubleshooting

  • If you removed one of your programs or changed its name, and scons install fails with "Source oldprogname not found, needed by target install", and you cleaned everything there was to clean but still get this message, remove RSFSRC/.sconsign*
  • If during scons install you get a DBAccessError : (13, 'Permission denied') in some reproducible papers, check permissions in your $DATAPATH directory. This is where SCons places database ".sconsign" files for its dependencies (according to the rules in rsf.proj and rsf.tex).
  • If scons or scons install fail due to an a bug introduced in a tool you are certain you will not use, a quick workaround for the problem is already built into scons: the -k option, which means "keep going". Thus, if you use scons -k or scons -k install, SCons will not be able to build the failed component, or anything that depends on it, but it will keep going and make everything else that it can.

Further support

Subscribe to the rsf-user mailing list.

Retrieved from "https://ahay.org/index.php?title=Advanced_Installation&oldid=3757"