Advanced Installation: Difference between revisions
Jenningsjwj (talk | contribs) |
Jenningsjwj (talk | contribs) |
||
| Line 449: | Line 449: | ||
However, it might be desirable to keep these components in separate places. For example, if you keep your private programs in RSFSRC/user you will have to remember to make a copy somewhere else if you ever want to delete the Madagascar installation to perform a fresh install. Yup, I deleted all my programs that way once. Good thing I had a back up! Fortunately, it is easy to keep each of these components in a separate place if desired. | However, it might be desirable to keep these components in separate places. For example, if you keep your private programs in RSFSRC/user you will have to remember to make a copy somewhere else if you ever want to delete the Madagascar installation to perform a fresh install. Yup, I deleted all my programs that way once. Good thing I had a back up! Fortunately, it is easy to keep each of these components in a separate place if desired. | ||
==Keeping programs separate== | |||
==Keeping recipes separate== | |||
==Keeping examples separate== | |||
==Keeping data separate== | |||
==Keeping locked figures separate== | |||
=Capturing error and warning messages= | =Capturing error and warning messages= | ||
Revision as of 02:31, 18 June 2010
Before reading this document, please familiarize yourself with the short Installation guide.
What the installation process does
The term "installation" in the title is used for brevity, and it actually covers all three steps: configuration, build and install.
- Configure: determine what tools are available on the system and how they should be used to built the software. Creates a layer of abstraction so that the build is platform-independent. Should ideally either solve or flag all problems, so that the build either works, or does not proceed at all.
- Build: compiles the software and documentation using RSFSRC/build as a "workplace"
- Install: moves the compiled executables and the documentation to the final locations in $RSFROOT, sometimes changing filenames. Kept separate from build so that it can be done by root, and to avoid build failures leaving junk files all over the system.
A successful installation will have created in $RSFROOT the following directories:
- bin/: executable programs
- doc/: auto-generated HTML documentation
- include/: header files with info on library procedures; fonts
- lib/: libraries and Python modules
Prerequisites
Basic prerequisites are described in the short Installation guide. Here are some additional details.
Python and SCons
As described below under Platform-specific installation advice, Madagascar supports the oldest non-deprecated Python version currently supported by the latest stable version of SCons. If your version of Python is older and you experience problems you should probably upgrade.
Madagascar includes the latest stable version of SCons and the configure scripts will try to install it for you in RSFROOT if you don't have it already. However, if you have an older version of SCons the configure scripts will not try to install the newer version. Your older version might work fine, but Madagascar attempts to support only the latest stable version of SCons, so if you have problems you should upgrade.
To install the SCons bundled with Madagascar go to RSFSRC/scons, unpack the tar file, and type
<bash> python setup.py install </bash>
This will install SCons in the standard location. You might need root privileges. If you don't have root privileges, or you don't want to interfere with the system SCons you can install it somewhere else with a --prefix option. A logical choice is to put it in RSFROOT like this:
<bash> python setup.py install --prefix=$RSFROOT </bash>
Location
As long as you set the environment variables and directory permissions correctly, it does not matter in what part of your filesystem you place the install. If you have the luxury of installing anywhere, it is good practice to follow the Filesystem Hierarchy Standard and either:
- Install everything (including figs if you do testing) under /usr/local/rsf, with the source tree in /usr/local/rsf/src, OR
- Put the source tree in /usr/local/src/rsf, and specify RSFROOT=/usr/local, so that header files and binaries go in /usr/local/bin and /usr/local/include. To follow the standard, before installing set RSFDOC=/usr/local/share/rsf/doc and create the appropriate directories. The auto-generated HTML documentation will get put there. Also, if installed, the figs directory for testing should be /usr/local/share/rsf/figs/.
- Package Madagascar (i.e. build a RPM, etc) and install it in the default locations. For RPMs, those are as like the ones from the previous option, just directly in the /usr/ hierarchy, instead of in the /usr/local/ one.
Disk space
At present (Feb 2007, r2530), the source directory containing the build tree from the development version was approx. 200Mb, the full installation (bin/, doc/, include/ and lib) is 31Mb, and figs/ (the optional directory if you want to do testing) is about 10 Gb. The stable version is significantly smaller.
The only Madagascar-related directory where disk space can be an issue is $DATAPATH. Real 3-D seismic datasets can be measured in Terabytes. Buggy programs/processing flows can fill up $DATAPATH. A real problem are "disk memory leaks" -- removing header files with anything else than sfrm will leave the binaries intact. Crashed jobs which start to write to binary but never get to write the header also produce "leaks". Experience has shown that over time $DATAPATH inexorably fills up. You may need to
- keep irreplaceable data and expensive results in a separate place;
- remove the oldest files in $DATAPATH whenever the amount of free space declines under a preset threshold.
Dependencies
Some platforms feature complete lists of dependencies. See Platform-specific installation advice for details.
C++ API
A C++ compiler. SCons is smart and will try to find it for you. If it does not work specify the path to your compiler in the CXX environment variable (can be passed as an option to the configuration script, like the API one).
F77 API
A Fortran 77 compiler. If SCons does not find one, then you can either specify its path through the F77 variable, or if the executable is in your path, add its name to the list of F77 compilers in RSFSRC/configure.py .
F90 API
Same as for Fortran 77 – just substitute F90. If using the gfortran compiler, make sure to get the latest version. If you have more than one compiler installed on your system, specify the desired one at configuration time: <bash> ./configure API=f90 F90=/path/to/preferred/compiler </bash>
Matlab API
Besides Matlab itself, you need Mex, which compiles C code into regular Matlab functions. Use the MATLAB and MEX environment variables to specify their paths if they are installed, but not found.
Octave API
The Octave function compiler (mkoctfile) is sometimes bundled in a separate package, so it may be missing from the Octave installation.
Python API
This API requires SWIG and numpy. Numpy requires Python 2.4 or newer (i.e. RHEL 5 or newer). However, these dependencies are unnecessary for the common case when Python is just used as glue to create chains of programs, and it only needs to read the RSF header, and not the binary. To allow Python metaprograms in madagascar to function, and programming in this style to be done, a fallback development kit implementing only the header-related functionality will be installed in the lack of these dependencies.
Python modules in user space
Python is an evolving language. Many large systems have old versions for stability reasons, and administrators of such large systems tend to not install all software users may wish, and to not allow access to rpm either. To install a module in your user space, download the tarball, unzip it, cd into the directory and run:
python setup.py install --prefix=/path/to/your/place
The installer will create a subdirectory named lib, or lib64 under the directory above. These lib* dirs will have a directory named python, or python2.3 for example, and those will have a subdirectory named site-packages. Add all paths to these site-packages subdirectories in your PYTHONPATH environment variable. Some (numpy) may create a bin directory that needs to be added to PATH.
Environment variables
Besides RSFROOT, DATAPATH and PYTHONPATH (described in the short Installation guide), Madagascar programs may read the variables below. They usually have reasonable defaults and were introduced just to provide more power to the advanced user.
For future documentation writers: the environment variables read by Madagascar that have not been documented below can be found by running the script $RSFSRC/admin/find_env_var.py. If the script does not exist or does not work, a summary of all environment variable calls can be obtained by going to $RSFSRC, temporarily moving the directory build/ outside RSFSRC, and typing <bash> grep environ.get *.py */*.py */*/*.py */*/*/*.py grep getenv */*.c */*/*.c */*/*/*.c </bash>
Used by the Madagascar core
| Variables introduced by Madagascar's non-graphic programs | ||
|---|---|---|
| Name | Default | Meaning |
| RSF_DATASERVER | ftp://egl.beg.utexas.edu/ | Data server for benchmark datasets |
| RSFDOC | $RSFROOT/doc | Directory for the HTML self-doc |
| RSFFIGS | $RSFROOT/figs | Directory with figures for testing examples in $RSFSRC/book |
| RSFALTFIGS | $RSFFIGS | Alternate directory with figures for testing examples not in $RSFSRC/book |
| RSFMEMSIZE | 100 | Maximum RAM (Mb) to be used by some programs |
| RSFSRC | undefined | Root of the Madagascar source tree |
| TMPDATAPATH | $DATAPATH | Datapath for temporary files on local disk. |
| LATEX2HTML | undefined | LateX2HTML customization directory |
| Variables introduced by Madagascar graphics programs | ||
|---|---|---|
| Name | Default | Meaning |
| DEFAULT_PAPER_SIZE | "letter" | For pspen. Other options: legal, a3, a4, a5. |
| FATMULT | ? | Fatness multiplication factor. |
| GIFBORDER | 0.25 | For vplot2gif (spacing) |
| GIFDELAY | 100 | For vplot2gif (for animations) |
| IMAGE_TYPE | 'png' | Icon type for LateX2HTML |
| PATTERNMULT | None | Pattern multiplication factor |
| PLOTSTYLE | None | Used in vplot |
| PPI | 75 | For vplot2gif (screen resolution) |
| PPMSCALE | 1 | For vplot2gif |
| PSBORDER | 0.05 | For vplot2eps (border around the plot) |
| PSPRINTER | postscript or colorps | For pspen |
| PSTEXPENOPTS | color=n fat=1 fatmult=1.5 invras=y | Other vplot2eps options |
| VPLOTFONTDIR | ? | Dir with backup fonts in case the runtime-loaded vplot fonts are not found |
| VPLOTSPOOLDIR | /tmp | Where to put vplot tmp files |
| WSTYPE | "default" | Workstation type. |
| Variables set by OS/other apps, read-only to Madagascar | |
|---|---|
| Name | Primarily used/set by |
| CWPROOT | Seismic Unix |
| DISPLAY | Operating System (OS) |
| HOME | OS |
| LD_LIBRARY_PATH | linker |
| MATLABPATH | Matlab |
| XAUTHORITY | X-Windows |
Used by the Madagascar build process
Type scons -h in RSFSRC to get a list of environment variables that affect the build process, with explanations, defaults and actual values. Below are more detailed explanations for some of them:
- RSF_CLUSTER: used by pscons
Used by the Matlab API
To use the Matlab API, you need to add $RSFROOT/lib to MATLABPATH
Used by the Octave API
To use the Octave API, you need to add $RSFROOT/lib to Octave's path. Determine Octave's version with <bash> octave -v | head -1 </bash> If your version is lower than 2.9.6, type at a Unix command line: <bash> echo 'LOADPATH = "::$RSFROOT/lib/octave"' >> ~/.octaverc </bash> For later versions, use: <bash> echo 'addpath([getenv("RSFROOT") "/lib/octave"])' >> ~/.octaverc </bash>
Heterogeneous networks with user home directories shared through NFS are quite common in many institutions. In addition, even when the architecture is the same (i.e. 64-bit) and the operating system is the same (i.e. RHEL), the difference between operating system versions may be very significant because clusters may run legacy versions, while desktop workstations may run the latest-and-greatest (even beta), and entirely different Madagascar versions may be needed to support both.
One possible solution of detecting the distribution version and architecture and setting RSFROOT appropriately is shown below. In the example network, all RHEL4 machines have the same architecture, but there are RHEL 3 machines with several architectures: <bash> REDHAT_RELEASE=`awk -F'release' '{ print $2 }' /etc/redhat-release | awk -F' ' '{ print $1 }'`
RSFROOT=/usr/local/rsf/rhel$REDHAT_RELEASE
if [ $REDHAT_RELEASE == '4' ] ; then
export RSFROOT
elif [ $REDHAT_RELEASE == '3' ] ; then
export RSFROOT=$RSFROOT/$ARCH
fi </bash> Of course, the Madagascar administrator will have to download appropriate versions of Madagascar to each $RSFROOT, and compile them on the appropriate system.
If you have many kinds of systems to maintain, with multiple versions of Madagascar, and users have more than one shell, you may find it easy to outsource the complex logic to the easy-to-debug Python, i.e.:
<bash> export RSFROOT=`$M8R_SETUP/get_rsfroot.py` export PYTHONPATH=`$M8R_SETUP/edit_pythonpath.py` export PATH=`$M8R_SETUP/edit_path.py` </bash>
and similarly for (t)csh. The Python scripts determine the operating system and its version, determine the machine name, and simply print to stdout the desired string.
Eclipse + Pydev
If you use Eclipse with Pydev, configure the interpreter by adding $RSFROOT/lib to the PYTHONPATH for your chosen interpreter.
Platform-specific installation advice
Supported platforms
Madagascar attempts to support any POSIX-compliant operating system demanded by users. For systems that bundle Python (i.e. Linux distributions, BSDs), backwards compatibility will attempt to cover those systems that were bundled with the oldest non-deprecated Python version currently supported by the latest stable version of SCons. For example, in early 2009 the stable SCons release (1.2) supported Python 2.2 or newer. Python 2.2 was bundled by RHEL3, so RHEL 3 and newer are supported.
Attempts for backward compatibility with a given operating system are also stopped if the operating system itself becomes unsupported. For example, Python 2.2 was bundled by Fedora 1 and newer, but in January 2010 only Fedora 11 and 12 are actively maintained. Thus, in January 2010 Madagascar was not attempting to support Fedora 1, even though it included Python 2.2.
Please keep in mind that the above statements constitute only general guidelines for what will be attempted, and do not constitute in any way a warranty of support. An application of the above guidelines to some Linux distributions follows:
Support info
| Distribution | Life Cycle | Supported versions |
|---|---|---|
| CentOS/RHEL | 7 years |
|
| Fedora | Release X maintained until one month after the release of X+2 |
|
| Ubuntu | Releases every 6 mo, maintained for 1.5 yrs; LTS versions every 2 yrs, maintained for 5 yrs |
|
| Debian | Usually: stable releases every 1.5 yrs, release X maintained 1 yr after release X+1 | 5 (Lenny) until its end of life, or until SCons stops supporting Python 2.5, whatever comes sooner. |
| openSUSE | openSUSE releases Lifetime of 2 years |
|
Ubuntu
In Ubuntu 10.04 Lucid Lynx, you can install all of Madagascar's dependencies by running
sudo apt-get install freeglut3-dev g++ gfortran libgd2-xpm-dev libglew1.5-dev libjpeg62-dev libx11-dev \ libxaw7-dev libnetpbm10-dev swig python-dev python-scipy python-numpy libtiff4-dev scons units libblas-dev \ libcairo2-dev libavcodec-dev libplplot-dev
In the previous version (9.04) the corresponding command is
sudo apt-get install freeglut3-dev g++ gfortran libc6-dev libgd2-xpm-dev libglew1.5-dev libjpeg62-dev \ libx11-dev libxaw7-dev libnetpbm10-dev swig python-dev python-scipy python-numpy libtiff4-dev scons units
Earlier versions may work with
sudo apt-get install mesa-libGL-devel g++ g77 libc6-dev libgd2-xpm-dev libglew-dev libjpeg62-dev \ libx11-dev libxaw7-dev libnetpbm10-dev swig python-dev python-scipy python-numpy libtiff4-dev scons units
If working with the development version, you will also need subversion.
Fedora 11, 12
Dependency package names, sorted by Linux distribution and m8r feature they provide. Packages that are not included in the standard distro repositories are hyperlinked to their providers. The tables below cover build dependencies.
Note: In the future, we should distinguish the subset of runtime dependencies (necessary to run already-compiled binaries installed through a package manager)
Note: In the future, it should be possible for the configuration scripts to output the dependency tables below, so that they are guaranteed to be in synch with a given m8r version
Minimal install ("Core"), publishing and development
| Core | LaTeX | Development version | C++ API | F77 API, F90 API | Python API | Java API | Octave API | |
|---|---|---|---|---|---|---|---|---|
| Fedora 11, 12 | binutils, gcc, glibc-headers, scons | texlive-latex | subversion | gcc-c++ | gcc-gfortran | numpy, python, swig | Java (Sun's? IcedTea?), Mines JTK | octave, octave-devel |
Numerical and file manipulation utilities
| OpenMP | MPI | BLAS/ATLAS | SciPy | pyct (?) | sfunits | |
|---|---|---|---|---|---|---|
| Fedora 11, 12 | libgomp | openmpi, openmpi-devel, openmpi-libs | blas, blas-devel, atlas, atlas-devel | scipy | pyct | units |
Graphics and visualization
| Some sort of movies? | TIFF output | JPEG output | PLplot graphics | OpenGL graphics | X11 graphics | ppm (?) | |
|---|---|---|---|---|---|---|---|
| Fedora 11, 12 | ffmpeg-devel | libtiff-devel | libjpeg-devel | plplot-devel | mesa-libGL-devel, freeglut, freeglut-devel | libXaw-devel | netpbm-devel |
Command to install all dependencies present in the public repositories
Usually package management software will not install again a package that is already installed, so it should be safe to copy and paste the command below to a command line (remember to delete newlines):
| Fedora 11, 12 | yum -y install binutils gcc glibc-headers scons texlive-latex subversion gcc-c++ gcc-gfortran numpy python swig octave octave-devel libgomp openmpi openmpi-devel openmpi-libs blas blas-devel atlas atlas-devel scipy pyct units ffmpeg-devel libtiff-devel libjpeg-devel plplot-devel mesa-libGL-devel freeglut freeglut-devel libXaw-devel netpbm-devel |
|---|
CentOS 3, 4, 5
To install and maintain the crucial SCons dependency using package management, you first need to add RPMforge's RHEL5 repository to your local list.
Dependencies can be installed from the repositories with:
yum -y install binutils freeglut freeglut-devel gcc gcc-c++ gcc-gfortran \ glibc-headers libjpeg-devel libXaw-devel netpbm-devel scons units
A few dependencies ( octave, octave-devel, numpy, scipy and units) are not in the repositories, so they must be searched for and installed manually if you need them. Fortunately, none of the missing packages is crucial enough to stop the core Madagascar installation; they are needed only by optional components.
Debian 5
Specific dependencies:
- Debian 5.0 ("Lenny"): Please make sure you have the libc6-dev package before trying to compile from source. The libXaw7-dev package might be a dependency for xtpen (was in Debian 4.0)
openSUSE 11.*
The development version of m8r has dependencies beyond the packages installed by the default openSUSE 11.0 DVD install:
- To download the development version, you need subversion
- To compile the package, you need scons and gcc
- To install the C++ API, you need gcc-c++
- To install the F90 API, you need gcc-fortran
- To install the Octave API, you need octave
- To install sfbyte2jpg and sfjpg2byte you need libjpeg-devel
- To install the vplot graphics, you need xorg-x11-devel
- To install LaTeX for building papers and books, you need texlive-latex
- To install some graphics programs, you need cairo-devel, gd-devel, glew-devel and libtiff-devel
Packages can be installed by running as root zypper install <package name>. The graphical front-end to Zypper integrated with YaST2 can of course be used as well (Geeko button... Applications... System... Configuration... Install Software)
Yellow Dog Linux 6.1 on Sony PlayStation 3
See W. Burnett's guide (PowerPoint)
Mac OS X
First of all, a Mac OS X precompiled binary package of the latest Madagascar stable release is available for download from SourceForge.
If you want to install a development version of Madagascar, the following might help.
- C compiler for Mac OS X. You can download the precompiled binary package of Xcode tools, including the gcc compiler, from Apple. Other sources are:
- Subversion client for Mac OS X. There are two methods to install Subversion in Mac OS X: you can use MacPorts or Fink to update Subversion client package or the precompiled binary. Some useful information can be found on the Wikihow website and Collab. You can use Subversion to download the development version of the Madagascar source code. Next, follow Installation instructions to install.
- SEGTeX, a LaTeX package for geophysical publications. To use SEGTeX, you may need TeX Live. MacPorts and Fink provide an easy way to install it with commands sudo port install texlive or sudo fink install texlive.
MS Windows
Due to its size, this topic has been assigned its own Wiki page.
How to adapt Madagascar to a new platform
The most laborious part of adapting madagascar to a new platform is finding the proper dependency names. This usually proceeds as follows: dependency X fails with a "missing file" error either as a header file in config.log, or a missing library during the build step. Possible package names are found through an internet search for the missing file name and the distribution name or by using specific rpm search tools. Packages are installed and the configure (and, if necessary) build processes are repeated until the error goes away.
Multi-user installs
Some organizations may find it desirable to deny write access of some users to all RSFSRC/RSFROOT except their own user directory. Fortunately, this can be easily done by placing the restricted user dirs outside RSFSRC/RSFROOT, i.e. in their home dirs, say /home/joe/rsfsrc. In order to move a user's directory out of RSFSRC, you must:
- "tell" the SConstruct in the user's dir where to find RSFSRC so that when the user builds in his directory, it can import configure.py and config.py You do that by setting the environment variable RSFSRC to the absolute path of the Madagascar source root, and by making sure that lines 2 and 3 in the users' SConstruct files are
<python> srcroot = os.environ.get('RSFSRC', '../..') sys.path.append(srcroot) </python> and then replace ../.. throughout the SConstruct using os.path.join and the srcroot variable.
- "tell" the build scripts about the user's dir, so that it is included in the builds launched from RSFSRC. You do that with a symbolic link:
<bash> ln -s /home/joe/rsfsrc $RSFSRC/user/joe </bash> When the link exists, those of Joe's programs that are mentioned in the "prog" string in SConstruct get included in the distribution, complete with self-doc. If Joe is just learning how to code and his stuff breaks the build, just remove the symbolic link. Even if build+installs are done after the link is removed, his stable programs and self-doc will continue to remain installed system-wide as long as the admin does not type scons -c install (not likely).
- point the user's RSFDOC environment variable to a location where the user has write access
- edit the users' SConstruct so that it uses the RSF library and headers already installed in $RSFROOT/lib and $RSFROOT/include , instead of building again the whole librsf with user-specific flags in RSFSRC/filt/lib/. To do that, replace in the user's SConstruct the env.Prepend statement with
<python> rsfroot = os.environ.get('RSFROOT','/usr/local/rsf')
env.Prepend(CPPPATH=[os.path.join(rsfroot,'include')],
LIBPATH=[os.path.join(rsfroot,'lib')],
LIBS=['rsf'])
</python>
- If the link from RSFSRC to Joe's directory was not made, add Joe's directory to his own path so that he can execute his own binaries.
To understand how $DATAPATH disk space issues may become an issue in a multi-user environment, refer to the Disk Space subsection at the beginning of this document.
Keeping your stuff separate
A user may add his own programs and recipes to the Madagascar system. He may also create his own computational examples, data, and locked figures for testing. All of these components can be placed in their default locations, but it is not necessary to make them public. To keep these items private simply do not add them to the repository.
However, it might be desirable to keep these components in separate places. For example, if you keep your private programs in RSFSRC/user you will have to remember to make a copy somewhere else if you ever want to delete the Madagascar installation to perform a fresh install. Yup, I deleted all my programs that way once. Good thing I had a back up! Fortunately, it is easy to keep each of these components in a separate place if desired.
Keeping programs separate
Keeping recipes separate
Keeping examples separate
Keeping data separate
Keeping locked figures separate
Capturing error and warning messages
The messages during configuration are few and their importance quite high, so they should be watched "in person". A complete log of the configuration process is recorded in RSFSRC/configure.log
Console messages generated during the build step can be captured to a log file and observed at the same time with a command like this (tcsh): <bash> nice +10 nohup /usr/bin/time -p scons -k |& tee log_build.asc </bash> The log file can be of course named otherwise than log_build.asc. The file can be later grepped for error and warnings with commands such as: <bash> grep -c error log_build.asc grep error log_build.asc | awk '/error.c/ {next}; /error.h/ {next}; /error.o/ {next}; {print}' grep -c warning log_build.asc grep warning log_build.asc | awk '/imaginary constants are a GCC extension/ {next}; {print}' </bash>
Advanced troubleshooting
- If you removed one of your programs or changed its name, and scons install fails with "Source oldprogname not found, needed by target install", and you cleaned everything there was to clean but still get this message, remove RSFSRC/.sconsign*
- If during scons install you get a DBAccessError : (13, 'Permission denied') in some reproducible papers, check permissions in your $DATAPATH directory. This is where SCons places database ".sconsign" files for its dependencies (according to the rules in rsfproj and rsftex).
- If scons or scons install fail due to an a bug introduced in a tool you are certain you will not use, a quick workaround for the problem is already built into scons: the -k option, which means "keep going". Thus, if you use scons -k or scons -k install, SCons will not be able to build the failed component, or anything that depends on it, but it will keep going and make everything else that it can.
Further support
Subscribe to the rsf-user mailing list.