In order to build AMG4PSBLAS it is necessary to set up a Makefile with appropriate system-dependent variables; this is done by means of the configure script. The distribution also includes the autoconf and automake sources employed to generate the script, but usually this is not needed to build the software.
AMG4PSBLAS is implemented almost entirely in Fortran 2003, with some interfaces to external libraries in C; the Fortran compiler must support the Fortran 2003 standard plus the extension MOLD= feature, which enhances the usability of ALLOCATE. Most Fortran compilers provide this feature; in particular, this is supported by the GNU Fortran compiler, for which we recommend to use at least version 4.8. The software defines data types and interfaces for real and complex data, in both single and double precision.
Building AMG4PSBLAS requires some base libraries (see Section 3.1); interfaces to optional third-party libraries, which extend the functionalities of AMG4PSBLAS (see Section 3.2), are also available. A number of Linux distributions (e.g., Ubuntu, Fedora, CentOS) provide precompiled packages for the prerequisite and optional software. In many cases these packages are split between a runtime part and a “developer” part; in order to build AMG4PSBLAS you need both. A description of the base and optional software used by AMG4PSBLAS is given in the next sections.
The following base libraries are needed:
BLAS
[18, 19, 26] Many vendors provide optimized versions of BLAS; if no vendor version is available for a given platform, the ATLAS software (math-atlas.sourceforge .net) may be employed. The reference BLAS from Netlib (www.netlib.org/blas) are meant to define the standard behaviour of the BLAS interface, so they are not optimized for any particular platform, and should only be used as a last resort. Note that BLAS computations form a relatively small part of the AMG4PSBLAS/PSBLAS; however they are critical when using preconditioners based on the MUMPS, UMFPACK or SuperLU third party libraries. UMFPACK requires a full LAPACK library; our experience is that configuring ATLAS for building full LAPACK does not always work in the expected way. Our advice is first to download the LAPACK tarfile from www.netlib.org/lapack and install it independently of ATLAS. In this case, you need to modify the OPTS and NOOPT definitions for including -fPIC compilation option in the make.inc file of the LAPACK library.
MPI
[25, 32] A version of MPI is available on most high-performance computing systems.
PSBLAS
[21, 23] Parallel Sparse BLAS (PSBLAS) is available from psctoolkit.github.io/ products/psblas/; version 3.7.0 (or later) is required. Indeed, all the prerequisites listed so far are also prerequisites of PSBLAS.
Please note that the four previous libraries must have Fortran interfaces compatible with AMG4PSBLAS; usually this means that they should all be built with the same compiler being used for AMG4PSBLAS.
If you want to use the PSBLAS support for NVIDIA GPUs, you will also need a working version of the CUDA Toolkit that is compatible with the compiler choice made to compile PSBLAS and AMG4PSBLAS. After that you will need to have configured and compiled the PSBLAS library with the options:
./configure --enable-cuda --with-cudadir=${CUDA_HOME} --with-cudacc=xx,yy,zz
Previous versions required you to have the auxiliary libraries SPGPU and PSBLAS-EXT compiled, this is no longer necessary because they have been integrated into PSBLAS and are compiled by activating the previous flags during configuration. See also Sec 4.2.
We provide interfaces to the following third-party software libraries; note that these are optional, but if you enable them some defaults for multilevel preconditioners may change to reflect their presence.
UMFPACK
[16] A sparse LU factorization package included in the SuiteSparse library, available from faculty.cse.tamu.edu/davis/suitesparse.html; it provides sequential factorization and triangular system solution for double precision real and complex data. We tested version 4.5.4 of SuiteSparse. Note that for configuring SuiteSparse you should provide the right path to the BLAS and LAPACK libraries in the SuiteSparse_config/SuiteSparse_config.mk file.
MUMPS
[2] A sparse LU factorization package available from mumps.enseeiht.fr; it provides sequential and parallel factorizations and triangular system solution for single and double precision, real and complex data. We tested versions 4.10.0 and 5.0.1.
SuperLU
[17] A sparse LU factorization package available from crd.lbl.gov/~xiaoye/SuperLU/; it provides sequential factorization and triangular system solution for single and double precision, real and complex data. We tested versions 4.3 and 5.0. If you installed BLAS from ATLAS, remember to define the BLASLIB variable in the make.inc file.
SuperLU_Dist
[28] A sparse LU factorization package available from the same site as SuperLU; it provides parallel factorization and triangular system solution for double precision real and complex data. We tested versions 3.3 and 4.2. If you installed BLAS from ATLAS, remember to define the BLASLIB variable in the make.inc file and to add the -std=c99 option to the C compiler options. Note that this library requires the ParMETIS library for parallel graph partitioning and fill-reducing matrix ordering, available from glaros.dtc.umn.edu/gkhome/metis/parmetis/overview.
In order to build AMG4PSBLAS, the first step is to use the configure script in the main directory to generate the necessary makefile.
As a minimal example consider the following:
./configure --with-psblas=PSB-INSTALL-DIR
which assumes that the various MPI compilers and support libraries are available in the standard directories on the system, and specifies only the PSBLAS install directory (note that the latter directory must be specified with an absolute path). The full set of options may be looked at by issuing the command ./configure --help, which produces:
‘configure’ configures AMG4PSBLAS 1.0.0 to adapt to many kinds of systems. Usage: ./configure [OPTION]... [VAR=VALUE]... To assign environment variables (e.g., CC, CFLAGS...), specify them as VAR=VALUE. See below for descriptions of some of the useful variables. Defaults for the options are specified in brackets. Configuration: -h, --help display this help and exit --help=short display options specific to this package --help=recursive display the short help of all the included packages -V, --version display version information and exit -q, --quiet, --silent do not print ‘checking ...’ messages --cache-file=FILE cache test results in FILE [disabled] -C, --config-cache alias for ‘--cache-file=config.cache’ -n, --no-create do not create output files --srcdir=DIR find the sources in DIR [configure dir or ‘..’] Installation directories: --prefix=PREFIX install architecture-independent files in PREFIX [/usr/local] --exec-prefix=EPREFIX install architecture-dependent files in EPREFIX [PREFIX] By default, ‘make install’ will install all the files in ‘/usr/local/bin’, ‘/usr/local/lib’ etc. You can specify an installation prefix other than ‘/usr/local’ using ‘--prefix’, for instance ‘--prefix=$HOME’. For better control, use the options below. Fine tuning of the installation directories: --bindir=DIR user executables [EPREFIX/bin] --sbindir=DIR system admin executables [EPREFIX/sbin] --libexecdir=DIR program executables [EPREFIX/libexec] --sysconfdir=DIR read-only single-machine data [PREFIX/etc] --sharedstatedir=DIR modifiable architecture-independent data [PREFIX/com] --localstatedir=DIR modifiable single-machine data [PREFIX/var] --libdir=DIR object code libraries [EPREFIX/lib] --includedir=DIR C header files [PREFIX/include] --oldincludedir=DIR C header files for non-gcc [/usr/include] --datarootdir=DIR read-only arch.-independent data root [PREFIX/share] --datadir=DIR read-only architecture-independent data [DATAROOTDIR] --infodir=DIR info documentation [DATAROOTDIR/info] --localedir=DIR locale-dependent data [DATAROOTDIR/locale] --mandir=DIR man documentation [DATAROOTDIR/man] --docdir=DIR documentation root [DATAROOTDIR/doc/amg4psblas] --htmldir=DIR html documentation [DOCDIR] --dvidir=DIR dvi documentation [DOCDIR] --pdfdir=DIR pdf documentation [DOCDIR] --psdir=DIR ps documentation [DOCDIR] Program names: --program-prefix=PREFIX prepend PREFIX to installed program names --program-suffix=SUFFIX append SUFFIX to installed program names --program-transform-name=PROGRAM run sed PROGRAM on installed program names Optional Features: --disable-option-checking ignore unrecognized --enable/--with options --disable-FEATURE do not include FEATURE (same as --enable-FEATURE=no) --enable-FEATURE[=ARG] include FEATURE [ARG=yes] --enable-silent-rules less verbose build output (undo: "make V=1") --disable-silent-rules verbose build output (undo: "make V=0") --enable-dependency-tracking do not reject slow dependency extractors --disable-dependency-tracking speeds up one-time build --enable-serial Specify whether to enable a fake mpi library to run in serial mode. Optional Packages: --with-PACKAGE[=ARG] use PACKAGE [ARG=yes] --without-PACKAGE do not use PACKAGE (same as --with-PACKAGE=no) --with-psblas=DIR The install directory for PSBLAS, for example, --with-psblas=/opt/packages/psblas-3.5 --with-psblas-incdir=DIR Specify the directory for PSBLAS C includes. --with-psblas-moddir=DIR Specify the directory for PSBLAS Fortran modules. --with-psblas-libdir=DIR Specify the directory for PSBLAS library. --with-ccopt additional [CCOPT] flags to be added: will prepend to [CCOPT] --with-fcopt additional [FCOPT] flags to be added: will prepend to [FCOPT] --with-libs List additional link flags here. For example, --with-libs=-lspecial_system_lib or --with-libs=-L/path/to/libs --with-clibs additional [CLIBS] flags to be added: will prepend to [CLIBS] --with-flibs additional [FLIBS] flags to be added: will prepend to [FLIBS] --with-library-path additional [LIBRARYPATH] flags to be added: will prepend to [LIBRARYPATH] --with-include-path additional [INCLUDEPATH] flags to be added: will prepend to [INCLUDEPATH] --with-module-path additional [MODULE_PATH] flags to be added: will prepend to [MODULE_PATH] --with-extra-libs List additional link flags here. For example, --with-extra-libs=-lspecial_system_lib or --with-extra-libs=-L/path/to/libs --with-blas=<lib> use BLAS library <lib> --with-blasdir=<dir> search for BLAS library in <dir> --with-lapack=<lib> use LAPACK library <lib> --with-mumps=LIBNAME Specify the libname for MUMPS. Default: autodetect with minimum "-lmumps_common -lpord" --with-mumpsdir=DIR Specify the directory for MUMPS library and includes. Note: you will need to add auxiliary libraries with --extra-libs; this depends on how MUMPS was configured and installed, at a minimum you will need SCALAPACK and BLAS --with-mumpsincdir=DIR Specify the directory for MUMPS includes. --with-mumpsmoddir=DIR Specify the directory for MUMPS Fortran modules. --with-mumpslibdir=DIR Specify the directory for MUMPS library. --with-umfpack=LIBNAME Specify the library name for UMFPACK and its support libraries. Default: "-lumfpack -lamd" --with-umfpackdir=DIR Specify the directory for UMFPACK library and includes. --with-umfpackincdir=DIR Specify the directory for UMFPACK includes. --with-umfpacklibdir=DIR Specify the directory for UMFPACK library. --with-superlu=LIBNAME Specify the library name for SUPERLU library. Default: "-lsuperlu" --with-superludir=DIR Specify the directory for SUPERLU library and includes. --with-superluincdir=DIR Specify the directory for SUPERLU includes. --with-superlulibdir=DIR Specify the directory for SUPERLU library. --with-superludist=LIBNAME Specify the libname for SUPERLUDIST library. Requires you also specify SuperLU. Default: "-lsuperlu_dist" --with-superludistdir=DIR Specify the directory for SUPERLUDIST library and includes. --with-superludistincdir=DIR Specify the directory for SUPERLUDIST includes. --with-superludistlibdir=DIR Specify the directory for SUPERLUDIST library. Some influential environment variables: FC Fortran compiler command FCFLAGS Fortran compiler flags LDFLAGS linker flags, e.g. -L<lib dir> if you have libraries in a nonstandard directory <lib dir> LIBS libraries to pass to the linker, e.g. -l<library> CC C compiler command CFLAGS C compiler flags CPPFLAGS (Objective) C/C++ preprocessor flags, e.g. -I<include dir> if you have headers in a nonstandard directory <include dir> MPICC MPI C compiler command MPIFC MPI Fortran compiler command CPP C preprocessor Use these variables to override the choices made by ‘configure’ or to help it to find libraries and programs with nonstandard names/locations. Report bugs to <https://github.com/psctoolkit/psctoolkit/issues>.
For instance, if a user has built and installed PSBLAS 3.7 under the /opt directory and is using the SuiteSparse package (which includes UMFPACK), then AMG4PSBLAS might be configured with:
./configure --with-psblas=/opt/psblas-3.7/ \ --with-umfpackincdir=/usr/include/suitesparse/
Once the configure script has completed execution, it will have generated the file Make.inc which will then be used by all Makefiles in the directory tree; this file will be copied in the install directory under the name Make.inc.AMG4PSBLAS.
To use the MUMPS solver package, the user has to add the appropriate options to the configure script; by default we are looking for the libraries -ldmumps -lsmumps -lzmumps -lcmumps -mumps_common -lpord. MUMPS often uses additional packages such as ScaLAPACK, ParMETIS, SCOTCH, as well as enabling OpenMP; in such cases it is necessary to add linker options with the --with-extra-libs configure option.
To build the library the user will now enter
make
followed (optionally) by
make install
If you find any bugs in our codes, please report them through our issues page
on
https://github.com/psctoolkit/psctoolkit/issues
To enable us to track the bug, please provide a log from the failing application, the test conditions, and ideally a self-contained test program reproducing the issue.
The package contains a samples directory, divided in two subdirs simple and advanced; both of them are further divided into fileread and pdegen subdirectories. Their purpose is as follows:
simple
contains a set of simple example programs with a predefined choice of preconditioners, selectable via integer values. These are intended to get acquainted with the multilevel preconditioners available in AMG4PSBLAS.
advanced
contains a set of more sophisticated examples that will allow the user, via the input files in the runs subdirectories, to experiment with the full range of preconditioners implemented in the package.
The fileread directories contain sample programs that read sparse matrices from files, according to the Matrix Market or the Harwell-Boeing storage format; the pdegen programs generate matrices in full parallel mode from the discretization of a sample partial differential equation.