|
|
|
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
|
|
|
|
|
|
|
|
|
|
<!--Converted with LaTeX2HTML 2008 (1.71)
|
|
|
|
|
original version by: Nikos Drakos, CBLU, University of Leeds
|
|
|
|
|
* revised and updated by: Marcus Hennecke, Ross Moore, Herb Swan
|
|
|
|
|
* with significant contributions from:
|
|
|
|
|
Jens Lippmann, Marek Rouchal, Martin Wilck and others -->
|
|
|
|
|
<HTML>
|
|
|
|
|
<HEAD>
|
|
|
|
|
<TITLE>General overview</TITLE>
|
|
|
|
|
<META NAME="description" CONTENT="General overview">
|
|
|
|
|
<META NAME="keywords" CONTENT="userhtml">
|
|
|
|
|
<META NAME="resource-type" CONTENT="document">
|
|
|
|
|
<META NAME="distribution" CONTENT="global">
|
|
|
|
|
|
|
|
|
|
<META NAME="Generator" CONTENT="LaTeX2HTML v2008">
|
|
|
|
|
<META HTTP-EQUIV="Content-Style-Type" CONTENT="text/css">
|
|
|
|
|
|
|
|
|
|
<LINK REL="STYLESHEET" HREF="userhtml.css">
|
|
|
|
|
|
|
|
|
|
<LINK REL="next" HREF="node8.html">
|
|
|
|
|
<LINK REL="previous" HREF="node2.html">
|
|
|
|
|
<LINK REL="up" HREF="userhtml.html">
|
|
|
|
|
<LINK REL="next" HREF="node4.html">
|
|
|
|
|
</HEAD>
|
|
|
|
|
|
|
|
|
|
<BODY >
|
|
|
|
|
<!--Navigation Panel-->
|
|
|
|
|
<A NAME="tex2html194"
|
|
|
|
|
HREF="node4.html">
|
|
|
|
|
<IMG WIDTH="37" HEIGHT="24" ALIGN="BOTTOM" BORDER="0" ALT="next" SRC="next.png"></A>
|
|
|
|
|
<A NAME="tex2html190"
|
|
|
|
|
HREF="userhtml.html">
|
|
|
|
|
<IMG WIDTH="26" HEIGHT="24" ALIGN="BOTTOM" BORDER="0" ALT="up" SRC="up.png"></A>
|
|
|
|
|
<A NAME="tex2html184"
|
|
|
|
|
HREF="node2.html">
|
|
|
|
|
<IMG WIDTH="63" HEIGHT="24" ALIGN="BOTTOM" BORDER="0" ALT="previous" SRC="prev.png"></A>
|
|
|
|
|
<A NAME="tex2html192"
|
|
|
|
|
HREF="node1.html">
|
|
|
|
|
<IMG WIDTH="65" HEIGHT="24" ALIGN="BOTTOM" BORDER="0" ALT="contents" SRC="contents.png"></A>
|
|
|
|
|
<BR>
|
|
|
|
|
<B> Next:</B> <A NAME="tex2html195"
|
|
|
|
|
HREF="node4.html">Basic Nomenclature</A>
|
|
|
|
|
<B> Up:</B> <A NAME="tex2html191"
|
|
|
|
|
HREF="userhtml.html">userhtml</A>
|
|
|
|
|
<B> Previous:</B> <A NAME="tex2html185"
|
|
|
|
|
HREF="node2.html">Introduction</A>
|
|
|
|
|
<B> <A NAME="tex2html193"
|
|
|
|
|
HREF="node1.html">Contents</A></B>
|
|
|
|
|
<BR>
|
|
|
|
|
<BR>
|
|
|
|
|
<!--End of Navigation Panel-->
|
|
|
|
|
|
|
|
|
|
<H1><A NAME="SECTION00030000000000000000"></A>
|
|
|
|
|
<A NAME="sec:overview"></A>
|
|
|
|
|
<BR>
|
|
|
|
|
General overview
|
|
|
|
|
</H1>
|
|
|
|
|
The PSBLAS library is designed to handle the implementation of
|
|
|
|
|
iterative solvers for sparse linear systems on distributed memory
|
|
|
|
|
parallel computers. The system coefficient matrix <IMG
|
|
|
|
|
WIDTH="16" HEIGHT="14" ALIGN="BOTTOM" BORDER="0"
|
|
|
|
|
SRC="img1.png"
|
|
|
|
|
ALT="$A$"> must be square;
|
|
|
|
|
it may be real or complex, nonsymmetric, and its sparsity pattern
|
|
|
|
|
needs not to be symmetric. The serial computation parts are based on
|
|
|
|
|
the serial sparse BLAS, so that any extension made to the data
|
|
|
|
|
structures of the serial kernels is available to the parallel
|
|
|
|
|
version. The overall design and parallelization strategy have been
|
|
|
|
|
influenced by the structure of the ScaLAPACK parallel
|
|
|
|
|
library. The layered structure of the PSBLAS library
|
|
|
|
|
is shown in figure <A HREF="#fig:psblas">1</A>; lower layers of the library
|
|
|
|
|
indicate an encapsulation relationship with upper layers. The ongoing
|
|
|
|
|
discussion focuses on the Fortran 2003 layer immediately below the
|
|
|
|
|
application layer.
|
|
|
|
|
The serial parts of the computation on each process are executed through
|
|
|
|
|
calls to the serial sparse BLAS subroutines.
|
|
|
|
|
In a similar way, the inter-process message exchanges are encapsulated
|
|
|
|
|
in an applicaiton layer that has been strongly inspired by the Basic
|
|
|
|
|
Linear Algebra Communication Subroutines (BLACS) library [<A
|
|
|
|
|
HREF="node107.html#BLACS">7</A>].
|
|
|
|
|
Usually there is no need to deal directly with MPI; however, in some
|
|
|
|
|
cases, MPI routines are used directly to improve efficiency. For
|
|
|
|
|
further details on our communication layer see Sec. <A HREF="node72.html#sec:parenv">7</A>.
|
|
|
|
|
|
|
|
|
|
<P>
|
|
|
|
|
|
|
|
|
|
<DIV ALIGN="CENTER"><A NAME="fig:psblas"></A><A NAME="226"></A>
|
|
|
|
|
<TABLE>
|
|
|
|
|
<CAPTION ALIGN="BOTTOM"><STRONG>Figure 1:</STRONG>
|
|
|
|
|
PSBLAS library components hierarchy.</CAPTION>
|
|
|
|
|
<TR><TD>
|
|
|
|
|
<DIV ALIGN="CENTER">
|
|
|
|
|
<IMG
|
|
|
|
|
WIDTH="203" HEIGHT="294" ALIGN="BOTTOM" BORDER="0"
|
|
|
|
|
SRC="img3.png"
|
|
|
|
|
ALT="\includegraphics[scale=0.65]{figures/psblas.eps}">
|
|
|
|
|
|
|
|
|
|
<!-- MATH
|
|
|
|
|
$\rotatebox{-90}{\includegraphics[scale=0.65]{figures/psblas}}$
|
|
|
|
|
-->
|
|
|
|
|
<IMG
|
|
|
|
|
WIDTH="37" HEIGHT="2" ALIGN="BOTTOM" BORDER="0"
|
|
|
|
|
SRC="img2.png"
|
|
|
|
|
ALT="\rotatebox{-90}{\includegraphics[scale=0.65]{figures/psblas}}">
|
|
|
|
|
|
|
|
|
|
</DIV></TD></TR>
|
|
|
|
|
</TABLE>
|
|
|
|
|
</DIV>
|
|
|
|
|
|
|
|
|
|
<P>
|
|
|
|
|
The type of linear system matrices that we address typically arise in the
|
|
|
|
|
numerical solution of PDEs; in such a context,
|
|
|
|
|
it is necessary to pay special attention to the
|
|
|
|
|
structure of the problem from which the application originates.
|
|
|
|
|
The nonzero pattern of a matrix arising from the
|
|
|
|
|
discretization of a PDE is influenced by various factors, such as the
|
|
|
|
|
shape of the domain, the discretization strategy, and
|
|
|
|
|
the equation/unknown ordering. The matrix itself can be interpreted as
|
|
|
|
|
the adjacency matrix of the graph associated with the discretization
|
|
|
|
|
mesh.
|
|
|
|
|
|
|
|
|
|
<P>
|
|
|
|
|
The distribution of the coefficient matrix for the linear system is
|
|
|
|
|
based on the ``owner computes'' rule:
|
|
|
|
|
the variable associated to each mesh point is assigned to a process
|
|
|
|
|
that will own the corresponding row in the coefficient matrix and
|
|
|
|
|
will carry out all related computations. This allocation strategy
|
|
|
|
|
is equivalent to a partition of the discretization mesh into <EM>sub-domains</EM>.
|
|
|
|
|
Our library supports any distribution that keeps together
|
|
|
|
|
the coefficients of each matrix row; there are no other constraints on
|
|
|
|
|
the variable assignment.
|
|
|
|
|
This choice is consistent with simple data distributions
|
|
|
|
|
such as <code>CYCLIC(N)</code> and <code>BLOCK</code>,
|
|
|
|
|
as well as completely arbitrary assignments of
|
|
|
|
|
equation indices to processes.
|
|
|
|
|
In particular it is consistent with the
|
|
|
|
|
usage of graph partitioning tools commonly available in the
|
|
|
|
|
literature, e.g. METIS [<A
|
|
|
|
|
HREF="node107.html#METIS">13</A>].
|
|
|
|
|
Dense vectors conform to sparse
|
|
|
|
|
matrices, that is, the entries of a vector follow the same distribution
|
|
|
|
|
of the matrix rows.
|
|
|
|
|
|
|
|
|
|
<P>
|
|
|
|
|
We assume that the sparse matrix is built in parallel, where each
|
|
|
|
|
process generates its own portion. We never require that the entire
|
|
|
|
|
matrix be available on a single node. However, it is possible
|
|
|
|
|
to hold the entire matrix in one process and distribute it
|
|
|
|
|
explicitly<A NAME="tex2html2"
|
|
|
|
|
HREF="footnode.html#foot167"><SUP>1</SUP></A>, even though the resulting
|
|
|
|
|
bottleneck would make this option unattractive in most cases.
|
|
|
|
|
|
|
|
|
|
<P>
|
|
|
|
|
<BR><HR>
|
|
|
|
|
<!--Table of Child-Links-->
|
|
|
|
|
<A NAME="CHILD_LINKS"><STRONG>Subsections</STRONG></A>
|
|
|
|
|
|
|
|
|
|
<UL>
|
|
|
|
|
<LI><A NAME="tex2html196"
|
|
|
|
|
HREF="node4.html">Basic Nomenclature</A>
|
|
|
|
|
<LI><A NAME="tex2html197"
|
|
|
|
|
HREF="node5.html">Library contents</A>
|
|
|
|
|
<LI><A NAME="tex2html198"
|
|
|
|
|
HREF="node6.html">Application structure</A>
|
|
|
|
|
<LI><A NAME="tex2html199"
|
|
|
|
|
HREF="node7.html">Programming model</A>
|
|
|
|
|
</UL>
|
|
|
|
|
<!--End of Table of Child-Links-->
|
|
|
|
|
<HR>
|
|
|
|
|
<!--Navigation Panel-->
|
|
|
|
|
<A NAME="tex2html194"
|
|
|
|
|
HREF="node4.html">
|
|
|
|
|
<IMG WIDTH="37" HEIGHT="24" ALIGN="BOTTOM" BORDER="0" ALT="next" SRC="next.png"></A>
|
|
|
|
|
<A NAME="tex2html190"
|
|
|
|
|
HREF="userhtml.html">
|
|
|
|
|
<IMG WIDTH="26" HEIGHT="24" ALIGN="BOTTOM" BORDER="0" ALT="up" SRC="up.png"></A>
|
|
|
|
|
<A NAME="tex2html184"
|
|
|
|
|
HREF="node2.html">
|
|
|
|
|
<IMG WIDTH="63" HEIGHT="24" ALIGN="BOTTOM" BORDER="0" ALT="previous" SRC="prev.png"></A>
|
|
|
|
|
<A NAME="tex2html192"
|
|
|
|
|
HREF="node1.html">
|
|
|
|
|
<IMG WIDTH="65" HEIGHT="24" ALIGN="BOTTOM" BORDER="0" ALT="contents" SRC="contents.png"></A>
|
|
|
|
|
<BR>
|
|
|
|
|
<B> Next:</B> <A NAME="tex2html195"
|
|
|
|
|
HREF="node4.html">Basic Nomenclature</A>
|
|
|
|
|
<B> Up:</B> <A NAME="tex2html191"
|
|
|
|
|
HREF="userhtml.html">userhtml</A>
|
|
|
|
|
<B> Previous:</B> <A NAME="tex2html185"
|
|
|
|
|
HREF="node2.html">Introduction</A>
|
|
|
|
|
<B> <A NAME="tex2html193"
|
|
|
|
|
HREF="node1.html">Contents</A></B>
|
|
|
|
|
<!--End of Navigation Panel-->
|
|
|
|
|
|
|
|
|
|
</BODY>
|
|
|
|
|
</HTML>
|
