@ -10,7 +10,7 @@
< link rel = "stylesheet" type = "text/css" href = "userhtml.css" >
< / head > < body
>
<!-- l. 1 29 --> < div class = "crosslinks" > < p class = "noindent" > [< a
<!-- l. 1 31 --> < div class = "crosslinks" > < p class = "noindent" > [< a
href="userhtmlse6.html" >next< / a > ] [< a
href="userhtmlse1.html" >prev< / a > ] [< a
href="userhtmlse1.html#tailuserhtmlse1.html" >prev-tail< / a > ] [< a
@ -18,7 +18,7 @@ href="#tailuserhtmlse2.html">tail</a>] [<a
href="userhtml.html#userhtmlse2.html" >up< / a > ] < / p > < / div >
< h3 class = "sectionHead" > < span class = "titlemark" > 2 < / span > < a
id="x5-40002">< / a > General overview< / h3 >
<!-- l. 13 1 --> < p class = "noindent" > The PSBLAS library is designed to handle the implementation of iterative solvers for
<!-- l. 13 3 --> < p class = "noindent" > The PSBLAS library is designed to handle the implementation of iterative solvers for
sparse linear systems on distributed memory parallel computers. The system
coefficient matrix < span
class="zplmr7m-">A < / span > must be square; it may be real or complex, nonsymmetric, and
@ -33,14 +33,14 @@ The ongoing discussion focuses on the Fortran 2008 layer immediately
below the application layer. The serial parts of the computation on each
process are executed through calls to the serial sparse BLAS subroutines. In a
similar way, the inter-process message exchanges are encapsulated in an
applicai ton layer that has been strongly inspired by the Basic Linear Algebra
applicati on layer that has been strongly inspired by the Basic Linear Algebra
Communication Subroutines (BLACS) library  < span class = "cite" > [< a
href="userhtmlli3.html#XBLACS">7< / a > ]< / span > . Usually there is no need to deal
directly with MPI; however, in some cases, MPI routines are used directly
to improve efficiency. For further details on our communication layer see
Sec.  < a
href="userhtmlse7.html#x13-1060007">7<!-- tex4ht:ref: sec:parenv --> < / a > .
<!-- l. 1 58 --> < p class = "indent" > < hr class = "figure" > < div class = "figure"
<!-- l. 1 60 --> < p class = "indent" > < hr class = "figure" > < div class = "figure"
>
@ -52,8 +52,8 @@ href="userhtmlse7.html#x13-1060007">7<!--tex4ht:ref: sec:parenv --></a>.
< div class = "center"
>
<!-- l. 1 59 --> < p class = "noindent" >
<!-- l. 16 1 --> < p class = "noindent" > < img
<!-- l. 1 61 --> < p class = "noindent" >
<!-- l. 16 3 --> < p class = "noindent" > < img
src="psblas.png" alt="PIC"
width="46" height="46" >< / div >
< br / > < div class = "caption"
@ -62,8 +62,8 @@ class="content">PSBLAS library components hierarchy.</span></div><!--tex4ht:labe
<!-- l. 16 7 --> < p class = "indent" > < / div > < hr class = "endfigure" >
<!-- l. 17 0 --> < p class = "indent" > The type of linear system matrices that we address typically arise in
<!-- l. 16 9 --> < p class = "indent" > < / div > < hr class = "endfigure" >
<!-- l. 17 2 --> < p class = "indent" > The type of linear system matrices that we address typically arise in
the numerical solution of PDEs; in such a context, it is necessary to pay
special attention to the structure of the problem from which the application
originates. The nonzero pattern of a matrix arising from the discretization of a
@ -71,12 +71,12 @@ PDE is influenced by various factors, such as the shape of the domain, the
discretization strategy, and the equation/unknown ordering. The matrix itself can be
interpreted as the adjacency matrix of the graph associated with the discretization
mesh.
<!-- l. 18 1 --> < p class = "indent" > The distribution of the coefficient matrix for the linear system is based on the
<!-- l. 18 3 --> < p class = "indent" > The distribution of the coefficient matrix for the linear system is based on the
“ owner computes” rule: the variable associated to each mesh point is assigned to a
process that will own the corresponding row in the coefficient matrix and will
carry out all related computations. This allocation strategy is equivalent to a
partition of the discretization mesh into < span
class="pplri7t-">sub-domains< / span > . O ur library supports any
class="pplri7t-">sub-domains< / span > ; o ur library supports any
distribution that keeps together the coefficients of each matrix row; there are no
other constraints on the variable assignment. This choice is consistent with
simple data distributions such as < span class = "obeylines-h" > < span class = "verb" > < span
@ -88,9 +88,10 @@ the literature, e.g. METIS <span class="cite">[<a
href="userhtmlli3.html#XMETIS">14< / a > ]< / span > . Dense vectors conform to sparse matrices,
that is, the entries of a vector follow the same distribution of the matrix
rows.
<!-- l. 203 --> < p class = "indent" > We assume that the sparse matrix is built in parallel, where each process generates
its own portion. We never require that the entire matrix be available on a single
node. However, it is possible to hold the entire matrix in one process and distribute it
<!-- l. 204 --> < p class = "indent" > We assume that the sparse matrix is built in parallel, where each process generates
its own portion: we never < span
class="pplri7t-">require < / span > that the entire matrix be available on a single node.
However, it is possible to hold the entire matrix in one process and distribute it
explicitly< span class = "footnote-mark" > < a
href="userhtml6.html#fn1x0">< sup class = "textsuperscript" > 1< / sup > < / a > < / span > < a
id="x5-4002f1">< / a > ,
@ -98,10 +99,10 @@ even though the resulting memory bottleneck would make this option unattractive
in most cases.
< h4 class = "subsectionHead" > < span class = "titlemark" > 2.1 < / span > < a
id="x5-50002.1">< / a > Basic Nomenclature< / h4 >
<!-- l. 21 5 --> < p class = "noindent" > Our computational model implies that the data allocation on the parallel distributed
<!-- l. 21 6 --> < p class = "noindent" > Our computational model implies that the data allocation on the parallel distributed
memory machine is guided by the structure of the physical model, and specifically
by the discretization mesh of the PDE.
<!-- l. 22 0 --> < p class = "indent" > Each point of the discretization mesh will have (at least) one associated
<!-- l. 22 1 --> < p class = "indent" > Each point of the discretization mesh will have (at least) one associated
equation/variable, and therefore one index. We say that point < span
class="zplmr7m-">i < / span > < span
class="pplri7t-">depends < / span > on point < span
@ -117,56 +118,57 @@ class="pplri7t-">sub-domains </span>assigned
to the parallel processes, we classify the points of a given sub-domain as
following.
< dl class = "description" > < dt class = "description" >
<!-- l. 2 29 --> < p class = "noindent" >
<!-- l. 2 30 --> < p class = "noindent" >
< span
class="pplb7t-">Internal.< / span > < / dt > < dd
class="description">
<!-- l. 2 29 --> < p class = "noindent" > An internal point of a given domain < span
class="pplri7t-">depends < / span > only on points of the same
domain. If all points of a domain are assigned to one process, then
a computational step (e.g., a matrix-vector product) of the equations
<!-- l. 2 30 --> < p class = "noindent" > An internal point of a given sub- domain < span
class="pplri7t-">depends < / span > only on points of the
same sub- domain. If all points of a sub- domain are assigned to one
process, then a computational step (e.g., a matrix-vector product) of the
associated with the internal points requires no data items from other
domains and no communications.
equations associated with the internal points requires no data items from
other sub- domains and no communications.
< / dd > < dt class = "description" >
<!-- l. 238 --> < p class = "noindent" >
< span
class="pplb7t-">Boundary.< / span > < / dt > < dd
class="description">
<!-- l. 238 --> < p class = "noindent" > A point of a given domain is a boundary point if it < span
class="pplri7t-">depends < / span > on points
belonging to other domains.
<!-- l. 238 --> < p class = "noindent" > A point of a given sub-domain is a boundary point if it < span
class="pplri7t-">depends < / span > on points
belonging to other sub- domains.
< / dd > < dt class = "description" >
<!-- l. 24 2 --> < p class = "noindent" >
<!-- l. 24 1 --> < p class = "noindent" >
< span
class="pplb7t-">Halo.< / span > < / dt > < dd
class="description">
<!-- l. 242 --> < p class = "noindent" > A halo point for a given domain is a point belonging to another domain
such that there is a boundary point which < span
class="pplri7t-">depends < / span > on it. Whenever performing
a computational step, such as a matrix-vector product, the values associated
with halo points are requested from other domains. A boundary point of
a given domain is usually a halo point for some other domain< span class = "footnote-mark" > < a
<!-- l. 241 --> < p class = "noindent" > A halo point for a given sub-domain is a point belonging to another
sub-domain such that there is a boundary point which < span
class="pplri7t-">depends < / span > on it.
Whenever performing a computational step, such as a matrix-vector product,
the values associated with halo points are requested from other sub-domains.
A boundary point of a given sub-domain is usually a halo point for some
other sub-domain< span class = "footnote-mark" > < a
href="userhtml7.html#fn2x0">< sup class = "textsuperscript" > 2< / sup > < / a > < / span > < a
id="x5-5001f2">< / a > ;
therefore the cardinality of the boundary points set determines the amount
of data sent to other domains.
of data sent to other sub- domains.
< / dd > < dt class = "description" >
<!-- l. 25 5 --> < p class = "noindent" >
<!-- l. 25 4 --> < p class = "noindent" >
< span
class="pplb7t-">Overlap.< / span > < / dt > < dd
class="description">
<!-- l. 25 5 --> < p class = "noindent" > An overlap point is a boundary point assigned to multiple domains. Any
operation that involves an overlap point has to be replicated for each
<!-- l. 25 4 --> < p class = "noindent" > An overlap point is a boundary point assigned to multiple sub- domains.
Any operation that involves an overlap point has to be replicated for each
assignment.< / dd > < / dl >
<!-- l. 25 9 --> < p class = "noindent" > Overlap points do not usually exist in the basic data distributions; however they are a
<!-- l. 25 8 --> < p class = "noindent" > Overlap points do not usually exist in the basic data distributions; however they are a
feature of Domain Decomposition Schwarz preconditioners which are the subject of
related research work  < span class = "cite" > [< a
href="userhtmlli3.html#X2007c">4< / a > ,  < a
href="userhtmlli3.html#X2007d">3< / a > ]< / span > .
<!-- l. 26 4 --> < p class = "indent" > We denote the sets of internal, boundary and halo points for a given subdomain
<!-- l. 26 3 --> < p class = "indent" > We denote the sets of internal, boundary and halo points for a given subdomain
by < span
class="zplmr7y-">< img
src="zplmr7y-49.png" alt="I" class="x-x-49" />< / span > , < span
@ -203,7 +205,7 @@ class="zplmr7y-">|<img
src="zplmr7y-48.png" alt="H" class="x-x-48" />< / span > < sub > < span
class="zplmr7m-x-x-76">i< / span > < / sub > < span
class="zplmr7y-">|< / span > .
<!-- l. 27 4 --> < p class = "indent" > < hr class = "figure" > < div class = "figure"
<!-- l. 27 3 --> < p class = "indent" > < hr class = "figure" > < div class = "figure"
>
@ -215,8 +217,8 @@ class="zplmr7y-">|</span>.
< div class = "center"
>
<!-- l. 27 5 --> < p class = "noindent" >
<!-- l. 27 8 --> < p class = "noindent" > < img
<!-- l. 27 4 --> < p class = "noindent" >
<!-- l. 27 7 --> < p class = "noindent" > < img
src="points.png" alt="PIC"
width="46" height="46" >< / div >
< br / > < div class = "caption"
@ -225,113 +227,113 @@ class="content">Point classfication.</span></div><!--tex4ht:label?: x5-5003r2 --
<!-- l. 28 4 --> < p class = "indent" > < / div > < hr class = "endfigure" >
<!-- l. 28 6 --> < p class = "indent" > This classification of mesh points guides the naming scheme that we adopted in
<!-- l. 28 3 --> < p class = "indent" > < / div > < hr class = "endfigure" >
<!-- l. 28 5 --> < p class = "indent" > This classification of mesh points guides the naming scheme that we adopted in
the library internals and in the data structures. We explicitly note that “ Halo” points
are also often called “ ghost” points in the literature.
< h4 class = "subsectionHead" > < span class = "titlemark" > 2.2 < / span > < a
id="x5-60002.2">< / a > Library contents< / h4 >
<!-- l. 29 5 --> < p class = "noindent" > The PSBLAS library consists of various classes of subroutines:
<!-- l. 29 4 --> < p class = "noindent" > The PSBLAS library consists of various classes of subroutines:
< dl class = "description" > < dt class = "description" >
<!-- l. 29 7 --> < p class = "noindent" >
<!-- l. 29 6 --> < p class = "noindent" >
< span
class="pplb7t-">Computational routines< / span > < / dt > < dd
class="description">
<!-- l. 29 7 --> < p class = "noindent" > comprising:
<!-- l. 29 6 --> < p class = "noindent" > comprising:
< ul class = "itemize1" >
< li class = "itemize" >
<!-- l. 29 9 --> < p class = "noindent" > Sparse matrix by dense matrix product;
<!-- l. 29 8 --> < p class = "noindent" > Sparse matrix by dense matrix product;
< / li >
< li class = "itemize" >
<!-- l. 300 --> < p class = "noindent" > Sparse triangular systems solution for block diagonal matrices;
<!-- l. 299 --> < p class = "noindent" > Sparse triangular systems solution for block diagonal matrices;
< / li >
< li class = "itemize" >
<!-- l. 30 2 --> < p class = "noindent" > Vector and matrix norms;
<!-- l. 30 1 --> < p class = "noindent" > Vector and matrix norms;
< / li >
< li class = "itemize" >
<!-- l. 30 3 --> < p class = "noindent" > Dense matrix sums;
<!-- l. 30 2 --> < p class = "noindent" > Dense matrix sums;
< / li >
< li class = "itemize" >
<!-- l. 30 4 --> < p class = "noindent" > Dot products.< / li > < / ul >
<!-- l. 30 3 --> < p class = "noindent" > Dot products.< / li > < / ul >
< / dd > < dt class = "description" >
<!-- l. 30 6 --> < p class = "noindent" >
<!-- l. 30 5 --> < p class = "noindent" >
< span
class="pplb7t-">Communication routines< / span > < / dt > < dd
class="description">
<!-- l. 30 6 --> < p class = "noindent" > handling halo and overlap communications;
<!-- l. 30 5 --> < p class = "noindent" > handling halo and overlap communications;
< / dd > < dt class = "description" >
<!-- l. 30 8 --> < p class = "noindent" >
<!-- l. 30 7 --> < p class = "noindent" >
< span
class="pplb7t-">Data management and auxiliary routines< / span > < / dt > < dd
class="description">
<!-- l. 30 8 --> < p class = "noindent" > including:
<!-- l. 30 7 --> < p class = "noindent" > including:
< ul class = "itemize1" >
< li class = "itemize" >
<!-- l. 3 1 0--> < p class = "noindent" > Parallel environment management
<!-- l. 3 09 --> < p class = "noindent" > Parallel environment management
< / li >
< li class = "itemize" >
<!-- l. 31 1 --> < p class = "noindent" > Communication descriptors allocation;
<!-- l. 31 0 --> < p class = "noindent" > Communication descriptors allocation;
< / li >
< li class = "itemize" >
<!-- l. 31 2 --> < p class = "noindent" > Dense and sparse matrix allocation;
<!-- l. 31 1 --> < p class = "noindent" > Dense and sparse matrix allocation;
< / li >
< li class = "itemize" >
<!-- l. 31 3 --> < p class = "noindent" > Dense and sparse matrix build and update;
<!-- l. 31 2 --> < p class = "noindent" > Dense and sparse matrix build and update;
< / li >
< li class = "itemize" >
<!-- l. 31 4 --> < p class = "noindent" > Sparse matrix and data distribution preprocessing.< / li > < / ul >
<!-- l. 31 3 --> < p class = "noindent" > Sparse matrix and data distribution preprocessing.< / li > < / ul >
< / dd > < dt class = "description" >
<!-- l. 31 6 --> < p class = "noindent" >
<!-- l. 31 5 --> < p class = "noindent" >
< span
class="pplb7t-">Preconditioner routines< / span > < / dt > < dd
class="description">
<!-- l. 31 6 --> < p class = "noindent" >
<!-- l. 31 5 --> < p class = "noindent" >
< / dd > < dt class = "description" >
<!-- l. 31 7 --> < p class = "noindent" >
<!-- l. 31 6 --> < p class = "noindent" >
< span
class="pplb7t-">Iterative methods< / span > < / dt > < dd
class="description">
<!-- l. 31 7 --> < p class = "noindent" > a subset of classical and Krylov subspace iterative methods< / dd > < / dl >
<!-- l. 3 20 --> < p class = "noindent" > The following naming scheme has been adopted for all the symbols internally defined
<!-- l. 31 6 --> < p class = "noindent" > a subset of classical and Krylov subspace iterative methods< / dd > < / dl >
<!-- l. 3 19 --> < p class = "noindent" > The following naming scheme has been adopted for all the symbols internally defined
in the PSBLAS software package:
< ul class = "itemize1" >
< li class = "itemize" >
<!-- l. 32 3 --> < p class = "noindent" > all symbols (i.e. subroutine names, data types...) are prefixed by < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 32 2 --> < p class = "noindent" > all symbols (i.e. subroutine names, data types...) are prefixed by < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_< / span > < / span > < / span >
< / li >
< li class = "itemize" >
<!-- l. 32 5 --> < p class = "noindent" > all data type names are suffixed by < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 32 4 --> < p class = "noindent" > all data type names are suffixed by < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">_type< / span > < / span > < / span >
< / li >
< li class = "itemize" >
<!-- l. 32 6 --> < p class = "noindent" > all constants are suffixed by < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 32 5 --> < p class = "noindent" > all constants are suffixed by < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">_< / span > < / span > < / span >
< / li >
< li class = "itemize" >
<!-- l. 32 7 --> < p class = "noindent" > all top-level subroutine names follow the rule < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 32 6 --> < p class = "noindent" > all top-level subroutine names follow the rule < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_xxname< / span > < / span > < / span > where < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">xx< / span > < / span > < / span > can be
either:
< ul class = "itemize2" >
< li class = "itemize" >
<!-- l. 3 30 --> < p class = "noindent" > < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 3 29 --> < p class = "noindent" > < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">ge< / span > < / span > < / span > : the routine is related to dense data,
< / li >
< li class = "itemize" >
<!-- l. 33 1 --> < p class = "noindent" > < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 33 0 --> < p class = "noindent" > < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">sp< / span > < / span > < / span > : the routine is related to sparse data,
< / li >
< li class = "itemize" >
<!-- l. 33 2 --> < p class = "noindent" > < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 33 1 --> < p class = "noindent" > < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">cd< / span > < / span > < / span > : the routine is related to communication descriptor (see  < a
href="userhtmlse3.html#x9-100003">3<!-- tex4ht:ref: sec:datastruct --> < / a > ).< / li > < / ul >
<!-- l. 33 5 --> < p class = "noindent" > For example the < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 33 4 --> < p class = "noindent" > For example the < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_geins< / span > < / span > < / span > , < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_spins< / span > < / span > < / span > and < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_cdins< / span > < / span > < / span > perform the same
@ -339,33 +341,33 @@ class="cmtt-10">psb_cdins</span></span></span> perform the same
href="userhtmlse6.html#x12-780006">6<!-- tex4ht:ref: sec:toolsrout --> < / a > ) on dense matrices, sparse matrices and communication
descriptors respectively. Interface overloading allows the usage of the same
subroutine names for both real and complex data.< / li > < / ul >
<!-- l. 34 2 --> < p class = "noindent" > In the description of the subroutines, arguments or argument entries are classified
<!-- l. 34 1 --> < p class = "noindent" > In the description of the subroutines, arguments or argument entries are classified
as:
< dl class = "description" > < dt class = "description" >
<!-- l. 34 5 --> < p class = "noindent" >
<!-- l. 34 4 --> < p class = "noindent" >
< span
class="pplb7t-">global< / span > < / dt > < dd
class="description">
<!-- l. 34 5 --> < p class = "noindent" > For input arguments, the value must be the same on all processes
<!-- l. 34 4 --> < p class = "noindent" > For input arguments, the value must be the same on all processes
participating in the subroutine call; for output arguments the value is
guaranteed to be the same.
< / dd > < dt class = "description" >
<!-- l. 34 8 --> < p class = "noindent" >
<!-- l. 34 7 --> < p class = "noindent" >
< span
class="pplb7t-">local< / span > < / dt > < dd
class="description">
<!-- l. 34 8 --> < p class = "noindent" > Each process has its own value(s) independently.< / dd > < / dl >
<!-- l. 3 50 --> < p class = "noindent" > To finish our general description, we define a version string with the constant
<!-- l. 34 7 --> < p class = "noindent" > Each process has its own value(s) independently.< / dd > < / dl >
<!-- l. 3 49 --> < p class = "noindent" > To finish our general description, we define a version string with the constant
< div class = "math-display" >
< img
src="userhtml0x.png" alt="psb_version_string_
" class="math-display" >< / div >
<!-- l. 35 2 --> < p class = "nopar" > whose current value is < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 35 1 --> < p class = "nopar" > whose current value is < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">3.9.0< / span > < / span > < / span >
<!-- l. 35 5 --> < p class = "noindent" >
<!-- l. 35 4 --> < p class = "noindent" >
< h4 class = "subsectionHead" > < span class = "titlemark" > 2.3 < / span > < a
id="x5-70002.3">< / a > Application structure< / h4 >
<!-- l. 35 8 --> < p class = "noindent" > The main underlying principle of the PSBLAS library is that the library objects are
<!-- l. 35 7 --> < p class = "noindent" > The main underlying principle of the PSBLAS library is that the library objects are
created and exist with reference to a discretized space to which there corresponds
an index space and a matrix sparsity pattern. As an example, consider a
cell-centered finite-volume discretization of the Navier-Stokes equations on a
@ -375,13 +377,13 @@ class="zplmr7m-">n </span>is isomorphic to the set of cell centers,
whereas the pattern of the associated linear system matrix is isomorphic to the
adjacency graph imposed on the discretization mesh by the discretization
stencil.
<!-- l. 36 8 --> < p class = "indent" > Thus the first order of business is to establish an index space, and this is done
<!-- l. 36 7 --> < p class = "indent" > Thus the first order of business is to establish an index space, and this is done
with a call to < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_cdall< / span > < / span > < / span > in which we specify the size of the index space < span
class="zplmr7m-">n < / span > and the
allocation of the elements of the index space to the various processes making up the
MPI (virtual) parallel machine.
<!-- l. 37 4 --> < p class = "indent" > The index space is partitioned among processes, and this creates a mapping from
<!-- l. 37 3 --> < p class = "indent" > The index space is partitioned among processes, and this creates a mapping from
the “ global” numbering 1< span
class="zplmr7m-">… < / span > < span
class="zplmr7m-">n < / span > to a numbering “ local” to each process; each process < span
@ -393,14 +395,14 @@ class="zplmr7m-x-x-60">i</span></sub></sub>, each element of which corresponds t
element of 1< span
class="zplmr7m-">… < / span > < span
class="zplmr7m-">n< / span > . The user does not set explicitly this mapping; when the application
needs to indicate to which element of the index space a certain item is related,
such as the row and column index of a matrix coefficient, it does so in the
needs to indicate to which element of the index space a certain item is related, such
as the row and column index of a matrix coefficient, it usually does so in the
“ global” numbering, and the library will translate into the appropriate “ local”
numbering.
<!-- l. 38 4 --> < p class = "indent" > For a given index space 1< span
<!-- l. 38 3 --> < p class = "indent" > For a given index space 1< span
class="zplmr7m-">… < / span > < span
class="zplmr7m-">n < / span > there are many possible associated topologies, i.e.
many different discretization stencils; thus the description of the index space is not
@ -423,51 +425,51 @@ class="zplmr7m-">n</span><sub>col<sub>
class="zplmr7m-x-x-60">i< / span > < / sub > < / sub > ,
denoting elements of the index space that are < span
class="pplri7t-">not < / span > assigned to process < span
class="zplmr7m-">i< / span > ; however the
variables associated with them are needed to complete computations associated with
class="zplmr7m-">i< / span > ; the variables
associated with them are needed to complete computations associated with
the sparse matrix < span
class="zplmr7m-">A< / span > , and thus they have to be fetched from (neighbouring)
processes. The descriptor of the index space is built exactly for the purpose
of properly sequencing the communication steps required to achieve this
objective.
<!-- l. 400 --> < p class = "indent" > A simple application structure will walk through the index space allocation,
<!-- l. 399 --> < p class = "indent" > A simple application structure will walk through the index space allocation,
matrix/vector creation and linear system solution as follows:
< ol class = "enumerate1" >
< li
class="enumerate" id="x5-7002x1">
<!-- l. 40 4 --> < p class = "noindent" > Initialize parallel environment with < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 40 3 --> < p class = "noindent" > Initialize parallel environment with < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_init< / span > < / span > < / span > ;
< / li >
< li
class="enumerate" id="x5-7004x2">
<!-- l. 40 5 --> < p class = "noindent" > Initialize index space with < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 40 4 --> < p class = "noindent" > Initialize index space with < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_cdall< / span > < / span > < / span > ;
< / li >
< li
class="enumerate" id="x5-7006x3">
<!-- l. 40 6 --> < p class = "noindent" > Allocate sparse matrix and dense vectors with < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 40 5 --> < p class = "noindent" > Allocate sparse matrix and dense vectors with < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_spall< / span > < / span > < / span > and < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_geall< / span > < / span > < / span > ;
< / li >
< li
class="enumerate" id="x5-7008x4">
<!-- l. 40 8 --> < p class = "noindent" > Loop over all local rows, generate matrix and vector entries, and insert
<!-- l. 40 7 --> < p class = "noindent" > Loop over all local rows, generate matrix and vector entries, and insert
them with < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_spins< / span > < / span > < / span > and < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_geins< / span > < / span > < / span >
< / li >
< li
class="enumerate" id="x5-7010x5">
<!-- l. 4 1 0--> < p class = "noindent" > Assemble the various entities:
<!-- l. 4 09 --> < p class = "noindent" > Assemble the various entities:
< ol class = "enumerate2" >
< li
class="enumerate" id="x5-7012x1">
<!-- l. 41 2 --> < p class = "noindent" > < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 41 1 --> < p class = "noindent" > < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_cdasb< / span > < / span > < / span > ,
< / li >
< li
class="enumerate" id="x5-7014x2">
<!-- l. 41 3 --> < p class = "noindent" > < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 41 2 --> < p class = "noindent" > < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_spasb< / span > < / span > < / span > ,
@ -475,12 +477,12 @@ class="cmtt-10">psb_spasb</span></span></span>,
< / li >
< li
class="enumerate" id="x5-7016x3">
<!-- l. 41 4 --> < p class = "noindent" > < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 41 3 --> < p class = "noindent" > < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_geasb< / span > < / span > < / span > ;< / li > < / ol >
< / li >
< li
class="enumerate" id="x5-7018x6">
<!-- l. 41 6 --> < p class = "noindent" > Choose the preconditioner to be used with < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 41 5 --> < p class = "noindent" > Choose the preconditioner to be used with < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">prec%init< / span > < / span > < / span > and < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">prec%set< / span > < / span > < / span > , and build it with
< span class = "obeylines-h" > < span class = "verb" > < span
@ -490,39 +492,39 @@ href="userhtml8.html#fn3x0"><sup class="textsuperscript">3</sup></a></span><a
< / li >
< li
class="enumerate" id="x5-7022x7">
<!-- l. 42 1 --> < p class = "noindent" > Call one of the iterative drivers with the method of choice, e.g. < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 42 0 --> < p class = "noindent" > Call one of the iterative drivers with the method of choice, e.g. < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_krylov< / span > < / span > < / span >
with < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">bicgstab< / span > < / span > < / span > .< / li > < / ol >
<!-- l. 42 4 --> < p class = "noindent" > This is the structure of the sample programs in the directory < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 42 3 --> < p class = "noindent" > This is the structure of the sample programs in the directory < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">test/pargen/< / span > < / span > < / span > .
<!-- l. 42 7 --> < p class = "indent" > For a simulation in which the same discretization mesh is used over multiple
<!-- l. 42 6 --> < p class = "indent" > For a simulation in which the same discretization mesh is used over multiple
time steps, the following structure may be more appropriate:
< ol class = "enumerate1" >
< li
class="enumerate" id="x5-7024x1">
<!-- l. 4 30 --> < p class = "noindent" > Initialize parallel environment with < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 4 29 --> < p class = "noindent" > Initialize parallel environment with < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_init< / span > < / span > < / span >
< / li >
< li
class="enumerate" id="x5-7026x2">
<!-- l. 43 1 --> < p class = "noindent" > Initialize index space with < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 43 0 --> < p class = "noindent" > Initialize index space with < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_cdall< / span > < / span > < / span >
< / li >
< li
class="enumerate" id="x5-7028x3">
<!-- l. 43 2 --> < p class = "noindent" > Loop over the topology of the discretization mesh and build the
<!-- l. 43 1 --> < p class = "noindent" > Loop over the topology of the discretization mesh and build the
descriptor with < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_cdins< / span > < / span > < / span > ;
< / li >
< li
class="enumerate" id="x5-7030x4">
<!-- l. 43 4 --> < p class = "noindent" > Assemble the descriptor with < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 43 3 --> < p class = "noindent" > Assemble the descriptor with < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_cdasb< / span > < / span > < / span > ;
< / li >
< li
class="enumerate" id="x5-7032x5">
<!-- l. 43 5 --> < p class = "noindent" > Allocate the sparse matrices and dense vectors with; < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 43 4 --> < p class = "noindent" > Allocate the sparse matrices and dense vectors with; < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_spall< / span > < / span > < / span > and
< span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_geall< / span > < / span > < / span > ;
@ -532,34 +534,34 @@ class="cmtt-10">psb_geall</span></span></span>;
< / li >
< li
class="enumerate" id="x5-7034x6">
<!-- l. 43 7 --> < p class = "noindent" > Loop over the time steps:
<!-- l. 43 6 --> < p class = "noindent" > Loop over the time steps:
< ol class = "enumerate2" >
< li
class="enumerate" id="x5-7036x1">
<!-- l. 43 9 --> < p class = "noindent" > If after first time step, reinitialize the sparse matrix with < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 43 8 --> < p class = "noindent" > If after first time step, reinitialize the sparse matrix with < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_sprn< / span > < / span > < / span > ;
also zero out the dense vectors;
< / li >
< li
class="enumerate" id="x5-7038x2">
<!-- l. 44 2 --> < p class = "noindent" > Loop over the mesh, generate the coefficients and insert/update
<!-- l. 44 1 --> < p class = "noindent" > Loop over the mesh, generate the coefficients and insert/update
them with < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_spins< / span > < / span > < / span > and < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_geins< / span > < / span > < / span > ;
< / li >
< li
class="enumerate" id="x5-7040x3">
<!-- l. 44 4 --> < p class = "noindent" > Assemble with < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 44 3 --> < p class = "noindent" > Assemble with < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_spasb< / span > < / span > < / span > and < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_geasb< / span > < / span > < / span > ;
< / li >
< li
class="enumerate" id="x5-7042x4">
<!-- l. 44 5 --> < p class = "noindent" >
<!-- l. 44 4 --> < p class = "noindent" >
< / li >
< li
class="enumerate" id="x5-7044x5">
<!-- l. 44 5 --> < p class = "noindent" > Choose the preconditioner to be used with < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 44 4 --> < p class = "noindent" > Choose the preconditioner to be used with < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">prec%init< / span > < / span > < / span > and
< span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">prec%set< / span > < / span > < / span > , and build it with < span class = "obeylines-h" > < span class = "verb" > < span
@ -567,21 +569,21 @@ class="cmtt-10">prec%build</span></span></span>;
< / li >
< li
class="enumerate" id="x5-7046x6">
<!-- l. 44 8 --> < p class = "noindent" > Call one of the iterative drivers with the method of choice, e.g.
<!-- l. 44 7 --> < p class = "noindent" > Call one of the iterative drivers with the method of choice, e.g.
< span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_krylov< / span > < / span > < / span > with < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">bicgstab< / span > < / span > < / span > .< / li > < / ol >
< / li > < / ol >
<!-- l. 45 2 --> < p class = "noindent" > The insertion routines will be called as many times as needed; they only need to be
<!-- l. 45 1 --> < p class = "noindent" > The insertion routines will be called as many times as needed; they only need to be
called on the data that is actually allocated to the current process, i.e. each process
generates its own data.
<!-- l. 45 7 --> < p class = "indent" > In principle there is no specific order in the calls to < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 45 6 --> < p class = "indent" > In principle there is no specific order in the calls to < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_spins< / span > < / span > < / span > , nor is there a
requirement to build a matrix row in its entirety before calling the routine; this
allows the application programmer to walk through the discretization mesh element
by element, generating the main part of a given matrix row but also contributions to
the rows corresponding to neighbouring elements.
<!-- l. 46 4 --> < p class = "indent" > From a functional point of view it is even possible to execute one call for each
<!-- l. 46 3 --> < p class = "indent" > From a functional point of view it is even possible to execute one call for each
nonzero coefficient; however this would have a substantial computational
overhead. It is therefore advisable to pack a certain amount of data into each
call to the insertion routine, say touching on a few tens of rows; the best
@ -595,23 +597,23 @@ process and pass it in a single call to <span class="obeylines-h"><span class="v
class="cmtt-10">psb_spins< / span > < / span > < / span > ; this, however, would entail a
doubling of memory occupation, and thus would be almost always far from
optimal.
<!-- l. 47 7 --> < p class = "noindent" >
<!-- l. 47 6 --> < p class = "noindent" >
< h5 class = "subsubsectionHead" > < span class = "titlemark" > 2.3.1 < / span > < a
id="x5-80002.3.1">< / a > User-defined index mappings< / h5 >
<!-- l. 47 9 --> < p class = "noindent" > PSBLAS supports user-defined global to local index mappings, subject to the
<!-- l. 47 8 --> < p class = "noindent" > PSBLAS supports user-defined global to local index mappings, subject to the
constraints outlined in sec.  < a
href="#x5-70002.3">2.3<!-- tex4ht:ref: sec:appstruct --> < / a > :
< ol class = "enumerate1" >
< li
class="enumerate" id="x5-8002x1">
<!-- l. 48 2 --> < p class = "noindent" > The set of indices owned locally must be mapped to the set 1< span
<!-- l. 48 1 --> < p class = "noindent" > The set of indices owned locally must be mapped to the set 1< span
class="zplmr7m-">… < / span > < span
class="zplmr7m-">n< / span > < sub > row< sub > < span
class="zplmr7m-x-x-60">i< / span > < / sub > < / sub > ;
< / li >
< li
class="enumerate" id="x5-8004x2">
<!-- l. 48 4 --> < p class = "noindent" > The set of halo points must be mapped to the set < span
<!-- l. 48 3 --> < p class = "noindent" > The set of halo points must be mapped to the set < span
class="zplmr7m-">n< / span > < sub > row< sub > < span
class="zplmr7m-x-x-60">i< / span > < / sub > < / sub > < span
class="zplmr7t-">+ < / span > 1< span
@ -619,14 +621,14 @@ class="zplmr7m-">…</span><span
class="zplmr7m-">n< / span > < sub > col< sub >
< span
class="zplmr7m-x-x-60">i< / span > < / sub > < / sub > ;< / li > < / ol >
<!-- l. 48 7 --> < p class = "noindent" > but otherwise the mapping is arbitrary. The user application is responsible to ensure
<!-- l. 48 6 --> < p class = "noindent" > but otherwise the mapping is arbitrary. The user application is responsible to ensure
consistency of this mapping; some errors may be caught by the library, but
this is not guaranteed. The application structure to support this usage is as
follows:
< ol class = "enumerate1" >
< li
class="enumerate" id="x5-8006x1">
<!-- l. 49 3 --> < p class = "noindent" > Initialize index
<!-- l. 49 2 --> < p class = "noindent" > Initialize index
space with < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_cdall(ictx,desc,info,vl=vl,lidx=lidx)< / span > < / span > < / span > passing the
vectors < span class = "obeylines-h" > < span class = "verb" > < span
@ -636,7 +638,7 @@ class="cmtt-10">lidx(:)</span></span></span> containing the corresponding local
< / li >
< li
class="enumerate" id="x5-8008x2">
<!-- l. 49 8 --> < p class = "noindent" > Add the halo points < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 49 7 --> < p class = "noindent" > Add the halo points < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">ja(:)< / span > < / span > < / span > and their associated local indices < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">lidx(:)< / span > < / span > < / span >
with a(some) call(s) to < span class = "obeylines-h" > < span class = "verb" > < span
@ -644,7 +646,7 @@ class="cmtt-10">psb_cdins(nz,ja,desc,info,lidx=lidx)</span></span></span>;
< / li >
< li
class="enumerate" id="x5-8010x3">
<!-- l. 50 1 --> < p class = "noindent" > Assemble the descriptor with < span class = "obeylines-h" > < span class = "verb" > < span
<!-- l. 50 0 --> < p class = "noindent" > Assemble the descriptor with < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_cdasb< / span > < / span > < / span > ;
< / li >
< li
@ -652,7 +654,7 @@ class="cmtt-10">psb_cdasb</span></span></span>;
<!-- l. 50 2 --> < p class = "noindent" > Build the sparse matrices and vectors, optionally making use in
<!-- l. 50 1 --> < p class = "noindent" > Build the sparse matrices and vectors, optionally making use in
< span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_spins< / span > < / span > < / span > and < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">psb_geins< / span > < / span > < / span > of the < span class = "obeylines-h" > < span class = "verb" > < span
@ -661,41 +663,41 @@ class="cmtt-10">local</span></span></span> argument specifying that the
class="cmtt-10">ia< / span > < / span > < / span > , < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">ja< / span > < / span > < / span > and < span class = "obeylines-h" > < span class = "verb" > < span
class="cmtt-10">irw< / span > < / span > < / span > , respectively, are already local indices.< / li > < / ol >
<!-- l. 50 9 --> < p class = "noindent" >
<!-- l. 50 8 --> < p class = "noindent" >
< h4 class = "subsectionHead" > < span class = "titlemark" > 2.4 < / span > < a
id="x5-90002.4">< / a > Programming model< / h4 >
<!-- l. 51 1 --> < p class = "noindent" > The PSBLAS librarary is based on the Single Program Multiple Data (SPMD)
<!-- l. 51 0 --> < p class = "noindent" > The PSBLAS librarary is based on the Single Program Multiple Data (SPMD)
programming model: each process participating in the computation performs the
same actions on a chunk of data. Parallelism is thus data-driven.
<!-- l. 51 6 --> < p class = "indent" > Because of this structure, many subroutines coordinate their action across the
<!-- l. 51 5 --> < p class = "indent" > Because of this structure, many subroutines coordinate their action across the
various processes, thus providing an implicit synchronization point, and therefore
< span
class="pplri7t-">must < / span > be called simultaneously by all processes participating in the computation. This
is certainly true for the data allocation and assembly routines, for all the
computational routines and for some of the tools routines.
<!-- l. 52 4 --> < p class = "indent" > However there are many cases where no synchronization, and indeed no
communication among processes, is implied; for instance, all the routines in sec.  < a
href="userhtmlse3.html#x9-100003">3<!-- tex4ht:ref: sec:datastruct --> < / a >
are only acting on the local data structures, and thus may be called independently.
The most important case is that of the coefficient insertion routines: since the number
of coefficients in the sparse and dense matrices varies among the processors, and
since the user is free to choose an arbitrary order in builiding the matrix entries,
these routines cannot imply a synchronization.
<!-- l. 53 4 --> < p class = "indent" > Throughout this user’ s guide each subroutine will be clearly indicated
<!-- l. 52 3 --> < p class = "indent" > However there are cases where no synchronization, and indeed no communication
among processes, is implied; for instance, all the routines in sec.  < a
href="userhtmlse3.html#x9-100003">3<!-- tex4ht:ref: sec:datastruct --> < / a > are only acting on
the local data structures, and thus may be called independently. The most important
case is that of the coefficient insertion routines: since the number of coefficients in the
sparse and dense matrices varies among the processors, and since the user is free to
choose an arbitrary order in builiding the matrix entries, these routines cannot imply
a synchronization.
<!-- l. 53 3 --> < p class = "indent" > Throughout this user’ s guide each subroutine will be clearly indicated
as:
< dl class = "description" > < dt class = "description" >
<!-- l. 53 7 --> < p class = "noindent" >
<!-- l. 53 6 --> < p class = "noindent" >
< span
class="pplb7t-">Synchronous:< / span > < / dt > < dd
class="description">
<!-- l. 53 7 --> < p class = "noindent" > must be called simultaneously by all the processes in the relevant
<!-- l. 53 6 --> < p class = "noindent" > must be called simultaneously by all the processes in the relevant
communication context;
< / dd > < dt class = "description" >
<!-- l. 53 9 --> < p class = "noindent" >
<!-- l. 53 8 --> < p class = "noindent" >
< span
class="pplb7t-">Asynchronous:< / span > < / dt > < dd
class="description">
<!-- l. 53 9 --> < p class = "noindent" > may be called in a totally independent manner.< / dd > < / dl >
<!-- l. 53 8 --> < p class = "noindent" > may be called in a totally independent manner.< / dd > < / dl >
sfilippone
5 months ago
