

\section{Parallel environment routines}
\label{sec:parenv}

\clearpage\subsection{psb\_init --- Initializes PSBLAS parallel
  environment}

\begin{verbatim}
call psb_init(ctxt, np, basectxt, ids, extcomm)
\end{verbatim}

This subroutine initializes the PSBLAS parallel environment, defining
a virtual parallel machine.
\begin{description}
\item[Type:] Synchronous.
\item[\bf  On Entry ]
\item[np] Number of processes in the PSBLAS virtual parallel machine.\\
Scope: {\bf global}.\\
Type: {\bf optional}.\\
Intent: {\bf in}.\\
Specified as: an integer value. \
Default: use all available processes.
\item[basectxt] the initial PSBLAS communication context. The new context
  will be defined from the processes participating in the initial one.\\
  Scope: {\bf global}.\\
Type: {\bf optional}.\\
Intent: {\bf in}.\\
Specified as: an integer value. \
Default: use MPI\_COMM\_WORLD.
\item[ids] Identities of the processes to use for the new context; the
  argument is ignored when \verb|np| is not specified. This allows the
  processes in the new environment to be in an order different from the
  original one.\\
  Scope: {\bf global}.\\
Type: {\bf optional}.\\
Intent: {\bf in}.\\
Specified as: an integer array. \
Default: use the indices $(0\dots np-1)$.  
\item[extcomm] an alternative initial MPI communicator. The new context
  will be defined from the processes participating in the initial one.\\
  Scope: {\bf global}.\\
Type: {\bf optional}.\\
Intent: {\bf in}.\\
Specified as: an integer value. \
Default: use MPI\_COMM\_WORLD.
\end{description}

\begin{description}
\item[\bf On Return]
\item[ctxt] the communication context identifying the virtual
  parallel machine, type \verb|psb_ctxt_type|. Note that this is always a duplicate of
  \verb|basectxt|, so that library communications are completely
  separated from other communication operations.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf out}.\\
Specified as: an integer variable.
\end{description}


{\par\noindent\large\bfseries Notes}
\begin{enumerate}
\item A call to this routine must precede any other PSBLAS call. 
\item It is an error to specify a value for $np$ greater than the
  number of processes available in the underlying base parallel
  environment. 
\end{enumerate}


\clearpage\subsection{psb\_info --- Return information about  PSBLAS parallel
  environment}

\begin{verbatim}
call psb_info(ctxt, iam, np)
\end{verbatim}

This subroutine returns information about  the PSBLAS parallel environment, defining
a virtual parallel machine.
\begin{description}
\item[Type:] Asynchronous.
\item[\bf  On Entry ]
\item[ctxt] the communication context identifying the virtual
  parallel machine.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf in}.\\
Specified as: an integer variable.
\end{description}

\begin{description}
\item[\bf On Return]
\item[iam] Identifier of current  process in the PSBLAS virtual parallel machine.\\
Scope: {\bf local}.\\
Type: {\bf required}.\\
Intent: {\bf out}.\\
Specified as: an integer value. $-1 \le iam \le np-1$\
\item[np] Number of processes in the PSBLAS virtual parallel machine.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf out}.\\
Specified as: an integer variable. \
\end{description}


{\par\noindent\large\bfseries Notes}
\begin{enumerate}
\item For processes in the virtual parallel machine  the identifier
  will satisfy $0 \le iam \le np-1$;
\item If the user has requested on \verb|psb_init| a number of
  processes less than the total available in the parallel execution
  environment, the remaining processes will have on return $iam=-1$;
  the only call involving \verb|ctxt| that any such process may
  execute is  to \verb|psb_exit|. 
\end{enumerate}


\clearpage\subsection{psb\_exit --- Exit from  PSBLAS parallel
  environment}

\begin{verbatim}
call psb_exit(ctxt)
call psb_exit(ctxt,close)
\end{verbatim}

This subroutine exits from the  PSBLAS parallel virtual  machine.
\begin{description}
\item[Type:] Synchronous.
\item[\bf  On Entry ]
\item[ctxt] the communication context identifying the virtual
  parallel machine.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf in}.\\
Specified as: an integer variable.
\item[close] Whether to close all data structures related to the
  virtual parallel machine, besides those associated with ctxt.\\
Scope: {\bf global}.\\
Type: {\bf optional}.\\
Intent: {\bf in}.\\
Specified as: a logical  variable, default value: true.
\end{description}

{\par\noindent\large\bfseries Notes}
\begin{enumerate}
\item This routine may be called even if a previous call to
  \verb|psb_info| has returned with $iam=-1$; indeed, it it is the only
  routine that may be called with argument \verb|ctxt| in this
  situation.
\item A call to this routine with \verb|close=.true.| implies a call
  to \verb|MPI_Finalize|, after which no parallel routine may be called.
\item If the user whishes to use multiple communication contexts in the
  same program, or to enter and exit multiple times into the parallel
  environment, this routine may be called to 
  selectively close the contexts with \verb|close=.false.|, while on
  the last call it should be called with \verb|close=.true.| to
  shutdown in a clean way the entire parallel environment.
\end{enumerate}


\clearpage\subsection{psb\_get\_mpi\_comm --- Get the MPI communicator}

\begin{verbatim}
icomm = psb_get_mpi_comm(ctxt)
\end{verbatim}

This function returns the MPI communicator associated with a PSBLAS context
\begin{description}
\item[Type:] Asynchronous.
\item[\bf  On Entry ]
\item[ctxt] the communication context identifying the virtual
  parallel machine.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf in}.\\
Specified as: an integer variable.
\end{description}

\begin{description}
\item[\bf On Return]
\item[Function value] The MPI communicator associated with the  PSBLAS virtual parallel machine.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf out}.\\
\end{description}

{\par\noindent\large\bfseries Notes}
The subroutine version \verb|psb_get_mpicomm| is still available but
is deprecated. 

\clearpage\subsection{psb\_get\_mpi\_rank --- Get the MPI rank}

\begin{verbatim}
rank = psb_get_mpi_rank(ctxt, id)
\end{verbatim}

This function returns the MPI rank of the  PSBLAS process $id$
\begin{description}
\item[Type:] Asynchronous.
\item[\bf  On Entry ]
\item[ctxt] the communication context identifying the virtual
  parallel machine.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf in}.\\
Specified as: an integer variable.
\item[id] Identifier of a   process in the PSBLAS virtual parallel machine.\\
Scope: {\bf local}.\\
Type: {\bf required}.\\
Intent: {\bf in}.\\
Specified as: an integer value. $0 \le id \le np-1$\
\end{description}

\begin{description}
\item[\bf On Return]
\item[Funciton value] The MPI rank associated with the  PSBLAS process $id$.\\
Scope: {\bf local}.\\
Type: {\bf required}.\\
Intent: {\bf out}.\\
\end{description}

{\par\noindent\large\bfseries Notes}
The subroutine version \verb|psb_get_rank| is still available but is
deprecated. 



\clearpage\subsection{psb\_wtime --- Wall clock timing}

\begin{verbatim}
time = psb_wtime()
\end{verbatim}

This function returns a wall clock timer. The resolution of the timer
is dependent on the underlying parallel environment implementation.
\begin{description}
\item[Type:] Asynchronous.
\item[\bf  On Exit ]
\item[Function value] the elapsed time in seconds.\\
Returned  as: a  \verb|real(psb_dpk_)|  variable.
\end{description}


\clearpage\subsection{psb\_barrier --- Sinchronization point  parallel
  environment}

\begin{verbatim}
call psb_barrier(ctxt)
\end{verbatim}

This subroutine acts as an explicit synchronization point for  the  PSBLAS
parallel virtual  machine. 
\begin{description}
\item[Type:] Synchronous.
\item[\bf  On Entry ]
\item[ctxt] the communication context identifying the virtual
  parallel machine.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf in}.\\
Specified as: an integer variable.
\end{description}


\clearpage\subsection{psb\_abort --- Abort a computation}

\begin{verbatim}
call psb_abort(ctxt)
\end{verbatim}

This subroutine aborts computation on the parallel virtual machine. 
\begin{description}
\item[Type:] Asynchronous.
\item[\bf  On Entry ]
\item[ctxt] the communication context identifying the virtual
  parallel machine.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf in}.\\
Specified as: an integer variable.
\end{description}





\clearpage\subsection{psb\_bcast --- Broadcast data}

\begin{verbatim}
call psb_bcast(ctxt, dat [, root, mode, request])
\end{verbatim}

This subroutine implements a broadcast operation based on the
underlying communication library. 
\begin{description}
\item[Type:] Synchronous.
\item[\bf  On Entry ]
\item[ctxt] the communication context identifying the virtual
  parallel machine.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf in}.\\
Specified as: an integer variable.
\item[dat] On the root process, the data to be broadcast.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf inout}.\\
Specified as: an integer, real or complex variable, which may be a
scalar, or a rank 1 or 2 array, or a character or logical variable,
which may be a scalar or rank 1 array. \
Type, kind, rank and size must agree on all processes.
\item[root] Root process holding data to be broadcast.\\
Scope: {\bf global}.\\
Type: {\bf optional}.\\
Intent: {\bf in}.\\
Specified as: an integer value $0<= root <= np-1$, default 0 \
\item[mode] Whether the call is started in non-blocking mode and completed
  later, or is executed synchronously.\\
Scope: {\bf global}.\\
Type: {\bf optional}.\\
Intent: {\bf in}.\\
Specified as: an integer value. The action to be taken is determined
by its bit fields, which can be set with bitwise \verb|OR|. Basic
action values are \verb|psb_collective_start_|, \verb|psb_collective_end_|.
Default:  both fields are selected (i.e. require synchronous completion).\\
\item[request] A request variable to check for operation completion.\\
Scope: {\bf local}.\\
Type: {\bf optional}.\\
Intent: {\bf inout}.\\
If \verb|mode| does not specify synchronous completion, then this
variable must be present.
\end{description}


\begin{description}
\item[\bf On Return]
\item[dat] On all processes other than  root, the broadcasted data.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf inout}.\\
Specified as: an integer, real or complex variable, which may be a
scalar, or a rank 1 or 2 array, or a character or logical scalar. \
Type, kind, rank and size must agree on all processes.
\item[request] A request variable to check for operation completion.\\
Scope: {\bf local}.\\
Type: {\bf optional}.\\
Intent: {\bf inout}.\\
If \verb|mode| does not specify synchronous completion, then this
variable must be present.
\end{description}

{\par\noindent\large\bfseries Notes}
\begin{enumerate}
\item The \verb|dat| argument is both input and output, and its
  value may be changed even on processes different from the final
  result destination.
\item The \verb|mode| argument can be built with the bitwise
  \verb|IOR()| operator; in the following example, the   argument is
  forcing immediate completion, hence the \verb|request| argument
  needs not be specified: 
\ifpdf
\begin{minted}[breaklines=true,bgcolor=bg,fontsize=\small]{fortran}
  call psb_bcast(ctxt,dat,&
       & mode=ior(psb_collective_start_,psb_collective_end_))
\end{minted}
\else
\begin{center}
    \begin{minipage}[tl]{0.9\textwidth}
\begin{verbatim} 
  call psb_bcast(ctxt,dat,mode=ior(psb_collective_start_,psb_collective_end_))
\end{verbatim}
    \end{minipage}
  \end{center}
\fi
  
\item When splitting the operation in two calls, the \verb|dat|
  argument  \emph{must not} be accessed between calls:
\ifpdf
\begin{minted}[breaklines=true,bgcolor=bg,fontsize=\small]{fortran}
  call psb_bcast(ctxt,dat,mode=psb_collective_start_,&
       & request=bcast_request)
  ....... ! Do not access dat 
  call psb_bcast(ctxt,dat,mode=psb_collective_end_,&
       & request=bcast_request)
\end{minted}
\else
\begin{center}
    \begin{minipage}[tl]{0.9\textwidth}
\begin{verbatim} 
  call psb_bcast(ctxt,dat,mode=psb_collective_start_,request=bcast_request)
  ....... ! Do not access dat 
  call psb_bcast(ctxt,dat,mode=psb_collective_end_, request=bcast_request)
\end{verbatim}
    \end{minipage}
  \end{center}
\fi
\end{enumerate}




\clearpage\subsection{psb\_sum --- Global sum}

\begin{verbatim}
call psb_sum(ctxt, dat [, root, mode, request])
\end{verbatim}

This subroutine implements a sum reduction  operation based on the
underlying communication library. 
\begin{description}
\item[Type:] Synchronous.
\item[\bf  On Entry ]
\item[ctxt] the communication context identifying the virtual
  parallel machine.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf in}.\\
Specified as: an integer variable.
\item[dat] The local contribution to the global sum.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf inout}.\\
Specified as: an integer, real or complex variable, which may be a
scalar, or a rank 1 or 2 array. \
Type, kind, rank and size must agree on all processes.
\item[root] Process to hold the final sum, or $-1$ to make it available
  on all processes.\\
Scope: {\bf global}.\\
Type: {\bf optional}.\\
Intent: {\bf in}.\\
Specified as: an integer value $-1<= root <= np-1$, default -1. \
\item[mode] Whether the call is started in non-blocking mode and completed
  later, or is executed synchronously.\\
Scope: {\bf global}.\\
Type: {\bf optional}.\\
Intent: {\bf in}.\\
Specified as: an integer value. The action to be taken is determined
by its bit fields, which can be set with bitwise \verb|OR|. Basic
action values are \verb|psb_collective_start_|, \verb|psb_collective_end_|.
Default:  both fields are selected (i.e. require synchronous completion).\\
\item[request] A request variable to check for operation completion.\\
Scope: {\bf local}.\\
Type: {\bf optional}.\\
Intent: {\bf inout}.\\
If \verb|mode| does not specify synchronous completion, then this
variable must be present.
\end{description}

\begin{description}
\item[\bf On Return]
\item[dat] On destination process(es), the result of the sum operation.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf inout}.\\
Specified as: an integer, real or complex variable, which may be a
scalar, or a rank 1 or 2 array. \\
Type, kind, rank and size must agree on all processes.
\item[request] A request variable to check for operation completion.\\
Scope: {\bf local}.\\
Type: {\bf optional}.\\
Intent: {\bf inout}.\\
If \verb|mode| does not specify synchronous completion, then this
variable must be present.
\end{description}

{\par\noindent\large\bfseries Notes}
\begin{enumerate}
\item The \verb|dat| argument is both input and output, and its
  value may be changed even on processes different from the final
  result destination.
\item The \verb|mode| argument can be built with the bitwise
  \verb|IOR()| operator; in the following example, the   argument is
  forcing immediate completion, hence the \verb|request| argument
  needs not be specified: 
\ifpdf
\begin{minted}[breaklines=true,bgcolor=bg,fontsize=\small]{fortran}
  call psb_sum(ctxt,dat,&
       & mode=ior(psb_collective_start_,psb_collective_end_))
\end{minted}
\else
\begin{center}
    \begin{minipage}[tl]{0.9\textwidth}
\begin{verbatim} 
  call psb_sum(ctxt,dat,mode=ior(psb_collective_start_,psb_collective_end_))
\end{verbatim}
    \end{minipage}
  \end{center}
\fi
  
\item When splitting the operation in two calls, the \verb|dat|
  argument  \emph{must not} be accessed between calls:
\ifpdf
\begin{minted}[breaklines=true,bgcolor=bg,fontsize=\small]{fortran}
  call psb_sum(ctxt,dat,mode=psb_collective_start_,&
       & request=sum_request)
  ....... ! Do not access dat 
  call psb_sum(ctxt,dat,mode=psb_collective_end_,&
       & request=sum_request)
\end{minted}
\else
\begin{center}
    \begin{minipage}[tl]{0.9\textwidth}
\begin{verbatim} 
  call psb_sum(ctxt,dat,mode=psb_collective_start_,request=sum_request)
  ....... ! Do not access dat 
  call psb_sum(ctxt,dat,mode=psb_collective_end_,request=sum_request)
\end{verbatim}
    \end{minipage}
  \end{center}
\fi
\end{enumerate}



\clearpage\subsection{psb\_max --- Global maximum}

\begin{verbatim}
call psb_max(ctxt, dat [, root, mode, request])
\end{verbatim}

This subroutine implements a maximum valuereduction
operation based on the underlying communication library. 
\begin{description}
\item[Type:] Synchronous.
\item[\bf  On Entry ]
\item[ctxt] the communication context identifying the virtual
  parallel machine.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf in}.\\
Specified as: an integer variable.
\item[dat] The local contribution to the global maximum.\\
Scope: {\bf local}.\\
Type: {\bf required}.\\
Intent: {\bf inout}.\\
Specified as: an integer or  real variable, which may be a
scalar, or a rank 1 or 2 array. \
Type, kind, rank and size must agree on all processes.
\item[root] Process to hold the final maximum, or $-1$ to make it available
  on all processes.\\
Scope: {\bf global}.\\
Type: {\bf optional}.\\
Intent: {\bf in}.\\
Specified as: an integer value $-1<= root <= np-1$, default -1. \\
\item[mode] Whether the call is started in non-blocking mode and completed
  later, or is executed synchronously.\\
Scope: {\bf global}.\\
Type: {\bf optional}.\\
Intent: {\bf in}.\\
Specified as: an integer value. The action to be taken is determined
by its bit fields, which can be set with bitwise \verb|OR|. Basic
action values are \verb|psb_collective_start_|, \verb|psb_collective_end_|.
Default:  both fields are selected (i.e. require synchronous completion).\\
\item[request] A request variable to check for operation completion.\\
Scope: {\bf local}.\\
Type: {\bf optional}.\\
Intent: {\bf inout}.\\
If \verb|mode| does not specify synchronous completion, then this
variable must be present.
\end{description}


\begin{description}
\item[\bf On Return]
\item[dat] On destination process(es), the result of the maximum operation.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf in}.\\
Specified as: an integer or  real variable, which may be a
scalar, or a rank 1 or 2 array. \
Type, kind, rank and size must agree on all processes.
\item[request] A request variable to check for operation completion.\\
Scope: {\bf local}.\\
Type: {\bf optional}.\\
Intent: {\bf inout}.\\
If \verb|mode| does not specify synchronous completion, then this
variable must be present.
\end{description}


{\par\noindent\large\bfseries Notes}
\begin{enumerate}
\item The \verb|dat| argument is both input and output, and its
  value may be changed even on processes different from the final
  result destination.
\item The \verb|mode| argument can be built with the bitwise
  \verb|IOR()| operator; in the following example, the   argument is
  forcing immediate completion, hence the \verb|request| argument
  needs not be specified: 
\ifpdf
\begin{minted}[breaklines=true,bgcolor=bg,fontsize=\small]{fortran}
  call psb_max(ctxt,dat,&
       & mode=ior(psb_collective_start_,psb_collective_end_))
\end{minted}
\else
\begin{center}
    \begin{minipage}[tl]{0.9\textwidth}
\begin{verbatim} 
  call psb_max(ctxt,dat,mode=ior(psb_collective_start_,psb_collective_end_))
\end{verbatim}
    \end{minipage}
  \end{center}
\fi
  
\item When splitting the operation in two calls, the \verb|dat|
  argument  \emph{must not} be accessed between calls:
\ifpdf
\begin{minted}[breaklines=true,bgcolor=bg,fontsize=\small]{fortran}
  call psb_max(ctxt,dat,mode=psb_collective_start_,&
       & request=max_request)
  ....... ! Do not access dat 
  call psb_max(ctxt,dat,mode=psb_collective_end_,&
       & request=max_request)
\end{minted}
\else
\begin{center}
    \begin{minipage}[tl]{0.9\textwidth}
\begin{verbatim} 
  call psb_max(ctxt,dat,mode=psb_collective_start_,request=max_request)
  ....... ! Do not access dat 
  call psb_max(ctxt,dat,mode=psb_collective_end_,request=max_request)
\end{verbatim}
    \end{minipage}
  \end{center}
\fi
\end{enumerate}

\clearpage\subsection{psb\_min --- Global minimum}

\begin{verbatim}
call psb_min(ctxt, dat [, root, mode, request])
\end{verbatim}

This subroutine implements a minimum value reduction
operation based on the underlying communication library. 
\begin{description}
\item[Type:] Synchronous.
\item[\bf  On Entry ]
\item[ctxt] the communication context identifying the virtual
  parallel machine.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf in}.\\
Specified as: an integer variable.
\item[dat] The local contribution to the global minimum.\\
Scope: {\bf local}.\\
Type: {\bf required}.\\
Intent: {\bf inout}.\\
Specified as: an integer  or real variable, which may be a
scalar, or a rank 1 or 2 array. \
Type, kind, rank and size must agree on all processes.
\item[root] Process to hold the final value, or $-1$ to make it available
  on all processes.\\
Scope: {\bf global}.\\
Type: {\bf optional}.\\
Intent: {\bf in}.\\
Specified as: an integer value $-1<= root <= np-1$, default -1. \\
\item[mode] Whether the call is started in non-blocking mode and completed
  later, or is executed synchronously.\\
Scope: {\bf global}.\\
Type: {\bf optional}.\\
Intent: {\bf in}.\\
Specified as: an integer value. The action to be taken is determined
by its bit fields, which can be set with bitwise \verb|OR|. Basic
action values are \verb|psb_collective_start_|, \verb|psb_collective_end_|.
Default:  both fields are selected (i.e. require synchronous completion).\\
\item[request] A request variable to check for operation completion.\\
Scope: {\bf local}.\\
Type: {\bf optional}.\\
Intent: {\bf inout}.\\
If \verb|mode| does not specify synchronous completion, then this
variable must be present.
\end{description}


\begin{description}
\item[\bf On Return]
\item[dat] On destination process(es), the result of the minimum operation.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf inout}.\\
Specified as: an integer  or  real variable, which may be a
scalar, or a rank 1 or 2 array. \\
Type, kind, rank and size must agree on all processes.
\item[request] A request variable to check for operation completion.\\
Scope: {\bf local}.\\
Type: {\bf optional}.\\
Intent: {\bf inout}.\\
If \verb|mode| does not specify synchronous completion, then this
variable must be present.
\end{description}


{\par\noindent\large\bfseries Notes}
\begin{enumerate}
\item The \verb|dat| argument is both input and output, and its
  value may be changed even on processes different from the final
  result destination.
\item The \verb|mode| argument can be built with the bitwise
  \verb|IOR()| operator; in the following example, the   argument is
  forcing immediate completion, hence the \verb|request| argument
  needs not be specified: 
\ifpdf
\begin{minted}[breaklines=true,bgcolor=bg,fontsize=\small]{fortran}
  call psb_min(ctxt,dat,&
       & mode=ior(psb_collective_start_,psb_collective_end_))
\end{minted}
\else
\begin{center}
    \begin{minipage}[tl]{0.9\textwidth}
\begin{verbatim} 
  call psb_min(ctxt,dat,mode=ior(psb_collective_start_,psb_collective_end_))
\end{verbatim}
    \end{minipage}
  \end{center}
\fi
  
\item When splitting the operation in two calls, the \verb|dat|
  argument  \emph{must not} be accessed between calls:
\ifpdf
\begin{minted}[breaklines=true,bgcolor=bg,fontsize=\small]{fortran}
  call psb_min(ctxt,dat,mode=psb_collective_start_,&
       & request=min_request)
  ....... ! Do not access dat 
  call psb_min(ctxt,dat,mode=psb_collective_end_,&
       & request=min_request)
\end{minted}
\else
\begin{center}
    \begin{minipage}[tl]{0.9\textwidth}
\begin{verbatim} 
  call psb_min(ctxt,dat,mode=psb_collective_start_,request=min_request)
  ....... ! Do not access dat 
  call psb_min(ctxt,dat,mode=psb_collective_end_,request=min_request)
\end{verbatim}
    \end{minipage}
  \end{center}
\fi
\end{enumerate}

\clearpage\subsection{psb\_amx --- Global maximum absolute value}

\begin{verbatim}
call psb_amx(ctxt, dat [, root, mode, request])
\end{verbatim}

This subroutine implements a maximum absolute value reduction
operation based on the underlying communication library. 
\begin{description}
\item[Type:] Synchronous.
\item[\bf  On Entry ]
\item[ctxt] the communication context identifying the virtual
  parallel machine.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf in}.\\
Specified as: an integer variable.
\item[dat] The local contribution to the global maximum.\\
Scope: {\bf local}.\\
Type: {\bf required}.\\
Intent: {\bf inout}.\\
Specified as: an integer, real or complex variable, which may be a
scalar, or a rank 1 or 2 array. \
Type, kind, rank and size must agree on all processes.
\item[root] Process to hold the final value, or $-1$ to make it available
  on all processes.\\
Scope: {\bf global}.\\
Type: {\bf optional}.\\
Intent: {\bf in}.\\
Specified as: an integer value $-1<= root <= np-1$, default -1. \\
\item[mode] Whether the call is started in non-blocking mode and completed
  later, or is executed synchronously.\\
Scope: {\bf global}.\\
Type: {\bf optional}.\\
Intent: {\bf in}.\\
Specified as: an integer value. The action to be taken is determined
by its bit fields, which can be set with bitwise \verb|OR|. Basic
action values are \verb|psb_collective_start_|, \verb|psb_collective_end_|.
Default:  both fields are selected (i.e. require synchronous completion).\\
\item[request] A request variable to check for operation completion.\\
Scope: {\bf local}.\\
Type: {\bf optional}.\\
Intent: {\bf inout}.\\
If \verb|mode| does not specify synchronous completion, then this
variable must be present.
\end{description}


\begin{description}
\item[\bf On Return]
\item[dat] On destination process(es), the result of the maximum operation.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf inout}.\\
Specified as: an integer, real or complex variable, which may be a
scalar, or a rank 1 or 2 array. \
Type, kind, rank and size must agree on all processes.
\item[request] A request variable to check for operation completion.\\
Scope: {\bf local}.\\
Type: {\bf optional}.\\
Intent: {\bf inout}.\\
If \verb|mode| does not specify synchronous completion, then this
variable must be present.
\end{description}


{\par\noindent\large\bfseries Notes}
\begin{enumerate}
\item The \verb|dat| argument is both input and output, and its
  value may be changed even on processes different from the final
  result destination.
\item The \verb|mode| argument can be built with the bitwise
  \verb|IOR()| operator; in the following example, the   argument is
  forcing immediate completion, hence the \verb|request| argument
  needs not be specified: 
\ifpdf
\begin{minted}[breaklines=true,bgcolor=bg,fontsize=\small]{fortran}
  call psb_amx(ctxt,dat,&
       & mode=ior(psb_collective_start_,psb_collective_end_))
\end{minted}
\else
\begin{center}
    \begin{minipage}[tl]{0.9\textwidth}
\begin{verbatim} 
  call psb_amx(ctxt,dat,mode=ior(psb_collective_start_,psb_collective_end_))
\end{verbatim}
    \end{minipage}
  \end{center}
\fi
  
\item When splitting the operation in two calls, the \verb|dat|
  argument  \emph{must not} be accessed between calls:
\ifpdf
\begin{minted}[breaklines=true,bgcolor=bg,fontsize=\small]{fortran}
call psb_amx(ctxt,dat,mode=psb_collective_start_,&
     & request=amx_request)
  ....... ! Do not access dat 
  call psb_amx(ctxt,dat,mode=psb_collective_end_,&
       & request=amx_request)
\end{minted}
\else
\begin{center}
    \begin{minipage}[tl]{0.9\textwidth}
\begin{verbatim} 
  call psb_amx(ctxt,dat,mode=psb_collective_start_,request=amx_request)
  ....... ! Do not access dat 
  call psb_amx(ctxt,dat,mode=psb_collective_end_,request=amx_request)
\end{verbatim}
    \end{minipage}
  \end{center}
\fi
\end{enumerate}

\clearpage\subsection{psb\_amn --- Global minimum absolute value}

\begin{verbatim}
call psb_amn(ctxt, dat [, root, mode, request])
\end{verbatim}

This subroutine implements a minimum absolute value reduction
operation based on the underlying communication library. 
\begin{description}
\item[Type:] Synchronous.
\item[\bf  On Entry ]
\item[ctxt] the communication context identifying the virtual
  parallel machine.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf in}.\\
Specified as: an integer variable.
\item[dat] The local contribution to the global minimum.\\
Scope: {\bf local}.\\
Type: {\bf required}.\\
Intent: {\bf inout}.\\
Specified as: an integer, real or complex variable, which may be a
scalar, or a rank 1 or 2 array. \
Type, kind, rank and size must agree on all processes.
\item[root] Process to hold the final value, or $-1$ to make it available
  on all processes.\\
Scope: {\bf global}.\\
Type: {\bf optional}.\\
Intent: {\bf in}.\\
Specified as: an integer value $-1<= root <= np-1$, default -1. \\
\item[mode] Whether the call is started in non-blocking mode and completed
  later, or is executed synchronously.\\
Scope: {\bf global}.\\
Type: {\bf optional}.\\
Intent: {\bf in}.\\
Specified as: an integer value. The action to be taken is determined
by its bit fields, which can be set with bitwise \verb|OR|. Basic
action values are \verb|psb_collective_start_|, \verb|psb_collective_end_|.
Default:  both fields are selected (i.e. require synchronous completion).\\
\item[request] A request variable to check for operation completion.\\
Scope: {\bf local}.\\
Type: {\bf optional}.\\
Intent: {\bf inout}.\\
If \verb|mode| does not specify synchronous completion, then this
variable must be present.
\end{description}


\begin{description}
\item[\bf On Return]
\item[dat] On destination process(es), the result of the minimum operation.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf inout}.\\
Specified as: an integer, real or complex variable, which may be a
scalar, or a rank 1 or 2 array. \\
Type, kind, rank and size must agree on all processes.
\item[request] A request variable to check for operation completion.\\
Scope: {\bf local}.\\
Type: {\bf optional}.\\
Intent: {\bf inout}.\\
If \verb|mode| does not specify synchronous completion, then this
variable must be present.
\end{description}


{\par\noindent\large\bfseries Notes}
\begin{enumerate}
\item The \verb|dat| argument is both input and output, and its
  value may be changed even on processes different from the final
  result destination.
\item The \verb|mode| argument can be built with the bitwise
  \verb|IOR()| operator; in the following example, the   argument is
  forcing immediate completion, hence the \verb|request| argument
  needs not be specified: 
\ifpdf
\begin{minted}[breaklines=true,bgcolor=bg,fontsize=\small]{fortran}
  call psb_amn(ctxt,dat,&
       & mode=ior(psb_collective_start_,psb_collective_end_))
\end{minted}
\else
\begin{center}
    \begin{minipage}[tl]{0.9\textwidth}
\begin{verbatim} 
  call psb_amn(ctxt,dat,mode=ior(psb_collective_start_,psb_collective_end_))
\end{verbatim}
    \end{minipage}
  \end{center}
\fi
  
\item When splitting the operation in two calls, the \verb|dat|
  argument  \emph{must not} be accessed between calls:
\ifpdf
\begin{minted}[breaklines=true,bgcolor=bg,fontsize=\small]{fortran}
  call psb_amn(ctxt,dat,mode=psb_collective_start_,&
       & request=amn_request)
  ....... ! Do not access dat 
  call psb_amn(ctxt,dat,mode=psb_collective_end_,&
       & request=amn_request)
\end{minted}
\else
\begin{center}
    \begin{minipage}[tl]{0.9\textwidth}
\begin{verbatim} 
  call psb_amn(ctxt,dat,mode=psb_collective_start_,request=amn_request)
  ....... ! Do not access dat 
  call psb_amn(ctxt,dat,mode=psb_collective_end_,request=amn_request)
\end{verbatim}
    \end{minipage}
  \end{center}
\fi
\end{enumerate}

\clearpage\subsection{psb\_nrm2 --- Global 2-norm reduction}

\begin{verbatim}
call psb_nrm2(ctxt, dat [, root, mode, request])
\end{verbatim}

This subroutine implements a 2-norm value reduction 
operation based on the underlying communication library. 
\begin{description}
\item[Type:] Synchronous.
\item[\bf  On Entry ]
\item[ctxt] the communication context identifying the virtual
  parallel machine.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf in}.\\
Specified as: an integer variable.
\item[dat] The local contribution to the global minimum.\\
Scope: {\bf local}.\\
Type: {\bf required}.\\
Intent: {\bf inout}.\\
Specified as: a real variable, which may be a
scalar, or a rank 1 array. \
Kind, rank and size must agree on all processes.
\item[root] Process to hold the final value, or $-1$ to make it available
  on all processes.\\
Scope: {\bf global}.\\
Type: {\bf optional}.\\
Intent: {\bf in}.\\
Specified as: an integer value $-1<= root <= np-1$, default -1. \\
\item[mode] Whether the call is started in non-blocking mode and completed
  later, or is executed synchronously.\\
Scope: {\bf global}.\\
Type: {\bf optional}.\\
Intent: {\bf in}.\\
Specified as: an integer value. The action to be taken is determined
by its bit fields, which can be set with bitwise \verb|OR|. Basic
action values are \verb|psb_collective_start_|, \verb|psb_collective_end_|.
Default:  both fields are selected (i.e. require synchronous completion).\\
\item[request] A request variable to check for operation completion.\\
Scope: {\bf local}.\\
Type: {\bf optional}.\\
Intent: {\bf inout}.\\
If \verb|mode| does not specify synchronous completion, then this
variable must be present.
\end{description}


\begin{description}
\item[\bf On Return]
\item[dat] On destination process(es), the result of the 2-norm reduction.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf inout}.\\
Specified as: a real variable, which may be a
scalar, or a rank 1 array. \\
Kind, rank and size must agree on all processes.
\item[request] A request variable to check for operation completion.\\
Scope: {\bf local}.\\
Type: {\bf optional}.\\
Intent: {\bf inout}.\\
If \verb|mode| does not specify synchronous completion, then this
variable must be present.
\end{description}


{\par\noindent\large\bfseries Notes}
\begin{enumerate}
\item This reduction is appropriate to compute the results of multiple
  (local) NRM2 operations at the same time. 
\item Denoting by $dat_i$ the value of the variable $dat$ on process
  $i$, the output $res$ is equivalent to the computation of
  \[ res = \sqrt{\sum_i dat_i^2},\]
  with care taken to avoid unnecessary overflow.
\item The \verb|dat| argument is both input and output, and its
  value may be changed even on processes different from the final
  result destination.
\item The \verb|mode| argument can be built with the bitwise
  \verb|IOR()| operator; in the following example, the   argument is
  forcing immediate completion, hence the \verb|request| argument
  needs not be specified: 
\ifpdf
\begin{minted}[breaklines=true,bgcolor=bg,fontsize=\small]{fortran}
  call psb_nrm2(ctxt,dat,&
       & mode=ior(psb_collective_start_,psb_collective_end_))
\end{minted}
\else
\begin{center}
    \begin{minipage}[tl]{0.9\textwidth}
\begin{verbatim} 
  call psb_nrm2(ctxt,dat,mode=ior(psb_collective_start_,psb_collective_end_))
\end{verbatim}
    \end{minipage}
  \end{center}
\fi
  
\item When splitting the operation in two calls, the \verb|dat|
  argument  \emph{must not} be accessed between calls:
\ifpdf
\begin{minted}[breaklines=true,bgcolor=bg,fontsize=\small]{fortran}
call psb_nrm2(ctxt,dat,mode=psb_collective_start_,&
     & request=nrm2_request)
  ....... ! Do not access dat 
  call psb_nrm2(ctxt,dat,mode=psb_collective_end_,&
       & request=nrm2_request)
\end{minted}
\else
\begin{center}
    \begin{minipage}[tl]{0.9\textwidth}
\begin{verbatim} 
  call psb_nrm2(ctxt,dat,mode=psb_collective_start_,request=nrm2_request)
  ....... ! Do not access dat 
  call psb_nrm2(ctxt,dat,mode=psb_collective_end_,request=nrm2_request)
\end{verbatim}
    \end{minipage}
  \end{center}
\fi
\end{enumerate}



\clearpage\subsection{psb\_snd --- Send data}

\begin{verbatim}
call psb_snd(ctxt, dat, dst, m)
\end{verbatim}

This subroutine sends a packet of data to a destination.
\begin{description}
\item[Type:] Synchronous: see usage notes. 
\item[\bf  On Entry ]
\item[ctxt] the communication context identifying the virtual
  parallel machine.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf in}.\\
Specified as: an integer variable.
\item[dat] The data to be sent.\\
Scope: {\bf local}.\\
Type: {\bf required}.\\
Intent: {\bf in}.\\
Specified as: an integer, real or complex variable, which may be a
scalar, or a rank 1 or 2 array, or a character or logical scalar. \
Type, kind and  rank must agree on sender and receiver process; if $m$ is
not specified, size must agree as well. 
\item[dst] Destination process.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf in}.\\
Specified as: an integer value $0<= dst <= np-1$. \\
\item[m] Number of rows.\\
Scope: {\bf global}.\\
Type: {\bf Optional}.\\
Intent: {\bf in}.\\
Specified as: an integer value $0<= m <= size(dat,1)$. \\
When $dat$ is a rank 2 array, specifies the number of rows to be sent
independently of the leading dimension $size(dat,1)$; must have the
same value on sending and receiving processes.
\end{description}


\begin{description}
\item[\bf On Return]
\end{description}

{\par\noindent\large\bfseries Notes}
\begin{enumerate}
\item This subroutine implies a synchronization, but only between the
  calling process and the destination  process $dst$.
\end{enumerate}


\clearpage\subsection{psb\_rcv --- Receive data}

\begin{verbatim}
call psb_rcv(ctxt, dat, src, m)
\end{verbatim}

This subroutine receives a packet of data to a destination.
\begin{description}
\item[Type:] Synchronous: see usage notes.
\item[\bf  On Entry ]
\item[ctxt] the communication context identifying the virtual
  parallel machine.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf in}.\\
Specified as: an integer variable.
\item[src] Source process.\\
Scope: {\bf global}.\\
Type: {\bf required}.\\
Intent: {\bf in}.\\
Specified as: an integer value $0<= src <= np-1$. \\
\item[m] Number of rows.\\
Scope: {\bf global}.\\
Type: {\bf Optional}.\\
Intent: {\bf in}.\\
Specified as: an integer value $0<= m <= size(dat,1)$. \\
When $dat$ is a rank 2 array, specifies the number of rows to be sent
independently of the leading dimension $size(dat,1)$; must have the
same value on sending and receiving processes.
\end{description}


\begin{description}
\item[\bf On Return]
\item[dat] The data to be received.\\
Scope: {\bf local}.\\
Type: {\bf required}.\\
Intent: {\bf inout}.\\
Specified as: an integer, real or complex variable, which may be a
scalar, or a rank 1 or 2 array, or a character or logical scalar. \
Type, kind and  rank must agree on sender and receiver process; if $m$ is
not specified, size must agree as well. 
\end{description}

{\par\noindent\large\bfseries Notes}
\begin{enumerate}
\item This subroutine implies a synchronization, but only between the
  calling process and the source  process $src$.
\end{enumerate}
