Paired A/B at 448 ranks, three replicas per arm inside one allocation: with
no_locks=true the aggregated MPI_Win_create cost 1463 s against 1450 s
without it, the two ranges fully overlapping. A probe with MPI_Win_get_info
confirmed this Open MPI retains the hint, so the null result belongs to the
hint and not to a failed measurement; same_disp_unit was silently ignored.
Removes the cached MPI_Info handles and their helpers, and returns the four
PSCW win_create calls to MPI_INFO_NULL.
Epoch handling. Both one-sided schemes opened and closed their access epoch
on every halo swap: push with lock_all/unlock_all over the window's whole
communicator, which is O(P) and not O(neighbours), pull with a
Win_lock/Win_unlock pair around every single Get. The epoch is now opened
once at window creation and closed by the handle teardown, which already
unlocked whenever window_open was set. Completion is explicit instead:
flush_all on the push side before the notifications go out, flush_local_all
on the pull side since a Get only needs to have landed in our own combuf, and
Win_sync before the scatter to reconcile the window's public and private
copies -- combuf is itself the exposed memory, so ordinary loads on it need
what the per-swap unlock used to provide implicitly.
448 ranks, dim 200, one scheme per process, 500 iterations, 3 replicas
rma_push 6.48e-2 -> 3.46e-4 (188x)
rma_pull 6.09e-4 -> 2.29e-4 (2.7x)
isend_irecv unchanged at 1.9e-4 (control)
rma_pull is now level with the point-to-point baseline while issuing no
two-sided traffic at all. rma_push stays at 1.8x because its completion
notification is still a point-to-point exchange.
The push loop is also split so the Puts are issued before any completion: a
flush per neighbour inside the loop serialised one round trip each. Measured
neutral on its own, kept because it removes one MPI call per neighbour.
Bound checks. y_nrows was assigned inside the do_send branch but read from
do_recv, so with split start/wait phases the second entry compared halo
indices against uninitialised stack memory: at 8 ranks and dim 200 that read
~32600 against legitimate indices up to 1040000 and aborted the run, while at
dim 40 the same garbage exceeded the halo and the check passed. The
minval/maxval index scans now run under the existing debug guard; they walked
the whole index list per neighbour per swap. The O(1) checks stay.
Dynamic windows (2a767a40) and the window pool (a3249c50) are kept in
history as measured negative results: no gain on window creation,
MPI_Get 49x slower, and a remote-access crash at 448 ranks. Back to
mpi_win_create, now with cached MPI_Info handles: same_disp_unit=true
always, no_locks=true only on the PSCW path in psi_dswapdata, the one
file where no passive-target call survives.
Validated at 8 ranks: psb_d_comm_test 9/9, and the AMG driver converges
in the same iterations to the same residual on all five schemes.
The complex halo exchange declared real MPI datatypes on complex buffers,
with counts in elements and never doubled, so it moved the wrong number
of bytes. On z every scheme transferred half the payload; on c the
baseline and persistent paths happened to work because complex(spk) and
real(dpk) are both 8 bytes, while the neighbour collective and the RMA
get/put moved half.
Regenerated from template-psblas after fixing @RMPI_TYPE@ to @MPI_TYPE@
there; the only change in these files is the datatype handle, no other
line differs.
Only c and z are regenerated on purpose. psi_sswapdata.F90 carries the
index bounds checks and psi_dswapdata.F90 the PSCW optimization, neither
of which has been folded back into the template yet, so regenerating
those two would silently drop them. Neither needs this fix anyway: for
s and d the real and complex macros resolve to the same handle.
Per-type halo test on 4 ranks, all five schemes:
before s 9/9 d 9/9 c 3/9 z 2/9
after s 9/9 d 9/9 c 9/9 z 9/9
Confirmed end to end by the CG driver, where all five schemes now reach
the same final error to the last digit in the same 62 iterations.
Extend the nested (MATNEST) matrix support to all the arithmetics: the
psb_{s,c,z}_nest_{mat,base_mat,tools,builder}_mod modules and the
psb_{s,c,z}_nest_mod umbrellas are generated from the template-psblas
X_nest_* templates; the d sources are regenerated byte-identical.
Preparatory changes to the d sources for clean templating: rowsum/arwsum and
colsum/aclsum no longer share a helper (for the complex arithmetics the
absolute sums are real-valued while the plain sums are complex-valued), the
transposed kernel forwards the actual 'T'/'C' character to the blocks
(conjugate transpose for the complex types), and the capacity helper takes a
type-neutral name.
Build hooks (autotools Makefile and CMakeLists) updated with the per-arith
objects, compile rules and dependencies. All four d tests keep passing.
Author: Simone Staccone (Stack-1)
Add get_owned_rows(i_field) and get_owned_row_count(i_field) to
psb_d_nest_matrix: the list of GLOBAL row indices of a field owned by the
calling process (i.e. the rows it is expected to insert through ins) and
their count. They replace the descriptor-level idiom
field_desc(i)%get_local_rows() / field_desc(i)%l2g(...) in user code, which
leaked descriptor jargon into the build loop.
The high-level tests (glob, rect, builder) are rewritten on the new queries;
the low-level CG test intentionally keeps the descriptor path. README updated
with the new queries and an example.
Author: Simone Staccone (Stack-1)
Complete the integration of the nested (MATNEST) operator into the standard
PSBLAS infrastructure:
- Preconditioners: implement get_diag and csgetrow on psb_d_nest_base_mat so
the stock one-level preconditioners build directly on the nested operator
(DIAG through the concatenated block diagonals, BJAC through the
format-agnostic csget path used by the ILU factorizations).
- Configurable block storage: psb_d_nest_rect_block and psb_d_nest_matrix%asb
accept an optional type ('CSR' default, 'CSC', 'COO') or mold (any class
extending psb_d_base_sparse_mat, e.g. the psb_ext ELL/HLL formats); the
operator is format-agnostic since every operation delegates to the blocks.
- Device-capable matvec: override vect_mv to gather/scatter through the
vectors' own gth/sct with encapsulated index vectors (device kernels on
device vectors) and to run each block through its vect_mv, so device block
formats execute their native kernels; bit-equivalent to csmv on host.
- Full psb_d_base_sparse_mat contract by delegation to the blocks: transposed
csmv (dedicated kernel, ghost contributions left to the transposed halo
exchange), multi-RHS csmm, cp_to_coo/mv_to_coo (unlocking cscnv, csclip,
tril/triu through the base generics), rowsum/arwsum/colsum/aclsum,
maxval/spnmi/spnm1, scal (left/right) and scals, clone (view semantics:
shared blocks, re-owned index maps), mold, sizeof. cp_from_coo/mv_from_coo,
csput and cssv/cssm are intentionally left to the base error (meaningless
for a block-operator view), documented in the type and in the README.
Tests: glob assembles the blocks in HLL (psb_ext) and rect in CSC, both still
bit-identical to the monolithic CSR oracle; the CG test solves under NONE,
DIAG and BJAC/ILU(0), requiring convergence to the exact solution for all of
them and DIAG bit-identical to NONE (exactness check of the nested get_diag).
README updated with the user API reference, the preconditioner section and
the implemented-contract section.
Author: Simone Staccone (Stack-1)
Propagate the latest development (via communication_v2) onto the nested branch:
brings the GMRES refactor, the stopping-criterion change and the restored work
parameter on top of the nested (MATNEST) matrix support. Clean merge, no
conflicts.
Realign communication_v2 with the latest development (10 commits, including the
GMRES refactor and the stopping-criterion change), keeping the communication_v2
work intact.
Conflict resolution:
- base/modules/Makefile (veryclean): keep communication_v2's '/bin/rm -f *.h'.
- linsolve/impl/psb_{c,d,s,z}rgmres.f90: keep development's variable
declaration (itmax_, naux), consistent with development's refactored GMRES
body which references itmax_.
Add a block-structured distributed operator that presents itself to Krylov
solvers and preconditioners as a single ordinary distributed matrix (the
PSBLAS analogue of PETSc MATNEST), targeting saddle-point systems
M = [[A, B^T], [B, 0]] with possibly rectangular sub-blocks.
Library (base/modules):
- psb_desc_nest_mod, psb_d_nest_mat_mod: grid of per-field descriptors and
per-block sparse storage.
- psb_d_nest_base_mat_mod: psb_d_nest_base_mat, the operator extending
psb_d_base_sparse_mat (local csmv, free, field-split hooks for a future
block preconditioner).
- psb_cd_nest_tools_mod / psb_d_nest_tools_mod: composed global descriptor
with union halo (psb_cd_nest_compose) and rectangular local block builder
(psb_d_nest_rect_block), plus the per-block assembly wrappers.
- psb_d_nest_builder_mod: psb_d_nest_matrix, the user frontend with the
init/ins/asb/free pattern hiding all descriptor/halo/compose/setup
boilerplate.
- psb_d_nest_mod: umbrella module (use psb_d_nest_mod).
Remove the earlier bespoke per-block prototype (comm/psblas/vect modules and
the pde_nest_psblas test) superseded by the single MATNEST design.
Tests (test/nested): glob (square operator vs monolithic CSR oracle), rect
(genuinely rectangular blocks), cg (low-level path, ill-conditioned SPD
red-black Laplacian solved with standard CG), builder (same solve via the
utility), plus a README describing the design and usage. All pass serially
and in parallel, with results invariant to the process count.
Build hooks updated (autotools Makefiles + CMakeLists); the nested tests are
relocated out of test/pdegen into test/nested.
Author: Simone Staccone (Stack-1)
The nested layer was imported from an older base where the vector psb_spsm
still took a work buffer. communication_v2 removed work from the psb_x_vect_type
routines, so psb_dspsv_vect has no work argument and the call failed generic
resolution. Remove work from psb_d_nest_spsm (signature, declaration, call).
base/CMakeLists.txt listed only psb_comm_rma_mod.F90 from comm_schemes,
omitting psb_comm_schemes_mod, psb_comm_baseline_mod, psb_comm_neighbor_impl_mod
and psb_comm_factory_mod, so their .mod files were never generated under CMake.
Also psb_comm_mod was listed as .F90 while the file is .f90 (breaks on
case-sensitive filesystems). The autotools Makefile was already correct.
Stack-1
sfilippone
jalmerol
Salvatore Filippone