mirror of
https://github.com/sfilippone/psblas3.git
synced 2026-10-16 10:52:08 +00:00
Stack-1
ea273e8e93
Matrix Market is ASCII with variable-length lines, so the byte offset of nonzero k cannot be computed without scanning the file: reading it in parallel is impossible without a redistribution pass. psb_mm2bin converts it once into a row-sorted binary layout, and psb_binmat_mod reads that layout in parallel. Layout, 8-byte native-endian throughout: a header carrying a magic number, the index and real kinds the file was written with, and the dimensions; then row_ptr(nrows+1), col_idx(nnz) and values(nnz). The kinds are recorded so a mismatched reader fails loudly instead of reinterpreting the bytes. Each rank reads the header, then only its own slice of row_ptr, and from that seeks straight to the nonzeros of the rows it owns. Because the block-row partition is implicit in the file order, psb_matdist is not needed at all -- that scatter alone accounted for 28 s of MPI time at 448 ranks. Geo_1438, 1.44M rows, 63.2M nonzeros: Matrix Market read, serial 27.0 s conversion, once 27.9 s binary read at 8 ranks, including assembly and one SpMV 0.98 s Verified: byte-for-byte layout check on a 4x4 symmetric matrix with the lower triangle expanded; A*1 identical at 1, 2, 3 and 4 ranks there, the 3-rank case covering the uneven split; sum(A*1) identical to sixteen digits at 1, 4 and 8 ranks on Geo_1438. Positioned stream reads rather than MPI-IO: the ranks touch disjoint byte ranges, so there is nothing to coordinate. If collective buffering turns out to matter at high rank counts, MPI_File_read_at_all is a change local to psb_binmat_mod.
Communication scheme tests
This directory contains tests created after adding multiple communication schemes to PSBLAS.
The goal is to exercise and compare communication patterns at different layers:
- direct halo exchange (
psi_swapdata) - overlap exchange for transpose/SpMV workflows (
psi_swaptran+psi_swapdata) - full Krylov solver runs (
CG) using different comm schemes.
Communication schemes covered in this area:
psb_comm_isend_irecv_(baseline point-to-point)psb_comm_ineighbor_alltoallv_(neighbor collective)psb_comm_persistent_ineighbor_alltoallv_(persistent neighbor collective)
See:
swapdata/for a direct halo-exchange test.spmv/for an overlap SpMV test that uses different communication schemes.cg/for a conjugate-gradient solve-time comparison across the three schemes.