Re: [mpich-discuss] Buffer corruption due to an excessive number of messages
Kurt, There is another common component between current MPICH and Open MPI: UCX, that is handling the low level communications. I suggest to try to change the communication substrate to see if your issue continues to exist. For OMPI add `--mca pml ob1 --mca btl self,sm,tcp' to your mpirun command. George. On Fri, Sep 15, 2023 at 11:20 AM Joachim Jenke via discuss < [email protected]> wrote:
Am 15.09.23 um 17:09 schrieb Tony Curtis via discuss:
On Sep 15, 2023, at 11:07 AM, Raffenetti, Ken via discuss <[email protected]> wrote:
1. Is there a way to detect this kind of overload with an MPI call?
If MPI detects an error at runtime, the default behavior is to abort the application. If you application does not abort (and you haven't changed the default error handler), then no error was detected by MPI.
There’s a tool called MUST that might help
MUST - RWTH AACHEN UNIVERSITY Lehrstuhl für Informatik 12 - Deutsch <https://www.i12.rwth-aachen.de/go/id/nrbe> i12.rwth-aachen.de <https://www.i12.rwth-aachen.de/go/id/nrbe> apple-touch-icon-180x180.png <
https://www.i12.rwth-aachen.de/go/id/nrbe>
The current release version can only detect conflicts in buffer usage at MPI API level. That means, it will only detect buffer conflicts for in-flight messages as in:
MPI_Irecv(buf, MPI_INT, 10, ..., req1); MPI_Irecv(&buf[9], MPI_INT, 10, ..., req2); MPI_Wait(req1,...); MPI_Wait(req2,...);
The upcoming release I was referencing in my other mail would detect conflicting accesses to in-flight buffers as in:
MPI_Irecv(buf, MPI_INT, 10, ..., req); buf[5]=5; MPI_Wait(req,...);
(Not affiliated, just happen to have been looking at it)
Happy to see that people look at the tool :D
- Joachim
Tony
_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
-- Dr. rer. nat. Joachim Jenke
IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 [email protected] www.itc.rwth-aachen.de
_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
participants (1)
-
George Bosilca