Hi Anatoly, I think the failure behavior is yet to be standardized, thus the behavior falls into the undefined behavior category. I suspect the different outcomes with MPICH 3.1 and 3.4.2 are from changes in the process manager. Could you try using hydra from 3.1 in your test with 3.4.2? -- Hui Zhou ________________________________ From: Anatoly G via discuss <[email protected]> Sent: Monday, December 13, 2021 7:18 AM To: [email protected] <[email protected]> Cc: Anatoly G <[email protected]> Subject: [mpich-discuss] Question regarding fault tolerance difference 3.1 & 3.4.2 Hi MPICH, I have a small program which has different outputs on MPICH 3.1 & MPICH 3.4.2 The program (code attached): 1. Master executes ping-pong with each one of the slaves (separately) 2. Each slave replies to master when it gets a message from master 3. One of the slaves simulates failre with command abort(). 4. Master recognizes that slave fails and continues to work with survived slaves only. I execute my program on ubuntu18. I use TCP as a transport layer. Execute command: mpiexec.hydra -genvall -disable-auto-cleanup -f MpiConfigMachines.txt -launcher=ssh -n 3 ft_ping_pong I expect that "ping pong" will continue till all 20 iterations will finish. MPICH 3.1 hydra works as expected only slave 1 fails and ping-pong continues between Master and slave2. MPICH 3.4.2 hydra process slave 2 fails together with slave 1. MPICH 3.1 configuration: ./configure --prefix="my directory" --enable-error-checking=runtime --enable-g=dbg CFLAGS=-fPIC CXXFLAGS=-fPIC FFLAGS=-fpic --enable-threads=runtime --enable-totalview --enable-static --disable-f77 --disable-fc MPICH 3.4.2 configuration: $ ./configure --prefix="my directory" --enable-error-checking=all --enable-g=dbg CFLAGS=-fPIC CXXFLAGS=-fPIC FFLAGS=-fpic --enable-threads=runtime --enable-totalview --enable-static --disable-f77 --disable-fc --disable-fortran --with-device=ch3:nemesis --enable-error-messages=all Should I use another device or the behavior was modified between versions? Regards, Anatoly.