April 2026 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] hydra_pmi_proxy sending signal 9 to successful processes
by Lana Deere 07 Apr '26

07 Apr '26

Thanks for the suggestion, I will try it. So far I am not seeing any sign that any of my processes exited abnormally. They all record success in their log files (except the ones killed by -9), and if I 'borrow' hydra_pmi_proxy from a different MPI they all exit cleanly, exit code 0. I can experiment with -disable-auto-cleanup while I'm debugging but in production use the auto cleanup is needed so if we do crash there aren't leftover processes hanging on the cluster. .. Lana (lana.deere(a)gmail.com) On Mon, Apr 6, 2026 at 11:36 AM Zhou, Hui <zhouh(a)anl.gov> wrote: > Hi Lana, > > You can try add -disable-auto-cleanup to mpiexec to prevent it kill > every other processes when one process exits abnormally. > > I usually use a .gdbinit script to get a backtrace for such cases. For > example, if you program is ./t, then > > mpirun gdb ./t > > Example .gdbinit: > ``` > set $_exitcode = -999 > run > if $_exitcode == -999 > backtrace > end > exit $_exitcode > ``` > > Hope that helps. > > -- > Hui Zhou > > > *From: *Lana Deere via discuss <discuss(a)mpich.org> > *Date: *Monday, April 6, 2026 at 10:24 AM > *To: *discuss(a)mpich.org <discuss(a)mpich.org> > *Cc: *Lana Deere <lana.deere(a)gmail.com> > *Subject: *[mpich-discuss] hydra_pmi_proxy sending signal 9 to successful > processes > > This Message Is From an External Sender > This message came from outside your organization. > > I've got several MPI programs here. The one which is the most complicated > started exiting, reporting that a process got signal 9 while cleaning up > after a run it reported was successful. Many of the other MPI processes > showed truncated outputs as if they too had received a signal 9. Only > that one program has this problem, the other programs don't. I tried > reducing the big program to a small testcase which reproduces the issue but > was unsuccessful. > > I did put a gdb onto the hydra_pmi_proxy and discovered that it is the > process sending the signal 9 to the various MPI processes, > > (gdb) where > #0 0x00007f4b17853d7e in killpg () from /lib64/libc.so.6 > #1 0x00000000004053e2 in PMIP_bcast_signal (sig=sig@entry=9) at > proxy/pmip_pg.c:259 > #2 0x0000000000406e60 in pmi_cb (fd=9, events=<optimized out>, > userp=<optimized out>) > at proxy/pmip_cb.c:326 > #3 0x0000000000421418 in HYDT_dmxu_poll_wait_for_event (wtime=<optimized > out>) > at lib/tools/demux/demux_poll.c:75 > #4 0x0000000000403ff5 in main (argc=<optimized out>, argv=<optimized > out>) at proxy/pmip.c:121 > > At that time I was using mpich 4.3.0, so I upgraded to 5.0.0 hoping the > problem would be resolved. 5.0.0 still showed the same symptom. This all > is happening on SUSE Linux 15.5. > > On CentOS7 and Rocky Linux 9 we use mvapich2 2.3.6, so for an experiment I > took the mpirun and hydra_pmi_proxy from 2.3.6 and used them instead of the > versions from the mpich 5.0.0 release. Now the program works without > difficulty. All of this suggests to me that the hydra_pmi_proxy has > incorrectly determined that one of the MPI processes exited with a signal. > Any suggestions about what's going on? > > > > .. Lana (lana.deere(a)gmail.com) > > >

1 0

Re: [mpich-discuss] hydra_pmi_proxy sending signal 9 to successful processes
by Zhou, Hui 06 Apr '26

06 Apr '26

Hi Lana, You can try add -disable-auto-cleanup to mpiexec to prevent it kill every other processes when one process exits abnormally. I usually use a .gdbinit script to get a backtrace for such cases. For example, if you program is ./t, then mpirun gdb ./t Example .gdbinit: ``` set $_exitcode = -999 run if $_exitcode == -999 backtrace end exit $_exitcode ``` Hope that helps. -- Hui Zhou From: Lana Deere via discuss <discuss(a)mpich.org> Date: Monday, April 6, 2026 at 10:24 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Lana Deere <lana.deere(a)gmail.com> Subject: [mpich-discuss] hydra_pmi_proxy sending signal 9 to successful processes This Message Is From an External Sender This message came from outside your organization. I've got several MPI programs here. The one which is the most complicated started exiting, reporting that a process got signal 9 while cleaning up after a run it reported was successful. Many of the other MPI processes showed truncated outputs as if they too had received a signal 9. Only that one program has this problem, the other programs don't. I tried reducing the big program to a small testcase which reproduces the issue but was unsuccessful. I did put a gdb onto the hydra_pmi_proxy and discovered that it is the process sending the signal 9 to the various MPI processes, (gdb) where #0 0x00007f4b17853d7e in killpg () from /lib64/libc.so.6 #1 0x00000000004053e2 in PMIP_bcast_signal (sig=sig@entry=9) at proxy/pmip_pg.c:259 #2 0x0000000000406e60 in pmi_cb (fd=9, events=<optimized out>, userp=<optimized out>) at proxy/pmip_cb.c:326 #3 0x0000000000421418 in HYDT_dmxu_poll_wait_for_event (wtime=<optimized out>) at lib/tools/demux/demux_poll.c:75 #4 0x0000000000403ff5 in main (argc=<optimized out>, argv=<optimized out>) at proxy/pmip.c:121 At that time I was using mpich 4.3.0, so I upgraded to 5.0.0 hoping the problem would be resolved. 5.0.0 still showed the same symptom. This all is happening on SUSE Linux 15.5. On CentOS7 and Rocky Linux 9 we use mvapich2 2.3.6, so for an experiment I took the mpirun and hydra_pmi_proxy from 2.3.6 and used them instead of the versions from the mpich 5.0.0 release. Now the program works without difficulty. All of this suggests to me that the hydra_pmi_proxy has incorrectly determined that one of the MPI processes exited with a signal. Any suggestions about what's going on? .. Lana (lana.deere(a)gmail.com<mailto:[email protected]>)

1 0

hydra_pmi_proxy sending signal 9 to successful processes
by Lana Deere 06 Apr '26

06 Apr '26

I've got several MPI programs here. The one which is the most complicated started exiting, reporting that a process got signal 9 while cleaning up after a run it reported was successful. Many of the other MPI processes showed truncated outputs as if they too had received a signal 9. Only that one program has this problem, the other programs don't. I tried reducing the big program to a small testcase which reproduces the issue but was unsuccessful. I did put a gdb onto the hydra_pmi_proxy and discovered that it is the process sending the signal 9 to the various MPI processes, (gdb) where #0 0x00007f4b17853d7e in killpg () from /lib64/libc.so.6 #1 0x00000000004053e2 in PMIP_bcast_signal (sig=sig@entry=9) at proxy/pmip_pg.c:259 #2 0x0000000000406e60 in pmi_cb (fd=9, events=<optimized out>, userp=<optimized out>) at proxy/pmip_cb.c:326 #3 0x0000000000421418 in HYDT_dmxu_poll_wait_for_event (wtime=<optimized out>) at lib/tools/demux/demux_poll.c:75 #4 0x0000000000403ff5 in main (argc=<optimized out>, argv=<optimized out>) at proxy/pmip.c:121 At that time I was using mpich 4.3.0, so I upgraded to 5.0.0 hoping the problem would be resolved. 5.0.0 still showed the same symptom. This all is happening on SUSE Linux 15.5. On CentOS7 and Rocky Linux 9 we use mvapich2 2.3.6, so for an experiment I took the mpirun and hydra_pmi_proxy from 2.3.6 and used them instead of the versions from the mpich 5.0.0 release. Now the program works without difficulty. All of this suggests to me that the hydra_pmi_proxy has incorrectly determined that one of the MPI processes exited with a signal. Any suggestions about what's going on? .. Lana (lana.deere(a)gmail.com)

1 0

Re: [mpich-discuss] MPI ABI in mpich 5.0*
by Alastair McKinstry 01 Apr '26

01 Apr '26

Hi Howard Thanks. I’m interested as the main packager of MPI for Debian; ie are we going to have another / definitive ABI for MPI “soon”, and testing it against the archive of applications. I plan to rebuild all MPI apps in Debian against mpi_abi (for both MPICH and OpenMPI when they add MPI-5 support), for now I’ll just test C apps on MPICH. Alastair From: Howard Pritchard via discuss <discuss(a)mpich.org> Date: Wednesday, 1 April 2026 at 16:10 To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Howard Pritchard <hppritcha(a)gmail.com>, Raffenetti, Ken <raffenet(a)anl.gov> Subject: Re: [mpich-discuss] MPI ABI in mpich 5.0* This Message Is From an External Sender This message came from outside your organization. Hi Alastair, You may want to try https://urldefense.us/v3/__https://github.com/eschnett/mpif__;!!G_uCfscf7eW… <https://urldefense.us/v3/__https://github.com/eschnett/mpif__;!!G_uCfscf7eW…> . Howard Am Mi., 1. Apr. 2026 um 08:32 Uhr schrieb Raffenetti, Ken via discuss <discuss(a)mpich.org<mailto:[email protected]>>: MPICH does not currently provide the mpifort_abi library and compile wrapper. I’ve created an issue to note this and track progress on implementation for the next major release. Thanks for the question/report. https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues/7759__;!… <https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues/7759__;!…> Ken From: Alastair McKinstry via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Tuesday, March 31, 2026 at 11:02 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Alastair McKinstry <mckinstry(a)debian.org<mailto:[email protected]>> Subject: [mpich-discuss] MPI ABI in mpich 5.0* This Message Is From an External Sender This message came from outside your organization. Hi, Does mpich 5.0.0 support the full MPI ABI standard ? I see it creates mpicc_abi which links against libmpi_abi, but no equivalent mpifort_abi. Is this planned? Best regards Alastair -- Alastair McKinstry, GPG: 82383CE9165B347C787081A2CBE6BB4E5D9AD3A5 e: mckinstry(a)debian.org<mailto:[email protected]>, im: @alastair:mckinstry.ie<https://urldefense.us/v3/__http://mckinstry.ie__;!!G_uCfscf7eWS!fNDCfjp-zbN…> https://urldefense.us/v3/__https://mastodon.ie/@amckinstry__;!!G_uCfscf7eWS… _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://urldefense.us/v3/__https://lists.mpich.org/mailman/listinfo/discuss… <https://urldefense.us/v3/__https://lists.mpich.org/mailman/listinfo/discuss…>

1 0

Re: [mpich-discuss] MPI ABI in mpich 5.0*
by Howard Pritchard 01 Apr '26

01 Apr '26

Hi Alastair, You may want to try https://urldefense.us/v3/__https://github.com/eschnett/mpif__;!!G_uCfscf7eW… . Howard Am Mi., 1. Apr. 2026 um 08:32 Uhr schrieb Raffenetti, Ken via discuss < discuss(a)mpich.org>: > MPICH does not currently provide the mpifort_abi library and compile > wrapper. I’ve created an issue to note this and track progress on > implementation for the next major release. Thanks for the question/report. > https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues/7759__;!… > <https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues/7759__;!…> > > Ken > > *From: *Alastair McKinstry via discuss <discuss(a)mpich.org> > *Date: *Tuesday, March 31, 2026 at 11:02 AM > *To: *discuss(a)mpich.org <discuss(a)mpich.org> > *Cc: *Alastair McKinstry <mckinstry(a)debian.org> > *Subject: *[mpich-discuss] MPI ABI in mpich 5.0* > > This Message Is From an External Sender > This message came from outside your organization. > > > Hi, > > Does mpich 5.0.0 support the full MPI ABI standard ? I see it creates > mpicc_abi which links against libmpi_abi, but no equivalent mpifort_abi. > Is this planned? > > Best regards > > Alastair > > -- > Alastair McKinstry, > GPG: 82383CE9165B347C787081A2CBE6BB4E5D9AD3A5 > e: mckinstry(a)debian.org, im: @alastair:mckinstry.iehttps://urldefense.us/v3/__https://mastodon.ie/@amckinstry__;!!G_uCfscf7eWS!bHMKIegHsbit0nQX1Sq-5QbOBnPhPrNfEql5J2PdjovBp1rTdjdMOsGb0lJRSMWd3SYZqyIOElK3x3qOIA$ > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://urldefense.us/v3/__https://lists.mpich.org/mailman/listinfo/discuss… >

1 0

Re: [mpich-discuss] MPI ABI in mpich 5.0*
by Raffenetti, Ken 01 Apr '26

01 Apr '26

MPICH does not currently provide the mpifort_abi library and compile wrapper. I’ve created an issue to note this and track progress on implementation for the next major release. Thanks for the question/report. https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues/7759__;!… Ken From: Alastair McKinstry via discuss <discuss(a)mpich.org> Date: Tuesday, March 31, 2026 at 11:02 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Alastair McKinstry <mckinstry(a)debian.org> Subject: [mpich-discuss] MPI ABI in mpich 5.0* This Message Is From an External Sender This message came from outside your organization. Hi, Does mpich 5.0.0 support the full MPI ABI standard ? I see it creates mpicc_abi which links against libmpi_abi, but no equivalent mpifort_abi. Is this planned? Best regards Alastair -- Alastair McKinstry, GPG: 82383CE9165B347C787081A2CBE6BB4E5D9AD3A5 e: mckinstry(a)debian.org, im: @alastair:mckinstry.ie https://urldefense.us/v3/__https://mastodon.ie/@amckinstry__;!!G_uCfscf7eWS…

1 0