February 2022 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] man(3) pages in 4.0.0?
by Zhou, Hui 14 Feb '22

14 Feb '22

Eric, Thanks for reporting. This has been fixed in the current development branch and will be fixed in the upcoming 4.0.1 release. -- Hui Zhou ________________________________ From: Eric Borisch via discuss <discuss(a)mpich.org> Sent: Monday, February 14, 2022 11:47 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Eric Borisch <eborisch(a)gmail.com> Subject: [mpich-discuss] man(3) pages in 4.0.0? Thank you for the excellent software! Not a critical issue, by any means, but just bringing to your attention that the man pages in 4.0.0 look like something didn't go quite right during generation; it has the function signature for QMPI_Send() rather than MPI_Send() and some extra preprocessor statements. [ From 4.0.0 .tar.gz from mpich.org<http://mpich.org> ] MPI_Send(3) MPI MPI_Send(3) NAME MPI_Send - Performs a blocking send SYNOPSIS #ifdef ENABLE_QMPI #ifndef MPICH_MPI_FROM_PMPI int QMPI_Send(QMPI_Context context, int tool_id, const void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm) MPICH_API_PUBLIC [...] [ From 3.4.3 .tar.gz from mpich.org<http://mpich.org> ] MPI_Send(3) MPI MPI_Send(3) NAME MPI_Send - Performs a blocking send SYNOPSIS int MPI_Send(const void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm) [...] You can see this in the mpich-hosted man pages, too: https://www.mpich.org/static/docs/latest/www3/MPI_Send.html Thanks, - Eric

1 0

man(3) pages in 4.0.0?
by Eric Borisch 14 Feb '22

14 Feb '22

Thank you for the excellent software! Not a critical issue, by any means, but just bringing to your attention that the man pages in 4.0.0 look like something didn't go quite right during generation; it has the function signature for QMPI_Send() rather than MPI_Send() and some extra preprocessor statements. [ From 4.0.0 .tar.gz from mpich.org ] MPI_Send(3) MPI MPI_Send(3) NAME MPI_Send - Performs a blocking send SYNOPSIS #ifdef ENABLE_QMPI #ifndef MPICH_MPI_FROM_PMPI int QMPI_Send(QMPI_Context context, int tool_id, const void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm) MPICH_API_PUBLIC [...] [ From 3.4.3 .tar.gz from mpich.org ] MPI_Send(3) MPI MPI_Send(3) NAME MPI_Send - Performs a blocking send SYNOPSIS int MPI_Send(const void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm) [...] You can see this in the mpich-hosted man pages, too: https://www.mpich.org/static/docs/latest/www3/MPI_Send.html Thanks, - Eric

1 0

Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries
by Zhou, Hui 07 Feb '22

07 Feb '22

Hi Kurt, As Ken mentioned, there is often the complication or confusion of several distinct combinations of setup. Mailinglist is especially bad at resolving such issues as we try different suggestions and the feedback is often all mixed up. Could you check https://github.com/pmodels/mpich/issues/5835 and provide (or re-provide) relevant information there? Particularly useful information are: how mpich is configured, how the job is invocated, the failure symptom, and the debug output by using mpiexec -verbose. As Ken listed, there are three different scenarios. Each should work. It probably will help if we focus on one scenario at a time. -- Hui Zhou [https://opengraph.githubassets.com/f73eca0aa1fd8cd3bc03c57525990e22e36e447b…]<https://github.com/pmodels/mpich/issues/5835> MPI_Comm_spawn in Slurm environment · Issue #5835 · pmodels/mpich<https://github.com/pmodels/mpich/issues/5835> Originated from user email https://lists.mpich.org/pipermail/discuss/2022-January/006360.html. MPICH + Hydra + PMI1 (crashes) MPICH + Hydra + PMI2 (works but ignores "hosts" info key) MPI... github.com ________________________________ From: Raffenetti, Ken via discuss <discuss(a)mpich.org> Sent: Monday, February 7, 2022 4:28 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Cc: Raffenetti, Ken <raffenet(a)anl.gov> Subject: Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries Darn. I'm creating an issue to track this since it will likely take some time and effort to investigate each configuration. https://github.com/pmodels/mpich/issues/5835 Ken On 2/7/22, 12:41 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, To review, I configured as follows: $ configure CFLAGS=-DUSE_PMI2_API LIBS=-lpmi2 --with-pm=none --with-pmi=slurm --with-slurm=/opt/slurm < ...> and ran srun with the argument --mpi=pmi2. The job is stil segfaulting in MPI_Comm_spawn in one process and returning an error from MPI_Barrier in the other. Error messages below: The MPI_Comm_spawn error: backtrace for error: backtrace after receiving signal SIGSEGV: /home/kmccall/Needles2/./NeedlesMpiMM() [0x45ab36] /lib64/libpthread.so.0(+0x12c20) [0x7f4833387c20] /lib64/libc.so.6(+0x15d6b7) [0x7f483310d6b7] /lib64/libc.so.6(__strdup+0x12) [0x7f4833039802] /lib64/libpmi2.so.0(+0x171c) [0x7f483294371c] /lib64/libpmi2.so.0(+0x185e) [0x7f483294385e] /lib64/libpmi2.so.0(PMI2_Job_Spawn+0x1a7) [0x7f48329453d8] /home/kmccall/mpich-slurm-install-4.0_2/lib/libmpi.so.12(+0x23a7db) [0x7f4834e0a7db] /home/kmccall/mpich-slurm-install-4.0_2/lib/libmpi.so.12(+0x1fc805) [0x7f4834dcc805] /home/kmccall/mpich-slurm-install-4.0_2/lib/libmpi.so.12(MPI_Comm_spawn+0x507) [0x7f4834cea9f7] The MPI_Barrier error: MPI_Barrier returned the error MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed Thanks, Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 4, 2022 4:08 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries :(. We need to link -lpmi2 instead of -lpmi. This really needs a patch in our configure script, but adding this to your configure is worth a shot: LIBS=-lpmi2 Ken On 2/4/22, 1:56 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: I added the CFLAGS argument and the configuration completed, but make ended with a link error. lib/.libs/libmpi.so: undefined reference to `PMI2_Abort' lib/.libs/libmpi.so: undefined reference to `PMI2_Info_GetJobAttr' lib/.libs/libmpi.so: undefined reference to `PMI2_Job_Spawn' lib/.libs/libmpi.so: undefined reference to `PMI2_Nameserv_publish' lib/.libs/libmpi.so: undefined reference to `PMI2_Finalize' lib/.libs/libmpi.so: undefined reference to `PMI2_KVS_Put' lib/.libs/libmpi.so: undefined reference to `PMI2_Info_GetNodeAttr' lib/.libs/libmpi.so: undefined reference to `PMI2_KVS_Get' lib/.libs/libmpi.so: undefined reference to `PMI2_KVS_Fence' lib/.libs/libmpi.so: undefined reference to `PMI2_Nameserv_unpublish' lib/.libs/libmpi.so: undefined reference to `PMI2_Info_PutNodeAttr' lib/.libs/libmpi.so: undefined reference to `PMI2_Job_GetId' lib/.libs/libmpi.so: undefined reference to `PMI2_Init' lib/.libs/libmpi.so: undefined reference to `PMI2_Nameserv_lookup' collect2: error: ld returned 1 exit status -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 4, 2022 1:23 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries I think I see a new issue. The Slurm website documentation says that their PMI library doesn't support PMI_Spawn_multiple from the PMI 1 API. We can try to force PMI 2 and see what happens. Try adding this to your configure line. CFLAGS=-DUSE_PMI2_API Ken On 2/4/22, 11:58 AM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Did that, and launched the job with "srun --mpi=none" and one of the processes failed when MPI_Comm_spawn was called. Note the : internal_Comm_spawn(101)......: MPI_Comm_spawn(command=NeedlesMpiMM, argv=0x226b030, maxprocs=1, info=0x9c000000, 0, MPI_COMM_SELF, intercomm=0x7ffffda9448c, array_of_errcodes=0x7ffffda94378) failed MPIDI_Comm_spawn_multiple(225): PMI_Spawn_multiple returned -1 The other process failed when MPI_Barrier was called: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed MPI manager 1 threw exception: MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 4, 2022 11:42 AM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries Yes, you should also use --with-pm=none. If using mpicc to build your application, you should not have to add -lpmi. The script will handle it for you. If using another method, you might have to add it. These days with shared libraries, linkers are often able to manage "inter-library" dependencies just fine. Static builds are a different story. Ken On 2/4/22, 11:35 AM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, >> configure --with-slurm=/opt/slurm --with-pmi=slurm That is similar to your first suggestion below. With the above, do I have to include --with-pm=none? I guess I also have to link my application with -lpmi, right? Thanks, Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 4, 2022 11:02 AM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries When running with srun you need to use the Slurm PMI library, not the embedded Simple PMI2 library. Simple PMI2 is API compatible, but uses a different wire protocol that the Slurm implementation. Try this instead: configure --with-slurm=/opt/slurm --with-pmi=slurm This will link the Slurm PMI library to MPICH. I do acknowledge how confusing this must be to users :). Probably a good FAQ topic for our Github discussions page. Ken On 2/3/22, 7:00 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I'm trying to build MPICH 4.0 in several ways, one of which will be what you suggested below. For this particular attempt suggested by the Slurm MPI guide, I built it with configure --with-slurm=/opt/slurm --with-pmi=pmi2/simple <etc> and invoked it with srun --mpi=pmi2 <etc> The job is crashing with this message. Any idea what is wrong? slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key <99>èþ^? in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key ´2¾ÿ^? in req slurmstepd: error: mpi/pmi2: no value for key ; in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: *** STEP 52227.0 ON n001 CANCELLED AT 2022-02-03T18:48:02 *** -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 3:15 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries On 1/28/22, 2:22 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I confirmed that MPI_Comm_spawn fails completely if I build MPICH without the PMI2 option. Dang, I thought that would work :(. Looking at the Slurm documentation https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.sch… it states "All MPI_comm_spawn work fine now going through hydra's PMI 1.1 interface." The full quote is below for reference. 1) how do I build MPICH to support hydra's PMI 1.1 interface? That is the default, so no extra configuration should be needed. One thing I notice in your log output is that the Slurm envvars seems to have changed name from what we have in our source. E.g. SLURM_JOB_NODELIST vs. SLURM_NODELIST. Do your initial processes launch on the right nodes? 2) Can you offer any guesses on how to build Slurm to do the same? (I realize this isn't a Slurm forum 😊) Hopefully you don't need to rebuild Slurm to do it. What you could try is configuring the Slurm PMI library when building MPICH. Add "--with-pm=none --with-pmi=slurm --with-slurm=<path/to/install>". Then use srun instead of mpiexec and see how it goes. Ken _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries
by Raffenetti, Ken 07 Feb '22

07 Feb '22

Darn. I'm creating an issue to track this since it will likely take some time and effort to investigate each configuration. https://github.com/pmodels/mpich/issues/5835 Ken On 2/7/22, 12:41 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, To review, I configured as follows: $ configure CFLAGS=-DUSE_PMI2_API LIBS=-lpmi2 --with-pm=none --with-pmi=slurm --with-slurm=/opt/slurm < ...> and ran srun with the argument --mpi=pmi2. The job is stil segfaulting in MPI_Comm_spawn in one process and returning an error from MPI_Barrier in the other. Error messages below: The MPI_Comm_spawn error: backtrace for error: backtrace after receiving signal SIGSEGV: /home/kmccall/Needles2/./NeedlesMpiMM() [0x45ab36] /lib64/libpthread.so.0(+0x12c20) [0x7f4833387c20] /lib64/libc.so.6(+0x15d6b7) [0x7f483310d6b7] /lib64/libc.so.6(__strdup+0x12) [0x7f4833039802] /lib64/libpmi2.so.0(+0x171c) [0x7f483294371c] /lib64/libpmi2.so.0(+0x185e) [0x7f483294385e] /lib64/libpmi2.so.0(PMI2_Job_Spawn+0x1a7) [0x7f48329453d8] /home/kmccall/mpich-slurm-install-4.0_2/lib/libmpi.so.12(+0x23a7db) [0x7f4834e0a7db] /home/kmccall/mpich-slurm-install-4.0_2/lib/libmpi.so.12(+0x1fc805) [0x7f4834dcc805] /home/kmccall/mpich-slurm-install-4.0_2/lib/libmpi.so.12(MPI_Comm_spawn+0x507) [0x7f4834cea9f7] The MPI_Barrier error: MPI_Barrier returned the error MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed Thanks, Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 4, 2022 4:08 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries :(. We need to link -lpmi2 instead of -lpmi. This really needs a patch in our configure script, but adding this to your configure is worth a shot: LIBS=-lpmi2 Ken On 2/4/22, 1:56 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: I added the CFLAGS argument and the configuration completed, but make ended with a link error. lib/.libs/libmpi.so: undefined reference to `PMI2_Abort' lib/.libs/libmpi.so: undefined reference to `PMI2_Info_GetJobAttr' lib/.libs/libmpi.so: undefined reference to `PMI2_Job_Spawn' lib/.libs/libmpi.so: undefined reference to `PMI2_Nameserv_publish' lib/.libs/libmpi.so: undefined reference to `PMI2_Finalize' lib/.libs/libmpi.so: undefined reference to `PMI2_KVS_Put' lib/.libs/libmpi.so: undefined reference to `PMI2_Info_GetNodeAttr' lib/.libs/libmpi.so: undefined reference to `PMI2_KVS_Get' lib/.libs/libmpi.so: undefined reference to `PMI2_KVS_Fence' lib/.libs/libmpi.so: undefined reference to `PMI2_Nameserv_unpublish' lib/.libs/libmpi.so: undefined reference to `PMI2_Info_PutNodeAttr' lib/.libs/libmpi.so: undefined reference to `PMI2_Job_GetId' lib/.libs/libmpi.so: undefined reference to `PMI2_Init' lib/.libs/libmpi.so: undefined reference to `PMI2_Nameserv_lookup' collect2: error: ld returned 1 exit status -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 4, 2022 1:23 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries I think I see a new issue. The Slurm website documentation says that their PMI library doesn't support PMI_Spawn_multiple from the PMI 1 API. We can try to force PMI 2 and see what happens. Try adding this to your configure line. CFLAGS=-DUSE_PMI2_API Ken On 2/4/22, 11:58 AM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Did that, and launched the job with "srun --mpi=none" and one of the processes failed when MPI_Comm_spawn was called. Note the : internal_Comm_spawn(101)......: MPI_Comm_spawn(command=NeedlesMpiMM, argv=0x226b030, maxprocs=1, info=0x9c000000, 0, MPI_COMM_SELF, intercomm=0x7ffffda9448c, array_of_errcodes=0x7ffffda94378) failed MPIDI_Comm_spawn_multiple(225): PMI_Spawn_multiple returned -1 The other process failed when MPI_Barrier was called: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed MPI manager 1 threw exception: MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 4, 2022 11:42 AM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries Yes, you should also use --with-pm=none. If using mpicc to build your application, you should not have to add -lpmi. The script will handle it for you. If using another method, you might have to add it. These days with shared libraries, linkers are often able to manage "inter-library" dependencies just fine. Static builds are a different story. Ken On 2/4/22, 11:35 AM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, >> configure --with-slurm=/opt/slurm --with-pmi=slurm That is similar to your first suggestion below. With the above, do I have to include --with-pm=none? I guess I also have to link my application with -lpmi, right? Thanks, Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 4, 2022 11:02 AM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries When running with srun you need to use the Slurm PMI library, not the embedded Simple PMI2 library. Simple PMI2 is API compatible, but uses a different wire protocol that the Slurm implementation. Try this instead: configure --with-slurm=/opt/slurm --with-pmi=slurm This will link the Slurm PMI library to MPICH. I do acknowledge how confusing this must be to users :). Probably a good FAQ topic for our Github discussions page. Ken On 2/3/22, 7:00 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I'm trying to build MPICH 4.0 in several ways, one of which will be what you suggested below. For this particular attempt suggested by the Slurm MPI guide, I built it with configure --with-slurm=/opt/slurm --with-pmi=pmi2/simple <etc> and invoked it with srun --mpi=pmi2 <etc> The job is crashing with this message. Any idea what is wrong? slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key <99>èþ^? in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key ´2¾ÿ^? in req slurmstepd: error: mpi/pmi2: no value for key ; in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: *** STEP 52227.0 ON n001 CANCELLED AT 2022-02-03T18:48:02 *** -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 3:15 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries On 1/28/22, 2:22 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I confirmed that MPI_Comm_spawn fails completely if I build MPICH without the PMI2 option. Dang, I thought that would work :(. Looking at the Slurm documentation https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.sch… it states "All MPI_comm_spawn work fine now going through hydra's PMI 1.1 interface." The full quote is below for reference. 1) how do I build MPICH to support hydra's PMI 1.1 interface? That is the default, so no extra configuration should be needed. One thing I notice in your log output is that the Slurm envvars seems to have changed name from what we have in our source. E.g. SLURM_JOB_NODELIST vs. SLURM_NODELIST. Do your initial processes launch on the right nodes? 2) Can you offer any guesses on how to build Slurm to do the same? (I realize this isn't a Slurm forum 😊) Hopefully you don't need to rebuild Slurm to do it. What you could try is configuring the Slurm PMI library when building MPICH. Add "--with-pm=none --with-pmi=slurm --with-slurm=<path/to/install>". Then use srun instead of mpiexec and see how it goes. Ken

1 0

Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries
by Mccall, Kurt E. (MSFC-EV41) 07 Feb '22

07 Feb '22

Ken, To review, I configured as follows: $ configure CFLAGS=-DUSE_PMI2_API LIBS=-lpmi2 --with-pm=none --with-pmi=slurm --with-slurm=/opt/slurm < ...> and ran srun with the argument --mpi=pmi2. The job is stil segfaulting in MPI_Comm_spawn in one process and returning an error from MPI_Barrier in the other. Error messages below: The MPI_Comm_spawn error: backtrace for error: backtrace after receiving signal SIGSEGV: /home/kmccall/Needles2/./NeedlesMpiMM() [0x45ab36] /lib64/libpthread.so.0(+0x12c20) [0x7f4833387c20] /lib64/libc.so.6(+0x15d6b7) [0x7f483310d6b7] /lib64/libc.so.6(__strdup+0x12) [0x7f4833039802] /lib64/libpmi2.so.0(+0x171c) [0x7f483294371c] /lib64/libpmi2.so.0(+0x185e) [0x7f483294385e] /lib64/libpmi2.so.0(PMI2_Job_Spawn+0x1a7) [0x7f48329453d8] /home/kmccall/mpich-slurm-install-4.0_2/lib/libmpi.so.12(+0x23a7db) [0x7f4834e0a7db] /home/kmccall/mpich-slurm-install-4.0_2/lib/libmpi.so.12(+0x1fc805) [0x7f4834dcc805] /home/kmccall/mpich-slurm-install-4.0_2/lib/libmpi.so.12(MPI_Comm_spawn+0x507) [0x7f4834cea9f7] The MPI_Barrier error: MPI_Barrier returned the error MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed Thanks, Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 4, 2022 4:08 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries :(. We need to link -lpmi2 instead of -lpmi. This really needs a patch in our configure script, but adding this to your configure is worth a shot: LIBS=-lpmi2 Ken On 2/4/22, 1:56 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: I added the CFLAGS argument and the configuration completed, but make ended with a link error. lib/.libs/libmpi.so: undefined reference to `PMI2_Abort' lib/.libs/libmpi.so: undefined reference to `PMI2_Info_GetJobAttr' lib/.libs/libmpi.so: undefined reference to `PMI2_Job_Spawn' lib/.libs/libmpi.so: undefined reference to `PMI2_Nameserv_publish' lib/.libs/libmpi.so: undefined reference to `PMI2_Finalize' lib/.libs/libmpi.so: undefined reference to `PMI2_KVS_Put' lib/.libs/libmpi.so: undefined reference to `PMI2_Info_GetNodeAttr' lib/.libs/libmpi.so: undefined reference to `PMI2_KVS_Get' lib/.libs/libmpi.so: undefined reference to `PMI2_KVS_Fence' lib/.libs/libmpi.so: undefined reference to `PMI2_Nameserv_unpublish' lib/.libs/libmpi.so: undefined reference to `PMI2_Info_PutNodeAttr' lib/.libs/libmpi.so: undefined reference to `PMI2_Job_GetId' lib/.libs/libmpi.so: undefined reference to `PMI2_Init' lib/.libs/libmpi.so: undefined reference to `PMI2_Nameserv_lookup' collect2: error: ld returned 1 exit status -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 4, 2022 1:23 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries I think I see a new issue. The Slurm website documentation says that their PMI library doesn't support PMI_Spawn_multiple from the PMI 1 API. We can try to force PMI 2 and see what happens. Try adding this to your configure line. CFLAGS=-DUSE_PMI2_API Ken On 2/4/22, 11:58 AM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Did that, and launched the job with "srun --mpi=none" and one of the processes failed when MPI_Comm_spawn was called. Note the : internal_Comm_spawn(101)......: MPI_Comm_spawn(command=NeedlesMpiMM, argv=0x226b030, maxprocs=1, info=0x9c000000, 0, MPI_COMM_SELF, intercomm=0x7ffffda9448c, array_of_errcodes=0x7ffffda94378) failed MPIDI_Comm_spawn_multiple(225): PMI_Spawn_multiple returned -1 The other process failed when MPI_Barrier was called: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed MPI manager 1 threw exception: MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 4, 2022 11:42 AM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries Yes, you should also use --with-pm=none. If using mpicc to build your application, you should not have to add -lpmi. The script will handle it for you. If using another method, you might have to add it. These days with shared libraries, linkers are often able to manage "inter-library" dependencies just fine. Static builds are a different story. Ken On 2/4/22, 11:35 AM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, >> configure --with-slurm=/opt/slurm --with-pmi=slurm That is similar to your first suggestion below. With the above, do I have to include --with-pm=none? I guess I also have to link my application with -lpmi, right? Thanks, Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 4, 2022 11:02 AM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries When running with srun you need to use the Slurm PMI library, not the embedded Simple PMI2 library. Simple PMI2 is API compatible, but uses a different wire protocol that the Slurm implementation. Try this instead: configure --with-slurm=/opt/slurm --with-pmi=slurm This will link the Slurm PMI library to MPICH. I do acknowledge how confusing this must be to users :). Probably a good FAQ topic for our Github discussions page. Ken On 2/3/22, 7:00 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I'm trying to build MPICH 4.0 in several ways, one of which will be what you suggested below. For this particular attempt suggested by the Slurm MPI guide, I built it with configure --with-slurm=/opt/slurm --with-pmi=pmi2/simple <etc> and invoked it with srun --mpi=pmi2 <etc> The job is crashing with this message. Any idea what is wrong? slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key <99>èþ^? in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key ´2¾ÿ^? in req slurmstepd: error: mpi/pmi2: no value for key ; in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: *** STEP 52227.0 ON n001 CANCELLED AT 2022-02-03T18:48:02 *** -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 3:15 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries On 1/28/22, 2:22 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I confirmed that MPI_Comm_spawn fails completely if I build MPICH without the PMI2 option. Dang, I thought that would work :(. Looking at the Slurm documentation https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.sch… it states "All MPI_comm_spawn work fine now going through hydra's PMI 1.1 interface." The full quote is below for reference. 1) how do I build MPICH to support hydra's PMI 1.1 interface? That is the default, so no extra configuration should be needed. One thing I notice in your log output is that the Slurm envvars seems to have changed name from what we have in our source. E.g. SLURM_JOB_NODELIST vs. SLURM_NODELIST. Do your initial processes launch on the right nodes? 2) Can you offer any guesses on how to build Slurm to do the same? (I realize this isn't a Slurm forum 😊) Hopefully you don't need to rebuild Slurm to do it. What you could try is configuring the Slurm PMI library when building MPICH. Add "--with-pm=none --with-pmi=slurm --with-slurm=<path/to/install>". Then use srun instead of mpiexec and see how it goes. Ken

1 0

Re: [mpich-discuss] Compile issue with QMPI
by Amit Ruhela 05 Feb '22

05 Feb '22

Thanks Hui, The patch worked very well. Thanks again for the prompt resolution. Best Regards, Amit Ruhela ________________________________ From: Zhou, Hui <zhouh(a)anl.gov> Sent: Saturday, February 5, 2022 10:25 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Amit Ruhela <aruhela(a)tacc.utexas.edu> Subject: Re: Compile issue with QMPI Hi Amit, Could you try this pull request: https://github.com/pmodels/mpich/pull/5831/files? You could apply the patch to release tarball, then run ``` python3 maint/gen_binding_c.py ``` to fix all the generated code. -- Hui Zhou [https://avatars.githubusercontent.com/u/1496702?s=400&v=4]<https://github.com/pmodels/mpich/pull/5831/files> qmpi: chop MPICH_API_PUBLIC from the definition by hzhou · Pull Request #5831 · pmodels/mpich<https://github.com/pmodels/mpich/pull/5831/files> Pull Request Description MPICH_API_PUBLIC is accidentally leaked into definitions, which isn't allowed. Fixes #5829 [skip warnings] Author Checklist Provide Description Particularly focus on ... github.com ________________________________ From: Amit Ruhela via discuss <discuss(a)mpich.org> Sent: Friday, February 4, 2022 5:34 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Amit Ruhela <aruhela(a)tacc.utexas.edu> Subject: [mpich-discuss] Compile issue with QMPI Hi Mpich team, I see the following issue with the latest 4.0 release of MPICH while adding QMPI support at configure time? make[2]: Entering directory `/scratch1/05231/aruhela/libs/mpich/tar-mpich-4.0' CC src/binding/c/attr/lib_libmpi_la-attr_delete.lo src/binding/c/attr/attr_delete.c(63): error: attributes are not permitted in a function definition int QMPI_Attr_delete(QMPI_Context context, int tool_id, MPI_Comm comm, int keyval) MPICH_API_PUBLIC ^ compilation aborted for src/binding/c/attr/attr_delete.c (code 2) make[2]: *** [src/binding/c/attr/lib_libmpi_la-attr_delete.lo] Error 1 Any suggestions to resolve the build error? Amit Ruhela, Ph.D. HPC Software Tools Group Texas Advanced Computing Center The University of Texas at Austin Email: aruhela(a)tacc.utexas.edu<mailto:[email protected]>

1 0

Re: [mpich-discuss] Compile issue with QMPI
by Zhou, Hui 05 Feb '22

05 Feb '22

Hi Amit, Could you try this pull request: https://github.com/pmodels/mpich/pull/5831/files? You could apply the patch to release tarball, then run ``` python3 maint/gen_binding_c.py ``` to fix all the generated code. -- Hui Zhou [https://avatars.githubusercontent.com/u/1496702?s=400&v=4]<https://github.com/pmodels/mpich/pull/5831/files> qmpi: chop MPICH_API_PUBLIC from the definition by hzhou · Pull Request #5831 · pmodels/mpich<https://github.com/pmodels/mpich/pull/5831/files> Pull Request Description MPICH_API_PUBLIC is accidentally leaked into definitions, which isn't allowed. Fixes #5829 [skip warnings] Author Checklist Provide Description Particularly focus on ... github.com ________________________________ From: Amit Ruhela via discuss <discuss(a)mpich.org> Sent: Friday, February 4, 2022 5:34 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Amit Ruhela <aruhela(a)tacc.utexas.edu> Subject: [mpich-discuss] Compile issue with QMPI Hi Mpich team, I see the following issue with the latest 4.0 release of MPICH while adding QMPI support at configure time? make[2]: Entering directory `/scratch1/05231/aruhela/libs/mpich/tar-mpich-4.0' CC src/binding/c/attr/lib_libmpi_la-attr_delete.lo src/binding/c/attr/attr_delete.c(63): error: attributes are not permitted in a function definition int QMPI_Attr_delete(QMPI_Context context, int tool_id, MPI_Comm comm, int keyval) MPICH_API_PUBLIC ^ compilation aborted for src/binding/c/attr/attr_delete.c (code 2) make[2]: *** [src/binding/c/attr/lib_libmpi_la-attr_delete.lo] Error 1 Any suggestions to resolve the build error? Amit Ruhela, Ph.D. HPC Software Tools Group Texas Advanced Computing Center The University of Texas at Austin Email: aruhela(a)tacc.utexas.edu<mailto:[email protected]>

1 0

Re: [mpich-discuss] Compile issue with QMPI
by Zhou, Hui 05 Feb '22

05 Feb '22

Hi Amit, Thanks for reporting. Tracked here: https://github.com/pmodels/mpich/issues/5829 We'll follow up. -- Hui Zhou [https://opengraph.githubassets.com/e4bc7b1f790349dcb375890726ef21718e853b6c…]<https://github.com/pmodels/mpich/issues/5829> qmpi: failed to build in mpich-4.0 · Issue #5829 · pmodels/mpich<https://github.com/pmodels/mpich/issues/5829> >From discuss mailinglist: [quote] Hi Mpich team, I see the following issue with the latest 4.0 release of MPICH while adding QMPI support at configure time? make[2]: Entering directory `/scratch1/0... github.com ________________________________ From: Amit Ruhela via discuss <discuss(a)mpich.org> Sent: Friday, February 4, 2022 5:34 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Amit Ruhela <aruhela(a)tacc.utexas.edu> Subject: [mpich-discuss] Compile issue with QMPI Hi Mpich team, I see the following issue with the latest 4.0 release of MPICH while adding QMPI support at configure time? make[2]: Entering directory `/scratch1/05231/aruhela/libs/mpich/tar-mpich-4.0' CC src/binding/c/attr/lib_libmpi_la-attr_delete.lo src/binding/c/attr/attr_delete.c(63): error: attributes are not permitted in a function definition int QMPI_Attr_delete(QMPI_Context context, int tool_id, MPI_Comm comm, int keyval) MPICH_API_PUBLIC ^ compilation aborted for src/binding/c/attr/attr_delete.c (code 2) make[2]: *** [src/binding/c/attr/lib_libmpi_la-attr_delete.lo] Error 1 Any suggestions to resolve the build error? Amit Ruhela, Ph.D. HPC Software Tools Group Texas Advanced Computing Center The University of Texas at Austin Email: aruhela(a)tacc.utexas.edu<mailto:[email protected]>

1 0

Compile issue with QMPI
by Amit Ruhela 04 Feb '22

04 Feb '22

Hi Mpich team, I see the following issue with the latest 4.0 release of MPICH while adding QMPI support at configure time? make[2]: Entering directory `/scratch1/05231/aruhela/libs/mpich/tar-mpich-4.0' CC src/binding/c/attr/lib_libmpi_la-attr_delete.lo src/binding/c/attr/attr_delete.c(63): error: attributes are not permitted in a function definition int QMPI_Attr_delete(QMPI_Context context, int tool_id, MPI_Comm comm, int keyval) MPICH_API_PUBLIC ^ compilation aborted for src/binding/c/attr/attr_delete.c (code 2) make[2]: *** [src/binding/c/attr/lib_libmpi_la-attr_delete.lo] Error 1 Any suggestions to resolve the build error? Amit Ruhela, Ph.D. HPC Software Tools Group Texas Advanced Computing Center The University of Texas at Austin Email: aruhela(a)tacc.utexas.edu<mailto:[email protected]>

1 0

Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries
by Raffenetti, Ken 04 Feb '22

04 Feb '22

:(. We need to link -lpmi2 instead of -lpmi. This really needs a patch in our configure script, but adding this to your configure is worth a shot: LIBS=-lpmi2 Ken On 2/4/22, 1:56 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: I added the CFLAGS argument and the configuration completed, but make ended with a link error. lib/.libs/libmpi.so: undefined reference to `PMI2_Abort' lib/.libs/libmpi.so: undefined reference to `PMI2_Info_GetJobAttr' lib/.libs/libmpi.so: undefined reference to `PMI2_Job_Spawn' lib/.libs/libmpi.so: undefined reference to `PMI2_Nameserv_publish' lib/.libs/libmpi.so: undefined reference to `PMI2_Finalize' lib/.libs/libmpi.so: undefined reference to `PMI2_KVS_Put' lib/.libs/libmpi.so: undefined reference to `PMI2_Info_GetNodeAttr' lib/.libs/libmpi.so: undefined reference to `PMI2_KVS_Get' lib/.libs/libmpi.so: undefined reference to `PMI2_KVS_Fence' lib/.libs/libmpi.so: undefined reference to `PMI2_Nameserv_unpublish' lib/.libs/libmpi.so: undefined reference to `PMI2_Info_PutNodeAttr' lib/.libs/libmpi.so: undefined reference to `PMI2_Job_GetId' lib/.libs/libmpi.so: undefined reference to `PMI2_Init' lib/.libs/libmpi.so: undefined reference to `PMI2_Nameserv_lookup' collect2: error: ld returned 1 exit status -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 4, 2022 1:23 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries I think I see a new issue. The Slurm website documentation says that their PMI library doesn't support PMI_Spawn_multiple from the PMI 1 API. We can try to force PMI 2 and see what happens. Try adding this to your configure line. CFLAGS=-DUSE_PMI2_API Ken On 2/4/22, 11:58 AM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Did that, and launched the job with "srun --mpi=none" and one of the processes failed when MPI_Comm_spawn was called. Note the : internal_Comm_spawn(101)......: MPI_Comm_spawn(command=NeedlesMpiMM, argv=0x226b030, maxprocs=1, info=0x9c000000, 0, MPI_COMM_SELF, intercomm=0x7ffffda9448c, array_of_errcodes=0x7ffffda94378) failed MPIDI_Comm_spawn_multiple(225): PMI_Spawn_multiple returned -1 The other process failed when MPI_Barrier was called: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed MPI manager 1 threw exception: MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 4, 2022 11:42 AM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries Yes, you should also use --with-pm=none. If using mpicc to build your application, you should not have to add -lpmi. The script will handle it for you. If using another method, you might have to add it. These days with shared libraries, linkers are often able to manage "inter-library" dependencies just fine. Static builds are a different story. Ken On 2/4/22, 11:35 AM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, >> configure --with-slurm=/opt/slurm --with-pmi=slurm That is similar to your first suggestion below. With the above, do I have to include --with-pm=none? I guess I also have to link my application with -lpmi, right? Thanks, Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 4, 2022 11:02 AM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries When running with srun you need to use the Slurm PMI library, not the embedded Simple PMI2 library. Simple PMI2 is API compatible, but uses a different wire protocol that the Slurm implementation. Try this instead: configure --with-slurm=/opt/slurm --with-pmi=slurm This will link the Slurm PMI library to MPICH. I do acknowledge how confusing this must be to users :). Probably a good FAQ topic for our Github discussions page. Ken On 2/3/22, 7:00 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I'm trying to build MPICH 4.0 in several ways, one of which will be what you suggested below. For this particular attempt suggested by the Slurm MPI guide, I built it with configure --with-slurm=/opt/slurm --with-pmi=pmi2/simple <etc> and invoked it with srun --mpi=pmi2 <etc> The job is crashing with this message. Any idea what is wrong? slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key <99>èþ^? in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key ´2¾ÿ^? in req slurmstepd: error: mpi/pmi2: no value for key ; in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: *** STEP 52227.0 ON n001 CANCELLED AT 2022-02-03T18:48:02 *** -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 3:15 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries On 1/28/22, 2:22 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I confirmed that MPI_Comm_spawn fails completely if I build MPICH without the PMI2 option. Dang, I thought that would work :(. Looking at the Slurm documentation https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.sch… it states "All MPI_comm_spawn work fine now going through hydra's PMI 1.1 interface." The full quote is below for reference. 1) how do I build MPICH to support hydra's PMI 1.1 interface? That is the default, so no extra configuration should be needed. One thing I notice in your log output is that the Slurm envvars seems to have changed name from what we have in our source. E.g. SLURM_JOB_NODELIST vs. SLURM_NODELIST. Do your initial processes launch on the right nodes? 2) Can you offer any guesses on how to build Slurm to do the same? (I realize this isn't a Slurm forum 😊) Hopefully you don't need to rebuild Slurm to do it. What you could try is configuring the Slurm PMI library when building MPICH. Add "--with-pm=none --with-pmi=slurm --with-slurm=<path/to/install>". Then use srun instead of mpiexec and see how it goes. Ken

1 0