Hi Kurt, As Ken mentioned, there is often the complication or confusion of several distinct combinations of setup. Mailinglist is especially bad at resolving such issues as we try different suggestions and the feedback is often all mixed up. Could you check https://github.com/pmodels/mpich/issues/5835 and provide (or re-provide) relevant information there? Particularly useful information are: how mpich is configured, how the job is invocated, the failure symptom, and the debug output by using mpiexec -verbose​. As Ken listed, there are three different scenarios. Each should work. It probably will help if we focus on one scenario at a time. -- Hui Zhou [https://opengraph.githubassets.com/f73eca0aa1fd8cd3bc03c57525990e22e36e447bdc6b0393f863e131273f0563/pmodels/mpich/issues/5835]<https://github.com/pmodels/mpich/issues/5835> MPI_Comm_spawn in Slurm environment · Issue #5835 · pmodels/mpich<https://github.com/pmodels/mpich/issues/5835> Originated from user email https://lists.mpich.org/pipermail/discuss/2022-January/006360.html. MPICH + Hydra + PMI1 (crashes) MPICH + Hydra + PMI2 (works but ignores "hosts" info key) MPI... github.com ________________________________ From: Raffenetti, Ken via discuss <[email protected]> Sent: Monday, February 7, 2022 4:28 PM To: Mccall, Kurt E. (MSFC-EV41) <[email protected]>; [email protected] <[email protected]> Cc: Raffenetti, Ken <[email protected]> Subject: Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries Darn. I'm creating an issue to track this since it will likely take some time and effort to investigate each configuration. https://github.com/pmodels/mpich/issues/5835 Ken On 2/7/22, 12:41 PM, "Mccall, Kurt E. (MSFC-EV41)" <[email protected]> wrote: Ken, To review, I configured as follows: $ configure CFLAGS=-DUSE_PMI2_API LIBS=-lpmi2 --with-pm=none --with-pmi=slurm --with-slurm=/opt/slurm < ...> and ran srun with the argument --mpi=pmi2. The job is stil segfaulting in MPI_Comm_spawn in one process and returning an error from MPI_Barrier in the other. Error messages below: The MPI_Comm_spawn error: backtrace for error: backtrace after receiving signal SIGSEGV: /home/kmccall/Needles2/./NeedlesMpiMM() [0x45ab36] /lib64/libpthread.so.0(+0x12c20) [0x7f4833387c20] /lib64/libc.so.6(+0x15d6b7) [0x7f483310d6b7] /lib64/libc.so.6(__strdup+0x12) [0x7f4833039802] /lib64/libpmi2.so.0(+0x171c) [0x7f483294371c] /lib64/libpmi2.so.0(+0x185e) [0x7f483294385e] /lib64/libpmi2.so.0(PMI2_Job_Spawn+0x1a7) [0x7f48329453d8] /home/kmccall/mpich-slurm-install-4.0_2/lib/libmpi.so.12(+0x23a7db) [0x7f4834e0a7db] /home/kmccall/mpich-slurm-install-4.0_2/lib/libmpi.so.12(+0x1fc805) [0x7f4834dcc805] /home/kmccall/mpich-slurm-install-4.0_2/lib/libmpi.so.12(MPI_Comm_spawn+0x507) [0x7f4834cea9f7] The MPI_Barrier error: MPI_Barrier returned the error MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed Thanks, Kurt -----Original Message----- From: Raffenetti, Ken <[email protected]> Sent: Friday, February 4, 2022 4:08 PM To: Mccall, Kurt E. (MSFC-EV41) <[email protected]>; [email protected] Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries :(. We need to link -lpmi2 instead of -lpmi. This really needs a patch in our configure script, but adding this to your configure is worth a shot: LIBS=-lpmi2 Ken On 2/4/22, 1:56 PM, "Mccall, Kurt E. (MSFC-EV41)" <[email protected]> wrote: I added the CFLAGS argument and the configuration completed, but make ended with a link error. lib/.libs/libmpi.so: undefined reference to `PMI2_Abort' lib/.libs/libmpi.so: undefined reference to `PMI2_Info_GetJobAttr' lib/.libs/libmpi.so: undefined reference to `PMI2_Job_Spawn' lib/.libs/libmpi.so: undefined reference to `PMI2_Nameserv_publish' lib/.libs/libmpi.so: undefined reference to `PMI2_Finalize' lib/.libs/libmpi.so: undefined reference to `PMI2_KVS_Put' lib/.libs/libmpi.so: undefined reference to `PMI2_Info_GetNodeAttr' lib/.libs/libmpi.so: undefined reference to `PMI2_KVS_Get' lib/.libs/libmpi.so: undefined reference to `PMI2_KVS_Fence' lib/.libs/libmpi.so: undefined reference to `PMI2_Nameserv_unpublish' lib/.libs/libmpi.so: undefined reference to `PMI2_Info_PutNodeAttr' lib/.libs/libmpi.so: undefined reference to `PMI2_Job_GetId' lib/.libs/libmpi.so: undefined reference to `PMI2_Init' lib/.libs/libmpi.so: undefined reference to `PMI2_Nameserv_lookup' collect2: error: ld returned 1 exit status -----Original Message----- From: Raffenetti, Ken <[email protected]> Sent: Friday, February 4, 2022 1:23 PM To: Mccall, Kurt E. (MSFC-EV41) <[email protected]>; [email protected] Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries I think I see a new issue. The Slurm website documentation says that their PMI library doesn't support PMI_Spawn_multiple from the PMI 1 API. We can try to force PMI 2 and see what happens. Try adding this to your configure line. CFLAGS=-DUSE_PMI2_API Ken On 2/4/22, 11:58 AM, "Mccall, Kurt E. (MSFC-EV41)" <[email protected]> wrote: Did that, and launched the job with "srun --mpi=none" and one of the processes failed when MPI_Comm_spawn was called. Note the : internal_Comm_spawn(101)......: MPI_Comm_spawn(command=NeedlesMpiMM, argv=0x226b030, maxprocs=1, info=0x9c000000, 0, MPI_COMM_SELF, intercomm=0x7ffffda9448c, array_of_errcodes=0x7ffffda94378) failed MPIDI_Comm_spawn_multiple(225): PMI_Spawn_multiple returned -1 The other process failed when MPI_Barrier was called: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed MPI manager 1 threw exception: MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed -----Original Message----- From: Raffenetti, Ken <[email protected]> Sent: Friday, February 4, 2022 11:42 AM To: Mccall, Kurt E. (MSFC-EV41) <[email protected]>; [email protected] Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries Yes, you should also use --with-pm=none. If using mpicc to build your application, you should not have to add -lpmi. The script will handle it for you. If using another method, you might have to add it. These days with shared libraries, linkers are often able to manage "inter-library" dependencies just fine. Static builds are a different story. Ken On 2/4/22, 11:35 AM, "Mccall, Kurt E. (MSFC-EV41)" <[email protected]> wrote: Ken, >> configure --with-slurm=/opt/slurm --with-pmi=slurm That is similar to your first suggestion below. With the above, do I have to include --with-pm=none? I guess I also have to link my application with -lpmi, right? Thanks, Kurt -----Original Message----- From: Raffenetti, Ken <[email protected]> Sent: Friday, February 4, 2022 11:02 AM To: Mccall, Kurt E. (MSFC-EV41) <[email protected]>; [email protected] Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries When running with srun you need to use the Slurm PMI library, not the embedded Simple PMI2 library. Simple PMI2 is API compatible, but uses a different wire protocol that the Slurm implementation. Try this instead: configure --with-slurm=/opt/slurm --with-pmi=slurm This will link the Slurm PMI library to MPICH. I do acknowledge how confusing this must be to users :). Probably a good FAQ topic for our Github discussions page. Ken On 2/3/22, 7:00 PM, "Mccall, Kurt E. (MSFC-EV41)" <[email protected]> wrote: Ken, I'm trying to build MPICH 4.0 in several ways, one of which will be what you suggested below. For this particular attempt suggested by the Slurm MPI guide, I built it with configure --with-slurm=/opt/slurm --with-pmi=pmi2/simple <etc> and invoked it with srun --mpi=pmi2 <etc> The job is crashing with this message. Any idea what is wrong? slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key <99>è­þ^? in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key ´2¾ÿ^? in req slurmstepd: error: mpi/pmi2: no value for key ; in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: *** STEP 52227.0 ON n001 CANCELLED AT 2022-02-03T18:48:02 *** -----Original Message----- From: Raffenetti, Ken <[email protected]> Sent: Friday, January 28, 2022 3:15 PM To: Mccall, Kurt E. (MSFC-EV41) <[email protected]>; [email protected] Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries On 1/28/22, 2:22 PM, "Mccall, Kurt E. (MSFC-EV41)" <[email protected]> wrote: Ken, I confirmed that MPI_Comm_spawn fails completely if I build MPICH without the PMI2 option. Dang, I thought that would work :(. Looking at the Slurm documentation https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Fmpi_guide.html%23intel_mpiexec_hydra&data=04%7C01%7Ckurt.e.mccall%40nasa.gov%7C9249f74763f1428b909908d9e82adeef%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637796093091225985%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=dcpAlSu4FLsL2o39rDH0t1jyzq4AYSOQlZXMReZmmJU%3D&reserved=0 it states "All MPI_comm_spawn work fine now going through hydra's PMI 1.1 interface." The full quote is below for reference. 1) how do I build MPICH to support hydra's PMI 1.1 interface? That is the default, so no extra configuration should be needed. One thing I notice in your log output is that the Slurm envvars seems to have changed name from what we have in our source. E.g. SLURM_JOB_NODELIST vs. SLURM_NODELIST. Do your initial processes launch on the right nodes? 2) Can you offer any guesses on how to build Slurm to do the same? (I realize this isn't a Slurm forum 😊) Hopefully you don't need to rebuild Slurm to do it. What you could try is configuring the Slurm PMI library when building MPICH. Add "--with-pm=none --with-pmi=slurm --with-slurm=<path/to/install>". Then use srun instead of mpiexec and see how it goes. Ken _______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss