February 2022 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries
by Mccall, Kurt E. (MSFC-EV41) 04 Feb '22

04 Feb '22

I added the CFLAGS argument and the configuration completed, but make ended with a link error. lib/.libs/libmpi.so: undefined reference to `PMI2_Abort' lib/.libs/libmpi.so: undefined reference to `PMI2_Info_GetJobAttr' lib/.libs/libmpi.so: undefined reference to `PMI2_Job_Spawn' lib/.libs/libmpi.so: undefined reference to `PMI2_Nameserv_publish' lib/.libs/libmpi.so: undefined reference to `PMI2_Finalize' lib/.libs/libmpi.so: undefined reference to `PMI2_KVS_Put' lib/.libs/libmpi.so: undefined reference to `PMI2_Info_GetNodeAttr' lib/.libs/libmpi.so: undefined reference to `PMI2_KVS_Get' lib/.libs/libmpi.so: undefined reference to `PMI2_KVS_Fence' lib/.libs/libmpi.so: undefined reference to `PMI2_Nameserv_unpublish' lib/.libs/libmpi.so: undefined reference to `PMI2_Info_PutNodeAttr' lib/.libs/libmpi.so: undefined reference to `PMI2_Job_GetId' lib/.libs/libmpi.so: undefined reference to `PMI2_Init' lib/.libs/libmpi.so: undefined reference to `PMI2_Nameserv_lookup' collect2: error: ld returned 1 exit status -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 4, 2022 1:23 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries I think I see a new issue. The Slurm website documentation says that their PMI library doesn't support PMI_Spawn_multiple from the PMI 1 API. We can try to force PMI 2 and see what happens. Try adding this to your configure line. CFLAGS=-DUSE_PMI2_API Ken On 2/4/22, 11:58 AM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Did that, and launched the job with "srun --mpi=none" and one of the processes failed when MPI_Comm_spawn was called. Note the : internal_Comm_spawn(101)......: MPI_Comm_spawn(command=NeedlesMpiMM, argv=0x226b030, maxprocs=1, info=0x9c000000, 0, MPI_COMM_SELF, intercomm=0x7ffffda9448c, array_of_errcodes=0x7ffffda94378) failed MPIDI_Comm_spawn_multiple(225): PMI_Spawn_multiple returned -1 The other process failed when MPI_Barrier was called: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed MPI manager 1 threw exception: MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 4, 2022 11:42 AM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries Yes, you should also use --with-pm=none. If using mpicc to build your application, you should not have to add -lpmi. The script will handle it for you. If using another method, you might have to add it. These days with shared libraries, linkers are often able to manage "inter-library" dependencies just fine. Static builds are a different story. Ken On 2/4/22, 11:35 AM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, >> configure --with-slurm=/opt/slurm --with-pmi=slurm That is similar to your first suggestion below. With the above, do I have to include --with-pm=none? I guess I also have to link my application with -lpmi, right? Thanks, Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 4, 2022 11:02 AM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries When running with srun you need to use the Slurm PMI library, not the embedded Simple PMI2 library. Simple PMI2 is API compatible, but uses a different wire protocol that the Slurm implementation. Try this instead: configure --with-slurm=/opt/slurm --with-pmi=slurm This will link the Slurm PMI library to MPICH. I do acknowledge how confusing this must be to users :). Probably a good FAQ topic for our Github discussions page. Ken On 2/3/22, 7:00 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I'm trying to build MPICH 4.0 in several ways, one of which will be what you suggested below. For this particular attempt suggested by the Slurm MPI guide, I built it with configure --with-slurm=/opt/slurm --with-pmi=pmi2/simple <etc> and invoked it with srun --mpi=pmi2 <etc> The job is crashing with this message. Any idea what is wrong? slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key <99>èþ^? in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key ´2¾ÿ^? in req slurmstepd: error: mpi/pmi2: no value for key ; in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: *** STEP 52227.0 ON n001 CANCELLED AT 2022-02-03T18:48:02 *** -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 3:15 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries On 1/28/22, 2:22 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I confirmed that MPI_Comm_spawn fails completely if I build MPICH without the PMI2 option. Dang, I thought that would work :(. Looking at the Slurm documentation https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.sch… it states "All MPI_comm_spawn work fine now going through hydra's PMI 1.1 interface." The full quote is below for reference. 1) how do I build MPICH to support hydra's PMI 1.1 interface? That is the default, so no extra configuration should be needed. One thing I notice in your log output is that the Slurm envvars seems to have changed name from what we have in our source. E.g. SLURM_JOB_NODELIST vs. SLURM_NODELIST. Do your initial processes launch on the right nodes? 2) Can you offer any guesses on how to build Slurm to do the same? (I realize this isn't a Slurm forum 😊) Hopefully you don't need to rebuild Slurm to do it. What you could try is configuring the Slurm PMI library when building MPICH. Add "--with-pm=none --with-pmi=slurm --with-slurm=<path/to/install>". Then use srun instead of mpiexec and see how it goes. Ken

1 0

Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries
by Raffenetti, Ken 04 Feb '22

04 Feb '22

I think I see a new issue. The Slurm website documentation says that their PMI library doesn't support PMI_Spawn_multiple from the PMI 1 API. We can try to force PMI 2 and see what happens. Try adding this to your configure line. CFLAGS=-DUSE_PMI2_API Ken On 2/4/22, 11:58 AM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Did that, and launched the job with "srun --mpi=none" and one of the processes failed when MPI_Comm_spawn was called. Note the : internal_Comm_spawn(101)......: MPI_Comm_spawn(command=NeedlesMpiMM, argv=0x226b030, maxprocs=1, info=0x9c000000, 0, MPI_COMM_SELF, intercomm=0x7ffffda9448c, array_of_errcodes=0x7ffffda94378) failed MPIDI_Comm_spawn_multiple(225): PMI_Spawn_multiple returned -1 The other process failed when MPI_Barrier was called: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed MPI manager 1 threw exception: MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 4, 2022 11:42 AM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries Yes, you should also use --with-pm=none. If using mpicc to build your application, you should not have to add -lpmi. The script will handle it for you. If using another method, you might have to add it. These days with shared libraries, linkers are often able to manage "inter-library" dependencies just fine. Static builds are a different story. Ken On 2/4/22, 11:35 AM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, >> configure --with-slurm=/opt/slurm --with-pmi=slurm That is similar to your first suggestion below. With the above, do I have to include --with-pm=none? I guess I also have to link my application with -lpmi, right? Thanks, Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 4, 2022 11:02 AM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries When running with srun you need to use the Slurm PMI library, not the embedded Simple PMI2 library. Simple PMI2 is API compatible, but uses a different wire protocol that the Slurm implementation. Try this instead: configure --with-slurm=/opt/slurm --with-pmi=slurm This will link the Slurm PMI library to MPICH. I do acknowledge how confusing this must be to users :). Probably a good FAQ topic for our Github discussions page. Ken On 2/3/22, 7:00 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I'm trying to build MPICH 4.0 in several ways, one of which will be what you suggested below. For this particular attempt suggested by the Slurm MPI guide, I built it with configure --with-slurm=/opt/slurm --with-pmi=pmi2/simple <etc> and invoked it with srun --mpi=pmi2 <etc> The job is crashing with this message. Any idea what is wrong? slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key <99>èþ^? in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key ´2¾ÿ^? in req slurmstepd: error: mpi/pmi2: no value for key ; in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: *** STEP 52227.0 ON n001 CANCELLED AT 2022-02-03T18:48:02 *** -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 3:15 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries On 1/28/22, 2:22 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I confirmed that MPI_Comm_spawn fails completely if I build MPICH without the PMI2 option. Dang, I thought that would work :(. Looking at the Slurm documentation https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.sch… it states "All MPI_comm_spawn work fine now going through hydra's PMI 1.1 interface." The full quote is below for reference. 1) how do I build MPICH to support hydra's PMI 1.1 interface? That is the default, so no extra configuration should be needed. One thing I notice in your log output is that the Slurm envvars seems to have changed name from what we have in our source. E.g. SLURM_JOB_NODELIST vs. SLURM_NODELIST. Do your initial processes launch on the right nodes? 2) Can you offer any guesses on how to build Slurm to do the same? (I realize this isn't a Slurm forum 😊) Hopefully you don't need to rebuild Slurm to do it. What you could try is configuring the Slurm PMI library when building MPICH. Add "--with-pm=none --with-pmi=slurm --with-slurm=<path/to/install>". Then use srun instead of mpiexec and see how it goes. Ken

1 0

Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries
by Mccall, Kurt E. (MSFC-EV41) 04 Feb '22

04 Feb '22

Did that, and launched the job with "srun --mpi=none" and one of the processes failed when MPI_Comm_spawn was called. Note the : internal_Comm_spawn(101)......: MPI_Comm_spawn(command=NeedlesMpiMM, argv=0x226b030, maxprocs=1, info=0x9c000000, 0, MPI_COMM_SELF, intercomm=0x7ffffda9448c, array_of_errcodes=0x7ffffda94378) failed MPIDI_Comm_spawn_multiple(225): PMI_Spawn_multiple returned -1 The other process failed when MPI_Barrier was called: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed MPI manager 1 threw exception: MPI runtime error: Unknown error class, error stack: internal_Barrier(84).......................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(91)......................: MPIR_Barrier_allcomm_auto(45)..............: MPIR_Barrier_intra_dissemination(39).......: MPIDI_CH3U_Complete_posted_with_error(1090): Process failed -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 4, 2022 11:42 AM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries Yes, you should also use --with-pm=none. If using mpicc to build your application, you should not have to add -lpmi. The script will handle it for you. If using another method, you might have to add it. These days with shared libraries, linkers are often able to manage "inter-library" dependencies just fine. Static builds are a different story. Ken On 2/4/22, 11:35 AM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, >> configure --with-slurm=/opt/slurm --with-pmi=slurm That is similar to your first suggestion below. With the above, do I have to include --with-pm=none? I guess I also have to link my application with -lpmi, right? Thanks, Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 4, 2022 11:02 AM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries When running with srun you need to use the Slurm PMI library, not the embedded Simple PMI2 library. Simple PMI2 is API compatible, but uses a different wire protocol that the Slurm implementation. Try this instead: configure --with-slurm=/opt/slurm --with-pmi=slurm This will link the Slurm PMI library to MPICH. I do acknowledge how confusing this must be to users :). Probably a good FAQ topic for our Github discussions page. Ken On 2/3/22, 7:00 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I'm trying to build MPICH 4.0 in several ways, one of which will be what you suggested below. For this particular attempt suggested by the Slurm MPI guide, I built it with configure --with-slurm=/opt/slurm --with-pmi=pmi2/simple <etc> and invoked it with srun --mpi=pmi2 <etc> The job is crashing with this message. Any idea what is wrong? slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key <99>èþ^? in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key ´2¾ÿ^? in req slurmstepd: error: mpi/pmi2: no value for key ; in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: *** STEP 52227.0 ON n001 CANCELLED AT 2022-02-03T18:48:02 *** -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 3:15 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries On 1/28/22, 2:22 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I confirmed that MPI_Comm_spawn fails completely if I build MPICH without the PMI2 option. Dang, I thought that would work :(. Looking at the Slurm documentation https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.sch… it states "All MPI_comm_spawn work fine now going through hydra's PMI 1.1 interface." The full quote is below for reference. 1) how do I build MPICH to support hydra's PMI 1.1 interface? That is the default, so no extra configuration should be needed. One thing I notice in your log output is that the Slurm envvars seems to have changed name from what we have in our source. E.g. SLURM_JOB_NODELIST vs. SLURM_NODELIST. Do your initial processes launch on the right nodes? 2) Can you offer any guesses on how to build Slurm to do the same? (I realize this isn't a Slurm forum 😊) Hopefully you don't need to rebuild Slurm to do it. What you could try is configuring the Slurm PMI library when building MPICH. Add "--with-pm=none --with-pmi=slurm --with-slurm=<path/to/install>". Then use srun instead of mpiexec and see how it goes. Ken

1 0

Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries
by Raffenetti, Ken 04 Feb '22

04 Feb '22

Yes, you should also use --with-pm=none. If using mpicc to build your application, you should not have to add -lpmi. The script will handle it for you. If using another method, you might have to add it. These days with shared libraries, linkers are often able to manage "inter-library" dependencies just fine. Static builds are a different story. Ken On 2/4/22, 11:35 AM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, >> configure --with-slurm=/opt/slurm --with-pmi=slurm That is similar to your first suggestion below. With the above, do I have to include --with-pm=none? I guess I also have to link my application with -lpmi, right? Thanks, Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 4, 2022 11:02 AM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries When running with srun you need to use the Slurm PMI library, not the embedded Simple PMI2 library. Simple PMI2 is API compatible, but uses a different wire protocol that the Slurm implementation. Try this instead: configure --with-slurm=/opt/slurm --with-pmi=slurm This will link the Slurm PMI library to MPICH. I do acknowledge how confusing this must be to users :). Probably a good FAQ topic for our Github discussions page. Ken On 2/3/22, 7:00 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I'm trying to build MPICH 4.0 in several ways, one of which will be what you suggested below. For this particular attempt suggested by the Slurm MPI guide, I built it with configure --with-slurm=/opt/slurm --with-pmi=pmi2/simple <etc> and invoked it with srun --mpi=pmi2 <etc> The job is crashing with this message. Any idea what is wrong? slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key <99>èþ^? in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key ´2¾ÿ^? in req slurmstepd: error: mpi/pmi2: no value for key ; in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: *** STEP 52227.0 ON n001 CANCELLED AT 2022-02-03T18:48:02 *** -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 3:15 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries On 1/28/22, 2:22 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I confirmed that MPI_Comm_spawn fails completely if I build MPICH without the PMI2 option. Dang, I thought that would work :(. Looking at the Slurm documentation https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.sch… it states "All MPI_comm_spawn work fine now going through hydra's PMI 1.1 interface." The full quote is below for reference. 1) how do I build MPICH to support hydra's PMI 1.1 interface? That is the default, so no extra configuration should be needed. One thing I notice in your log output is that the Slurm envvars seems to have changed name from what we have in our source. E.g. SLURM_JOB_NODELIST vs. SLURM_NODELIST. Do your initial processes launch on the right nodes? 2) Can you offer any guesses on how to build Slurm to do the same? (I realize this isn't a Slurm forum 😊) Hopefully you don't need to rebuild Slurm to do it. What you could try is configuring the Slurm PMI library when building MPICH. Add "--with-pm=none --with-pmi=slurm --with-slurm=<path/to/install>". Then use srun instead of mpiexec and see how it goes. Ken

1 0

Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries
by Mccall, Kurt E. (MSFC-EV41) 04 Feb '22

04 Feb '22

Ken, >> configure --with-slurm=/opt/slurm --with-pmi=slurm That is similar to your first suggestion below. With the above, do I have to include --with-pm=none? I guess I also have to link my application with -lpmi, right? Thanks, Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 4, 2022 11:02 AM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries When running with srun you need to use the Slurm PMI library, not the embedded Simple PMI2 library. Simple PMI2 is API compatible, but uses a different wire protocol that the Slurm implementation. Try this instead: configure --with-slurm=/opt/slurm --with-pmi=slurm This will link the Slurm PMI library to MPICH. I do acknowledge how confusing this must be to users :). Probably a good FAQ topic for our Github discussions page. Ken On 2/3/22, 7:00 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I'm trying to build MPICH 4.0 in several ways, one of which will be what you suggested below. For this particular attempt suggested by the Slurm MPI guide, I built it with configure --with-slurm=/opt/slurm --with-pmi=pmi2/simple <etc> and invoked it with srun --mpi=pmi2 <etc> The job is crashing with this message. Any idea what is wrong? slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key <99>èþ^? in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key ´2¾ÿ^? in req slurmstepd: error: mpi/pmi2: no value for key ; in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: *** STEP 52227.0 ON n001 CANCELLED AT 2022-02-03T18:48:02 *** -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 3:15 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries On 1/28/22, 2:22 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I confirmed that MPI_Comm_spawn fails completely if I build MPICH without the PMI2 option. Dang, I thought that would work :(. Looking at the Slurm documentation https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.sch… it states "All MPI_comm_spawn work fine now going through hydra's PMI 1.1 interface." The full quote is below for reference. 1) how do I build MPICH to support hydra's PMI 1.1 interface? That is the default, so no extra configuration should be needed. One thing I notice in your log output is that the Slurm envvars seems to have changed name from what we have in our source. E.g. SLURM_JOB_NODELIST vs. SLURM_NODELIST. Do your initial processes launch on the right nodes? 2) Can you offer any guesses on how to build Slurm to do the same? (I realize this isn't a Slurm forum 😊) Hopefully you don't need to rebuild Slurm to do it. What you could try is configuring the Slurm PMI library when building MPICH. Add "--with-pm=none --with-pmi=slurm --with-slurm=<path/to/install>". Then use srun instead of mpiexec and see how it goes. Ken

1 0

Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries
by Raffenetti, Ken 04 Feb '22

04 Feb '22

When running with srun you need to use the Slurm PMI library, not the embedded Simple PMI2 library. Simple PMI2 is API compatible, but uses a different wire protocol that the Slurm implementation. Try this instead: configure --with-slurm=/opt/slurm --with-pmi=slurm This will link the Slurm PMI library to MPICH. I do acknowledge how confusing this must be to users :). Probably a good FAQ topic for our Github discussions page. Ken On 2/3/22, 7:00 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I'm trying to build MPICH 4.0 in several ways, one of which will be what you suggested below. For this particular attempt suggested by the Slurm MPI guide, I built it with configure --with-slurm=/opt/slurm --with-pmi=pmi2/simple <etc> and invoked it with srun --mpi=pmi2 <etc> The job is crashing with this message. Any idea what is wrong? slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key <99>èþ^? in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key ´2¾ÿ^? in req slurmstepd: error: mpi/pmi2: no value for key ; in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: *** STEP 52227.0 ON n001 CANCELLED AT 2022-02-03T18:48:02 *** -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 3:15 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries On 1/28/22, 2:22 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I confirmed that MPI_Comm_spawn fails completely if I build MPICH without the PMI2 option. Dang, I thought that would work :(. Looking at the Slurm documentation https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.sch… it states "All MPI_comm_spawn work fine now going through hydra's PMI 1.1 interface." The full quote is below for reference. 1) how do I build MPICH to support hydra's PMI 1.1 interface? That is the default, so no extra configuration should be needed. One thing I notice in your log output is that the Slurm envvars seems to have changed name from what we have in our source. E.g. SLURM_JOB_NODELIST vs. SLURM_NODELIST. Do your initial processes launch on the right nodes? 2) Can you offer any guesses on how to build Slurm to do the same? (I realize this isn't a Slurm forum 😊) Hopefully you don't need to rebuild Slurm to do it. What you could try is configuring the Slurm PMI library when building MPICH. Add "--with-pm=none --with-pmi=slurm --with-slurm=<path/to/install>". Then use srun instead of mpiexec and see how it goes. Ken

1 0

Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries
by Mccall, Kurt E. (MSFC-EV41) 03 Feb '22

03 Feb '22

Ken, I'm trying to build MPICH 4.0 in several ways, one of which will be what you suggested below. For this particular attempt suggested by the Slurm MPI guide, I built it with configure --with-slurm=/opt/slurm --with-pmi=pmi2/simple <etc> and invoked it with srun --mpi=pmi2 <etc> The job is crashing with this message. Any idea what is wrong? slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key <99>èþ^? in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key ´2¾ÿ^? in req slurmstepd: error: mpi/pmi2: no value for key ; in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: *** STEP 52227.0 ON n001 CANCELLED AT 2022-02-03T18:48:02 *** -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 3:15 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries On 1/28/22, 2:22 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I confirmed that MPI_Comm_spawn fails completely if I build MPICH without the PMI2 option. Dang, I thought that would work :(. Looking at the Slurm documentation https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.sch… it states "All MPI_comm_spawn work fine now going through hydra's PMI 1.1 interface." The full quote is below for reference. 1) how do I build MPICH to support hydra's PMI 1.1 interface? That is the default, so no extra configuration should be needed. One thing I notice in your log output is that the Slurm envvars seems to have changed name from what we have in our source. E.g. SLURM_JOB_NODELIST vs. SLURM_NODELIST. Do your initial processes launch on the right nodes? 2) Can you offer any guesses on how to build Slurm to do the same? (I realize this isn't a Slurm forum 😊) Hopefully you don't need to rebuild Slurm to do it. What you could try is configuring the Slurm PMI library when building MPICH. Add "--with-pm=none --with-pmi=slurm --with-slurm=<path/to/install>". Then use srun instead of mpiexec and see how it goes. Ken

1 0

Re: [mpich-discuss] MPI_Cancel + MPI_Wait stalls when using ch4, not when using ch3
by Edric Ellis 02 Feb '22

02 Feb '22

Ok, thanks for explaining that. I think it should be possible for us to arrange things to avoid calling MPI_Cancel on send requests. We do this currently because we’re using MPI communication in an interactive environment where the user can tell us to stop at any time – the reproduction steps I posted were rather simplified compared to the actual case we were hitting. We already have mechanisms in place to “flush” messages at the receiving side when necessary, so that covers part of what we’ll need. Cheers, Edric. From: Zhou, Hui via discuss <discuss(a)mpich.org> Sent: 02 February 2022 00:16 To: discuss(a)mpich.org Cc: Zhou, Hui <zhouh(a)anl.gov>; Edric Ellis <eellis(a)mathworks.com> Subject: Re: [mpich-discuss] MPI_Cancel + MPI_Wait stalls when using ch4, not when using ch3 Hi Edric, After some pondering, we are closing https://github.com/pmodels/mpich/issues/5775<https://github.com/pmodels/mpich/issues/5775> as wontfix. For the following reasons: * Cancelling send is deprecated in MPI-4. * The potential fix will be very messy due to current design. * On the other hand, it is probably easier to work around in application. Example workaround could be to replace MPI_Issend with `MPI_Isend` plus separate MPI_Irecv for acknowledgement. You only need do this for the MPI_Issend that you may cancel, and I think it is a small inconvenience for more predictable behavior. Internally an MPI_Issend is not implemented as an MPI_Isend + MPI_Irecv to be more efficient and to exploit hardware acceleration where we can, but that also made cancelling it very complicated. If you have strong reasons to fix this issue, for example, no feasible work around, let's discuss. -- Hui Zhou ________________________________ From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Wednesday, January 19, 2022 12:28 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Edric Ellis <eellis(a)mathworks.com<mailto:[email protected]>> Subject: Re: MPI_Cancel + MPI_Wait stalls when using ch4, not when using ch3 Thanks for reporting. We'll fix this. Tracking issue: https://github.com/pmodels/mpich/issues/5775<https://github.com/pmodels/mpich/issues/5775> [https://opengraph.githubassets.com/abeb43664f034115c44e99f3e273e49dbfdeb9eb…]<https://github.com/pmodels/mpich/issues/5775> ch4: MPI_Cancel stalls MPI_Wait Issue #5775 pmodels/mpich<https://github.com/pmodels/mpich/issues/5775> Reported on discuss mailinglist: ---- quote ---- Hi, Running one of our test programs using MPICH 3.4.3 and ch4:ofi, I notice that MPI_Wait on an MPI_Request that has been MPI_Cancelled never compl... github.com ________________________________ From: Edric Ellis via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Wednesday, January 19, 2022 11:21 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Edric Ellis <eellis(a)mathworks.com<mailto:[email protected]>> Subject: [mpich-discuss] MPI_Cancel + MPI_Wait stalls when using ch4, not when using ch3 Hi, Running one of our test programs using MPICH 3.4.3 and ch4:ofi, I notice that MPI_Wait on an MPI_Request that has been MPI_Cancelled never completes (it does when using ch3). (The documentation for MPI_Cancel states "If a communication is marked for cancellation, then a MPI_WAIT call for that communication is guaranteed to return, irrespective of the activities of other processes (i.e., MPI_WAIT behaves as a local function)") Here's a simple example: #include <mpi.h> #include <stdio.h> #include <stdlib.h> void check(int const value) { if (value != MPI_SUCCESS) { fprintf(stderr, "Failed.\n"); exit(1); } } int main(int argc, char** argv) { MPI_Request r1; int payload = 42; int result; check(MPI_Init(0,0)); check(MPI_Issend(&payload, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &r1)); check(MPI_Test(&r1, &result, MPI_STATUS_IGNORE)); fprintf(stdout, "MPI_Test result: %d\n", result); check(MPI_Cancel(&r1)); check(MPI_Wait(&r1, MPI_STATUS_IGNORE)); MPI_Finalize(); return 0; } This stalls in MPI_Wait when executed using "mpiexec -n 1 ./a.out". Cheers, Edric. _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss<https://lists.mpich.org/mailman/listinfo/discuss>

1 0

Re: [mpich-discuss] MPI_Cancel + MPI_Wait stalls when using ch4, not when using ch3
by Jeff Hammond 01 Feb '22

01 Feb '22

Instead of stalling, it would be better to detect send cancel and throw an error. Jeff On Wed, Feb 2, 2022 at 2:16 AM Zhou, Hui via discuss <discuss(a)mpich.org> wrote: > Hi Edric, > > After some pondering, we are closing > https://github.com/pmodels/mpich/issues/5775 as wontfix. For the > following reasons: > > > - Cancelling send is deprecated in MPI-4. > - The potential fix will be very messy due to current design. > - On the other hand, it is probably easier to work around in > application. > > Example workaround could be to replace MPI_Issend with `MPI_Isend` plus > separate MPI_Irecv for acknowledgement. You only need do this for the > MPI_Issend that you may cancel, and I think it is a small inconvenience > for more predictable behavior. Internally an MPI_Issend is not > implemented as an MPI_Isend + MPI_Irecv to be more efficient and to > exploit hardware acceleration where we can, but that also made cancelling > it very complicated. > > If you have strong reasons to fix this issue, for example, no feasible > work around, let's discuss. > > -- > Hui Zhou > > ------------------------------ > *From:* Zhou, Hui <zhouh(a)anl.gov> > *Sent:* Wednesday, January 19, 2022 12:28 PM > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* Edric Ellis <eellis(a)mathworks.com> > *Subject:* Re: MPI_Cancel + MPI_Wait stalls when using ch4, not when > using ch3 > > Thanks for reporting. We'll fix this. Tracking issue: > https://github.com/pmodels/mpich/issues/5775 > <https://github.com/pmodels/mpich/issues/5775> > ch4: MPI_Cancel stalls MPI_Wait · Issue #5775 · pmodels/mpich > <https://github.com/pmodels/mpich/issues/5775> > Reported on discuss mailinglist: ---- quote ---- Hi, Running one of our > test programs using MPICH 3.4.3 and ch4:ofi, I notice that MPI_Wait on an > MPI_Request that has been MPI_Cancelled never compl... > github.com > > ------------------------------ > *From:* Edric Ellis via discuss <discuss(a)mpich.org> > *Sent:* Wednesday, January 19, 2022 11:21 AM > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* Edric Ellis <eellis(a)mathworks.com> > *Subject:* [mpich-discuss] MPI_Cancel + MPI_Wait stalls when using ch4, > not when using ch3 > > Hi, > > Running one of our test programs using MPICH 3.4.3 and ch4:ofi, I notice > that MPI_Wait on an MPI_Request that has been MPI_Cancelled never completes > (it does when using ch3). (The documentation for MPI_Cancel states "If a > communication is marked for cancellation, then a MPI_WAIT call for that > communication is guaranteed to return, irrespective of the activities of > other processes (i.e., MPI_WAIT behaves as a local function)") > > Here's a simple example: > > #include <mpi.h> > #include <stdio.h> > #include <stdlib.h> > > void check(int const value) { > if (value != MPI_SUCCESS) { > fprintf(stderr, "Failed.\n"); > exit(1); > } > } > > int main(int argc, char** argv) { > MPI_Request r1; > int payload = 42; > int result; > > check(MPI_Init(0,0)); > check(MPI_Issend(&payload, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &r1)); > check(MPI_Test(&r1, &result, MPI_STATUS_IGNORE)); > fprintf(stdout, "MPI_Test result: %d\n", result); > check(MPI_Cancel(&r1)); > check(MPI_Wait(&r1, MPI_STATUS_IGNORE)); > MPI_Finalize(); > return 0; > } > > This stalls in MPI_Wait when executed using "mpiexec -n 1 ./a.out". > > Cheers, > Edric. > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-discuss] MPI_Cancel + MPI_Wait stalls when using ch4, not when using ch3
by Zhou, Hui 01 Feb '22

01 Feb '22

Hi Edric, After some pondering, we are closing https://github.com/pmodels/mpich/issues/5775 as wontfix. For the following reasons: * Cancelling send is deprecated in MPI-4. * The potential fix will be very messy due to current design. * On the other hand, it is probably easier to work around in application. Example workaround could be to replace MPI_Issend with `MPI_Isend` plus separate MPI_Irecv for acknowledgement. You only need do this for the MPI_Issend that you may cancel, and I think it is a small inconvenience for more predictable behavior. Internally an MPI_Issend is not implemented as an MPI_Isend + MPI_Irecv to be more efficient and to exploit hardware acceleration where we can, but that also made cancelling it very complicated. If you have strong reasons to fix this issue, for example, no feasible work around, let's discuss. -- Hui Zhou ________________________________ From: Zhou, Hui <zhouh(a)anl.gov> Sent: Wednesday, January 19, 2022 12:28 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Edric Ellis <eellis(a)mathworks.com> Subject: Re: MPI_Cancel + MPI_Wait stalls when using ch4, not when using ch3 Thanks for reporting. We'll fix this. Tracking issue: https://github.com/pmodels/mpich/issues/5775 [https://opengraph.githubassets.com/abeb43664f034115c44e99f3e273e49dbfdeb9eb…]<https://github.com/pmodels/mpich/issues/5775> ch4: MPI_Cancel stalls MPI_Wait · Issue #5775 · pmodels/mpich<https://github.com/pmodels/mpich/issues/5775> Reported on discuss mailinglist: ---- quote ---- Hi, Running one of our test programs using MPICH 3.4.3 and ch4:ofi, I notice that MPI_Wait on an MPI_Request that has been MPI_Cancelled never compl... github.com ________________________________ From: Edric Ellis via discuss <discuss(a)mpich.org> Sent: Wednesday, January 19, 2022 11:21 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Edric Ellis <eellis(a)mathworks.com> Subject: [mpich-discuss] MPI_Cancel + MPI_Wait stalls when using ch4, not when using ch3 Hi, Running one of our test programs using MPICH 3.4.3 and ch4:ofi, I notice that MPI_Wait on an MPI_Request that has been MPI_Cancelled never completes (it does when using ch3). (The documentation for MPI_Cancel states "If a communication is marked for cancellation, then a MPI_WAIT call for that communication is guaranteed to return, irrespective of the activities of other processes (i.e., MPI_WAIT behaves as a local function)") Here's a simple example: #include <mpi.h> #include <stdio.h> #include <stdlib.h> void check(int const value) { if (value != MPI_SUCCESS) { fprintf(stderr, "Failed.\n"); exit(1); } } int main(int argc, char** argv) { MPI_Request r1; int payload = 42; int result; check(MPI_Init(0,0)); check(MPI_Issend(&payload, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &r1)); check(MPI_Test(&r1, &result, MPI_STATUS_IGNORE)); fprintf(stdout, "MPI_Test result: %d\n", result); check(MPI_Cancel(&r1)); check(MPI_Wait(&r1, MPI_STATUS_IGNORE)); MPI_Finalize(); return 0; } This stalls in MPI_Wait when executed using "mpiexec -n 1 ./a.out". Cheers, Edric. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0