- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPICH Hydra mpiexec and Slurm job allocation
by r900＠mail.com 05 Dec '19

05 Dec '19

On Thu, 5 Dec 2019 14:34:57 +0000 "Raffenetti, Kenneth J." <raffenet(a)mcs.anl.gov> wrote: > On 12/5/19 6:23 AM, Stefan via discuss wrote: > > Tested with nightly mpich-master-v3.4a1-405-gcb944ba and it works indeed, > > great! I've compiled 3.3.2 with the slurm_query_node_list.c file from > > master, and that seems to work fine too. Hope this makes it into the next > > 3.3 patch release. > > > > I've noticed that both the nodelist detection and slurm launcher work > > fine even without specifying the Slurm installation directory (Slurm is > > installed _outside_ of /usr and LD_LIBRARY_PATH is empty when I compile). > > > > Is there any benefit of using --with-slurm=/path/to/slurm ? > > > > > > I have another vaguely related question. When I configure with > > --with-pmix=/sw/pmix/2.2.3 > > it seems like Hydra is silently disabled. I don't get any warning or > > other message even when I explicitly specify > > --with-pm=hydra > > there is no mpiexec in the bin folder after make install. > > > > Is there a way to use PMIx and Hydra, or is this planned for the future? > > No, Hydra does not support PMIx. We do not have plans to support PMIx > with Hydra in the near future. > > If Slurm on your cluster is configured with PMIx support, you should be > able to launch MPICH+PMIx applications with srun instead of mpiexec. Yes that works for _most_ programs. We have however some programs written in Perl that use MPI, and only work properly when started with mpiexec. But I can live with having a non-PMIx build for these :-) /Stefan > Ken > > > > > /Stefan > > > > On Wed, 4 Dec 2019 23:05:46 +0000 > > "Congiu, Giuseppe" <gcongiu(a)anl.gov> wrote: > > > >> That is a bug that should be now fixed in current master. Additionally we added a short test program in `src/pm/hydra/maint` (slurm_nodelist_parse) that you can use to test your node list format to see whether it will cause problems with MPICH. If that fails please provide the output of such program in your message and we will look into it. > >> > >> Giuseppe Congiu > >> Postdoctoral Appointee > >> MCS Division > >> Argonne National Laboratory > >> 9700 South Cass Ave., Lemont, IL 60439 > >> > >> > >> > >> On Dec 4, 2019, at 4:28 PM, Stefan via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: > >> > >> Hi, > >> > >> I'm having some issues to make mpirun/mpiexec play nicely with Slurm > >> allocations. I'm using Slurm 19.05.4, and have configured MPICH with: > >> --enable-shared --enable-static --with-slurm=/sw/slurm/19.05.4 \ > >> --with-pm=hydra > >> > >> Now I request resources from Slurm with: > >> $ salloc -N 2 --ntasks-per-node 4 > >> > >> Then when I try to run a test binary: > >> $ mpiexec.hydra ./mpich_hello > >> Error: node list format not recognized. Try using '-hosts=<hostnames>'. > >> Aborted (core dumped) > >> > >> When I do the same with OpenMPI's mpirun/mpiexec it runs on the allocated > >> nodes. Am I missing something, or does MPICH simply not support this use case? > >> > >> Currently I'm working around this by using a script to translate Slurm > >> node allocations into a host list and run like this: > >> $ mpiexec.hydra -hosts $(mpich-host) ./mpich_hello > >> > >> That works fine, but I suppose this workaround should not be necessary. > >> Here is ltrace output which shows that mpiexec tries to process some Slurm > >> related environment variables but apparently fails to do so: > >> https://paste.ubuntu.com/p/327tGrTzq5/ > >> > >> I've also tried with salloc -N 1 -n 1, so that the environment variables > >> are simpler, e.g. > >> SLURM_NODELIST=node-b01 > >> SLURM_TASKS_PER_NODE=1 > >> but that did not change the way mpiexec fails. > >> > >> /Stefan > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org<mailto:[email protected]> > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > >> > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > >

1 0

Re: [mpich-discuss] MPICH Hydra mpiexec and Slurm job allocation
by Raffenetti, Kenneth J. 05 Dec '19

05 Dec '19

On 12/5/19 6:23 AM, Stefan via discuss wrote: > Tested with nightly mpich-master-v3.4a1-405-gcb944ba and it works indeed, > great! I've compiled 3.3.2 with the slurm_query_node_list.c file from > master, and that seems to work fine too. Hope this makes it into the next > 3.3 patch release. > > I've noticed that both the nodelist detection and slurm launcher work > fine even without specifying the Slurm installation directory (Slurm is > installed _outside_ of /usr and LD_LIBRARY_PATH is empty when I compile). > > Is there any benefit of using --with-slurm=/path/to/slurm ? > > > I have another vaguely related question. When I configure with > --with-pmix=/sw/pmix/2.2.3 > it seems like Hydra is silently disabled. I don't get any warning or > other message even when I explicitly specify > --with-pm=hydra > there is no mpiexec in the bin folder after make install. > > Is there a way to use PMIx and Hydra, or is this planned for the future? No, Hydra does not support PMIx. We do not have plans to support PMIx with Hydra in the near future. If Slurm on your cluster is configured with PMIx support, you should be able to launch MPICH+PMIx applications with srun instead of mpiexec. Ken > > /Stefan > > On Wed, 4 Dec 2019 23:05:46 +0000 > "Congiu, Giuseppe" <gcongiu(a)anl.gov> wrote: > >> That is a bug that should be now fixed in current master. Additionally we added a short test program in `src/pm/hydra/maint` (slurm_nodelist_parse) that you can use to test your node list format to see whether it will cause problems with MPICH. If that fails please provide the output of such program in your message and we will look into it. >> >> Giuseppe Congiu >> Postdoctoral Appointee >> MCS Division >> Argonne National Laboratory >> 9700 South Cass Ave., Lemont, IL 60439 >> >> >> >> On Dec 4, 2019, at 4:28 PM, Stefan via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: >> >> Hi, >> >> I'm having some issues to make mpirun/mpiexec play nicely with Slurm >> allocations. I'm using Slurm 19.05.4, and have configured MPICH with: >> --enable-shared --enable-static --with-slurm=/sw/slurm/19.05.4 \ >> --with-pm=hydra >> >> Now I request resources from Slurm with: >> $ salloc -N 2 --ntasks-per-node 4 >> >> Then when I try to run a test binary: >> $ mpiexec.hydra ./mpich_hello >> Error: node list format not recognized. Try using '-hosts=<hostnames>'. >> Aborted (core dumped) >> >> When I do the same with OpenMPI's mpirun/mpiexec it runs on the allocated >> nodes. Am I missing something, or does MPICH simply not support this use case? >> >> Currently I'm working around this by using a script to translate Slurm >> node allocations into a host list and run like this: >> $ mpiexec.hydra -hosts $(mpich-host) ./mpich_hello >> >> That works fine, but I suppose this workaround should not be necessary. >> Here is ltrace output which shows that mpiexec tries to process some Slurm >> related environment variables but apparently fails to do so: >> https://paste.ubuntu.com/p/327tGrTzq5/ >> >> I've also tried with salloc -N 1 -n 1, so that the environment variables >> are simpler, e.g. >> SLURM_NODELIST=node-b01 >> SLURM_TASKS_PER_NODE=1 >> but that did not change the way mpiexec fails. >> >> /Stefan >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org<mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] MPICH Hydra mpiexec and Slurm job allocation
by r900＠mail.com 05 Dec '19

05 Dec '19

Tested with nightly mpich-master-v3.4a1-405-gcb944ba and it works indeed, great! I've compiled 3.3.2 with the slurm_query_node_list.c file from master, and that seems to work fine too. Hope this makes it into the next 3.3 patch release. I've noticed that both the nodelist detection and slurm launcher work fine even without specifying the Slurm installation directory (Slurm is installed _outside_ of /usr and LD_LIBRARY_PATH is empty when I compile). Is there any benefit of using --with-slurm=/path/to/slurm ? I have another vaguely related question. When I configure with --with-pmix=/sw/pmix/2.2.3 it seems like Hydra is silently disabled. I don't get any warning or other message even when I explicitly specify --with-pm=hydra there is no mpiexec in the bin folder after make install. Is there a way to use PMIx and Hydra, or is this planned for the future? /Stefan On Wed, 4 Dec 2019 23:05:46 +0000 "Congiu, Giuseppe" <gcongiu(a)anl.gov> wrote: > That is a bug that should be now fixed in current master. Additionally we added a short test program in `src/pm/hydra/maint` (slurm_nodelist_parse) that you can use to test your node list format to see whether it will cause problems with MPICH. If that fails please provide the output of such program in your message and we will look into it. > > Giuseppe Congiu > Postdoctoral Appointee > MCS Division > Argonne National Laboratory > 9700 South Cass Ave., Lemont, IL 60439 > > > > On Dec 4, 2019, at 4:28 PM, Stefan via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: > > Hi, > > I'm having some issues to make mpirun/mpiexec play nicely with Slurm > allocations. I'm using Slurm 19.05.4, and have configured MPICH with: > --enable-shared --enable-static --with-slurm=/sw/slurm/19.05.4 \ > --with-pm=hydra > > Now I request resources from Slurm with: > $ salloc -N 2 --ntasks-per-node 4 > > Then when I try to run a test binary: > $ mpiexec.hydra ./mpich_hello > Error: node list format not recognized. Try using '-hosts=<hostnames>'. > Aborted (core dumped) > > When I do the same with OpenMPI's mpirun/mpiexec it runs on the allocated > nodes. Am I missing something, or does MPICH simply not support this use case? > > Currently I'm working around this by using a script to translate Slurm > node allocations into a host list and run like this: > $ mpiexec.hydra -hosts $(mpich-host) ./mpich_hello > > That works fine, but I suppose this workaround should not be necessary. > Here is ltrace output which shows that mpiexec tries to process some Slurm > related environment variables but apparently fails to do so: > https://paste.ubuntu.com/p/327tGrTzq5/ > > I've also tried with salloc -N 1 -n 1, so that the environment variables > are simpler, e.g. > SLURM_NODELIST=node-b01 > SLURM_TASKS_PER_NODE=1 > but that did not change the way mpiexec fails. > > /Stefan > _______________________________________________ > discuss mailing list discuss(a)mpich.org<mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] MPICH Hydra mpiexec and Slurm job allocation
by Congiu, Giuseppe 04 Dec '19

04 Dec '19

Just an example of how that works: $ ./slurm_nodelist_parse input compressed nodelist: node-b[01-10] expanded nodelist: node-b01,node-b02,node-b03,node-b04,node-b05,node-b06,node-b07,node-b08,node-b09,node-b10 Thus your node list format should be recognized currently Giuseppe Congiu Postdoctoral Appointee MCS Division Argonne National Laboratory 9700 South Cass Ave., Lemont, IL 60439 On Dec 4, 2019, at 5:05 PM, Congiu, Giuseppe via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: That is a bug that should be now fixed in current master. Additionally we added a short test program in `src/pm/hydra/maint` (slurm_nodelist_parse) that you can use to test your node list format to see whether it will cause problems with MPICH. If that fails please provide the output of such program in your message and we will look into it. Giuseppe Congiu Postdoctoral Appointee MCS Division Argonne National Laboratory 9700 South Cass Ave., Lemont, IL 60439 On Dec 4, 2019, at 4:28 PM, Stefan via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi, I'm having some issues to make mpirun/mpiexec play nicely with Slurm allocations. I'm using Slurm 19.05.4, and have configured MPICH with: --enable-shared --enable-static --with-slurm=/sw/slurm/19.05.4 \ --with-pm=hydra Now I request resources from Slurm with: $ salloc -N 2 --ntasks-per-node 4 Then when I try to run a test binary: $ mpiexec.hydra ./mpich_hello Error: node list format not recognized. Try using '-hosts=<hostnames>'. Aborted (core dumped) When I do the same with OpenMPI's mpirun/mpiexec it runs on the allocated nodes. Am I missing something, or does MPICH simply not support this use case? Currently I'm working around this by using a script to translate Slurm node allocations into a host list and run like this: $ mpiexec.hydra -hosts $(mpich-host) ./mpich_hello That works fine, but I suppose this workaround should not be necessary. Here is ltrace output which shows that mpiexec tries to process some Slurm related environment variables but apparently fails to do so: https://paste.ubuntu.com/p/327tGrTzq5/ I've also tried with salloc -N 1 -n 1, so that the environment variables are simpler, e.g. SLURM_NODELIST=node-b01 SLURM_TASKS_PER_NODE=1 but that did not change the way mpiexec fails. /Stefan _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH Hydra mpiexec and Slurm job allocation
by Congiu, Giuseppe 04 Dec '19

04 Dec '19

That is a bug that should be now fixed in current master. Additionally we added a short test program in `src/pm/hydra/maint` (slurm_nodelist_parse) that you can use to test your node list format to see whether it will cause problems with MPICH. If that fails please provide the output of such program in your message and we will look into it. Giuseppe Congiu Postdoctoral Appointee MCS Division Argonne National Laboratory 9700 South Cass Ave., Lemont, IL 60439 On Dec 4, 2019, at 4:28 PM, Stefan via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi, I'm having some issues to make mpirun/mpiexec play nicely with Slurm allocations. I'm using Slurm 19.05.4, and have configured MPICH with: --enable-shared --enable-static --with-slurm=/sw/slurm/19.05.4 \ --with-pm=hydra Now I request resources from Slurm with: $ salloc -N 2 --ntasks-per-node 4 Then when I try to run a test binary: $ mpiexec.hydra ./mpich_hello Error: node list format not recognized. Try using '-hosts=<hostnames>'. Aborted (core dumped) When I do the same with OpenMPI's mpirun/mpiexec it runs on the allocated nodes. Am I missing something, or does MPICH simply not support this use case? Currently I'm working around this by using a script to translate Slurm node allocations into a host list and run like this: $ mpiexec.hydra -hosts $(mpich-host) ./mpich_hello That works fine, but I suppose this workaround should not be necessary. Here is ltrace output which shows that mpiexec tries to process some Slurm related environment variables but apparently fails to do so: https://paste.ubuntu.com/p/327tGrTzq5/ I've also tried with salloc -N 1 -n 1, so that the environment variables are simpler, e.g. SLURM_NODELIST=node-b01 SLURM_TASKS_PER_NODE=1 but that did not change the way mpiexec fails. /Stefan _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH Hydra mpiexec and Slurm job allocation
by Stefan 04 Dec '19

04 Dec '19

Oh sorry, I thought I mentioned that. I'm using the latest stable release 3.3.2 /Stefan On December 4, 2019 11:43:04 PM GMT+01:00, "Raffenetti, Kenneth J." <raffenet(a)mcs.anl.gov> wrote: >Which version of MPICH are you using? > >Ken > >On 12/4/19 4:28 PM, Stefan via discuss wrote: >> Hi, >> >> I'm having some issues to make mpirun/mpiexec play nicely with Slurm >> allocations. I'm using Slurm 19.05.4, and have configured MPICH with: >> --enable-shared --enable-static --with-slurm=/sw/slurm/19.05.4 \ >> --with-pm=hydra >> >> Now I request resources from Slurm with: >> $ salloc -N 2 --ntasks-per-node 4 >> >> Then when I try to run a test binary: >> $ mpiexec.hydra ./mpich_hello >> Error: node list format not recognized. Try using >'-hosts=<hostnames>'. >> Aborted (core dumped) >> >> When I do the same with OpenMPI's mpirun/mpiexec it runs on the >allocated >> nodes. Am I missing something, or does MPICH simply not support this >use case? >> >> Currently I'm working around this by using a script to translate >Slurm >> node allocations into a host list and run like this: >> $ mpiexec.hydra -hosts $(mpich-host) ./mpich_hello >> >> That works fine, but I suppose this workaround should not be >necessary. >> Here is ltrace output which shows that mpiexec tries to process some >Slurm >> related environment variables but apparently fails to do so: >> https://paste.ubuntu.com/p/327tGrTzq5/ >> >> I've also tried with salloc -N 1 -n 1, so that the environment >variables >> are simpler, e.g. >> SLURM_NODELIST=node-b01 >> SLURM_TASKS_PER_NODE=1 >> but that did not change the way mpiexec fails. >> >> /Stefan >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >>

1 0

Re: [mpich-discuss] MPICH Hydra mpiexec and Slurm job allocation
by Raffenetti, Kenneth J. 04 Dec '19

04 Dec '19

Which version of MPICH are you using? Ken On 12/4/19 4:28 PM, Stefan via discuss wrote: > Hi, > > I'm having some issues to make mpirun/mpiexec play nicely with Slurm > allocations. I'm using Slurm 19.05.4, and have configured MPICH with: > --enable-shared --enable-static --with-slurm=/sw/slurm/19.05.4 \ > --with-pm=hydra > > Now I request resources from Slurm with: > $ salloc -N 2 --ntasks-per-node 4 > > Then when I try to run a test binary: > $ mpiexec.hydra ./mpich_hello > Error: node list format not recognized. Try using '-hosts=<hostnames>'. > Aborted (core dumped) > > When I do the same with OpenMPI's mpirun/mpiexec it runs on the allocated > nodes. Am I missing something, or does MPICH simply not support this use case? > > Currently I'm working around this by using a script to translate Slurm > node allocations into a host list and run like this: > $ mpiexec.hydra -hosts $(mpich-host) ./mpich_hello > > That works fine, but I suppose this workaround should not be necessary. > Here is ltrace output which shows that mpiexec tries to process some Slurm > related environment variables but apparently fails to do so: > https://paste.ubuntu.com/p/327tGrTzq5/ > > I've also tried with salloc -N 1 -n 1, so that the environment variables > are simpler, e.g. > SLURM_NODELIST=node-b01 > SLURM_TASKS_PER_NODE=1 > but that did not change the way mpiexec fails. > > /Stefan > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

MPICH Hydra mpiexec and Slurm job allocation
by r900＠mail.com 04 Dec '19

04 Dec '19

Hi, I'm having some issues to make mpirun/mpiexec play nicely with Slurm allocations. I'm using Slurm 19.05.4, and have configured MPICH with: --enable-shared --enable-static --with-slurm=/sw/slurm/19.05.4 \ --with-pm=hydra Now I request resources from Slurm with: $ salloc -N 2 --ntasks-per-node 4 Then when I try to run a test binary: $ mpiexec.hydra ./mpich_hello Error: node list format not recognized. Try using '-hosts=<hostnames>'. Aborted (core dumped) When I do the same with OpenMPI's mpirun/mpiexec it runs on the allocated nodes. Am I missing something, or does MPICH simply not support this use case? Currently I'm working around this by using a script to translate Slurm node allocations into a host list and run like this: $ mpiexec.hydra -hosts $(mpich-host) ./mpich_hello That works fine, but I suppose this workaround should not be necessary. Here is ltrace output which shows that mpiexec tries to process some Slurm related environment variables but apparently fails to do so: https://paste.ubuntu.com/p/327tGrTzq5/ I've also tried with salloc -N 1 -n 1, so that the environment variables are simpler, e.g. SLURM_NODELIST=node-b01 SLURM_TASKS_PER_NODE=1 but that did not change the way mpiexec fails. /Stefan

1 0

Re: [mpich-discuss] What is the best way to connect to a spawned process?
by Zhou, Hui 04 Dec '19

04 Dec '19

Hi Kurt, Let’s say you have a communicator of 10 processes (MPI_COMM_WORLD or any derivatives), and if you use that communicator to spawn process P, the result inter-communicator will be between your original comm (10 processes) and P’s MPI_COMM_WORLD (1 process), right? All original 10 processes should be already connected to P. Did I miss anything? — Hui Zhou On Dec 4, 2019, at 1:31 PM, Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi, My job creates 10 processes, and the rank 0 one spawns a distinct process P to function as a master. I’d like to share P’s inter-communicator with the other 9 processes, but it seems that I can’t pass it directly to them through function arguments to those processes (very weird errors happen involving the inter-communicator). What is an efficient way to connect all 10 processes to P? I’ve looked into MPI_Open_port / MPI_Comm_accept / MPI_Comm_connect, but was wondering if there is more streamlined way. (I can’t make rank 0 become the master directly, due to the peculiarities of my PBS/Torque node allocations). Thanks, Kurt _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

What is the best way to connect to a spawned process?
by Mccall, Kurt E. (MSFC-EV41) 04 Dec '19

04 Dec '19

Hi, My job creates 10 processes, and the rank 0 one spawns a distinct process P to function as a master. I'd like to share P's inter-communicator with the other 9 processes, but it seems that I can't pass it directly to them through function arguments to those processes (very weird errors happen involving the inter-communicator). What is an efficient way to connect all 10 processes to P? I've looked into MPI_Open_port / MPI_Comm_accept / MPI_Comm_connect, but was wondering if there is more streamlined way. (I can't make rank 0 become the master directly, due to the peculiarities of my PBS/Torque node allocations). Thanks, Kurt

1 0