December 2019 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Best practices for installation procedures of ib-drivers/ucx/pmix/mpich stack
by Raffenetti, Kenneth J. 12 Dec '19

12 Dec '19

Hi, Both PMIx and UCX are runtime dependencies. They must be available on all nodes where your application runs. Location is mostly a style decision. Many systems use /usr/local for packages installed manually (i.e. not by the system package manager). GPFS is also a fine option, which allows you to install only once. Ken On 12/12/19 8:54 AM, Sajid Ali via discuss wrote: > Hi MPICH-developers, > > I'm trying to help the sysadmins at my univ. cluster install mpich using > spack with ucx and pmix (previously we just let mpich interact with > slurm's pmi2 and as mentioned on the mailing list the old versions of > mpich either failed to read the nodelist properly or were (incorrectly) > configured with default TCP/IP transport protocols). > > After the sysadmins installed pmix-2.2.3 and pmix-3.1.3 from rpm's at > /usr/lib64/pmix-version (and configured slurm to use them), I tried > building mpich in my home folder (on GPFS) but it was giving a build > error as it was picking up some wrong libraries from /usr/lib64. When I > installed pmix using spack in my home directory (on GPFS), I was able to > build mpich-devel against it. > > What is the best location to install pmix and ucx? Does it have to be > available on every node and if yes would it be on the local file system > on each node or GPFS file-system ? Or is it okay to have it installed on > the slurm launcher node alone ? > > Hope this is the right place to ask this question and thanks in advance > for the help! > > > -- > Sajid Ali | PhD Candidate > Applied Physics > Northwestern University > s-sajid-ali.github.io <http://s-sajid-ali.github.io> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Best practices for installation procedures of ib-drivers/ucx/pmix/mpich stack
by Sajid Ali 12 Dec '19

12 Dec '19

Hi MPICH-developers, I'm trying to help the sysadmins at my univ. cluster install mpich using spack with ucx and pmix (previously we just let mpich interact with slurm's pmi2 and as mentioned on the mailing list the old versions of mpich either failed to read the nodelist properly or were (incorrectly) configured with default TCP/IP transport protocols). After the sysadmins installed pmix-2.2.3 and pmix-3.1.3 from rpm's at /usr/lib64/pmix-version (and configured slurm to use them), I tried building mpich in my home folder (on GPFS) but it was giving a build error as it was picking up some wrong libraries from /usr/lib64. When I installed pmix using spack in my home directory (on GPFS), I was able to build mpich-devel against it. What is the best location to install pmix and ucx? Does it have to be available on every node and if yes would it be on the local file system on each node or GPFS file-system ? Or is it okay to have it installed on the slurm launcher node alone ? Hope this is the right place to ask this question and thanks in advance for the help! -- Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io

1 0

Re: [mpich-discuss] too many ssh connections warning
by Reuti 08 Dec '19

08 Dec '19

Hi Kurt: > Am 07.12.2019 um 16:07 schrieb Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>: > > Thanks Reuti. I assume that by the "task manager interface without ssh", you mean > > $ mpiexec -launcher rsh ... No. If the default compilation doesn't include SLURM support in your case, one has to compile MPICH with: $ ./configure --with-slurm=[PATH] … It should look like: $ ps -e f … 16594 ? Sl 0:00 slurmstepd: [166806.batch] 16599 ? S 0:00 \_ /bin/bash /var/spool/slurm/d/job166806/slurm_script 16755 ? S 0:00 \_ mpiexec ./mpihello 16757 ? Ssl 0:00 \_ /bin/srun -N 2 -n 2 --input none /home/reuti/local/mpich-3.3.2/bin/hydra_pmi_pro 16758 ? S 0:00 \_ /bin/srun -N 2 -n 2 --input none /home/reuti/local/mpich-3.3.2/bin/hydra_pmi 16766 ? Sl 0:00 slurmstepd: [166806.0] 16772 ? S 0:00 \_ /home/reuti/local/mpich-3.3.2/bin/hydra_pmi_proxy --control-port node045:3 16773 ? Rs 0:09 \_ ./mpihello 16774 ? Rs 0:09 \_ ./mpihello … (and on the slave nodes only the second daemon is present) No SSH, no RSH. Hence it's a tight integration into the queuing system. https://slurm.schedmd.com/mpi_guide.html In addition, to change to `srun` as startup one might need (not used in the example above): https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Note_that…. It should work with a cluster which has no `ssh` to the nodes implemented at all (I allow `ssh` to the nodes only for admin staff). === Are the nodes on a private network, i.e. the nodes can't be reached from the Internet? Then one might also discuss pro and cons of allowing `rsh` or not. If even inside a private cluster all communication between the nodes has to be encrypted, I fear neither MPICH nor any other MPI implementation provides this. === AFAIR "fork" was used in times before Hydra as an alternative to start the MPI tasks local on a single machine only. I didn't check for "fork" or "smpd" for some time since Hydra appeared. -- Reuti > I can't use rsh on our cluster due to security concerns. Another launcher option is "fork", but when I tried it, the whole job froze. Does "fork" refer to a specific binary like ssh, or does it refer to the Linux system call? > > Thanks, > Kurt > >>> Sorry, I forgot to mention that I am starting the job under PBS/Torque with the qsub command. > >> Then it should be possible to use the task manager interface without `ssh`: > > -----Original Message----- > From: Reuti via discuss <discuss(a)mpich.org> > Sent: Tuesday, December 3, 2019 9:02 AM > To: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> > Cc: Reuti <reuti(a)staff.uni-marburg.de> > Subject: Re: [mpich-discuss] [EXTERNAL] Re: too many ssh connections warning > > Hi: > >> Am 03.12.2019 um 15:45 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org>: >> >> Reuti, >> >> Sorry, I forgot to mention that I am starting the job under PBS/Torque with the qsub command. > > Then it should be possible to use the task manager interface without `ssh`: > > https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.adaptivecomputing.… > > >> I'll check with our sysadmins to see if there are firewall issues. > > This could also be later an issue if MPICH will connect to other machines directly to talk to the already started daemons. > > >> What is PAM? > > https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org_wiki_… > > Several limits can be set here, depending on your distribution: > > ls /lib64/security/ > > will show the available ones which are installed by default and are then used/configured in /etc/pam.d > > -- Reuti > > >> Hui Zhou, >> >> What do you expect would be making multiple SSH connections to the node? The creation of inter-communicators? Individual MPI_Iprobe/MPI_Isend/MPI_IRecv commands? If you have a guess, that would help me know how to fix the problem. >> >> Kurt >> >> >> -----Original Message----- >> From: Reuti <reuti(a)staff.uni-marburg.de> >> Sent: Monday, December 2, 2019 3:20 PM >> To: discuss(a)mpich.org >> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> >> Subject: [EXTERNAL] Re: [mpich-discuss] too many ssh connections warning >> >> >>> Am 02.12.2019 um 22:14 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org>: >>> >>> My application uses mainly inter-communicators rather than intra-communicators for fault tolerance. A particular process might have 20 inter-communicators active at one time. I’m receiving the warning >>> >>> [mpiexec(a)n010.cluster.com] WARNING: too many ssh connections to n009.cluster.com; waiting 6 seconds >>> >>> What is the cause of this? I have several guesses: >>> >>> 1) MPICH has an internal limit on the number of connections >>> 2) I’m bumping up against a Linux limit on the number of connections >>> 3) Non-blocking communication using MPI_Isend() creates a temporary ssh connection (not likely) >> >> 4) Firewall or PAM settings on the target prevent to many logins in a certain timeframe. >> >> Are you using a queuing system and have the chance to skip SSH and startup MPICH by the queuing system? >> >> -- Reuti >> >> >>> The other question is, what are the consequences of “waiting 6 seconds”? Are some non-blocking messages dropped? >>> >>> I’m using MPICH 3.3.2, CentOS 3.10 and the Portland Group compiler pgc++ 19.5.0. >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma… >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma… > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…

1 0

Re: [mpich-discuss] too many ssh connections warning
by Mccall, Kurt E. (MSFC-EV41) 07 Dec '19

07 Dec '19

Thanks Reuti. I assume that by the "task manager interface without ssh", you mean $ mpiexec -launcher rsh ... I can't use rsh on our cluster due to security concerns. Another launcher option is "fork", but when I tried it, the whole job froze. Does "fork" refer to a specific binary like ssh, or does it refer to the Linux system call? Thanks, Kurt >> Sorry, I forgot to mention that I am starting the job under PBS/Torque with the qsub command. > Then it should be possible to use the task manager interface without `ssh`: -----Original Message----- From: Reuti via discuss <discuss(a)mpich.org> Sent: Tuesday, December 3, 2019 9:02 AM To: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Cc: Reuti <reuti(a)staff.uni-marburg.de> Subject: Re: [mpich-discuss] [EXTERNAL] Re: too many ssh connections warning Hi: > Am 03.12.2019 um 15:45 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org>: > > Reuti, > > Sorry, I forgot to mention that I am starting the job under PBS/Torque with the qsub command. Then it should be possible to use the task manager interface without `ssh`: https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.adaptivecomputing.… > I'll check with our sysadmins to see if there are firewall issues. This could also be later an issue if MPICH will connect to other machines directly to talk to the already started daemons. > What is PAM? https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org_wiki_… Several limits can be set here, depending on your distribution: ls /lib64/security/ will show the available ones which are installed by default and are then used/configured in /etc/pam.d -- Reuti > Hui Zhou, > > What do you expect would be making multiple SSH connections to the node? The creation of inter-communicators? Individual MPI_Iprobe/MPI_Isend/MPI_IRecv commands? If you have a guess, that would help me know how to fix the problem. > > Kurt > > > -----Original Message----- > From: Reuti <reuti(a)staff.uni-marburg.de> > Sent: Monday, December 2, 2019 3:20 PM > To: discuss(a)mpich.org > Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> > Subject: [EXTERNAL] Re: [mpich-discuss] too many ssh connections warning > > >> Am 02.12.2019 um 22:14 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org>: >> >> My application uses mainly inter-communicators rather than intra-communicators for fault tolerance. A particular process might have 20 inter-communicators active at one time. I’m receiving the warning >> >> [mpiexec(a)n010.cluster.com] WARNING: too many ssh connections to n009.cluster.com; waiting 6 seconds >> >> What is the cause of this? I have several guesses: >> >> 1) MPICH has an internal limit on the number of connections >> 2) I’m bumping up against a Linux limit on the number of connections >> 3) Non-blocking communication using MPI_Isend() creates a temporary ssh connection (not likely) > > 4) Firewall or PAM settings on the target prevent to many logins in a certain timeframe. > > Are you using a queuing system and have the chance to skip SSH and startup MPICH by the queuing system? > > -- Reuti > > >> The other question is, what are the consequences of “waiting 6 seconds”? Are some non-blocking messages dropped? >> >> I’m using MPICH 3.3.2, CentOS 3.10 and the Portland Group compiler pgc++ 19.5.0. >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma… > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma… _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…

1 0

Re: [mpich-discuss] What is the best way to connect to a spawned process?
by Mccall, Kurt E. (MSFC-EV41) 05 Dec '19

05 Dec '19

Hui, Thanks for the reply. I am spawning the new process using MPI_COMM_SELF instead of MPI_COMM_WORLD. Maybe that is my problem! Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov> Sent: Wednesday, December 4, 2019 1:43 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] What is the best way to connect to a spawned process? Hi Kurt, Let’s say you have a communicator of 10 processes (MPI_COMM_WORLD or any derivatives), and if you use that communicator to spawn process P, the result inter-communicator will be between your original comm (10 processes) and P’s MPI_COMM_WORLD (1 process), right? All original 10 processes should be already connected to P. Did I miss anything? — Hui Zhou On Dec 4, 2019, at 1:31 PM, Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi, My job creates 10 processes, and the rank 0 one spawns a distinct process P to function as a master. I’d like to share P’s inter-communicator with the other 9 processes, but it seems that I can’t pass it directly to them through function arguments to those processes (very weird errors happen involving the inter-communicator). What is an efficient way to connect all 10 processes to P? I’ve looked into MPI_Open_port / MPI_Comm_accept / MPI_Comm_connect, but was wondering if there is more streamlined way. (I can’t make rank 0 become the master directly, due to the peculiarities of my PBS/Torque node allocations). Thanks, Kurt _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…>

1 0

Re: [mpich-discuss] MPICH Hydra mpiexec and Slurm job allocation
by r900＠mail.com 05 Dec '19

05 Dec '19

On Thu, 5 Dec 2019 14:34:57 +0000 "Raffenetti, Kenneth J." <raffenet(a)mcs.anl.gov> wrote: > On 12/5/19 6:23 AM, Stefan via discuss wrote: > > Tested with nightly mpich-master-v3.4a1-405-gcb944ba and it works indeed, > > great! I've compiled 3.3.2 with the slurm_query_node_list.c file from > > master, and that seems to work fine too. Hope this makes it into the next > > 3.3 patch release. > > > > I've noticed that both the nodelist detection and slurm launcher work > > fine even without specifying the Slurm installation directory (Slurm is > > installed _outside_ of /usr and LD_LIBRARY_PATH is empty when I compile). > > > > Is there any benefit of using --with-slurm=/path/to/slurm ? > > > > > > I have another vaguely related question. When I configure with > > --with-pmix=/sw/pmix/2.2.3 > > it seems like Hydra is silently disabled. I don't get any warning or > > other message even when I explicitly specify > > --with-pm=hydra > > there is no mpiexec in the bin folder after make install. > > > > Is there a way to use PMIx and Hydra, or is this planned for the future? > > No, Hydra does not support PMIx. We do not have plans to support PMIx > with Hydra in the near future. > > If Slurm on your cluster is configured with PMIx support, you should be > able to launch MPICH+PMIx applications with srun instead of mpiexec. Yes that works for _most_ programs. We have however some programs written in Perl that use MPI, and only work properly when started with mpiexec. But I can live with having a non-PMIx build for these :-) /Stefan > Ken > > > > > /Stefan > > > > On Wed, 4 Dec 2019 23:05:46 +0000 > > "Congiu, Giuseppe" <gcongiu(a)anl.gov> wrote: > > > >> That is a bug that should be now fixed in current master. Additionally we added a short test program in `src/pm/hydra/maint` (slurm_nodelist_parse) that you can use to test your node list format to see whether it will cause problems with MPICH. If that fails please provide the output of such program in your message and we will look into it. > >> > >> Giuseppe Congiu > >> Postdoctoral Appointee > >> MCS Division > >> Argonne National Laboratory > >> 9700 South Cass Ave., Lemont, IL 60439 > >> > >> > >> > >> On Dec 4, 2019, at 4:28 PM, Stefan via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: > >> > >> Hi, > >> > >> I'm having some issues to make mpirun/mpiexec play nicely with Slurm > >> allocations. I'm using Slurm 19.05.4, and have configured MPICH with: > >> --enable-shared --enable-static --with-slurm=/sw/slurm/19.05.4 \ > >> --with-pm=hydra > >> > >> Now I request resources from Slurm with: > >> $ salloc -N 2 --ntasks-per-node 4 > >> > >> Then when I try to run a test binary: > >> $ mpiexec.hydra ./mpich_hello > >> Error: node list format not recognized. Try using '-hosts=<hostnames>'. > >> Aborted (core dumped) > >> > >> When I do the same with OpenMPI's mpirun/mpiexec it runs on the allocated > >> nodes. Am I missing something, or does MPICH simply not support this use case? > >> > >> Currently I'm working around this by using a script to translate Slurm > >> node allocations into a host list and run like this: > >> $ mpiexec.hydra -hosts $(mpich-host) ./mpich_hello > >> > >> That works fine, but I suppose this workaround should not be necessary. > >> Here is ltrace output which shows that mpiexec tries to process some Slurm > >> related environment variables but apparently fails to do so: > >> https://paste.ubuntu.com/p/327tGrTzq5/ > >> > >> I've also tried with salloc -N 1 -n 1, so that the environment variables > >> are simpler, e.g. > >> SLURM_NODELIST=node-b01 > >> SLURM_TASKS_PER_NODE=1 > >> but that did not change the way mpiexec fails. > >> > >> /Stefan > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org<mailto:[email protected]> > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > >> > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > >

1 0

Re: [mpich-discuss] MPICH Hydra mpiexec and Slurm job allocation
by Raffenetti, Kenneth J. 05 Dec '19

05 Dec '19

On 12/5/19 6:23 AM, Stefan via discuss wrote: > Tested with nightly mpich-master-v3.4a1-405-gcb944ba and it works indeed, > great! I've compiled 3.3.2 with the slurm_query_node_list.c file from > master, and that seems to work fine too. Hope this makes it into the next > 3.3 patch release. > > I've noticed that both the nodelist detection and slurm launcher work > fine even without specifying the Slurm installation directory (Slurm is > installed _outside_ of /usr and LD_LIBRARY_PATH is empty when I compile). > > Is there any benefit of using --with-slurm=/path/to/slurm ? > > > I have another vaguely related question. When I configure with > --with-pmix=/sw/pmix/2.2.3 > it seems like Hydra is silently disabled. I don't get any warning or > other message even when I explicitly specify > --with-pm=hydra > there is no mpiexec in the bin folder after make install. > > Is there a way to use PMIx and Hydra, or is this planned for the future? No, Hydra does not support PMIx. We do not have plans to support PMIx with Hydra in the near future. If Slurm on your cluster is configured with PMIx support, you should be able to launch MPICH+PMIx applications with srun instead of mpiexec. Ken > > /Stefan > > On Wed, 4 Dec 2019 23:05:46 +0000 > "Congiu, Giuseppe" <gcongiu(a)anl.gov> wrote: > >> That is a bug that should be now fixed in current master. Additionally we added a short test program in `src/pm/hydra/maint` (slurm_nodelist_parse) that you can use to test your node list format to see whether it will cause problems with MPICH. If that fails please provide the output of such program in your message and we will look into it. >> >> Giuseppe Congiu >> Postdoctoral Appointee >> MCS Division >> Argonne National Laboratory >> 9700 South Cass Ave., Lemont, IL 60439 >> >> >> >> On Dec 4, 2019, at 4:28 PM, Stefan via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: >> >> Hi, >> >> I'm having some issues to make mpirun/mpiexec play nicely with Slurm >> allocations. I'm using Slurm 19.05.4, and have configured MPICH with: >> --enable-shared --enable-static --with-slurm=/sw/slurm/19.05.4 \ >> --with-pm=hydra >> >> Now I request resources from Slurm with: >> $ salloc -N 2 --ntasks-per-node 4 >> >> Then when I try to run a test binary: >> $ mpiexec.hydra ./mpich_hello >> Error: node list format not recognized. Try using '-hosts=<hostnames>'. >> Aborted (core dumped) >> >> When I do the same with OpenMPI's mpirun/mpiexec it runs on the allocated >> nodes. Am I missing something, or does MPICH simply not support this use case? >> >> Currently I'm working around this by using a script to translate Slurm >> node allocations into a host list and run like this: >> $ mpiexec.hydra -hosts $(mpich-host) ./mpich_hello >> >> That works fine, but I suppose this workaround should not be necessary. >> Here is ltrace output which shows that mpiexec tries to process some Slurm >> related environment variables but apparently fails to do so: >> https://paste.ubuntu.com/p/327tGrTzq5/ >> >> I've also tried with salloc -N 1 -n 1, so that the environment variables >> are simpler, e.g. >> SLURM_NODELIST=node-b01 >> SLURM_TASKS_PER_NODE=1 >> but that did not change the way mpiexec fails. >> >> /Stefan >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org<mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] MPICH Hydra mpiexec and Slurm job allocation
by r900＠mail.com 05 Dec '19

05 Dec '19

Tested with nightly mpich-master-v3.4a1-405-gcb944ba and it works indeed, great! I've compiled 3.3.2 with the slurm_query_node_list.c file from master, and that seems to work fine too. Hope this makes it into the next 3.3 patch release. I've noticed that both the nodelist detection and slurm launcher work fine even without specifying the Slurm installation directory (Slurm is installed _outside_ of /usr and LD_LIBRARY_PATH is empty when I compile). Is there any benefit of using --with-slurm=/path/to/slurm ? I have another vaguely related question. When I configure with --with-pmix=/sw/pmix/2.2.3 it seems like Hydra is silently disabled. I don't get any warning or other message even when I explicitly specify --with-pm=hydra there is no mpiexec in the bin folder after make install. Is there a way to use PMIx and Hydra, or is this planned for the future? /Stefan On Wed, 4 Dec 2019 23:05:46 +0000 "Congiu, Giuseppe" <gcongiu(a)anl.gov> wrote: > That is a bug that should be now fixed in current master. Additionally we added a short test program in `src/pm/hydra/maint` (slurm_nodelist_parse) that you can use to test your node list format to see whether it will cause problems with MPICH. If that fails please provide the output of such program in your message and we will look into it. > > Giuseppe Congiu > Postdoctoral Appointee > MCS Division > Argonne National Laboratory > 9700 South Cass Ave., Lemont, IL 60439 > > > > On Dec 4, 2019, at 4:28 PM, Stefan via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: > > Hi, > > I'm having some issues to make mpirun/mpiexec play nicely with Slurm > allocations. I'm using Slurm 19.05.4, and have configured MPICH with: > --enable-shared --enable-static --with-slurm=/sw/slurm/19.05.4 \ > --with-pm=hydra > > Now I request resources from Slurm with: > $ salloc -N 2 --ntasks-per-node 4 > > Then when I try to run a test binary: > $ mpiexec.hydra ./mpich_hello > Error: node list format not recognized. Try using '-hosts=<hostnames>'. > Aborted (core dumped) > > When I do the same with OpenMPI's mpirun/mpiexec it runs on the allocated > nodes. Am I missing something, or does MPICH simply not support this use case? > > Currently I'm working around this by using a script to translate Slurm > node allocations into a host list and run like this: > $ mpiexec.hydra -hosts $(mpich-host) ./mpich_hello > > That works fine, but I suppose this workaround should not be necessary. > Here is ltrace output which shows that mpiexec tries to process some Slurm > related environment variables but apparently fails to do so: > https://paste.ubuntu.com/p/327tGrTzq5/ > > I've also tried with salloc -N 1 -n 1, so that the environment variables > are simpler, e.g. > SLURM_NODELIST=node-b01 > SLURM_TASKS_PER_NODE=1 > but that did not change the way mpiexec fails. > > /Stefan > _______________________________________________ > discuss mailing list discuss(a)mpich.org<mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] MPICH Hydra mpiexec and Slurm job allocation
by Congiu, Giuseppe 04 Dec '19

04 Dec '19

Just an example of how that works: $ ./slurm_nodelist_parse input compressed nodelist: node-b[01-10] expanded nodelist: node-b01,node-b02,node-b03,node-b04,node-b05,node-b06,node-b07,node-b08,node-b09,node-b10 Thus your node list format should be recognized currently Giuseppe Congiu Postdoctoral Appointee MCS Division Argonne National Laboratory 9700 South Cass Ave., Lemont, IL 60439 On Dec 4, 2019, at 5:05 PM, Congiu, Giuseppe via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: That is a bug that should be now fixed in current master. Additionally we added a short test program in `src/pm/hydra/maint` (slurm_nodelist_parse) that you can use to test your node list format to see whether it will cause problems with MPICH. If that fails please provide the output of such program in your message and we will look into it. Giuseppe Congiu Postdoctoral Appointee MCS Division Argonne National Laboratory 9700 South Cass Ave., Lemont, IL 60439 On Dec 4, 2019, at 4:28 PM, Stefan via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi, I'm having some issues to make mpirun/mpiexec play nicely with Slurm allocations. I'm using Slurm 19.05.4, and have configured MPICH with: --enable-shared --enable-static --with-slurm=/sw/slurm/19.05.4 \ --with-pm=hydra Now I request resources from Slurm with: $ salloc -N 2 --ntasks-per-node 4 Then when I try to run a test binary: $ mpiexec.hydra ./mpich_hello Error: node list format not recognized. Try using '-hosts=<hostnames>'. Aborted (core dumped) When I do the same with OpenMPI's mpirun/mpiexec it runs on the allocated nodes. Am I missing something, or does MPICH simply not support this use case? Currently I'm working around this by using a script to translate Slurm node allocations into a host list and run like this: $ mpiexec.hydra -hosts $(mpich-host) ./mpich_hello That works fine, but I suppose this workaround should not be necessary. Here is ltrace output which shows that mpiexec tries to process some Slurm related environment variables but apparently fails to do so: https://paste.ubuntu.com/p/327tGrTzq5/ I've also tried with salloc -N 1 -n 1, so that the environment variables are simpler, e.g. SLURM_NODELIST=node-b01 SLURM_TASKS_PER_NODE=1 but that did not change the way mpiexec fails. /Stefan _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH Hydra mpiexec and Slurm job allocation
by Congiu, Giuseppe 04 Dec '19

04 Dec '19

That is a bug that should be now fixed in current master. Additionally we added a short test program in `src/pm/hydra/maint` (slurm_nodelist_parse) that you can use to test your node list format to see whether it will cause problems with MPICH. If that fails please provide the output of such program in your message and we will look into it. Giuseppe Congiu Postdoctoral Appointee MCS Division Argonne National Laboratory 9700 South Cass Ave., Lemont, IL 60439 On Dec 4, 2019, at 4:28 PM, Stefan via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi, I'm having some issues to make mpirun/mpiexec play nicely with Slurm allocations. I'm using Slurm 19.05.4, and have configured MPICH with: --enable-shared --enable-static --with-slurm=/sw/slurm/19.05.4 \ --with-pm=hydra Now I request resources from Slurm with: $ salloc -N 2 --ntasks-per-node 4 Then when I try to run a test binary: $ mpiexec.hydra ./mpich_hello Error: node list format not recognized. Try using '-hosts=<hostnames>'. Aborted (core dumped) When I do the same with OpenMPI's mpirun/mpiexec it runs on the allocated nodes. Am I missing something, or does MPICH simply not support this use case? Currently I'm working around this by using a script to translate Slurm node allocations into a host list and run like this: $ mpiexec.hydra -hosts $(mpich-host) ./mpich_hello That works fine, but I suppose this workaround should not be necessary. Here is ltrace output which shows that mpiexec tries to process some Slurm related environment variables but apparently fails to do so: https://paste.ubuntu.com/p/327tGrTzq5/ I've also tried with salloc -N 1 -n 1, so that the environment variables are simpler, e.g. SLURM_NODELIST=node-b01 SLURM_TASKS_PER_NODE=1 but that did not change the way mpiexec fails. /Stefan _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0