December 2019 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPICH Hydra mpiexec and Slurm job allocation
by Stefan 04 Dec '19

04 Dec '19

Oh sorry, I thought I mentioned that. I'm using the latest stable release 3.3.2 /Stefan On December 4, 2019 11:43:04 PM GMT+01:00, "Raffenetti, Kenneth J." <raffenet(a)mcs.anl.gov> wrote: >Which version of MPICH are you using? > >Ken > >On 12/4/19 4:28 PM, Stefan via discuss wrote: >> Hi, >> >> I'm having some issues to make mpirun/mpiexec play nicely with Slurm >> allocations. I'm using Slurm 19.05.4, and have configured MPICH with: >> --enable-shared --enable-static --with-slurm=/sw/slurm/19.05.4 \ >> --with-pm=hydra >> >> Now I request resources from Slurm with: >> $ salloc -N 2 --ntasks-per-node 4 >> >> Then when I try to run a test binary: >> $ mpiexec.hydra ./mpich_hello >> Error: node list format not recognized. Try using >'-hosts=<hostnames>'. >> Aborted (core dumped) >> >> When I do the same with OpenMPI's mpirun/mpiexec it runs on the >allocated >> nodes. Am I missing something, or does MPICH simply not support this >use case? >> >> Currently I'm working around this by using a script to translate >Slurm >> node allocations into a host list and run like this: >> $ mpiexec.hydra -hosts $(mpich-host) ./mpich_hello >> >> That works fine, but I suppose this workaround should not be >necessary. >> Here is ltrace output which shows that mpiexec tries to process some >Slurm >> related environment variables but apparently fails to do so: >> https://paste.ubuntu.com/p/327tGrTzq5/ >> >> I've also tried with salloc -N 1 -n 1, so that the environment >variables >> are simpler, e.g. >> SLURM_NODELIST=node-b01 >> SLURM_TASKS_PER_NODE=1 >> but that did not change the way mpiexec fails. >> >> /Stefan >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >>

1 0

Re: [mpich-discuss] MPICH Hydra mpiexec and Slurm job allocation
by Raffenetti, Kenneth J. 04 Dec '19

04 Dec '19

Which version of MPICH are you using? Ken On 12/4/19 4:28 PM, Stefan via discuss wrote: > Hi, > > I'm having some issues to make mpirun/mpiexec play nicely with Slurm > allocations. I'm using Slurm 19.05.4, and have configured MPICH with: > --enable-shared --enable-static --with-slurm=/sw/slurm/19.05.4 \ > --with-pm=hydra > > Now I request resources from Slurm with: > $ salloc -N 2 --ntasks-per-node 4 > > Then when I try to run a test binary: > $ mpiexec.hydra ./mpich_hello > Error: node list format not recognized. Try using '-hosts=<hostnames>'. > Aborted (core dumped) > > When I do the same with OpenMPI's mpirun/mpiexec it runs on the allocated > nodes. Am I missing something, or does MPICH simply not support this use case? > > Currently I'm working around this by using a script to translate Slurm > node allocations into a host list and run like this: > $ mpiexec.hydra -hosts $(mpich-host) ./mpich_hello > > That works fine, but I suppose this workaround should not be necessary. > Here is ltrace output which shows that mpiexec tries to process some Slurm > related environment variables but apparently fails to do so: > https://paste.ubuntu.com/p/327tGrTzq5/ > > I've also tried with salloc -N 1 -n 1, so that the environment variables > are simpler, e.g. > SLURM_NODELIST=node-b01 > SLURM_TASKS_PER_NODE=1 > but that did not change the way mpiexec fails. > > /Stefan > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

MPICH Hydra mpiexec and Slurm job allocation
by r900＠mail.com 04 Dec '19

04 Dec '19

Hi, I'm having some issues to make mpirun/mpiexec play nicely with Slurm allocations. I'm using Slurm 19.05.4, and have configured MPICH with: --enable-shared --enable-static --with-slurm=/sw/slurm/19.05.4 \ --with-pm=hydra Now I request resources from Slurm with: $ salloc -N 2 --ntasks-per-node 4 Then when I try to run a test binary: $ mpiexec.hydra ./mpich_hello Error: node list format not recognized. Try using '-hosts=<hostnames>'. Aborted (core dumped) When I do the same with OpenMPI's mpirun/mpiexec it runs on the allocated nodes. Am I missing something, or does MPICH simply not support this use case? Currently I'm working around this by using a script to translate Slurm node allocations into a host list and run like this: $ mpiexec.hydra -hosts $(mpich-host) ./mpich_hello That works fine, but I suppose this workaround should not be necessary. Here is ltrace output which shows that mpiexec tries to process some Slurm related environment variables but apparently fails to do so: https://paste.ubuntu.com/p/327tGrTzq5/ I've also tried with salloc -N 1 -n 1, so that the environment variables are simpler, e.g. SLURM_NODELIST=node-b01 SLURM_TASKS_PER_NODE=1 but that did not change the way mpiexec fails. /Stefan

1 0

Re: [mpich-discuss] What is the best way to connect to a spawned process?
by Zhou, Hui 04 Dec '19

04 Dec '19

Hi Kurt, Let’s say you have a communicator of 10 processes (MPI_COMM_WORLD or any derivatives), and if you use that communicator to spawn process P, the result inter-communicator will be between your original comm (10 processes) and P’s MPI_COMM_WORLD (1 process), right? All original 10 processes should be already connected to P. Did I miss anything? — Hui Zhou On Dec 4, 2019, at 1:31 PM, Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi, My job creates 10 processes, and the rank 0 one spawns a distinct process P to function as a master. I’d like to share P’s inter-communicator with the other 9 processes, but it seems that I can’t pass it directly to them through function arguments to those processes (very weird errors happen involving the inter-communicator). What is an efficient way to connect all 10 processes to P? I’ve looked into MPI_Open_port / MPI_Comm_accept / MPI_Comm_connect, but was wondering if there is more streamlined way. (I can’t make rank 0 become the master directly, due to the peculiarities of my PBS/Torque node allocations). Thanks, Kurt _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

What is the best way to connect to a spawned process?
by Mccall, Kurt E. (MSFC-EV41) 04 Dec '19

04 Dec '19

Hi, My job creates 10 processes, and the rank 0 one spawns a distinct process P to function as a master. I'd like to share P's inter-communicator with the other 9 processes, but it seems that I can't pass it directly to them through function arguments to those processes (very weird errors happen involving the inter-communicator). What is an efficient way to connect all 10 processes to P? I've looked into MPI_Open_port / MPI_Comm_accept / MPI_Comm_connect, but was wondering if there is more streamlined way. (I can't make rank 0 become the master directly, due to the peculiarities of my PBS/Torque node allocations). Thanks, Kurt

1 0

Re: [mpich-discuss] Possible integer-overflow for MPI_COMM_WORLD in MPI_Iprobe
by Jeff Hammond 04 Dec '19

04 Dec '19

This was fixed in 2019 update 2. Jeff On Fri, Apr 26, 2019 at 8:09 AM Jeff Hammond <jeff.science(a)gmail.com> wrote: > For anyone who cares about this bug because of Intel MPI, I am told it is > fixed in Intel MPI 2019 update 3. > > Jeff > > On Mon, Jan 21, 2019 at 7:46 PM Jeff Hammond <jeff.science(a)gmail.com> > wrote: > >> I was able to reproduce with my own test ( >> https://github.com/jeffhammond/HPCInfo/blob/master/mpi/bugs/iprobe-overflow…) >> with Intel MPI 2019, so I will report that bug to the Intel MPI team. It >> should be easy enough for them to figure out if this bug is from MPICH or >> not. >> >> 2139000000 iterations, 627.143272 seconds >> >> 2140000000 iterations, 627.436206 seconds >> >> 2141000000 iterations, 627.729135 seconds >> >> 2142000000 iterations, 628.022049 seconds >> >> 2143000000 iterations, 628.315015 seconds >> >> 2144000000 iterations, 628.608066 seconds >> >> 2145000000 iterations, 628.901065 seconds >> >> 2146000000 iterations, 629.193992 seconds >> >> 2147000000 iterations, 629.488107 seconds >> >> Abort(738833413) on node 0 (rank 0 in comm 0): Fatal error in >> PMPI_Iprobe: Invalid communicator, error stack: >> >> PMPI_Iprobe(123): MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, >> MPI_COMM_WORLD, flag=0x7ffdf75a396c, status=0x7ffdf75a3970) failed >> >> PMPI_Iprobe(90).: Invalid communicator >> >> Jeff, who works for Intel but knows more about MPICH than Intel MPI >> >> On Mon, Jan 21, 2019 at 11:19 AM Joachim Protze via discuss < >> discuss(a)mpich.org> wrote: >> >>> Hi all, >>> >>> we detected the behavior with IntelMPI 2019 (which is based on MPICH >>> 3.3). Reproducing it with MPICH-3.3 was not yet successful. But I fear, >>> that our built of MPICH just not uses the necessary code path / build >>> flags. >>> >>> When calling MPI_Iprobe with the same communicator for ~2^31 times >>> (which can take 10-30 minutes), the execution stops with: >>> >>> Abort(201962501) on node 0 (rank 0 in comm 0): Fatal error in >>> PMPI_Iprobe: Invalid communicator, error stack: >>> PMPI_Iprobe(123): MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, >>> MPI_COMM_WORLD, flag=0x7ffd925056c0, status=0x7ffd92505694) failed >>> PMPI_Iprobe(90).: Invalid communicator >>> >>> From my understanding of the referenced MPICH code lines, I guess, that >>> the ref-count for MPI_COMM_WORLD overflows, which triggers this error >>> message. >>> >>> Best >>> Joachim >>> >>> -- >>> Dipl.-Inf. Joachim Protze >>> >>> IT Center >>> Group: High Performance Computing >>> Division: Computational Science and Engineering >>> RWTH Aachen University >>> Seffenter Weg 23 >>> D 52074 Aachen (Germany) >>> Tel: +49 241 80- 24765 >>> Fax: +49 241 80-624765 >>> protze(a)itc.rwth-aachen.de >>> www.itc.rwth-aachen.de >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> http://jeffhammond.github.io/ >> > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-discuss] [EXTERNAL] Re: too many ssh connections warning
by Reuti 03 Dec '19

03 Dec '19

Hi: > Am 03.12.2019 um 15:45 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org>: > > Reuti, > > Sorry, I forgot to mention that I am starting the job under PBS/Torque with the qsub command. Then it should be possible to use the task manager interface without `ssh`: http://docs.adaptivecomputing.com/torque/4-2-7/Content/topics/7-messagePass… > I'll check with our sysadmins to see if there are firewall issues. This could also be later an issue if MPICH will connect to other machines directly to talk to the already started daemons. > What is PAM? https://en.wikipedia.org/wiki/Linux_PAM Several limits can be set here, depending on your distribution: ls /lib64/security/ will show the available ones which are installed by default and are then used/configured in /etc/pam.d -- Reuti > Hui Zhou, > > What do you expect would be making multiple SSH connections to the node? The creation of inter-communicators? Individual MPI_Iprobe/MPI_Isend/MPI_IRecv commands? If you have a guess, that would help me know how to fix the problem. > > Kurt > > > -----Original Message----- > From: Reuti <reuti(a)staff.uni-marburg.de> > Sent: Monday, December 2, 2019 3:20 PM > To: discuss(a)mpich.org > Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> > Subject: [EXTERNAL] Re: [mpich-discuss] too many ssh connections warning > > >> Am 02.12.2019 um 22:14 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org>: >> >> My application uses mainly inter-communicators rather than intra-communicators for fault tolerance. A particular process might have 20 inter-communicators active at one time. I’m receiving the warning >> >> [mpiexec(a)n010.cluster.com] WARNING: too many ssh connections to n009.cluster.com; waiting 6 seconds >> >> What is the cause of this? I have several guesses: >> >> 1) MPICH has an internal limit on the number of connections >> 2) I’m bumping up against a Linux limit on the number of connections >> 3) Non-blocking communication using MPI_Isend() creates a temporary ssh connection (not likely) > > 4) Firewall or PAM settings on the target prevent to many logins in a certain timeframe. > > Are you using a queuing system and have the chance to skip SSH and startup MPICH by the queuing system? > > -- Reuti > > >> The other question is, what are the consequences of “waiting 6 seconds”? Are some non-blocking messages dropped? >> >> I’m using MPICH 3.3.2, CentOS 3.10 and the Portland Group compiler pgc++ 19.5.0. >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma… > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] [EXTERNAL] Re: too many ssh connections warning
by Mccall, Kurt E. (MSFC-EV41) 03 Dec '19

03 Dec '19

Reuti, Sorry, I forgot to mention that I am starting the job under PBS/Torque with the qsub command. I'll check with our sysadmins to see if there are firewall issues. What is PAM? Hui Zhou, What do you expect would be making multiple SSH connections to the node? The creation of inter-communicators? Individual MPI_Iprobe/MPI_Isend/MPI_IRecv commands? If you have a guess, that would help me know how to fix the problem. Kurt -----Original Message----- From: Reuti <reuti(a)staff.uni-marburg.de> Sent: Monday, December 2, 2019 3:20 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] too many ssh connections warning > Am 02.12.2019 um 22:14 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org>: > > My application uses mainly inter-communicators rather than intra-communicators for fault tolerance. A particular process might have 20 inter-communicators active at one time. I’m receiving the warning > > [mpiexec(a)n010.cluster.com] WARNING: too many ssh connections to n009.cluster.com; waiting 6 seconds > > What is the cause of this? I have several guesses: > > 1) MPICH has an internal limit on the number of connections > 2) I’m bumping up against a Linux limit on the number of connections > 3) Non-blocking communication using MPI_Isend() creates a temporary ssh connection (not likely) 4) Firewall or PAM settings on the target prevent to many logins in a certain timeframe. Are you using a queuing system and have the chance to skip SSH and startup MPICH by the queuing system? -- Reuti > The other question is, what are the consequences of “waiting 6 seconds”? Are some non-blocking messages dropped? > > I’m using MPICH 3.3.2, CentOS 3.10 and the Portland Group compiler pgc++ 19.5.0. > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…

1 0

Re: [mpich-discuss] too many ssh connections warning
by Zhou, Hui 02 Dec '19

02 Dec '19

The particular warning is issued by hydra, MPICH’s process manager. Following excerpt is the comment in that source code: ``` /* ssh has many types of security controls that do not allow a * user to ssh to the same node multiple times very * quickly. If this happens, the ssh daemons disables ssh * connections causing the job to fail. This is basically a * hack to slow down ssh connections to the same node. We * check for offset == 0 before applying this hack, so we only * slow down the cases where ssh is being used, and not the * cases where we fall back to fork. */ ``` — Hui Zhou On Dec 2, 2019, at 3:14 PM, Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: My application uses mainly inter-communicators rather than intra-communicators for fault tolerance. A particular process might have 20 inter-communicators active at one time. I’m receiving the warning [mpiexec(a)n010.cluster.com<mailto:[email protected]>] WARNING: too many ssh connections to n009.cluster.com<http://n009.cluster.com/>; waiting 6 seconds What is the cause of this? I have several guesses: 1) MPICH has an internal limit on the number of connections 2) I’m bumping up against a Linux limit on the number of connections 3) Non-blocking communication using MPI_Isend() creates a temporary ssh connection (not likely) The other question is, what are the consequences of “waiting 6 seconds”? Are some non-blocking messages dropped? I’m using MPICH 3.3.2, CentOS 3.10 and the Portland Group compiler pgc++ 19.5.0. _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] too many ssh connections warning
by Reuti 02 Dec '19

02 Dec '19

> Am 02.12.2019 um 22:14 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org>: > > My application uses mainly inter-communicators rather than intra-communicators for fault tolerance. A particular process might have 20 inter-communicators active at one time. I’m receiving the warning > > [mpiexec(a)n010.cluster.com] WARNING: too many ssh connections to n009.cluster.com; waiting 6 seconds > > What is the cause of this? I have several guesses: > > 1) MPICH has an internal limit on the number of connections > 2) I’m bumping up against a Linux limit on the number of connections > 3) Non-blocking communication using MPI_Isend() creates a temporary ssh connection (not likely) 4) Firewall or PAM settings on the target prevent to many logins in a certain timeframe. Are you using a queuing system and have the chance to skip SSH and startup MPICH by the queuing system? -- Reuti > The other question is, what are the consequences of “waiting 6 seconds”? Are some non-blocking messages dropped? > > I’m using MPICH 3.3.2, CentOS 3.10 and the Portland Group compiler pgc++ 19.5.0. > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0