- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Possible integer-overflow for MPI_COMM_WORLD in MPI_Iprobe
by Jeff Hammond 04 Dec '19

04 Dec '19

This was fixed in 2019 update 2. Jeff On Fri, Apr 26, 2019 at 8:09 AM Jeff Hammond <jeff.science(a)gmail.com> wrote: > For anyone who cares about this bug because of Intel MPI, I am told it is > fixed in Intel MPI 2019 update 3. > > Jeff > > On Mon, Jan 21, 2019 at 7:46 PM Jeff Hammond <jeff.science(a)gmail.com> > wrote: > >> I was able to reproduce with my own test ( >> https://github.com/jeffhammond/HPCInfo/blob/master/mpi/bugs/iprobe-overflow…) >> with Intel MPI 2019, so I will report that bug to the Intel MPI team. It >> should be easy enough for them to figure out if this bug is from MPICH or >> not. >> >> 2139000000 iterations, 627.143272 seconds >> >> 2140000000 iterations, 627.436206 seconds >> >> 2141000000 iterations, 627.729135 seconds >> >> 2142000000 iterations, 628.022049 seconds >> >> 2143000000 iterations, 628.315015 seconds >> >> 2144000000 iterations, 628.608066 seconds >> >> 2145000000 iterations, 628.901065 seconds >> >> 2146000000 iterations, 629.193992 seconds >> >> 2147000000 iterations, 629.488107 seconds >> >> Abort(738833413) on node 0 (rank 0 in comm 0): Fatal error in >> PMPI_Iprobe: Invalid communicator, error stack: >> >> PMPI_Iprobe(123): MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, >> MPI_COMM_WORLD, flag=0x7ffdf75a396c, status=0x7ffdf75a3970) failed >> >> PMPI_Iprobe(90).: Invalid communicator >> >> Jeff, who works for Intel but knows more about MPICH than Intel MPI >> >> On Mon, Jan 21, 2019 at 11:19 AM Joachim Protze via discuss < >> discuss(a)mpich.org> wrote: >> >>> Hi all, >>> >>> we detected the behavior with IntelMPI 2019 (which is based on MPICH >>> 3.3). Reproducing it with MPICH-3.3 was not yet successful. But I fear, >>> that our built of MPICH just not uses the necessary code path / build >>> flags. >>> >>> When calling MPI_Iprobe with the same communicator for ~2^31 times >>> (which can take 10-30 minutes), the execution stops with: >>> >>> Abort(201962501) on node 0 (rank 0 in comm 0): Fatal error in >>> PMPI_Iprobe: Invalid communicator, error stack: >>> PMPI_Iprobe(123): MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, >>> MPI_COMM_WORLD, flag=0x7ffd925056c0, status=0x7ffd92505694) failed >>> PMPI_Iprobe(90).: Invalid communicator >>> >>> From my understanding of the referenced MPICH code lines, I guess, that >>> the ref-count for MPI_COMM_WORLD overflows, which triggers this error >>> message. >>> >>> Best >>> Joachim >>> >>> -- >>> Dipl.-Inf. Joachim Protze >>> >>> IT Center >>> Group: High Performance Computing >>> Division: Computational Science and Engineering >>> RWTH Aachen University >>> Seffenter Weg 23 >>> D 52074 Aachen (Germany) >>> Tel: +49 241 80- 24765 >>> Fax: +49 241 80-624765 >>> protze(a)itc.rwth-aachen.de >>> www.itc.rwth-aachen.de >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> http://jeffhammond.github.io/ >> > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-discuss] [EXTERNAL] Re: too many ssh connections warning
by Reuti 03 Dec '19

03 Dec '19

Hi: > Am 03.12.2019 um 15:45 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org>: > > Reuti, > > Sorry, I forgot to mention that I am starting the job under PBS/Torque with the qsub command. Then it should be possible to use the task manager interface without `ssh`: http://docs.adaptivecomputing.com/torque/4-2-7/Content/topics/7-messagePass… > I'll check with our sysadmins to see if there are firewall issues. This could also be later an issue if MPICH will connect to other machines directly to talk to the already started daemons. > What is PAM? https://en.wikipedia.org/wiki/Linux_PAM Several limits can be set here, depending on your distribution: ls /lib64/security/ will show the available ones which are installed by default and are then used/configured in /etc/pam.d -- Reuti > Hui Zhou, > > What do you expect would be making multiple SSH connections to the node? The creation of inter-communicators? Individual MPI_Iprobe/MPI_Isend/MPI_IRecv commands? If you have a guess, that would help me know how to fix the problem. > > Kurt > > > -----Original Message----- > From: Reuti <reuti(a)staff.uni-marburg.de> > Sent: Monday, December 2, 2019 3:20 PM > To: discuss(a)mpich.org > Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> > Subject: [EXTERNAL] Re: [mpich-discuss] too many ssh connections warning > > >> Am 02.12.2019 um 22:14 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org>: >> >> My application uses mainly inter-communicators rather than intra-communicators for fault tolerance. A particular process might have 20 inter-communicators active at one time. I’m receiving the warning >> >> [mpiexec(a)n010.cluster.com] WARNING: too many ssh connections to n009.cluster.com; waiting 6 seconds >> >> What is the cause of this? I have several guesses: >> >> 1) MPICH has an internal limit on the number of connections >> 2) I’m bumping up against a Linux limit on the number of connections >> 3) Non-blocking communication using MPI_Isend() creates a temporary ssh connection (not likely) > > 4) Firewall or PAM settings on the target prevent to many logins in a certain timeframe. > > Are you using a queuing system and have the chance to skip SSH and startup MPICH by the queuing system? > > -- Reuti > > >> The other question is, what are the consequences of “waiting 6 seconds”? Are some non-blocking messages dropped? >> >> I’m using MPICH 3.3.2, CentOS 3.10 and the Portland Group compiler pgc++ 19.5.0. >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma… > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] [EXTERNAL] Re: too many ssh connections warning
by Mccall, Kurt E. (MSFC-EV41) 03 Dec '19

03 Dec '19

Reuti, Sorry, I forgot to mention that I am starting the job under PBS/Torque with the qsub command. I'll check with our sysadmins to see if there are firewall issues. What is PAM? Hui Zhou, What do you expect would be making multiple SSH connections to the node? The creation of inter-communicators? Individual MPI_Iprobe/MPI_Isend/MPI_IRecv commands? If you have a guess, that would help me know how to fix the problem. Kurt -----Original Message----- From: Reuti <reuti(a)staff.uni-marburg.de> Sent: Monday, December 2, 2019 3:20 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] too many ssh connections warning > Am 02.12.2019 um 22:14 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org>: > > My application uses mainly inter-communicators rather than intra-communicators for fault tolerance. A particular process might have 20 inter-communicators active at one time. I’m receiving the warning > > [mpiexec(a)n010.cluster.com] WARNING: too many ssh connections to n009.cluster.com; waiting 6 seconds > > What is the cause of this? I have several guesses: > > 1) MPICH has an internal limit on the number of connections > 2) I’m bumping up against a Linux limit on the number of connections > 3) Non-blocking communication using MPI_Isend() creates a temporary ssh connection (not likely) 4) Firewall or PAM settings on the target prevent to many logins in a certain timeframe. Are you using a queuing system and have the chance to skip SSH and startup MPICH by the queuing system? -- Reuti > The other question is, what are the consequences of “waiting 6 seconds”? Are some non-blocking messages dropped? > > I’m using MPICH 3.3.2, CentOS 3.10 and the Portland Group compiler pgc++ 19.5.0. > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…

1 0

Re: [mpich-discuss] too many ssh connections warning
by Zhou, Hui 02 Dec '19

02 Dec '19

The particular warning is issued by hydra, MPICH’s process manager. Following excerpt is the comment in that source code: ``` /* ssh has many types of security controls that do not allow a * user to ssh to the same node multiple times very * quickly. If this happens, the ssh daemons disables ssh * connections causing the job to fail. This is basically a * hack to slow down ssh connections to the same node. We * check for offset == 0 before applying this hack, so we only * slow down the cases where ssh is being used, and not the * cases where we fall back to fork. */ ``` — Hui Zhou On Dec 2, 2019, at 3:14 PM, Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: My application uses mainly inter-communicators rather than intra-communicators for fault tolerance. A particular process might have 20 inter-communicators active at one time. I’m receiving the warning [mpiexec(a)n010.cluster.com<mailto:[email protected]>] WARNING: too many ssh connections to n009.cluster.com<http://n009.cluster.com/>; waiting 6 seconds What is the cause of this? I have several guesses: 1) MPICH has an internal limit on the number of connections 2) I’m bumping up against a Linux limit on the number of connections 3) Non-blocking communication using MPI_Isend() creates a temporary ssh connection (not likely) The other question is, what are the consequences of “waiting 6 seconds”? Are some non-blocking messages dropped? I’m using MPICH 3.3.2, CentOS 3.10 and the Portland Group compiler pgc++ 19.5.0. _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] too many ssh connections warning
by Reuti 02 Dec '19

02 Dec '19

> Am 02.12.2019 um 22:14 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org>: > > My application uses mainly inter-communicators rather than intra-communicators for fault tolerance. A particular process might have 20 inter-communicators active at one time. I’m receiving the warning > > [mpiexec(a)n010.cluster.com] WARNING: too many ssh connections to n009.cluster.com; waiting 6 seconds > > What is the cause of this? I have several guesses: > > 1) MPICH has an internal limit on the number of connections > 2) I’m bumping up against a Linux limit on the number of connections > 3) Non-blocking communication using MPI_Isend() creates a temporary ssh connection (not likely) 4) Firewall or PAM settings on the target prevent to many logins in a certain timeframe. Are you using a queuing system and have the chance to skip SSH and startup MPICH by the queuing system? -- Reuti > The other question is, what are the consequences of “waiting 6 seconds”? Are some non-blocking messages dropped? > > I’m using MPICH 3.3.2, CentOS 3.10 and the Portland Group compiler pgc++ 19.5.0. > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

too many ssh connections warning
by Mccall, Kurt E. (MSFC-EV41) 02 Dec '19

02 Dec '19

My application uses mainly inter-communicators rather than intra-communicators for fault tolerance. A particular process might have 20 inter-communicators active at one time. I'm receiving the warning [mpiexec(a)n010.cluster.com] WARNING: too many ssh connections to n009.cluster.com; waiting 6 seconds What is the cause of this? I have several guesses: 1) MPICH has an internal limit on the number of connections 2) I'm bumping up against a Linux limit on the number of connections 3) Non-blocking communication using MPI_Isend() creates a temporary ssh connection (not likely) The other question is, what are the consequences of "waiting 6 seconds"? Are some non-blocking messages dropped? I'm using MPICH 3.3.2, CentOS 3.10 and the Portland Group compiler pgc++ 19.5.0.

1 0

Re: [mpich-discuss] UCX run-time warnings
by Sajid Ali 27 Nov '19

27 Nov '19

Hi Kenneth, I installed mpich-master with external ucx(a)1.6.0 and I don't get any of the UCX errors. I still have the slurmstepd error but that's a slurm error and probably has nothing to do with mpich. Thanks for your help! -- Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io

1 0

Re: [mpich-discuss] UCX run-time warnings
by Sajid Ali 27 Nov '19

27 Nov '19

Hi Kenneth, Thanks for the spack PR. I'll get back to you with the results with that soon. It looks like the links got scrubbed in the last email so I'll give them here. 1 -> https://github.com/s-sajid-ali/nu-quest-mpi/blob/master/new_mpi_bench/submi… 2 -> https://github.com/s-sajid-ali/nu-quest-mpi/tree/master/new_mpi_bench/mdb6i… 3 -> https://github.com/s-sajid-ali/nu-quest-mpi/blob/master/new_mpi_bench/mdb6i…

1 0

Re: [mpich-discuss] UCX run-time warnings
by Sajid Ali 27 Nov '19

27 Nov '19

Hi Kenneth, I have added a command to print the linking info for the benchmark executable for each mpi build as per [1]. The results for benchmarks built with mpich@develop with gcc(a)8.3.0 are here [2] where the mpi_config file [3] has the ldd info printed out and I’m confirming that the right mpi library is being used. I don’t see any hcoll linking and can confirm that it is picking up the right mpich version. I have also printed out the linking info for mpich itself and attach the output of the following along with this email : [sas4990@quser10 ~]$ ldd $(spack location -i mpich@develop%[email protected])/lib/libmpich.so &> mpich_ldd The only thing that I’m concerned about is that the libmpich.so has no link to libslurmfull.so (but openmpi does link to). Thank You, Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io

1 0

Re: [mpich-discuss] UCX run-time warnings
by Raffenetti, Kenneth J. 27 Nov '19

27 Nov '19

I have also opened a PR on spack to add a dependency on ucx for MPICH. Otherwise an embedded version is used, which *should* work but is not the preferred configuration. https://github.com/spack/spack/pull/13921 Ken On 11/27/19 2:08 PM, Raffenetti, Kenneth J. via discuss wrote: > Those warning messages are not coming from MPICH. Can you run ldd one of > the benchmark binaries and confirm all the expected libraries are linked > to it? Are you perhaps also linked against an hcoll library? > > Ken > > On 11/27/19 12:58 PM, Sajid Ali wrote: >> Hi Kenneth, >> >> Running the same test with mpich@develop, I still see the following >> errors in a typical outfile : >> >> |[sas4990@quser10 mdb6i63]$ cat lat_10 >> ---------------------------------------- ... >> ---------------------------------------- slurmstepd: error: mpi/pmi2: no >> value for key in req slurmstepd: error: mpi/pmi2: no value for key in >> req slurmstepd: error: mpi/pmi2: no value for key in req >> [1574880837.368994] [qnode5038:4743 :0] ucp_context.c:1131 UCX WARN UCP >> version is incompatible, required: 1.5, actual: 1.4 (release 0) >> slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: >> mpi/pmi2: no value for key in req [1574880837.377860] [qnode5039:4546 >> :0] ucp_context.c:1131 UCX WARN UCP version is incompatible, required: >> 1.5, actual: 1.4 (release 0) slurmstepd: error: mpi/pmi2: no value for >> key in req slurmstepd: error: mpi/pmi2: no value for key in req >> slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: >> mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value >> for key in req # OSU MPI_Get latency Test v5.6.1 # Window creation: >> MPI_Win_allocate # Synchronization: MPI_Win_flush # Size Latency (us) 1 >> 2.13 .... | >> >> >> Since UCX is a vendored dependency of MPICH, why is MPICH complaining >> about an incompatible UCX version ? >> >> >> Thank You, >> Sajid Ali | PhD Candidate >> Applied Physics >> Northwestern University >> s-sajid-ali.github.io <http://s-sajid-ali.github.io> >> > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0