My application uses mainly inter-communicators rather than intra-communicators for fault tolerance.    A particular process might have 20 inter-communicators active at one time.   I’m receiving the warning

[mpiexec@n010.cluster.com] WARNING: too many ssh connections to n009.cluster.com; waiting 6 seconds

What is the cause of this?   I have several guesses:

1) MPICH has an internal limit on the number of  connections

2) I’m bumping up against a Linux limit on the number of connections

3) Non-blocking communication using MPI_Isend() creates a temporary ssh connection (not likely)

The other question is, what are  the consequences of “waiting 6 seconds”?   Are some non-blocking messages dropped?

I’m using MPICH 3.3.2, CentOS 3.10 and the Portland Group compiler pgc++ 19.5.0.