My application uses mainly inter-communicators rather than intra-communicators for fault tolerance. A particular process might have 20 inter-communicators active at one time. I'm receiving the warning
[mpiexec(a)n010.cluster.com] WARNING: too many ssh connections to n009.cluster.com; waiting 6 seconds
What is the cause of this? I have several guesses:
1) MPICH has an internal limit on the number of connections
2) I'm bumping up against a Linux limit on the number of connections
3) Non-blocking communication using MPI_Isend() creates a temporary ssh connection (not likely)
The other question is, what are the consequences of "waiting 6 seconds"? Are some non-blocking messages dropped?
I'm using MPICH 3.3.2, CentOS 3.10 and the Portland Group compiler pgc++ 19.5.0.
