These lock-ups seem to be gone in 3.3a2. I do occasionally get the following though: Unknown error class, error stack: PMPI_Comm_accept(129).................: MPI_Comm_accept(port="tag#0$description#aaa.com$port#36230$ifname#192.168.42.99$", MPI_INFO_NULL, ro ot=180, comm=0x84000003, newcomm=0x7f3cf681842c) failed MPID_Comm_accept(153).................: MPIDI_Comm_accept(1244)...............: MPIR_Get_contextid_sparse_group(499)..: MPIR_Allreduce_impl(755)..............: MPIR_Allreduce_intra(414).............: MPIDU_Complete_posted_with_error(1137): Process failed What does this message mean? some process just exited/died (like with seg fault?) Thank you. -Dmitriy On Thu, Jan 12, 2017 at 11:55 AM, Dmitriy Lyubimov <[email protected]> wrote:

further debugging shows that it's not actually mergeIntercom that locks up but a pair of send/recv that two nodes decide to execute before MPI_intercom_merge.

so the total snapshot of the situation is that everyone waits on mergeIntercom except for two processes that wait in send/recv respectively, while majority of others already have entered collective barrier.

it would seem that this sort of assymetric logic would be acceptable, since the send/recv pair is balanced before the merge is to occur, but in practice it seems to lock up -- increasingly so as the number of participating processes increases. It almost like once collective barrier of certain cardinality is formed, point-to-point messages are not going thru any longer.

If this scenario begets any ideas, please let me know.

thank you! -Dmitriy

On Wed, Jan 11, 2017 at 9:38 AM, Dmitriy Lyubimov <[email protected]> wrote:

...

Maybe it has something to do with the fact that it is stepping thru JVM JNI and that somehow screws threading model of MPI, although it is a single threaded JVM process, and MPI mappings are known to have been done before (e.g., openmpi had an effort towards that).

Strange thing is that i never had lock up with # of processes under 120 but something changes after that, the spurious condition becomes much more common after that. By the time I am at 150 processes in the intercom, I am almost certain to have a merge lock-up.

On Wed, Jan 11, 2017 at 9:34 AM, Dmitriy Lyubimov <[email protected]> wrote:

...

Thanks. it would not be easy for me to do immediately as i am using proprietary scala binding api for MPI.

it would help me to know if there's a known problem like that in the past, or generally mergeIntercomm api is known to work on hundreds of processes. Sounds like there are no known issues with that.

On Tue, Jan 10, 2017 at 11:53 PM, Oden, Lena <[email protected]> wrote:

...

Hello Dmittiy,

can you maybe create a simple example-program to reproduce this failure? It is also often easier also to look at a code example to identify a problem.

Thanks, Lena

...

On Jan 11, 2017, at 2:45 AM, Dmitriy Lyubimov <[email protected]> wrote:

Hello,

(mpich 3.2)

I have a scenario when i add a few extra processes do existing intercom.

it works as a simple loop -- (1) n processes accept on n-intercom (2) 1 process connects (3) intracom is merged into n+1 intercom, intracom and n-intercom are closed (4) repeat 1-3 as needed.

Occasionally, i observe that step 3 spuriously locks up (once i get in the range of 100+ processes). From what i can tell, all processes in step 3 are accounted for, and are waiting on the merge, but nothing happens. the collective barrier locks up.

I really have trouble resolving this issue, any ideas are appreciated!

Thank you very much. -Dmitriy

_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss