On Fri, Apr 21, 2017 at 1:31 PM, Latham, Robert J. <[email protected]> wrote:

On Fri, 2017-04-21 at 16:56 +0000, Sewall, Jason wrote:

...

Folks,

I have been working on some code that does something akin to a 'manual' implementation of gatherv with derived types. I've run into a case where some requests never complete. I think it's a bug in MPICH.:

good news, maybe?

I can't reproduce this with today's MPICH . I get some debug-logging warnings that you didn't free two of your types, but it doesn't hang on my laptop. Those datatype-related allocations are the only valgrind error I see when I run "mpiexec -np 3 ./mpi-gather 256 256"

I was able to reproduce it with d8e9213647e782b77a1e72fde6a7638198ac3be5 on Mac, but only when I ran with MPIR_CVAR_NOLOCAL=1. Jeff

Is it possible a 256 by 256 grids could overflow an integer anywhere? I think master has some integer overflow fixes in the gather path that might not have made it into an MPICH release. Aside from that, I'd have to dig through the history to figure out what might be different.

==rob

...

My minimal code example is still ~300 lines long, so I've stuck it, as well as a run script and some example output here:

https://gist.github.com/jasonsewall/0d1fb12a93e157795786e560733cbf0c

I'm new to this list, so if there's a better place/practice for sharing code, please let me know.

The code mimics a decomposition of a multi-field grid in 2 dimensions. Each grid has a halo region allocated, and this gather is only interested in transmitting the interior of each grid.

The conditions, as far as I can tell, are related to:

1. nranks > 2 2. 'larger' messages: I haven't fully bisected it, but 256x128 grids work, and 256x256 don't. 3. Having halos. With the global integer halow = 2, the strides for the grids (and the resulting hvector derived types) exceed the lengths, and that seems to be important; if you set halow = 0 and run, it completes, independent of the other factors.

The test program uses Waitsome to figure out which requests are completing, and it looks like rank 0 fails to complete a recv from rank 2, despite rank 2 completing its send.

You can fiddle with the hacky 'mask' parameter to have a subset of the ranks send (line 326). -1ULL means all ranks are allowed in, but you get interesting results with 6ULL, which has just ranks 1 and 2 participate in sending, but rank 0 still hangs on the recv from rank 2. (Obviously the correctness test will fail here, since it expects results from all ranks).

I discovered this on Intel MPI, and thanks to my colleague Jeff Hammond's analysis, was able to reproduce the behavior on MPICH 3.2. OpenMPI 2.1.0 successfully completes every case I have tried.

The obvious caveat here is that there could very well be a bug in how I'm using the derived datatypes, but I have tried to carefully check that I'm not trashing random memory. Things seem correct.

Please let me know what I can do help pinpoint the problem. I'm not familiar with the implementation details of any MPI libraries.

Cheers, Jason

_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

-- Jeff Hammond [email protected] http://jeffhammond.github.io/ _______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss