In general, testing your code with more than one MPI implementation (any of MPICH, Open MPI, MVAPICH, Intel MPI, …) will help identify whether the bug is in the user code or the MPI implementation. Rajeev From: Brent Morgan via devel <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Thursday, February 18, 2021 at 1:02 AM To: "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, Robert Katona <[email protected]> Cc: Brent Morgan <[email protected]> Subject: Re: [mpich-devel] Fatal error in PMPI_Isend: Unknown error class Hi all, As usual, we resolved our own issue. It turns out there was an overloading of MPI_Isend/MPI_Irecv with sending/receiving a vector of ~100 elements. We improved this by playing with the grid configuration of sub-communicators, but that failed further down the road. We just switched to mpi_send/mpi_recv which we found to be more robust, but our experiments show this is slower (for reasons that differ between MPI_send and MPI_isend). The reason why we are using send/recv and not BCAST, is because BCAST led to undesired timing behavior. It turns out (that we learned on our own), that broadcast sends data down a tree-like arrangement of ranks (when messages are small and communication time is dominated by network latency) or down a ring of ranks (with large messages). If a rank on an intermediate tree level is late, it delays the broadcast to all other ranks beneath it in the sub-tree. Similarly, in the ring arrangement, a late rank delays all ranks after it in the ring. This can lead to delays in subgroups of ranks if the ranks are not aligned in time (like it is for us). Our team is wondering why there isn't any support on these email discussion boards- is this the official place to get help for MPICH? Are there any other places like github discussions? I would imagine NASA or Argonne National Labs or other government laboratories are needing to communicate issues that arise, or need to send/receive much more data than a vector of 100 elements without issues. The more we implement MPICH on our end, the less confident we are about this mpi implementation. Our errors we have observed have no documented error codes or error traces; we have encountered many issues thus far (and reported them, with no responses)- which led to us scanning mpich source code and trying random things to arrive at solutions, which is far from ideal in any software development environment. We've arrived in a position that if we continue to experience more issues, we are being forced to switch to another implementation (openmpi) because of all the questions we've submitted, none of them were answered or considered at all in neither [email protected]<mailto:[email protected]> or [email protected] and I see others' questions go unanswered as well. We don't understand why there isn't any documentation (at all) on some things we've encountered (and others as well). Best, Brent On Sun, Feb 14, 2021 at 12:20 AM Brent Morgan <[email protected]<mailto:[email protected]>> wrote: Hello all, I am seeing this: "Abort(105) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Isend: Unknown error class" This is after my float vector I am sending to the nodes has increased to a size of ~70 elements. I cannot find any documentation about what this means. I tracked down where this fails and upon sending it to the 300th process (of 600), the MPI_ISend() command dies and this error shows. Is there anything I can do to further diagnose the issue? Best, Brent