Apparently I was having some issues with respect to the libraries that were getting configured when I built MPICH versus the libraries that were available on the compute nodes. I’ve got that sorted out. I configure with --with-device=ch4: ofi
Apparently I was having some issues with respect to the libraries that were getting configured when I built MPICH versus the libraries that were available on the compute nodes. I’ve got that sorted out.
I configure with --with-device=ch4:ofi and got rid of the --without-ucx flag. I’m still getting a crash on MPI_Wait. If I don’t set FI_PROVIDER, then I get the error
Bad : modifier in $ (-).
Abort(944950159) on node 1: Fatal error in internal_Wait: Other MPI error, error stack:
internal_Wait(68205)..........: MPI_Wait(request=0x5e88690, status=0x7ffc63581a40) failed
MPIR_Wait(780)................:
MPIR_Wait_state(737)..........:
MPIDI_progress_test(134)......:
MPIDI_OFI_handle_cq_error(791): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error)
If I set FI_PROVIDER=verbs, I get
Abort(673331087) on node 5 (rank 5 in comm 496): Fatal error in internal_Recv: Other MPI error, error stack:
internal_Recv(61700)..........: MPI_Recv(buf=0x2b4e0fa00000, count=4194304, MPI_CHAR, 0, 27624, comm=0x84000001, status=0x7ffdb2e9bdd0) failed
MPIR_Wait(780)................:
MPIR_Wait_state(737)..........:
MPIDI_progress_test(134)......:
MPIDI_OFI_handle_cq_error(791): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Bad address)
This error appears on ranks 2 and 5 of a 6 processor job on 2 nodes, but the remaining nodes still appear to be failing on an MPI_Wait call.
From:
Zhou, Hui <zhouh@anl.gov>
Date: Friday, June 21, 2024 at 11:03 AM
To: Palmer, Bruce J <Bruce.Palmer@pnnl.gov>, discuss@mpich.org <discuss@mpich.org>
Subject: Re: Fail on MPI_Wait
>../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_decptn/install \
--with-device=ch4:ofi:sockets --with-libfabric=embedded \
--without-ucx CC=gcc CXX=g++
You are statically using "sockets" provider. Try --with-device=ch4:ofi
From: Palmer, Bruce J <Bruce.Palmer@pnnl.gov>
Sent: Friday, June 21, 2024 12:27 PM
To: Zhou, Hui <zhouh@anl.gov>; discuss@mpich.org <discuss@mpich.org>
Subject: RE: Fail on MPI_Wait
Hui, When I set FI_PROVIDER=tcp, the code crashes in MPI_Init. Specifically, this code will fail on one process: #include "mpi. h"
int main(int argc, char **argv) { MPI_Init(&argc, &argv); MPI_Finalize(); } I’m running on a
This Message Is From an External Sender
This message came from outside your organization.
Hui,
When I set FI_PROVIDER=tcp, the code crashes in MPI_Init. Specifically, this code will fail on one process:
#include "mpi.h"
int main(int argc, char **argv) {
MPI_Init(&argc, &argv);
MPI_Finalize();
}
I’m running on a system with the following modules
[d3g293@deception02 testing]$ module list
Currently Loaded Modulefiles:
1) gcc/11.2.0 3) python/3.7.0 5) mkl/2019u4
2) cmake/3.21.4 4) git/2.42.0(default) 6) cuda/11.8
and a home-built version of mpich-4.2.1 configured with
../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_decptn/install \
--with-device=ch4:ofi:sockets --with-libfabric=embedded \
--without-ucx CC=gcc CXX=g++
I thought it might have something to do with using a build configuration in my application build that is set up to include Cuda, but it also fails in MPI_Init
with a non-Cuda configuraton if I set the FI_PROVIDER variable.
Bruce
From: Zhou, Hui <zhouh@anl.gov>
Sent: Friday, June 14, 2024 9:27 AM
To: Palmer, Bruce J <Bruce.Palmer@pnnl.gov>; discuss@mpich.org
Subject: Re: Fail on MPI_Wait
Never mind. It is v4.2.1.
What is the mpich version, BTW?
You are using the sockets provider. Could try set
FI_PROVIDER=tcp to see if it makes a difference?
Meanwhile, if you can get a small reproducer – with the sockets provider or any provider, I'll try to debug it. It is difficult to guess the true source of the issue without a reproducer.
The output to standard out from running on 2 nodes and one process per node is attached. From: Zhou, Hui <zhouh@ anl. gov> Date: Tuesday, June 11, 2024 at 5: 49 PM To: discuss@ mpich. org <discuss@ mpich. org>
Cc: Palmer, Bruce J <Bruce. Palmer@ pnnl. gov>
This Message Is From an External Sender
This message came from outside your organization.
The output to standard out from running on 2 nodes and one process per node is attached.
>MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error)
This is an error coming from the libfabric provider. First we need find out which provider are you using. Try set environment variable
MPIR_CVAR_DEBUG_SUMMARY=1 and run a simple
MPI_INIT+MPI_Finalize test code. Could post its console output?
Hi, I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down
why. Currently, we are getting failures in MPI_Wait
This Message Is From an External Sender
This message came from outside your organization.
Hi,
I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down why. Currently, we are getting failures in
MPI_Wait and were wondering if anyone could provide some information on what exactly seems to be failing inside the wait call. The error we are getting is
Abort(206752655) on node 0: Fatal error in internal_Wait: Other MPI error, error stack:
internal_Wait(68205)..........: MPI_Wait(request=0x500847a0, status=0x7ffff9331800) failed
MPIR_Wait(780)................:
MPIR_Wait_state(737)..........:
MPIDI_progress_test(134)......:
MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error)
I’ve verified that the handle corresponding to
0x500847a0 is getting set earlier in the code in an MPI_Isend call and that no MPI_Wait or MPI_Test is called on the handle before it crashes with the above error message. I’m using MPICH 4.2.1 using gcc/8.3.0. The MPICH library was configured with
../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_newell/install \
--with-device=ch4:ofi:sockets --with-libfabric=embedded \
--without-ucx --enable-threads=multiple --with-slurm \
CC=gcc CXX=g+
I’ve tried building with UCX and gotten the same results.
Are these errors indicative of corruption of the request handle or problems with some internal MPI data structures or something else? Any information you can provide would be appreciated.
Thanks,
Bruce