Hi, I am using mpich on a hadoop cluster with yarn. It is not a smooth
integration but it is working.
Flowing an old post, I am using the manual launcher and the option
-disable-hostname-propagation
(https://lists.mpich.org/mailman/htdig/devel/2016-July/000717.html)
The command line to launch by binary is :
mpiexec -launcher manual -disable-hostname-propagation -n 6 -f ./hosts
/path/to/my/bin
And the file hosts contains
host1:2
host2:2
host3:2
I expect to have 2 processes on each host but most of the time it
launches 3 processes on host1, 1 on host2 and 2 on host3
Any ideas?
Bruno
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.
All,
The MPICH team will host a Birds-of-a-Feather session this Wednesday evening at 5:15pm CST as part of the SC22 proceedings. We hope to see you there to discuss the latest developments in MPICH, and to hear from partners Intel, HPE, ParTec, MVAPICH, NVIDIA, and Cornelis Networks. Visit the link below for more info.
https://github.com/pmodels/mpich/wiki/2022-Supercomputing-BoF
Cheers,
The MPICH team
Oops. I forgot to hit send on this after I wrote it. Sorry about the delay.
You can download the latest ga release at https://github.com/GlobalArrays/ga/releases/tag/v5.8.2 (this is what I’ve been testing with lately). I’ve been configuring with
../configure --enable-i4 --enable-cxx --with-mpi3\
--prefix=/people/d3g293/tmp/ga-hotfiix/build_rma_314 CC=mpicc CXX=mpicxx FC=mpif90 \
CFLAGS="-g" CXXFLAG="-g" FFLAGS="-g"
After configuring and building, you can run the test suite using
make check-ga MPIEXEC="mpirun -n 4 "
Our standard configuration for running the test suite is 4 processors on 2 nodes.
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Friday, November 4, 2022 at 2:09 PM
To: discuss(a)mpich.org <discuss(a)mpich.org>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>
Subject: Re: [mpich-discuss] Crash on MPI_Rput
Hi Bruce,
Is the test suite available for us to checkout and test?
--
Hui
________________________________
From: Palmer, Bruce J via discuss <discuss(a)mpich.org>
Sent: Friday, November 4, 2022 4:03 PM
To: discuss(a)mpich.org <discuss(a)mpich.org>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>
Subject: Re: [mpich-discuss] Crash on MPI_Rput
I kind of dropped this for a while but I’d like to pick it back up. I did some more testing using different versions of mpich and got the following results for the RMA runtime
MPICH-3.1.4 configured with
./configure --prefix=/people/d3g293/mpich/mpich-3.1.4/install --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++
2/80 tests fail in GA test suite
MPICH-4.0.2 configured with
unset F90
./configure --prefix=/people/d3g293/mpich/mpich-4.0.2/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++
25/80 tests fail in GA test suite
Running with MPICH-3.3.2 seems to lead to around 8 failures, but my notes on this aren’t that good.
If I run with OpenMPI 4.1.4, everything passes. Any reason for why I’m seeing this? I haven’t really done much to this runtime in the last few years.
Bruce
From: Palmer, Bruce J via discuss <discuss(a)mpich.org>
Date: Wednesday, September 28, 2022 at 12:30 PM
To: 'Thakur, Rajeev' <thakur(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>, Zhou, Hui <zhouh(a)anl.gov>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>
Subject: Re: [mpich-discuss] Crash on MPI_Rput
Check twice before you click! This email originated from outside PNNL.
I think the MPI-RMA runtime was mostly (maybe completely) working with 3.2-3.4. It may have even been working earlier with 4.0. I think there is a pretty good chance that the problem is a system configuration problem at our end and I was hoping that you might have some insight into what it might be based on the errors I’m seeing. I can try running with a few earlier versions of mpich and see if any of them work better.
Bruce
From: Thakur, Rajeev <thakur(a)anl.gov>
Sent: Wednesday, September 28, 2022 12:24 PM
To: discuss(a)mpich.org; Zhou, Hui <zhouh(a)anl.gov>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>
Subject: Re: [mpich-discuss] Crash on MPI_Rput
Was it working with an earlier version of MPICH? If so, which one?
Rajeev
From: "Palmer, Bruce J via discuss" <discuss(a)mpich.org<mailto:[email protected]>>
Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>>
Date: Wednesday, September 28, 2022 at 2:20 PM
To: "Zhou, Hui" <zhouh(a)anl.gov<mailto:[email protected]>>, "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>>
Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>
Subject: Re: [mpich-discuss] Crash on MPI_Rput
I upgraded to mpich-4.0.2 (latest stable release) and get pretty much the same result. This failure is reproducible, I get the same error on multiple runs so it doesn’t look like an unexpected process failure.
One other feature that I forgot to mention earlier is that I’m running this test on 4 processors distributed over 2 nodes. If I run 4 processes on 1 node, the code runs without error.
Bruce
From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>
Date: Tuesday, September 27, 2022 at 2:55 PM
To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>
Subject: Re: Crash on MPI_Rput
Hi Bruce,
* srun: error: node003: task 1: Exited with exit code 7
Looks like one of the process crashed unexpectedly.
--
Hui Zhou
From: Palmer, Bruce J via discuss <discuss(a)mpich.org<mailto:[email protected]>>
Date: Tuesday, September 27, 2022 at 3:32 PM
To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>
Subject: [mpich-discuss] Crash on MPI_Rput
Hi,
I’m testing the MPI-RMA runtime in Global Arrays and I’m getting a lot more crashes than I’ve seen in the past. The MPI-RMA runtime code is fairly stable and hasn’t been modified much recently and all the tests used to pass using one of the more recent MPICH releases. However, I’m getting significant crashes at this point. One of them occurs in a program designed to test non-blocking communication. It creates an MPI window, using MPI_Alloc_mem followed by MPI_Win_create and then calls MPI_Win_lock_all on the window. The code currently crashes when it gets to an MPI_Rput call. I’m trying to see if there is something different in the environment that might be causing this.
I’m currently up to MPICH-4.0b1 configured with
./configure --prefix=/people/d3g293/mpich/mpich-4.0b1/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++
#./configure --prefix=/people/d3g293/mpich/mpich-3.4.1/install-newell-nocuda --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++
I’ve tried other recent vintages of MPICH, but I get similar results. The error I’m seeing when the program crashes is
[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:899): assert (!closed) failed
[proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
srun: error: node003: task 1: Exited with exit code 7
[proxy:0:[email protected]] main (pm/pmiserv/pmip.c:169): demux engine error waiting for event
[mpiexec(a)node002.local] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:74): one of the processes terminated badly; aborting
[mpiexec(a)node002.local] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion
[mpiexec(a)node002.local] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:179): launcher returned error waiting for completion
[mpiexec(a)node002.local] main (ui/mpich/mpiexec.c:325): process manager error waiting for completion
Any suggestions about what might be going wrong here? It could be a problem with the machine configuration, since this code seemed to be running fine a while ago and has not been modified since then. I’ll try building the latest stable release and see if that fixes anything, but as I mentioned none of the recent releases seems to work.
Bruce Palmer
Computer Scientist
Pacific Northwest National Laboratory
(509) 375-3899
Hi Bruce,
Is the test suite available for us to checkout and test?
--
Hui
________________________________
From: Palmer, Bruce J via discuss <discuss(a)mpich.org>
Sent: Friday, November 4, 2022 4:03 PM
To: discuss(a)mpich.org <discuss(a)mpich.org>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>
Subject: Re: [mpich-discuss] Crash on MPI_Rput
I kind of dropped this for a while but I’d like to pick it back up. I did some more testing using different versions of mpich and got the following results for the RMA runtime
MPICH-3.1.4 configured with
./configure --prefix=/people/d3g293/mpich/mpich-3.1.4/install --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++
2/80 tests fail in GA test suite
MPICH-4.0.2 configured with
unset F90
./configure --prefix=/people/d3g293/mpich/mpich-4.0.2/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++
25/80 tests fail in GA test suite
Running with MPICH-3.3.2 seems to lead to around 8 failures, but my notes on this aren’t that good.
If I run with OpenMPI 4.1.4, everything passes. Any reason for why I’m seeing this? I haven’t really done much to this runtime in the last few years.
Bruce
From: Palmer, Bruce J via discuss <discuss(a)mpich.org>
Date: Wednesday, September 28, 2022 at 12:30 PM
To: 'Thakur, Rajeev' <thakur(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>, Zhou, Hui <zhouh(a)anl.gov>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>
Subject: Re: [mpich-discuss] Crash on MPI_Rput
Check twice before you click! This email originated from outside PNNL.
I think the MPI-RMA runtime was mostly (maybe completely) working with 3.2-3.4. It may have even been working earlier with 4.0. I think there is a pretty good chance that the problem is a system configuration problem at our end and I was hoping that you might have some insight into what it might be based on the errors I’m seeing. I can try running with a few earlier versions of mpich and see if any of them work better.
Bruce
From: Thakur, Rajeev <thakur(a)anl.gov>
Sent: Wednesday, September 28, 2022 12:24 PM
To: discuss(a)mpich.org; Zhou, Hui <zhouh(a)anl.gov>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>
Subject: Re: [mpich-discuss] Crash on MPI_Rput
Was it working with an earlier version of MPICH? If so, which one?
Rajeev
From: "Palmer, Bruce J via discuss" <discuss(a)mpich.org<mailto:[email protected]>>
Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>>
Date: Wednesday, September 28, 2022 at 2:20 PM
To: "Zhou, Hui" <zhouh(a)anl.gov<mailto:[email protected]>>, "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>>
Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>
Subject: Re: [mpich-discuss] Crash on MPI_Rput
I upgraded to mpich-4.0.2 (latest stable release) and get pretty much the same result. This failure is reproducible, I get the same error on multiple runs so it doesn’t look like an unexpected process failure.
One other feature that I forgot to mention earlier is that I’m running this test on 4 processors distributed over 2 nodes. If I run 4 processes on 1 node, the code runs without error.
Bruce
From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>
Date: Tuesday, September 27, 2022 at 2:55 PM
To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>
Subject: Re: Crash on MPI_Rput
Hi Bruce,
* srun: error: node003: task 1: Exited with exit code 7
Looks like one of the process crashed unexpectedly.
--
Hui Zhou
From: Palmer, Bruce J via discuss <discuss(a)mpich.org<mailto:[email protected]>>
Date: Tuesday, September 27, 2022 at 3:32 PM
To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>
Subject: [mpich-discuss] Crash on MPI_Rput
Hi,
I’m testing the MPI-RMA runtime in Global Arrays and I’m getting a lot more crashes than I’ve seen in the past. The MPI-RMA runtime code is fairly stable and hasn’t been modified much recently and all the tests used to pass using one of the more recent MPICH releases. However, I’m getting significant crashes at this point. One of them occurs in a program designed to test non-blocking communication. It creates an MPI window, using MPI_Alloc_mem followed by MPI_Win_create and then calls MPI_Win_lock_all on the window. The code currently crashes when it gets to an MPI_Rput call. I’m trying to see if there is something different in the environment that might be causing this.
I’m currently up to MPICH-4.0b1 configured with
./configure --prefix=/people/d3g293/mpich/mpich-4.0b1/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++
#./configure --prefix=/people/d3g293/mpich/mpich-3.4.1/install-newell-nocuda --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++
I’ve tried other recent vintages of MPICH, but I get similar results. The error I’m seeing when the program crashes is
[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:899): assert (!closed) failed
[proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
srun: error: node003: task 1: Exited with exit code 7
[proxy:0:[email protected]] main (pm/pmiserv/pmip.c:169): demux engine error waiting for event
[mpiexec(a)node002.local] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:74): one of the processes terminated badly; aborting
[mpiexec(a)node002.local] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion
[mpiexec(a)node002.local] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:179): launcher returned error waiting for completion
[mpiexec(a)node002.local] main (ui/mpich/mpiexec.c:325): process manager error waiting for completion
Any suggestions about what might be going wrong here? It could be a problem with the machine configuration, since this code seemed to be running fine a while ago and has not been modified since then. I’ll try building the latest stable release and see if that fixes anything, but as I mentioned none of the recent releases seems to work.
Bruce Palmer
Computer Scientist
Pacific Northwest National Laboratory
(509) 375-3899
I kind of dropped this for a while but I’d like to pick it back up. I did some more testing using different versions of mpich and got the following results for the RMA runtime
MPICH-3.1.4 configured with
./configure --prefix=/people/d3g293/mpich/mpich-3.1.4/install --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++
2/80 tests fail in GA test suite
MPICH-4.0.2 configured with
unset F90
./configure --prefix=/people/d3g293/mpich/mpich-4.0.2/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++
25/80 tests fail in GA test suite
Running with MPICH-3.3.2 seems to lead to around 8 failures, but my notes on this aren’t that good.
If I run with OpenMPI 4.1.4, everything passes. Any reason for why I’m seeing this? I haven’t really done much to this runtime in the last few years.
Bruce
From: Palmer, Bruce J via discuss <discuss(a)mpich.org>
Date: Wednesday, September 28, 2022 at 12:30 PM
To: 'Thakur, Rajeev' <thakur(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>, Zhou, Hui <zhouh(a)anl.gov>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>
Subject: Re: [mpich-discuss] Crash on MPI_Rput
Check twice before you click! This email originated from outside PNNL.
I think the MPI-RMA runtime was mostly (maybe completely) working with 3.2-3.4. It may have even been working earlier with 4.0. I think there is a pretty good chance that the problem is a system configuration problem at our end and I was hoping that you might have some insight into what it might be based on the errors I’m seeing. I can try running with a few earlier versions of mpich and see if any of them work better.
Bruce
From: Thakur, Rajeev <thakur(a)anl.gov>
Sent: Wednesday, September 28, 2022 12:24 PM
To: discuss(a)mpich.org; Zhou, Hui <zhouh(a)anl.gov>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>
Subject: Re: [mpich-discuss] Crash on MPI_Rput
Was it working with an earlier version of MPICH? If so, which one?
Rajeev
From: "Palmer, Bruce J via discuss" <discuss(a)mpich.org<mailto:[email protected]>>
Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>>
Date: Wednesday, September 28, 2022 at 2:20 PM
To: "Zhou, Hui" <zhouh(a)anl.gov<mailto:[email protected]>>, "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>>
Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>
Subject: Re: [mpich-discuss] Crash on MPI_Rput
I upgraded to mpich-4.0.2 (latest stable release) and get pretty much the same result. This failure is reproducible, I get the same error on multiple runs so it doesn’t look like an unexpected process failure.
One other feature that I forgot to mention earlier is that I’m running this test on 4 processors distributed over 2 nodes. If I run 4 processes on 1 node, the code runs without error.
Bruce
From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>
Date: Tuesday, September 27, 2022 at 2:55 PM
To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>
Subject: Re: Crash on MPI_Rput
Hi Bruce,
* srun: error: node003: task 1: Exited with exit code 7
Looks like one of the process crashed unexpectedly.
--
Hui Zhou
From: Palmer, Bruce J via discuss <discuss(a)mpich.org<mailto:[email protected]>>
Date: Tuesday, September 27, 2022 at 3:32 PM
To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>
Subject: [mpich-discuss] Crash on MPI_Rput
Hi,
I’m testing the MPI-RMA runtime in Global Arrays and I’m getting a lot more crashes than I’ve seen in the past. The MPI-RMA runtime code is fairly stable and hasn’t been modified much recently and all the tests used to pass using one of the more recent MPICH releases. However, I’m getting significant crashes at this point. One of them occurs in a program designed to test non-blocking communication. It creates an MPI window, using MPI_Alloc_mem followed by MPI_Win_create and then calls MPI_Win_lock_all on the window. The code currently crashes when it gets to an MPI_Rput call. I’m trying to see if there is something different in the environment that might be causing this.
I’m currently up to MPICH-4.0b1 configured with
./configure --prefix=/people/d3g293/mpich/mpich-4.0b1/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++
#./configure --prefix=/people/d3g293/mpich/mpich-3.4.1/install-newell-nocuda --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++
I’ve tried other recent vintages of MPICH, but I get similar results. The error I’m seeing when the program crashes is
[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:899): assert (!closed) failed
[proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
srun: error: node003: task 1: Exited with exit code 7
[proxy:0:[email protected]] main (pm/pmiserv/pmip.c:169): demux engine error waiting for event
[mpiexec(a)node002.local] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:74): one of the processes terminated badly; aborting
[mpiexec(a)node002.local] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion
[mpiexec(a)node002.local] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:179): launcher returned error waiting for completion
[mpiexec(a)node002.local] main (ui/mpich/mpiexec.c:325): process manager error waiting for completion
Any suggestions about what might be going wrong here? It could be a problem with the machine configuration, since this code seemed to be running fine a while ago and has not been modified since then. I’ll try building the latest stable release and see if that fixes anything, but as I mentioned none of the recent releases seems to work.
Bruce Palmer
Computer Scientist
Pacific Northwest National Laboratory
(509) 375-3899
Hi Bruce,
Can you can get a small reproducer and share the code?
>From the error messages, it is saying that somehow the process is terminated, possibly killed by `srun`. Could you try `mpirun -v` to see if there are more clues?
--
Hui Zhou
From: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>
Date: Wednesday, September 28, 2022 at 2:19 PM
To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>
Subject: Re: Crash on MPI_Rput
I upgraded to mpich-4.0.2 (latest stable release) and get pretty much the same result. This failure is reproducible, I get the same error on multiple runs so it doesn’t look like an unexpected process failure.
One other feature that I forgot to mention earlier is that I’m running this test on 4 processors distributed over 2 nodes. If I run 4 processes on 1 node, the code runs without error.
Bruce
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Tuesday, September 27, 2022 at 2:55 PM
To: discuss(a)mpich.org <discuss(a)mpich.org>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>
Subject: Re: Crash on MPI_Rput
Hi Bruce,
Ø srun: error: node003: task 1: Exited with exit code 7
Looks like one of the process crashed unexpectedly.
--
Hui Zhou
From: Palmer, Bruce J via discuss <discuss(a)mpich.org>
Date: Tuesday, September 27, 2022 at 3:32 PM
To: discuss(a)mpich.org <discuss(a)mpich.org>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>
Subject: [mpich-discuss] Crash on MPI_Rput
Hi,
I’m testing the MPI-RMA runtime in Global Arrays and I’m getting a lot more crashes than I’ve seen in the past. The MPI-RMA runtime code is fairly stable and hasn’t been modified much recently and all the tests used to pass using one of the more recent MPICH releases. However, I’m getting significant crashes at this point. One of them occurs in a program designed to test non-blocking communication. It creates an MPI window, using MPI_Alloc_mem followed by MPI_Win_create and then calls MPI_Win_lock_all on the window. The code currently crashes when it gets to an MPI_Rput call. I’m trying to see if there is something different in the environment that might be causing this.
I’m currently up to MPICH-4.0b1 configured with
./configure --prefix=/people/d3g293/mpich/mpich-4.0b1/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++
#./configure --prefix=/people/d3g293/mpich/mpich-3.4.1/install-newell-nocuda --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++
I’ve tried other recent vintages of MPICH, but I get similar results. The error I’m seeing when the program crashes is
[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:899): assert (!closed) failed
[proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
srun: error: node003: task 1: Exited with exit code 7
[proxy:0:[email protected]] main (pm/pmiserv/pmip.c:169): demux engine error waiting for event
[mpiexec(a)node002.local] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:74): one of the processes terminated badly; aborting
[mpiexec(a)node002.local] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion
[mpiexec(a)node002.local] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:179): launcher returned error waiting for completion
[mpiexec(a)node002.local] main (ui/mpich/mpiexec.c:325): process manager error waiting for completion
Any suggestions about what might be going wrong here? It could be a problem with the machine configuration, since this code seemed to be running fine a while ago and has not been modified since then. I’ll try building the latest stable release and see if that fixes anything, but as I mentioned none of the recent releases seems to work.
Bruce Palmer
Computer Scientist
Pacific Northwest National Laboratory
(509) 375-3899
I think the MPI-RMA runtime was mostly (maybe completely) working with 3.2-3.4. It may have even been working earlier with 4.0. I think there is a pretty good chance that the problem is a system configuration problem at our end and I was hoping that you might have some insight into what it might be based on the errors I’m seeing. I can try running with a few earlier versions of mpich and see if any of them work better.
Bruce
From: Thakur, Rajeev <thakur(a)anl.gov>
Sent: Wednesday, September 28, 2022 12:24 PM
To: discuss(a)mpich.org; Zhou, Hui <zhouh(a)anl.gov>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>
Subject: Re: [mpich-discuss] Crash on MPI_Rput
Was it working with an earlier version of MPICH? If so, which one?
Rajeev
From: "Palmer, Bruce J via discuss" <discuss(a)mpich.org<mailto:[email protected]>>
Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>>
Date: Wednesday, September 28, 2022 at 2:20 PM
To: "Zhou, Hui" <zhouh(a)anl.gov<mailto:[email protected]>>, "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>>
Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>
Subject: Re: [mpich-discuss] Crash on MPI_Rput
I upgraded to mpich-4.0.2 (latest stable release) and get pretty much the same result. This failure is reproducible, I get the same error on multiple runs so it doesn’t look like an unexpected process failure.
One other feature that I forgot to mention earlier is that I’m running this test on 4 processors distributed over 2 nodes. If I run 4 processes on 1 node, the code runs without error.
Bruce
From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>
Date: Tuesday, September 27, 2022 at 2:55 PM
To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>
Subject: Re: Crash on MPI_Rput
Hi Bruce,
Ø srun: error: node003: task 1: Exited with exit code 7
Looks like one of the process crashed unexpectedly.
--
Hui Zhou
From: Palmer, Bruce J via discuss <discuss(a)mpich.org<mailto:[email protected]>>
Date: Tuesday, September 27, 2022 at 3:32 PM
To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>
Subject: [mpich-discuss] Crash on MPI_Rput
Hi,
I’m testing the MPI-RMA runtime in Global Arrays and I’m getting a lot more crashes than I’ve seen in the past. The MPI-RMA runtime code is fairly stable and hasn’t been modified much recently and all the tests used to pass using one of the more recent MPICH releases. However, I’m getting significant crashes at this point. One of them occurs in a program designed to test non-blocking communication. It creates an MPI window, using MPI_Alloc_mem followed by MPI_Win_create and then calls MPI_Win_lock_all on the window. The code currently crashes when it gets to an MPI_Rput call. I’m trying to see if there is something different in the environment that might be causing this.
I’m currently up to MPICH-4.0b1 configured with
./configure --prefix=/people/d3g293/mpich/mpich-4.0b1/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++
#./configure --prefix=/people/d3g293/mpich/mpich-3.4.1/install-newell-nocuda --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++
I’ve tried other recent vintages of MPICH, but I get similar results. The error I’m seeing when the program crashes is
[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:899): assert (!closed) failed
[proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
srun: error: node003: task 1: Exited with exit code 7
[proxy:0:[email protected]] main (pm/pmiserv/pmip.c:169): demux engine error waiting for event
[mpiexec(a)node002.local] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:74): one of the processes terminated badly; aborting
[mpiexec(a)node002.local] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion
[mpiexec(a)node002.local] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:179): launcher returned error waiting for completion
[mpiexec(a)node002.local] main (ui/mpich/mpiexec.c:325): process manager error waiting for completion
Any suggestions about what might be going wrong here? It could be a problem with the machine configuration, since this code seemed to be running fine a while ago and has not been modified since then. I’ll try building the latest stable release and see if that fixes anything, but as I mentioned none of the recent releases seems to work.
Bruce Palmer
Computer Scientist
Pacific Northwest National Laboratory
(509) 375-3899
Was it working with an earlier version of MPICH? If so, which one?
Rajeev
From: "Palmer, Bruce J via discuss" <discuss(a)mpich.org>
Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org>
Date: Wednesday, September 28, 2022 at 2:20 PM
To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org>
Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov>
Subject: Re: [mpich-discuss] Crash on MPI_Rput
I upgraded to mpich-4.0.2 (latest stable release) and get pretty much the same result. This failure is reproducible, I get the same error on multiple runs so it doesn’t look like an unexpected process failure.
One other feature that I forgot to mention earlier is that I’m running this test on 4 processors distributed over 2 nodes. If I run 4 processes on 1 node, the code runs without error.
Bruce
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Tuesday, September 27, 2022 at 2:55 PM
To: discuss(a)mpich.org <discuss(a)mpich.org>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>
Subject: Re: Crash on MPI_Rput
Hi Bruce,
* srun: error: node003: task 1: Exited with exit code 7
Looks like one of the process crashed unexpectedly.
--
Hui Zhou
From: Palmer, Bruce J via discuss <discuss(a)mpich.org>
Date: Tuesday, September 27, 2022 at 3:32 PM
To: discuss(a)mpich.org <discuss(a)mpich.org>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>
Subject: [mpich-discuss] Crash on MPI_Rput
Hi,
I’m testing the MPI-RMA runtime in Global Arrays and I’m getting a lot more crashes than I’ve seen in the past. The MPI-RMA runtime code is fairly stable and hasn’t been modified much recently and all the tests used to pass using one of the more recent MPICH releases. However, I’m getting significant crashes at this point. One of them occurs in a program designed to test non-blocking communication. It creates an MPI window, using MPI_Alloc_mem followed by MPI_Win_create and then calls MPI_Win_lock_all on the window. The code currently crashes when it gets to an MPI_Rput call. I’m trying to see if there is something different in the environment that might be causing this.
I’m currently up to MPICH-4.0b1 configured with
./configure --prefix=/people/d3g293/mpich/mpich-4.0b1/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++
#./configure --prefix=/people/d3g293/mpich/mpich-3.4.1/install-newell-nocuda --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++
I’ve tried other recent vintages of MPICH, but I get similar results. The error I’m seeing when the program crashes is
[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:899): assert (!closed) failed
[proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
srun: error: node003: task 1: Exited with exit code 7
[proxy:0:[email protected]] main (pm/pmiserv/pmip.c:169): demux engine error waiting for event
[mpiexec(a)node002.local] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:74): one of the processes terminated badly; aborting
[mpiexec(a)node002.local] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion
[mpiexec(a)node002.local] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:179): launcher returned error waiting for completion
[mpiexec(a)node002.local] main (ui/mpich/mpiexec.c:325): process manager error waiting for completion
Any suggestions about what might be going wrong here? It could be a problem with the machine configuration, since this code seemed to be running fine a while ago and has not been modified since then. I’ll try building the latest stable release and see if that fixes anything, but as I mentioned none of the recent releases seems to work.
Bruce Palmer
Computer Scientist
Pacific Northwest National Laboratory
(509) 375-3899
I upgraded to mpich-4.0.2 (latest stable release) and get pretty much the same result. This failure is reproducible, I get the same error on multiple runs so it doesn't look like an unexpected process failure.
One other feature that I forgot to mention earlier is that I'm running this test on 4 processors distributed over 2 nodes. If I run 4 processes on 1 node, the code runs without error.
Bruce
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Tuesday, September 27, 2022 at 2:55 PM
To: discuss(a)mpich.org <discuss(a)mpich.org>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>
Subject: Re: Crash on MPI_Rput
Hi Bruce,
? srun: error: node003: task 1: Exited with exit code 7
Looks like one of the process crashed unexpectedly.
--
Hui Zhou
From: Palmer, Bruce J via discuss <discuss(a)mpich.org>
Date: Tuesday, September 27, 2022 at 3:32 PM
To: discuss(a)mpich.org <discuss(a)mpich.org>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>
Subject: [mpich-discuss] Crash on MPI_Rput
Hi,
I'm testing the MPI-RMA runtime in Global Arrays and I'm getting a lot more crashes than I've seen in the past. The MPI-RMA runtime code is fairly stable and hasn't been modified much recently and all the tests used to pass using one of the more recent MPICH releases. However, I'm getting significant crashes at this point. One of them occurs in a program designed to test non-blocking communication. It creates an MPI window, using MPI_Alloc_mem followed by MPI_Win_create and then calls MPI_Win_lock_all on the window. The code currently crashes when it gets to an MPI_Rput call. I'm trying to see if there is something different in the environment that might be causing this.
I'm currently up to MPICH-4.0b1 configured with
./configure --prefix=/people/d3g293/mpich/mpich-4.0b1/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++
#./configure --prefix=/people/d3g293/mpich/mpich-3.4.1/install-newell-nocuda --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++
I've tried other recent vintages of MPICH, but I get similar results. The error I'm seeing when the program crashes is
[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:899): assert (!closed) failed
[proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
srun: error: node003: task 1: Exited with exit code 7
[proxy:0:[email protected]] main (pm/pmiserv/pmip.c:169): demux engine error waiting for event
[mpiexec(a)node002.local] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:74): one of the processes terminated badly; aborting
[mpiexec(a)node002.local] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion
[mpiexec(a)node002.local] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:179): launcher returned error waiting for completion
[mpiexec(a)node002.local] main (ui/mpich/mpiexec.c:325): process manager error waiting for completion
Any suggestions about what might be going wrong here? It could be a problem with the machine configuration, since this code seemed to be running fine a while ago and has not been modified since then. I'll try building the latest stable release and see if that fixes anything, but as I mentioned none of the recent releases seems to work.
Bruce Palmer
Computer Scientist
Pacific Northwest National Laboratory
(509) 375-3899
Hi Bruce,
* srun: error: node003: task 1: Exited with exit code 7
Looks like one of the process crashed unexpectedly.
--
Hui Zhou
From: Palmer, Bruce J via discuss <discuss(a)mpich.org>
Date: Tuesday, September 27, 2022 at 3:32 PM
To: discuss(a)mpich.org <discuss(a)mpich.org>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>
Subject: [mpich-discuss] Crash on MPI_Rput
Hi,
I’m testing the MPI-RMA runtime in Global Arrays and I’m getting a lot more crashes than I’ve seen in the past. The MPI-RMA runtime code is fairly stable and hasn’t been modified much recently and all the tests used to pass using one of the more recent MPICH releases. However, I’m getting significant crashes at this point. One of them occurs in a program designed to test non-blocking communication. It creates an MPI window, using MPI_Alloc_mem followed by MPI_Win_create and then calls MPI_Win_lock_all on the window. The code currently crashes when it gets to an MPI_Rput call. I’m trying to see if there is something different in the environment that might be causing this.
I’m currently up to MPICH-4.0b1 configured with
./configure --prefix=/people/d3g293/mpich/mpich-4.0b1/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++
#./configure --prefix=/people/d3g293/mpich/mpich-3.4.1/install-newell-nocuda --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++
I’ve tried other recent vintages of MPICH, but I get similar results. The error I’m seeing when the program crashes is
[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:899): assert (!closed) failed
[proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
srun: error: node003: task 1: Exited with exit code 7
[proxy:0:[email protected]] main (pm/pmiserv/pmip.c:169): demux engine error waiting for event
[mpiexec(a)node002.local] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:74): one of the processes terminated badly; aborting
[mpiexec(a)node002.local] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion
[mpiexec(a)node002.local] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:179): launcher returned error waiting for completion
[mpiexec(a)node002.local] main (ui/mpich/mpiexec.c:325): process manager error waiting for completion
Any suggestions about what might be going wrong here? It could be a problem with the machine configuration, since this code seemed to be running fine a while ago and has not been modified since then. I’ll try building the latest stable release and see if that fixes anything, but as I mentioned none of the recent releases seems to work.
Bruce Palmer
Computer Scientist
Pacific Northwest National Laboratory
(509) 375-3899
