- discuss - mailman3.cels.anl.gov

Mpich failure with external libfabric on macOS: OFI resource bind failed
by Michka Popoff 01 May '21

01 May '21

Hi Homebrew maintainer here (https://github.com/Homebrew) Homebrew ships mpich as a package on both MacOS and Linux. The issue below was found with version 3.4.1 but might have been there for longer. We noticed that mpich was building it’s own internal libfabric dependency. After reading https://lists.mpich.org/pipermail/discuss/2021-January/006092.html, we added `--with-device=ch4:ofi` and set the libfabric path with the --with-libfabric= flag, to use our own version. The build is fine. We have a small test to check if mpich is still working fine; #include <mpi.h> #include <stdio.h> int main() { int size, rank, nameLen; char name[MPI_MAX_PROCESSOR_NAME]; MPI_Init(NULL, NULL); MPI_Comm_size(MPI_COMM_WORLD, &size); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Get_processor_name(name, &nameLen); printf("[%d/%d] Hello, world! My name is %s.\\n", rank, size, name); MPI_Finalize(); return 0; } Executing the test fails with a weird error: /usr/local/Cellar/mpich/3.4.1_2/bin/mpicc hello.c -o hello ./hello Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(152).......: MPID_Init(597)..............: MPIDI_OFI_mpi_init_hook(674): create_vni_context(964).....: OFI resource bind failed (ofi_init.c:964:create_vni_context:No message available on STREAM) [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247 : system msg for write_line failure : Bad file descriptor Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(152).......: MPID_Init(597)..............: MPIDI_OFI_mpi_init_hook(674): create_vni_context(964).....: OFI resource bind failed (ofi_init.c:964:create_vni_context:No message available on STREAM) [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247 : system msg for write_line failure : Bad file descriptor This test passes fine on Linux, and fails only on MacOS. Using the internal libfabric is fine on both platforms. Here is the related discussion: https://github.com/Homebrew/homebrew-core/pull/73062 Maybe you could help us debug this issue? Regards Michka

1 0

Re: [mpich-discuss] Efficient message size for MPI_Bcast
by Latham, Robert J. 23 Apr '21

23 Apr '21

On Fri, 2021-04-23 at 19:31 +0000, Mccall, Kurt E. (MSFC-EV41) via discuss wrote: > I have a file containing multiple Mb of data that I need to be read > by all of my processes. I’m assuming it is more efficient to have > one process read it in and then broadcast it to the rest, rather than > have all processes (about 20) hammer our NFS server (?). You could have all processes read the file collectively. Normally MPICH won't try to optimize such a simple workload, so additionally you'd have to set the "romio_cb_read" hint to "enable". Did your eyes glaze over? Yeah, that's a little inside-baseball for sure. Read and broadcast is not a bad solution, either. What's the best size to use? Depends on your network. You could run a message size benchmark: just about every network shows an s shape, where small messages perform worse than larger messages up to a point and then things level off. MPI_Bcast is going to behave a bit differently than a point-to-point workload, but the same general pattern should hold: Broadcasting a bunch of 8 byte regions is going to be bad, 1 MiB regions significantly better, 8 MiB at a time might be a few percentage points better but will probably perfom the same. Some benchmarks to help answer your question on your specific hardware if you want a more precise number: - Intel MPI benchmarks https://software.intel.com/content/www/us/en/develop/articles/intel-mpi-ben… - LLNL's 'mpiBench' https://github.com/LLNL/mpiBench - OSU MPI Benchmarks https://mvapich.cse.ohio-state.edu/benchmarks/ (though the broadcast benchmark reports latency: I think the inverse will give you bandwidth) Reading from NFS in this way is about as good as you can expect. Please do not expect anything more than "best effort" when writing to NFS in parallel. It is not a parallel file system and in fact goes out of its way to make life hard for MPI writes (cache behavior is unpredictable, blocks are falsely shared among processes, and the single nfs server is a communication bottlenck) ==rob

1 0

Efficient message size for MPI_Bcast
by Mccall, Kurt E. (MSFC-EV41) 23 Apr '21

23 Apr '21

I have a file containing multiple Mb of data that I need to be read by all of my processes. I'm assuming it is more efficient to have one process read it in and then broadcast it to the rest, rather than have all processes (about 20) hammer our NFS server (?). I can potentially break the data into multiple broadcast messages. What is the recommended maximum message size for MPI_Bcast? Thanks, Kurt

1 0

Re: [mpich-discuss] Supporting > 64K ranks in CH4/UCX netmod
by Zhou, Hui 21 Apr '21

21 Apr '21

Hi Min, Following up, we have a pull request, https://github.com/pmodels/mpich/pull/5201, should add the configure option --with-ch4-ucx-rankbits=<N> . You can configure the rankbits for ch4:ucx to between 16 and 32. We are expected to merge this PR soon. -- Hui Zhou From: Zhou, Hui <zhouh(a)anl.gov> Date: Thursday, April 8, 2021 at 10:18 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: M Xie <xmxmxie(a)gmail.com> Subject: Re: [mpich-discuss] Supporting > 64K ranks in CH4/UCX netmod Hi Min, I think we can do something about it. We’ll follow-up when we have updates. -- Hui Zhou From: M Xie via discuss <discuss(a)mpich.org> Date: Thursday, April 8, 2021 at 12:30 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: M Xie <xmxmxie(a)gmail.com> Subject: [mpich-discuss] Supporting > 64K ranks in CH4/UCX netmod Hi, I am using MPICH on CH4/UCX netmod, the version is mpich-3.4.1. I noticed that there is a configure parameter "--with-ch4-rank-bits" which can set the value of CH4_RANK_BITS, but seems CH4_RANK_BITS is not used in the code. And I also find in the netmod/ucx/ucx_impl.h, _UCX_init_tag()/_UCX_recv_tag() use only 16 bits to set MPI rank in the ucp_tag, but this cannot differentiate correct ucp_tag when MPI ranks exceed 64K. In Open MPI, 20 bits is used in pml/ucx module to set rank in ucp_tag, 20 bits for context, 24 bits for MPI tag, thus the maximum ranks in Open MPI can be 1M. Is there any plan to support > 64K ranks in MPICH/CH4/UCX? Thanks. Min

1 0

Re: [mpich-discuss] MPI_Comm_Spawn + UCX error
by Zhou, Hui 21 Apr '21

21 Apr '21

Dynamic process is currently not supported with ch4:ucx. -- Hui Zhou From: Iker Martín Álvarez via discuss <discuss(a)mpich.org> Date: Wednesday, April 21, 2021 at 6:22 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Iker Martín Álvarez <martini(a)uji.es> Subject: [mpich-discuss] MPI_Comm_Spawn + UCX error Hello, I have been working with the MPI_Comm_spawn function, which was working fine for a simple compiled version of MPICH 3.4.1, where in the configure step it only has the "--prefix" argument. However, when this function was called with another compiled version of MPICH 3.4.1 which uses Infiniband, an error arised. Am I missing some arguments in the compilation step of MPICH when using UCX? Here is the output of mpichversion: $ mpichversion MPICH Version: 3.4.1 MPICH Release date: Fri Jan 22 14:17:48 CST 2021 MPICH Device: ch4:ucx MPICH configure: --prefix=/soft/gnu/mpich-3.4.1-ucx --with-device=ch4:ucx --with-ucx=/soft/gnu/ucx-1.11 MPICH CC: gcc -O2 MPICH CXX: g++ -O2 MPICH F77: gfortran -O2 MPICH FC: gfortran -O2 MPICH Custom Information: The following is the information about the minimal code which arises the error Source: https://www.rookiehpc.com/mpi/docs/mpi_comm_spawn.php Compiling: mpicc mpi_spawn.c Running: mpirun -np 2 ./a.out We are processes spawned directly by you, we now spawn a new instance of an MPI application. We are processes spawned directly by you, we now spawn a new instance of an MPI application. Assertion failed in file src/mpid/ch4/src/ch4_init.c at line 651: MPIR_Process.comm_parent != NULL /soft/gnu/mpich-3.4.1-ucx/lib/libmpi.so.12(MPL_backtrace_show+0x39) [0x7fe15d506d41] /soft/gnu/mpich-3.4.1-ucx/lib/libmpi.so.12(+0x32eaa8) [0x7fe15d4a6aa8] /soft/gnu/mpich-3.4.1-ucx/lib/libmpi.so.12(+0x3602f8) [0x7fe15d4d82f8] /soft/gnu/mpich-3.4.1-ucx/lib/libmpi.so.12(+0x225803) [0x7fe15d39d803] /soft/gnu/mpich-3.4.1-ucx/lib/libmpi.so.12(PMPI_Init+0xa8) [0x7fe15d39d598] ./a.out(+0x123e) [0x55ece110a23e] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fe15cfa10b3] ./a.out(+0x114e) [0x55ece110a14e] Abort(1) on node 0: Internal error

1 0

MPI_Comm_Spawn + UCX error
by Iker Martín Álvarez 21 Apr '21

21 Apr '21

Hello, I have been working with the MPI_Comm_spawn function, which was working fine for a simple compiled version of MPICH 3.4.1, where in the *configure* step it only has the "--prefix" argument. However, when this function was called with another compiled version of MPICH 3.4.1 which uses Infiniband, an error arised. Am I missing some arguments in the compilation step of MPICH when using UCX? Here is the output of *mpichversion*: $ mpichversion MPICH Version: 3.4.1 MPICH Release date: Fri Jan 22 14:17:48 CST 2021 MPICH Device: ch4:ucx MPICH configure: --prefix=/soft/gnu/mpich-3.4.1-ucx --with-device=ch4:ucx --with-ucx=/soft/gnu/ucx-1.11 MPICH CC: gcc -O2 MPICH CXX: g++ -O2 MPICH F77: gfortran -O2 MPICH FC: gfortran -O2 MPICH Custom Information: The following is the information about the minimal code which arises the error Source: https://www.rookiehpc.com/mpi/docs/mpi_comm_spawn.php Compiling: mpicc mpi_spawn.c Running: mpirun -np 2 ./a.out We are processes spawned directly by you, we now spawn a new instance of an MPI application. We are processes spawned directly by you, we now spawn a new instance of an MPI application. Assertion failed in file src/mpid/ch4/src/ch4_init.c at line 651: MPIR_Process.comm_parent != NULL /soft/gnu/mpich-3.4.1-ucx/lib/libmpi.so.12(MPL_backtrace_show+0x39) [0x7fe15d506d41] /soft/gnu/mpich-3.4.1-ucx/lib/libmpi.so.12(+0x32eaa8) [0x7fe15d4a6aa8] /soft/gnu/mpich-3.4.1-ucx/lib/libmpi.so.12(+0x3602f8) [0x7fe15d4d82f8] /soft/gnu/mpich-3.4.1-ucx/lib/libmpi.so.12(+0x225803) [0x7fe15d39d803] /soft/gnu/mpich-3.4.1-ucx/lib/libmpi.so.12(PMPI_Init+0xa8) [0x7fe15d39d598] ./a.out(+0x123e) [0x55ece110a23e] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fe15cfa10b3] ./a.out(+0x114e) [0x55ece110a14e] Abort(1) on node 0: Internal error

1 0

Re: [mpich-discuss] Supporting > 64K ranks in CH4/UCX netmod
by Amit Ruhela 08 Apr '21

08 Apr '21

We have seen this issue with Intel MPI as well and the solution was to set the following two variables. export MPIR_CVAR_CH4_OFI_RANK_BITS=17 export MPIR_CVAR_CH4_OFI_TAG_BITS=$((41 - MPIR_CVAR_CH4_OFI_RANK_BITS)) The total length is 41 and can be adjusted for desired ranks and tags count. Thanks, Amit Ruhela ________________________________ From: Zhou, Hui via discuss <discuss(a)mpich.org> Sent: Thursday, April 8, 2021 10:18 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Zhou, Hui <zhouh(a)anl.gov> Subject: Re: [mpich-discuss] Supporting > 64K ranks in CH4/UCX netmod Hi Min, I think we can do something about it. We’ll follow-up when we have updates. -- Hui Zhou From: M Xie via discuss <discuss(a)mpich.org> Date: Thursday, April 8, 2021 at 12:30 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: M Xie <xmxmxie(a)gmail.com> Subject: [mpich-discuss] Supporting > 64K ranks in CH4/UCX netmod Hi, I am using MPICH on CH4/UCX netmod, the version is mpich-3.4.1. I noticed that there is a configure parameter "--with-ch4-rank-bits" which can set the value of CH4_RANK_BITS, but seems CH4_RANK_BITS is not used in the code. And I also find in the netmod/ucx/ucx_impl.h, _UCX_init_tag()/_UCX_recv_tag() use only 16 bits to set MPI rank in the ucp_tag, but this cannot differentiate correct ucp_tag when MPI ranks exceed 64K. In Open MPI, 20 bits is used in pml/ucx module to set rank in ucp_tag, 20 bits for context, 24 bits for MPI tag, thus the maximum ranks in Open MPI can be 1M. Is there any plan to support > 64K ranks in MPICH/CH4/UCX? Thanks. Min

1 0

Re: [mpich-discuss] Supporting > 64K ranks in CH4/UCX netmod
by Zhou, Hui 08 Apr '21

08 Apr '21

Hi Min, I think we can do something about it. We’ll follow-up when we have updates. -- Hui Zhou From: M Xie via discuss <discuss(a)mpich.org> Date: Thursday, April 8, 2021 at 12:30 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: M Xie <xmxmxie(a)gmail.com> Subject: [mpich-discuss] Supporting > 64K ranks in CH4/UCX netmod Hi, I am using MPICH on CH4/UCX netmod, the version is mpich-3.4.1. I noticed that there is a configure parameter "--with-ch4-rank-bits" which can set the value of CH4_RANK_BITS, but seems CH4_RANK_BITS is not used in the code. And I also find in the netmod/ucx/ucx_impl.h, _UCX_init_tag()/_UCX_recv_tag() use only 16 bits to set MPI rank in the ucp_tag, but this cannot differentiate correct ucp_tag when MPI ranks exceed 64K. In Open MPI, 20 bits is used in pml/ucx module to set rank in ucp_tag, 20 bits for context, 24 bits for MPI tag, thus the maximum ranks in Open MPI can be 1M. Is there any plan to support > 64K ranks in MPICH/CH4/UCX? Thanks. Min

1 0

Supporting > 64K ranks in CH4/UCX netmod
by M Xie 08 Apr '21

08 Apr '21

Hi, I am using MPICH on CH4/UCX netmod, the version is mpich-3.4.1. I noticed that there is a configure parameter "--with-ch4-rank-bits" which can set the value of CH4_RANK_BITS, but seems CH4_RANK_BITS is not used in the code. And I also find in the netmod/ucx/ucx_impl.h, _UCX_init_tag()/_UCX_recv_tag() use only 16 bits to set MPI rank in the ucp_tag, but this cannot differentiate correct ucp_tag when MPI ranks exceed 64K. In Open MPI, 20 bits is used in pml/ucx module to set rank in ucp_tag, 20 bits for context, 24 bits for MPI tag, thus the maximum ranks in Open MPI can be 1M. Is there any plan to support > 64K ranks in MPICH/CH4/UCX? Thanks. Min

1 0

Re: [mpich-discuss] MPICH taking MUCH longer to run same job compared to srun
by Zhou, Hui 30 Mar '21

30 Mar '21

PMI2 is different from PMI. Could you try configure mpich with `--with-pmi=pmi2/simple`? -- Hui Zhou From: Anne Hammond via discuss <discuss(a)mpich.org> Date: Tuesday, March 30, 2021 at 5:51 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Anne Hammond <hammond(a)txcorp.com> Subject: [mpich-discuss] MPICH taking MUCH longer to run same job compared to srun Is pmi2 different from pmi?

1 0