May 2021 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Mpich failure with external libfabric on macOS: OFI resource bind failed
by Zhou, Hui 07 May '21

07 May '21

Hi Michka, Thanks for the information. That is helpful. This seems to be an issue with the latest development in libfabric. It is selecting the `udp;ofi_rxd` provider, which has never been tested with MPICH. You can try force the `sockets` provider selection by setting `MPIR_CVAR_OFI_USE_PROVIDER=sockets` or setting `FI_PROVIDER=sockets`. Alternatively, you may want force the sockets provider during configure with `--with-device=ch4:ofi:sockets`. Or you may consider disable the `ofi_rxd` provider in libfabric. I admit either seem non-optimum. The better solution probably should be either fix or tune the provider selection logic in mpich. FYI, the current embedded libfabric is on version 1.10.1 and we are still working on upgrading that to version 1.11.2. -- Hui Zhou From: Michka Popoff <michkapopoff(a)gmail.com> Date: Friday, May 7, 2021 at 4:37 PM To: Zhou, Hui <zhouh(a)anl.gov> Cc: discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [mpich-discuss] Mpich failure with external libfabric on macOS: OFI resource bind failed The libfabric version is 1.12.1 Here is the log as asked: ==== Capability set configuration ==== libfabric provider: udp;ofi_rxd MPIDI_OFI_ENABLE_AV_TABLE: 1 MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS: 0 MPIDI_OFI_ENABLE_SHARED_CONTEXTS: 0 MPIDI_OFI_ENABLE_MR_SCALABLE: 0 MPIDI_OFI_ENABLE_MR_VIRT_ADDRESS: 0 MPIDI_OFI_ENABLE_MR_ALLOCATED: 0 MPIDI_OFI_ENABLE_MR_PROV_KEY: 0 MPIDI_OFI_ENABLE_TAGGED: 1 MPIDI_OFI_ENABLE_AM: 1 MPIDI_OFI_ENABLE_RMA: 1 MPIDI_OFI_ENABLE_ATOMICS: 1 MPIDI_OFI_FETCH_ATOMIC_IOVECS: 1 MPIDI_OFI_ENABLE_DATA_AUTO_PROGRESS: 0 MPIDI_OFI_ENABLE_CONTROL_AUTO_PROGRESS: 0 MPIDI_OFI_ENABLE_PT2PT_NOPACK: 1 MPIDI_OFI_ENABLE_HMEM: 0 MPIDI_OFI_NUM_AM_BUFFERS: 8 MPIDI_OFI_CONTEXT_BITS: 20 MPIDI_OFI_SOURCE_BITS: 0 MPIDI_OFI_TAG_BITS: 31 ====================================== MAXIMUM SUPPORTED RANKS: 4294967296 MAXIMUM TAG: 2147483648 ====================================== Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(152).......: MPID_Init(597)..............: MPIDI_OFI_mpi_init_hook(674): create_vni_context(964).....: OFI resource bind failed (ofi_init.c:964:create_vni_context:No message available on STREAM) … Michka On 4 May 2021, at 00:53, Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> wrote: Hi Michka, Which libfabric version are you using? Could you try setting `MPIR_CVAR_CH4_OFI_CAPABILITY_SETS_DEBUG=` to see if there is more debug messages? -- Hui Zhou ________________________________ From: Michka Popoff via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Saturday, May 1, 2021 4:33 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Michka Popoff <michkapopoff(a)gmail.com<mailto:[email protected]>> Subject: [mpich-discuss] Mpich failure with external libfabric on macOS: OFI resource bind failed Hi Homebrew maintainer here (https://github.com/Homebrew) Homebrew ships mpich as a package on both MacOS and Linux. The issue below was found with version 3.4.1 but might have been there for longer. We noticed that mpich was building it’s own internal libfabric dependency. After reading https://lists.mpich.org/pipermail/discuss/2021-January/006092.html, we added `--with-device=ch4:ofi` and set the libfabric path with the --with-libfabric= flag, to use our own version. The build is fine. We have a small test to check if mpich is still working fine; #include <mpi.h> #include <stdio.h> int main() { int size, rank, nameLen; char name[MPI_MAX_PROCESSOR_NAME]; MPI_Init(NULL, NULL); MPI_Comm_size(MPI_COMM_WORLD, &size); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Get_processor_name(name, &nameLen); printf("[%d/%d] Hello, world! My name is %s.\\n", rank, size, name); MPI_Finalize(); return 0; } Executing the test fails with a weird error: /usr/local/Cellar/mpich/3.4.1_2/bin/mpicc hello.c -o hello ./hello Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(152).......: MPID_Init(597)..............: MPIDI_OFI_mpi_init_hook(674): create_vni_context(964).....: OFI resource bind failed (ofi_init.c:964:create_vni_context:No message available on STREAM) [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247 : system msg for write_line failure : Bad file descriptor Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(152).......: MPID_Init(597)..............: MPIDI_OFI_mpi_init_hook(674): create_vni_context(964).....: OFI resource bind failed (ofi_init.c:964:create_vni_context:No message available on STREAM) [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247 : system msg for write_line failure : Bad file descriptor This test passes fine on Linux, and fails only on MacOS. Using the internal libfabric is fine on both platforms. Here is the related discussion: https://github.com/Homebrew/homebrew-core/pull/73062 Maybe you could help us debug this issue? Regards Michka

1 0

Re: [mpich-discuss] Mpich failure with external libfabric on macOS: OFI resource bind failed
by Michka Popoff 07 May '21

07 May '21

The libfabric version is 1.12.1 Here is the log as asked: ==== Capability set configuration ==== libfabric provider: udp;ofi_rxd MPIDI_OFI_ENABLE_AV_TABLE: 1 MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS: 0 MPIDI_OFI_ENABLE_SHARED_CONTEXTS: 0 MPIDI_OFI_ENABLE_MR_SCALABLE: 0 MPIDI_OFI_ENABLE_MR_VIRT_ADDRESS: 0 MPIDI_OFI_ENABLE_MR_ALLOCATED: 0 MPIDI_OFI_ENABLE_MR_PROV_KEY: 0 MPIDI_OFI_ENABLE_TAGGED: 1 MPIDI_OFI_ENABLE_AM: 1 MPIDI_OFI_ENABLE_RMA: 1 MPIDI_OFI_ENABLE_ATOMICS: 1 MPIDI_OFI_FETCH_ATOMIC_IOVECS: 1 MPIDI_OFI_ENABLE_DATA_AUTO_PROGRESS: 0 MPIDI_OFI_ENABLE_CONTROL_AUTO_PROGRESS: 0 MPIDI_OFI_ENABLE_PT2PT_NOPACK: 1 MPIDI_OFI_ENABLE_HMEM: 0 MPIDI_OFI_NUM_AM_BUFFERS: 8 MPIDI_OFI_CONTEXT_BITS: 20 MPIDI_OFI_SOURCE_BITS: 0 MPIDI_OFI_TAG_BITS: 31 ====================================== MAXIMUM SUPPORTED RANKS: 4294967296 MAXIMUM TAG: 2147483648 ====================================== Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(152).......: MPID_Init(597)..............: MPIDI_OFI_mpi_init_hook(674): create_vni_context(964).....: OFI resource bind failed (ofi_init.c:964:create_vni_context:No message available on STREAM) … Michka > On 4 May 2021, at 00:53, Zhou, Hui <zhouh(a)anl.gov> wrote: > > Hi Michka, > > Which libfabric version are you using? Could you try setting `MPIR_CVAR_CH4_OFI_CAPABILITY_SETS_DEBUG=` to see if there is more debug messages? > > -- > Hui Zhou > From: Michka Popoff via discuss <discuss(a)mpich.org> > Sent: Saturday, May 1, 2021 4:33 PM > To: discuss(a)mpich.org <discuss(a)mpich.org> > Cc: Michka Popoff <michkapopoff(a)gmail.com> > Subject: [mpich-discuss] Mpich failure with external libfabric on macOS: OFI resource bind failed > > Hi > > Homebrew maintainer here (https://github.com/Homebrew <https://github.com/Homebrew>). > Homebrew ships mpich as a package on both MacOS and Linux. > > The issue below was found with version 3.4.1 but might have been there for longer. > > We noticed that mpich was building it’s own internal libfabric dependency. > After reading https://lists.mpich.org/pipermail/discuss/2021-January/006092.html <https://lists.mpich.org/pipermail/discuss/2021-January/006092.html>, > we added `--with-device=ch4:ofi` and set the libfabric path with the --with-libfabric= flag, > to use our own version. > > The build is fine. We have a small test to check if mpich is still working fine; > > #include <mpi.h> > #include <stdio.h> > int main() > { > int size, rank, nameLen; > char name[MPI_MAX_PROCESSOR_NAME]; > MPI_Init(NULL, NULL); > MPI_Comm_size(MPI_COMM_WORLD, &size); > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > MPI_Get_processor_name(name, &nameLen); > printf("[%d/%d] Hello, world! My name is %s.\\n", rank, size, name); > MPI_Finalize(); > return 0; > } > > Executing the test fails with a weird error: > > /usr/local/Cellar/mpich/3.4.1_2/bin/mpicc hello.c -o hello > ./hello > Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in MPI_Init: Other MPI error, error stack: > MPIR_Init_thread(152).......: > MPID_Init(597)..............: > MPIDI_OFI_mpi_init_hook(674): > create_vni_context(964).....: OFI resource bind failed (ofi_init.c:964:create_vni_context:No message available on STREAM) > [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247 > : > system msg for write_line failure : Bad file descriptor > Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in MPI_Init: Other MPI error, error stack: > MPIR_Init_thread(152).......: > MPID_Init(597)..............: > MPIDI_OFI_mpi_init_hook(674): > create_vni_context(964).....: OFI resource bind failed (ofi_init.c:964:create_vni_context:No message available on STREAM) > [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247 > : > system msg for write_line failure : Bad file descriptor > > This test passes fine on Linux, and fails only on MacOS. Using the internal libfabric is fine on both platforms. > > Here is the related discussion: > https://github.com/Homebrew/homebrew-core/pull/73062 <https://github.com/Homebrew/homebrew-core/pull/73062> > > Maybe you could help us debug this issue? > > Regards > > Michka

1 0

Re: [mpich-discuss] Problems to install MPICH 3.4.1 in Ubuntu 20.04.2 LTS 64bits.
by Jean Schuster 06 May '21

06 May '21

Hi Hui, it worked! Thank you very much. Have a nice day. Best regards, Jean. Em qui., 6 de mai. de 2021 às 16:00, Zhou, Hui <zhouh(a)anl.gov> escreveu: > Hi Jean, > > > > Please try > > > > ./configure --prefix=/home/jean/mpich-install –with-device=ch4:ofi > > > > -- > Hui Zhou > > > > > > *From: *Jean Schuster via discuss <discuss(a)mpich.org> > *Date: *Thursday, May 6, 2021 at 1:44 PM > *To: *discuss(a)mpich.org <discuss(a)mpich.org> > *Cc: *Jean Schuster <jeanjschuster(a)gmail.com> > *Subject: *[mpich-discuss] Problems to install MPICH 3.4.1 in Ubuntu > 20.04.2 LTS 64bits. > > Hi everyone. > > I hope you are doing well. > > > > My name is Jean Schuster and I'm from Brazil. > > > > I'm facing some difficulties in installing Mpich 3.4.1 in my Ubuntu > 20.04.2 LTS 64bits machine following the install guide. I'm getting stuck > in the 6th step on the section 2.2, where it suppose to build the Mpich > through the "make" command (make 2>&1 | tee m.txt) when it returns mi te > following error: > > > > jean@Prandtl:/tmp/jean/mpich-3.4.1$ make 2>&1 | tee m.txt > make: *** No targets specified and no makefile found. Stop. > > > > After running ./configure --prefix=/home/jean/mpich-install 2>&1 | tee > c.txt I get this message in the very end: > > > > configure: error: no ch4 netmod selected > > The default ch4 device could not detect a preferred network > library. Supported options are ofi (libfabric) and ucx: > > --with-device=ch4:ofi or --with-device=ch4:ucx > > Configure will use an embedded copy of libfabric or ucx if one is > not found in the user environment. An installation can be specified > by adding > > --with-libfabric=<path/to/install> or --with-ucx=<path/to/install> > > to the configuration. > > The previous MPICH default device (ch3) is also available and > supported with option: > > --with-device=ch3 > > > > And no Makefile is created on the source folder. The installation folder > /home/jean/mpich-install remains empty. > > > > I'm a relatively new Ubuntu user, and despite the installation guide goes > step-by-step, I could not go through this error. > > Attached follows the c.txt and config files from the temporary folder. > > I hope you can kindly help me to get it to work. > > > > Thank you in advance. > > > > Best regards, > > Jean. > > >

1 0

Re: [mpich-discuss] Problems to install MPICH 3.4.1 in Ubuntu 20.04.2 LTS 64bits.
by Zhou, Hui 06 May '21

06 May '21

Hi Jean, Please try ./configure --prefix=/home/jean/mpich-install –with-device=ch4:ofi -- Hui Zhou From: Jean Schuster via discuss <discuss(a)mpich.org> Date: Thursday, May 6, 2021 at 1:44 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Jean Schuster <jeanjschuster(a)gmail.com> Subject: [mpich-discuss] Problems to install MPICH 3.4.1 in Ubuntu 20.04.2 LTS 64bits. Hi everyone. I hope you are doing well. My name is Jean Schuster and I'm from Brazil. I'm facing some difficulties in installing Mpich 3.4.1 in my Ubuntu 20.04.2 LTS 64bits machine following the install guide. I'm getting stuck in the 6th step on the section 2.2, where it suppose to build the Mpich through the "make" command (make 2>&1 | tee m.txt) when it returns mi te following error: jean@Prandtl:/tmp/jean/mpich-3.4.1$ make 2>&1 | tee m.txt make: *** No targets specified and no makefile found. Stop. After running ./configure --prefix=/home/jean/mpich-install 2>&1 | tee c.txt I get this message in the very end: configure: error: no ch4 netmod selected The default ch4 device could not detect a preferred network library. Supported options are ofi (libfabric) and ucx: --with-device=ch4:ofi or --with-device=ch4:ucx Configure will use an embedded copy of libfabric or ucx if one is not found in the user environment. An installation can be specified by adding --with-libfabric=<path/to/install> or --with-ucx=<path/to/install> to the configuration. The previous MPICH default device (ch3) is also available and supported with option: --with-device=ch3 And no Makefile is created on the source folder. The installation folder /home/jean/mpich-install remains empty. I'm a relatively new Ubuntu user, and despite the installation guide goes step-by-step, I could not go through this error. Attached follows the c.txt and config files from the temporary folder. I hope you can kindly help me to get it to work. Thank you in advance. Best regards, Jean.

1 0

Problems to install MPICH 3.4.1 in Ubuntu 20.04.2 LTS 64bits.
by Jean Schuster 06 May '21

06 May '21

Hi everyone. I hope you are doing well. My name is Jean Schuster and I'm from Brazil. I'm facing some difficulties in installing Mpich 3.4.1 in my Ubuntu 20.04.2 LTS 64bits machine following the install guide. I'm getting stuck in the 6th step on the section 2.2, where it suppose to build the Mpich through the "make" command (make 2>&1 | tee m.txt) when it returns mi te following error: jean@Prandtl:/tmp/jean/mpich-3.4.1$ make 2>&1 | tee m.txt make: *** No targets specified and no makefile found. Stop. After running ./configure --prefix=/home/jean/mpich-install 2>&1 | tee c.txt I get this message in the very end: configure: error: no ch4 netmod selected The default ch4 device could not detect a preferred network library. Supported options are ofi (libfabric) and ucx: --with-device=ch4:ofi or --with-device=ch4:ucx Configure will use an embedded copy of libfabric or ucx if one is not found in the user environment. An installation can be specified by adding --with-libfabric=<path/to/install> or --with-ucx=<path/to/install> to the configuration. The previous MPICH default device (ch3) is also available and supported with option: --with-device=ch3 And no Makefile is created on the source folder. The installation folder /home/jean/mpich-install remains empty. I'm a relatively new Ubuntu user, and despite the installation guide goes step-by-step, I could not go through this error. Attached follows the c.txt and config files from the temporary folder. I hope you can kindly help me to get it to work. Thank you in advance. Best regards, Jean.

1 0

Re: [mpich-discuss] Mpich failure with external libfabric on macOS: OFI resource bind failed
by Zhou, Hui 03 May '21

03 May '21

Hi Michka, Which libfabric version are you using? Could you try setting `MPIR_CVAR_CH4_OFI_CAPABILITY_SETS_DEBUG=` to see if there is more debug messages? -- Hui Zhou ________________________________ From: Michka Popoff via discuss <discuss(a)mpich.org> Sent: Saturday, May 1, 2021 4:33 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Michka Popoff <michkapopoff(a)gmail.com> Subject: [mpich-discuss] Mpich failure with external libfabric on macOS: OFI resource bind failed Hi Homebrew maintainer here (https://github.com/Homebrew) Homebrew ships mpich as a package on both MacOS and Linux. The issue below was found with version 3.4.1 but might have been there for longer. We noticed that mpich was building it’s own internal libfabric dependency. After reading https://lists.mpich.org/pipermail/discuss/2021-January/006092.html, we added `--with-device=ch4:ofi` and set the libfabric path with the --with-libfabric= flag, to use our own version. The build is fine. We have a small test to check if mpich is still working fine; #include <mpi.h> #include <stdio.h> int main() { int size, rank, nameLen; char name[MPI_MAX_PROCESSOR_NAME]; MPI_Init(NULL, NULL); MPI_Comm_size(MPI_COMM_WORLD, &size); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Get_processor_name(name, &nameLen); printf("[%d/%d] Hello, world! My name is %s.\\n", rank, size, name); MPI_Finalize(); return 0; } Executing the test fails with a weird error: /usr/local/Cellar/mpich/3.4.1_2/bin/mpicc hello.c -o hello ./hello Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(152).......: MPID_Init(597)..............: MPIDI_OFI_mpi_init_hook(674): create_vni_context(964).....: OFI resource bind failed (ofi_init.c:964:create_vni_context:No message available on STREAM) [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247 : system msg for write_line failure : Bad file descriptor Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(152).......: MPID_Init(597)..............: MPIDI_OFI_mpi_init_hook(674): create_vni_context(964).....: OFI resource bind failed (ofi_init.c:964:create_vni_context:No message available on STREAM) [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247 : system msg for write_line failure : Bad file descriptor This test passes fine on Linux, and fails only on MacOS. Using the internal libfabric is fine on both platforms. Here is the related discussion: https://github.com/Homebrew/homebrew-core/pull/73062 Maybe you could help us debug this issue? Regards Michka

1 0

Mpich failure with external libfabric on macOS: OFI resource bind failed
by Michka Popoff 01 May '21

01 May '21

Hi Homebrew maintainer here (https://github.com/Homebrew) Homebrew ships mpich as a package on both MacOS and Linux. The issue below was found with version 3.4.1 but might have been there for longer. We noticed that mpich was building it’s own internal libfabric dependency. After reading https://lists.mpich.org/pipermail/discuss/2021-January/006092.html, we added `--with-device=ch4:ofi` and set the libfabric path with the --with-libfabric= flag, to use our own version. The build is fine. We have a small test to check if mpich is still working fine; #include <mpi.h> #include <stdio.h> int main() { int size, rank, nameLen; char name[MPI_MAX_PROCESSOR_NAME]; MPI_Init(NULL, NULL); MPI_Comm_size(MPI_COMM_WORLD, &size); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Get_processor_name(name, &nameLen); printf("[%d/%d] Hello, world! My name is %s.\\n", rank, size, name); MPI_Finalize(); return 0; } Executing the test fails with a weird error: /usr/local/Cellar/mpich/3.4.1_2/bin/mpicc hello.c -o hello ./hello Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(152).......: MPID_Init(597)..............: MPIDI_OFI_mpi_init_hook(674): create_vni_context(964).....: OFI resource bind failed (ofi_init.c:964:create_vni_context:No message available on STREAM) [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247 : system msg for write_line failure : Bad file descriptor Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(152).......: MPID_Init(597)..............: MPIDI_OFI_mpi_init_hook(674): create_vni_context(964).....: OFI resource bind failed (ofi_init.c:964:create_vni_context:No message available on STREAM) [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247 : system msg for write_line failure : Bad file descriptor This test passes fine on Linux, and fails only on MacOS. Using the internal libfabric is fine on both platforms. Here is the related discussion: https://github.com/Homebrew/homebrew-core/pull/73062 Maybe you could help us debug this issue? Regards Michka

1 0