June 2026 - discuss - mailman3.cels.anl.gov

None
by None 25 Jun '26

25 Jun '26

ld you elaborate on why the dynamic process model is not sufficient for you= r needs? Halim </div></div><a href=3D"http: //www.mcs.anl.gov/~aamer" target=3D"_blank" rel= =3D"noreferrer">www.mcs.anl.gov/~aamer</a> <<a href=3D"http: //www.mcs.an= l.gov/~aamer" target=3D"_blank" rel=3D"noreferrer">http: //www.mcs.anl.gov/~= aamer</a>> <<a href=3D"http: //www.mcs.anl.gov/~aamer" target=3D"= _blank" rel=3D"noreferrer">http: //www.mcs.anl.gov/~aamer</a>> <div class=3D"HOEnZb"><div class=3D"h5"> On 5/26/17 9: 11 AM, sanjeev s wrote: Hi mpich, I have a requirement where in we need to add start stop application instances on the fly before starting a job.Is there any mpich service available. I looked through dynamic process model, but its not sufficing<br= > our need. More precisely my requirement is suppose I started 4 instances of my application. Now I want to add one more instance dynamically to this set<br= > Is there any tool which MPICH supports for fault tolerance behavior? Thanks Sanjeev _______________________________________________ discuss mailing list     <a href=3D"mailto: discuss(a)mpich.org= ">discuss(a)mpich.org</a> To manage subscription options or unsubscribe: <a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" target=3D"_bla= nk" rel=3D"noreferrer">https: //lists.mpich.org/mailman/listinfo/discus= s</a> _______________________________________________ discuss mailing list     <a href=3D"mailto: discuss(a)mpich.org= ">discuss(a)mpich.org</a> To manage subscription options or unsubscribe: <a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" target=3D"_bla= nk" rel=3D"noreferrer">https: //lists.mpich.org/mailman/listinfo/discus= s</a> _______________________________________________ discuss mailing list     <a href=3D"mailto: discuss(a)mpich.org= ">discuss(a)mpich.org</a> To manage subscription options or unsubscribe: <a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" target=3D"_bla= nk" rel=3D"noreferrer">https: //lists.mpich.org/mailman/listinfo/discus= s</a> _______________________________________________ discuss mailing list     <a href=3D"mailto: discuss(a)mpich.org= ">discuss(a)mpich.org</a> To manage subscription options or unsubscribe: <a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" target=3D"_bla= nk" rel=3D"noreferrer">https: //lists.mpich.org/mailman/listinfo/discus= s</a></div></div></blockquote></div> </div> --001a113d859a7bd8fe0550df4ee5-- --===============8098711719025613836== Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https: //lists.mpich.org/mailman/listinfo/discuss --===============8098711719025613836==-- Message-ID: <sanitized-94(a)migration.local>

1 0

None
by None 25 Jun '26

25 Jun '26

ld you elaborate on why the dynamic process model is not sufficient for you= r needs? Halim </div></div><a href=3D"http: //www.mcs.anl.gov/~aamer" target=3D"_blank" rel= =3D"noreferrer">www.mcs.anl.gov/~aamer</a> <<a href=3D"http: //www.mcs.an= l.gov/~aamer" target=3D"_blank" rel=3D"noreferrer">http: //www.mcs.anl.gov/~= aamer</a>> <<a href=3D"http: //www.mcs.anl.gov/~aamer" target=3D"= _blank" rel=3D"noreferrer">http: //www.mcs.anl.gov/~aamer</a>> <<= a href=3D"http: //www.mcs.anl.gov/~aamer" target=3D"_blank" rel=3D"noreferre= r">http: //www.mcs.anl.gov/~aamer</a>> <div class=3D"HOEnZb"><div class=3D"h5"> On 5/26/17 9: 11 AM, sanjeev s wrote: Hi mpich, I have a requirement where in we need to add start stop application instances on the fly before starting a job.Is there any mpich service available. I looked through dynamic process model, but its not sufficing<br= > our need. More precisely my requirement is suppose I started 4 instances of my application. Now I want to add one more instance dynamically to this set<br= > Is there any tool which MPICH supports for fault tolerance behavior? Thanks Sanjeev _______________________________________________ discuss mailing list     <a href=3D"mailto: discuss(a)mpich.org= ">discuss(a)mpich.org</a> To manage subscription options or unsubscribe: <a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" target=3D"_bla= nk" rel=3D"noreferrer">https: //lists.mpich.org/mailman/listinfo/discus= s</a> _______________________________________________ discuss mailing list     <a href=3D"mailto: discuss(a)mpich.org= ">discuss(a)mpich.org</a> To manage subscription options or unsubscribe: <a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" target=3D"_bla= nk" rel=3D"noreferrer">https: //lists.mpich.org/mailman/listinfo/discus= s</a> _______________________________________________ discuss mailing list     <a href=3D"mailto: discuss(a)mpich.org= ">discuss(a)mpich.org</a> To manage subscription options or unsubscribe: <a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" target=3D"_bla= nk" rel=3D"noreferrer">https: //lists.mpich.org/mailman/listinfo/discus= s</a> _______________________________________________ discuss mailing list     <a href=3D"mailto: discuss(a)mpich.org= ">discuss(a)mpich.org</a> To manage subscription options or unsubscribe: <a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" target=3D"_bla= nk" rel=3D"noreferrer">https: //lists.mpich.org/mailman/listinfo/discus= s</a> _______________________________________________ discuss mailing list     <a href=3D"mailto: discuss(a)mpich.org= ">discuss(a)mpich.org</a> To manage subscription options or unsubscribe: <a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" target=3D"_bla= nk" rel=3D"noreferrer">https: //lists.mpich.org/mailman/listinfo/discus= s</a></div></div></blockquote></div> </div> --001a11473f4a441a450550f239c9-- --===============5627409794820161001== Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https: //lists.mpich.org/mailman/listinfo/discuss --===============5627409794820161001==-- Message-ID: <sanitized-87(a)migration.local>

1 0

Re: [mpich-discuss] How to get srun/mpich to use the right interface?
by Zhou, Hui 23 Jun '26

23 Jun '26

Yes, simply "ofi" or "ucx". All the CVARs can be found by running mpivars. Unfortunately, the details are limited in the description. It's on our TODO list to furnish our documentation. -- Hui ________________________________ From: John Cary <cary(a)colorado.edu> Sent: Monday, June 22, 2026 4:19 PM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [mpich-discuss] How to get srun/mpich to use the right interface? Thanks, Hui. What are the values for MPIR_CVAR_CH4_NETMOD? Simply ofi or ucx? Where do I find this information in the manual? Thx. . . John On 6/22/26 2: 50 PM, Zhou, Hui wrote: [External email - use caution] * Both ch4: ucx,ofi and ch4: ofi,ucx build ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thanks, Hui. What are the values for MPIR_CVAR_CH4_NETMOD? Simply ofi or ucx? Where do I find this information in the manual? Thx...John On 6/22/26 2:50 PM, Zhou, Hui wrote: [External email - use caution] * Both ch4:ucx,ofi and ch4:ofi,ucx build with both the UCX and Libfabric, and one can use the environment variable "MPIR_CVAR_CH4_NETMOD" to choose which netmod (ofi or ucx) at runtime. The order between ucx an ofi determines the default. On single mode, shared memory will be used unless it is disabled. "ch4:ucx" disables shared memory by default since UCX's shared memory performance is quite decent. When shared memory is used, which FI_PROVIDER doesn't matter. Hope this answers your question. Cheers, Hui ________________________________ From: John Cary <cary(a)colorado.edu><mailto:[email protected]> Sent: Monday, June 22, 2026 2:21 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org><mailto:[email protected]>; Zhou, Hui <zhouh(a)anl.gov><mailto:[email protected]> Subject: Re: [mpich-discuss] How to get srun/mpich to use the right interface? Hello Hui, We are getting some mysterious results. We have four configurations: --with-device=ch4: ucx --with-device=ch4: ofi --with-device=ch4: ucx,ofi --with-device=ch4: ofi,ucx Is there any difference between the last two? Are the last two valid ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello Hui, We are getting some mysterious results. We have four configurations: --with-device=ch4:ucx --with-device=ch4:ofi --with-device=ch4:ucx,ofi --with-device=ch4:ofi,ucx Is there any difference between the last two? Are the last two valid configurations? If one is on a single node, does FI_PROVIDER=verbs make any sense? Or perhaps on a single node, shared memory is always used? Thanks....John On 6/11/26 10:33 AM, John Cary wrote: Thanks, Hui! I am trying to have one build that works for our cluster and also for AWS, which uses ofi. Is that possible? Thx....John On 6/11/26 10:06 AM, Zhou, Hui wrote: [External email - use caution] HI John, Try using the UCX instead of libfabric. You can configure MPICH with ./configure --with-device=ch4:ucx to use UCX. With libfabric, could you try set environment variable FI_PROVIDER=verbs ? If you still have issue, try run a dummy MPI program setting `MPIR_CVAR_DEBUG_SUMMARY=1`. That will provide more logging details on which libfabric provider is being selected. -- Hui Zhou ________________________________ From: John Cary via discuss <discuss(a)mpich.org><mailto:[email protected]> Sent: Thursday, June 11, 2026 8:50 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org><mailto:[email protected]> Cc: John Cary <cary(a)colorado.edu><mailto:[email protected]> Subject: [mpich-discuss] How to get srun/mpich to use the right interface? How to get mpich to use the right interface? mpich configured and built with libfabric as shown below. It is run using slurm (srun). The result is [1781142435. 559314543] ne07: rank64. vorpal: Failed to modify UD QP to INIT on mlx5_0: Operation ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd How to get mpich to use the right interface? mpich configured and built with libfabric as shown below. It is run using slurm (srun). The result is [1781142435.559314543] ne07:rank64.vorpal: Failed to modify UD QP to INIT on mlx5_0: Operation not permitted [1781142435.563376291] ne07:rank66.vorpal: Failed to modify UD QP to INIT on mlx5_0: Operation not permitted Abort(203572367): Fatal error in internal_Init: Other MPI error, error stack ... which I think means that mpich is txrying to run over the mlx5_0 interface, which does not exist. The interfaces are eno1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 eno2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 and we want to use ib0, infiniband. We tried export HYDRA_IFACE=ib0 srun ... and got the same error. How can srun/mpich be instructed to use the ib0 interface by default? Also, how can one see which interface mpich is choosing? Thx... '/user/builds-linux-centos8-zen2/gvxsimall/mpich-5.0.1/configure' \ --prefix=/home/research/user/installs/linux-centos8-zen2/contrib-gcc1140/mpich-5.0.1-shared \ --enable-shared \ --disable-static \ CC='/home/common/gcc-11.4.0/bin/gcc' \ CXX='/home/common/gcc-11.4.0/bin/g++' \ FC='/home/common/gcc-11.4.0/bin/gfortran' \ F77='/home/common/gcc-11.4.0/bin/gfortran' \ CFLAGS='-pthread -pipe -fPIC' \ CXXFLAGS='-pthread -Wno-deprecated-declarations -pipe -fPIC' \ FFLAGS='-fallow-argument-mismatch -pipe -fPIC' \ FCFLAGS='-fallow-argument-mismatch -pipe -fPIC' \ LDFLAGS='-L/home/common/gcc-11.4.0/lib64 -Wl,-rpath,/home/common/gcc-11.4.0/lib64' \ LDSHARED='-L/home/common/gcc-11.4.0/lib64 -Wl,-rpath,/home/common/gcc-11.4.0/lib64' \ --with-libfabric=install \ --with-device=ch4:ofi \ --without-cuda \ --disable-gl

1 0

Re: [mpich-discuss] How to get srun/mpich to use the right interface?
by John Cary 22 Jun '26

22 Jun '26

Thanks, Hui. What are the values for MPIR_CVAR_CH4_NETMOD? Simply ofi or ucx? Where do I find this information in the manual? Thx...John On 6/22/26 2:50 PM, Zhou, Hui wrote: > [External email - use caution] > > * Both |ch4:ucx,ofi| and |ch4:ofi,ucx build with both the UCX and > Libfabric, and one can use the environment variable > "MPIR_CVAR_CH4_NETMOD" to choose which netmod (ofi or ucx) at runtime.| > > The order between ucx an ofi determines the default. > > On single mode, shared memory will be used unless it is disabled. > "ch4:ucx" disables shared memory by default since UCX's shared memory > performance is quite decent. When shared memory is used, which > FI_PROVIDER doesn't matter. > > Hope this answers your question. > > Cheers, > Hui > ------------------------------------------------------------------------ > *From:* John Cary <cary(a)colorado.edu> > *Sent:* Monday, June 22, 2026 2:21 PM > *To:* discuss(a)mpich.org <discuss(a)mpich.org>; Zhou, Hui <zhouh(a)anl.gov> > *Subject:* Re: [mpich-discuss] How to get srun/mpich to use the right > interface? > Hello Hui, We are getting some mysterious results. We have four > configurations: --with-device=ch4: ucx --with-device=ch4: ofi > --with-device=ch4: ucx,ofi --with-device=ch4: ofi,ucx Is there any > difference between the last two? Are the last two valid > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > ZjQcmQRYFpfptBannerEnd > Hello Hui, > > We are getting some mysterious results. We have four configurations: > > --with-device=ch4:ucx > --with-device=ch4:ofi > --with-device=ch4:ucx,ofi > --with-device=ch4:ofi,ucx > > Is there any difference between the last two? Are the last two valid > configurations? > > If one is on a single node, does FI_PROVIDER=verbs make any sense? > Or perhaps on a single node, shared memory is always used? > > Thanks....John > > > > On 6/11/26 10:33 AM, John Cary wrote: >> Thanks, Hui! >> >> I am trying to have one build that works for our cluster and also for >> AWS, which uses ofi. Is that possible? >> >> Thx....John >> >> >> >> On 6/11/26 10:06 AM, Zhou, Hui wrote: >>> [External email - use caution] >>> >>> HI John, >>> >>> Try using the UCX instead of libfabric. You can configure MPICH with >>> |./configure --with-device=ch4:ucx| to use UCX. With libfabric, >>> could you try set environment variable |FI_PROVIDER=verbs ?| >>> >>> If you still have issue, try run a dummy MPI program setting >>> `MPIR_CVAR_DEBUG_SUMMARY=1`. That will provide more logging details >>> on which libfabric provider is being selected. >>> >>> -- >>> Hui Zhou >>> ------------------------------------------------------------------------ >>> *From:* John Cary via discuss <discuss(a)mpich.org> >>> <mailto:[email protected]> >>> *Sent:* Thursday, June 11, 2026 8:50 AM >>> *To:* discuss(a)mpich.org <mailto:[email protected]> >>> <discuss(a)mpich.org> <mailto:[email protected]> >>> *Cc:* John Cary <cary(a)colorado.edu> <mailto:[email protected]> >>> *Subject:* [mpich-discuss] How to get srun/mpich to use the right >>> interface? >>> How to get mpich to use the right interface? mpich configured and >>> built with libfabric as shown below. It is run using slurm >>> (srun). The result is [1781142435. 559314543] >>> ne07: rank64. vorpal: Failed to modify UD QP to INIT on mlx5_0: >>> Operation >>> ZjQcmQRYFpfptBannerStart >>> This Message Is From an External Sender >>> This message came from outside your organization. >>> ZjQcmQRYFpfptBannerEnd >>> How to get mpich to use the right interface? >>> >>> mpich configured and built with libfabric as shown below. It is run >>> using slurm (srun). The result is >>> >>> [1781142435.559314543] ne07:rank64.vorpal: Failed to modify UD QP to >>> INIT on mlx5_0: Operation not permitted >>> [1781142435.563376291] ne07:rank66.vorpal: Failed to modify UD QP to >>> INIT on mlx5_0: Operation not permitted >>> Abort(203572367): Fatal error in internal_Init: Other MPI error, error stack >>> ... >>> >>> which I think means that mpich is txrying to run over the mlx5_0 >>> interface, which does not exist. The interfaces are >>> >>> eno1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 >>> eno2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >>> ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044 >>> lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 >>> >>> and we want to use ib0, infiniband. We tried >>> >>> export HYDRA_IFACE=ib0 >>> srun ... >>> >>> and got the same error. >>> >>> How can srun/mpich be instructed to use the ib0 interface by default? >>> >>> Also, how can one see which interface mpich is choosing? >>> >>> Thx... >>> >>> >>> >>> '/user/builds-linux-centos8-zen2/gvxsimall/mpich-5.0.1/configure' \ >>> --prefix=/home/research/user/installs/linux-centos8-zen2/contrib-gcc1140/mpich-5.0.1-shared >>> \ >>> --enable-shared \ >>> --disable-static \ >>> CC='/home/common/gcc-11.4.0/bin/gcc' \ >>> CXX='/home/common/gcc-11.4.0/bin/g++' \ >>> FC='/home/common/gcc-11.4.0/bin/gfortran' \ >>> F77='/home/common/gcc-11.4.0/bin/gfortran' \ >>> CFLAGS='-pthread -pipe -fPIC' \ >>> CXXFLAGS='-pthread -Wno-deprecated-declarations -pipe -fPIC' \ >>> FFLAGS='-fallow-argument-mismatch -pipe -fPIC' \ >>> FCFLAGS='-fallow-argument-mismatch -pipe -fPIC' \ >>> LDFLAGS='-L/home/common/gcc-11.4.0/lib64 >>> -Wl,-rpath,/home/common/gcc-11.4.0/lib64' \ >>> LDSHARED='-L/home/common/gcc-11.4.0/lib64 >>> -Wl,-rpath,/home/common/gcc-11.4.0/lib64' \ >>> --with-libfabric=install \ >>> --with-device=ch4:ofi \ >>> --without-cuda \ >>> --disable-gl >> >

1 0

Re: [mpich-discuss] How to get srun/mpich to use the right interface?
by Zhou, Hui 22 Jun '26

22 Jun '26

* Both ch4:ucx,ofi and ch4:ofi,ucx build with both the UCX and Libfabric, and one can use the environment variable "MPIR_CVAR_CH4_NETMOD" to choose which netmod (ofi or ucx) at runtime. The order between ucx an ofi determines the default. On single mode, shared memory will be used unless it is disabled. "ch4:ucx" disables shared memory by default since UCX's shared memory performance is quite decent. When shared memory is used, which FI_PROVIDER doesn't matter. Hope this answers your question. Cheers, Hui ________________________________ From: John Cary <cary(a)colorado.edu> Sent: Monday, June 22, 2026 2:21 PM To: discuss(a)mpich.org <discuss(a)mpich.org>; Zhou, Hui <zhouh(a)anl.gov> Subject: Re: [mpich-discuss] How to get srun/mpich to use the right interface? Hello Hui, We are getting some mysterious results. We have four configurations: --with-device=ch4: ucx --with-device=ch4: ofi --with-device=ch4: ucx,ofi --with-device=ch4: ofi,ucx Is there any difference between the last two? Are the last two valid ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello Hui, We are getting some mysterious results. We have four configurations: --with-device=ch4:ucx --with-device=ch4:ofi --with-device=ch4:ucx,ofi --with-device=ch4:ofi,ucx Is there any difference between the last two? Are the last two valid configurations? If one is on a single node, does FI_PROVIDER=verbs make any sense? Or perhaps on a single node, shared memory is always used? Thanks....John On 6/11/26 10:33 AM, John Cary wrote: Thanks, Hui! I am trying to have one build that works for our cluster and also for AWS, which uses ofi. Is that possible? Thx....John On 6/11/26 10:06 AM, Zhou, Hui wrote: [External email - use caution] HI John, Try using the UCX instead of libfabric. You can configure MPICH with ./configure --with-device=ch4:ucx to use UCX. With libfabric, could you try set environment variable FI_PROVIDER=verbs ? If you still have issue, try run a dummy MPI program setting `MPIR_CVAR_DEBUG_SUMMARY=1`. That will provide more logging details on which libfabric provider is being selected. -- Hui Zhou ________________________________ From: John Cary via discuss <discuss(a)mpich.org><mailto:[email protected]> Sent: Thursday, June 11, 2026 8:50 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org><mailto:[email protected]> Cc: John Cary <cary(a)colorado.edu><mailto:[email protected]> Subject: [mpich-discuss] How to get srun/mpich to use the right interface? How to get mpich to use the right interface? mpich configured and built with libfabric as shown below. It is run using slurm (srun). The result is [1781142435. 559314543] ne07: rank64. vorpal: Failed to modify UD QP to INIT on mlx5_0: Operation ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd How to get mpich to use the right interface? mpich configured and built with libfabric as shown below. It is run using slurm (srun). The result is [1781142435.559314543] ne07:rank64.vorpal: Failed to modify UD QP to INIT on mlx5_0: Operation not permitted [1781142435.563376291] ne07:rank66.vorpal: Failed to modify UD QP to INIT on mlx5_0: Operation not permitted Abort(203572367): Fatal error in internal_Init: Other MPI error, error stack ... which I think means that mpich is txrying to run over the mlx5_0 interface, which does not exist. The interfaces are eno1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 eno2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 and we want to use ib0, infiniband. We tried export HYDRA_IFACE=ib0 srun ... and got the same error. How can srun/mpich be instructed to use the ib0 interface by default? Also, how can one see which interface mpich is choosing? Thx... '/user/builds-linux-centos8-zen2/gvxsimall/mpich-5.0.1/configure' \ --prefix=/home/research/user/installs/linux-centos8-zen2/contrib-gcc1140/mpich-5.0.1-shared \ --enable-shared \ --disable-static \ CC='/home/common/gcc-11.4.0/bin/gcc' \ CXX='/home/common/gcc-11.4.0/bin/g++' \ FC='/home/common/gcc-11.4.0/bin/gfortran' \ F77='/home/common/gcc-11.4.0/bin/gfortran' \ CFLAGS='-pthread -pipe -fPIC' \ CXXFLAGS='-pthread -Wno-deprecated-declarations -pipe -fPIC' \ FFLAGS='-fallow-argument-mismatch -pipe -fPIC' \ FCFLAGS='-fallow-argument-mismatch -pipe -fPIC' \ LDFLAGS='-L/home/common/gcc-11.4.0/lib64 -Wl,-rpath,/home/common/gcc-11.4.0/lib64' \ LDSHARED='-L/home/common/gcc-11.4.0/lib64 -Wl,-rpath,/home/common/gcc-11.4.0/lib64' \ --with-libfabric=install \ --with-device=ch4:ofi \ --without-cuda \ --disable-gl

1 0

Re: [mpich-discuss] How to get srun/mpich to use the right interface?
by John Cary 22 Jun '26

22 Jun '26

Hello Hui, We are getting some mysterious results. We have four configurations: --with-device=ch4:ucx --with-device=ch4:ofi --with-device=ch4:ucx,ofi --with-device=ch4:ofi,ucx Is there any difference between the last two? Are the last two valid configurations? If one is on a single node, does FI_PROVIDER=verbs make any sense? Or perhaps on a single node, shared memory is always used? Thanks....John On 6/11/26 10:33 AM, John Cary wrote: > Thanks, Hui! > > I am trying to have one build that works for our cluster and also for > AWS, which uses ofi. Is that possible? > > Thx....John > > > > On 6/11/26 10:06 AM, Zhou, Hui wrote: >> [External email - use caution] >> >> HI John, >> >> Try using the UCX instead of libfabric. You can configure MPICH with >> |./configure --with-device=ch4:ucx| to use UCX. With libfabric, could >> you try set environment variable |FI_PROVIDER=verbs ?| >> >> If you still have issue, try run a dummy MPI program setting >> `MPIR_CVAR_DEBUG_SUMMARY=1`. That will provide more logging details >> on which libfabric provider is being selected. >> >> -- >> Hui Zhou >> ------------------------------------------------------------------------ >> *From:* John Cary via discuss <discuss(a)mpich.org> >> *Sent:* Thursday, June 11, 2026 8:50 AM >> *To:* discuss(a)mpich.org <discuss(a)mpich.org> >> *Cc:* John Cary <cary(a)colorado.edu> >> *Subject:* [mpich-discuss] How to get srun/mpich to use the right >> interface? >> How to get mpich to use the right interface? mpich configured and >> built with libfabric as shown below. It is run using slurm >> (srun). The result is [1781142435. 559314543] ne07: rank64. vorpal: >> Failed to modify UD QP to INIT on mlx5_0: Operation >> ZjQcmQRYFpfptBannerStart >> This Message Is From an External Sender >> This message came from outside your organization. >> ZjQcmQRYFpfptBannerEnd >> How to get mpich to use the right interface? >> >> mpich configured and built with libfabric as shown below. It is run >> using slurm (srun). The result is >> >> [1781142435.559314543] ne07:rank64.vorpal: Failed to modify UD QP to >> INIT on mlx5_0: Operation not permitted >> [1781142435.563376291] ne07:rank66.vorpal: Failed to modify UD QP to >> INIT on mlx5_0: Operation not permitted >> Abort(203572367): Fatal error in internal_Init: Other MPI error, error stack >> ... >> >> which I think means that mpich is txrying to run over the mlx5_0 >> interface, which does not exist. The interfaces are >> >> eno1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 >> eno2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >> ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044 >> lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 >> >> and we want to use ib0, infiniband. We tried >> >> export HYDRA_IFACE=ib0 >> srun ... >> >> and got the same error. >> >> How can srun/mpich be instructed to use the ib0 interface by default? >> >> Also, how can one see which interface mpich is choosing? >> >> Thx... >> >> >> >> '/user/builds-linux-centos8-zen2/gvxsimall/mpich-5.0.1/configure' \ >> --prefix=/home/research/user/installs/linux-centos8-zen2/contrib-gcc1140/mpich-5.0.1-shared >> \ >> --enable-shared \ >> --disable-static \ >> CC='/home/common/gcc-11.4.0/bin/gcc' \ >> CXX='/home/common/gcc-11.4.0/bin/g++' \ >> FC='/home/common/gcc-11.4.0/bin/gfortran' \ >> F77='/home/common/gcc-11.4.0/bin/gfortran' \ >> CFLAGS='-pthread -pipe -fPIC' \ >> CXXFLAGS='-pthread -Wno-deprecated-declarations -pipe -fPIC' \ >> FFLAGS='-fallow-argument-mismatch -pipe -fPIC' \ >> FCFLAGS='-fallow-argument-mismatch -pipe -fPIC' \ >> LDFLAGS='-L/home/common/gcc-11.4.0/lib64 >> -Wl,-rpath,/home/common/gcc-11.4.0/lib64' \ >> LDSHARED='-L/home/common/gcc-11.4.0/lib64 >> -Wl,-rpath,/home/common/gcc-11.4.0/lib64' \ >> --with-libfabric=install \ >> --with-device=ch4:ofi \ >> --without-cuda \ >> --disable-gl >

1 0

Re: [mpich-discuss] How to get srun/mpich to use the right interface?
by Zhou, Hui 11 Jun '26

11 Jun '26

Sure. Try set `FI_PROVIDER=verbs` to see if it works. And also try set `MPIR_CVAR_DEBUG_SUMMARY=1` and send me the console log. Hui ________________________________ From: John Cary <cary(a)colorado.edu> Sent: Thursday, June 11, 2026 11:33 AM To: discuss(a)mpich.org <discuss(a)mpich.org>; Zhou, Hui <zhouh(a)anl.gov> Subject: Re: [mpich-discuss] How to get srun/mpich to use the right interface? Thanks, Hui! I am trying to have one build that works for our cluster and also for AWS, which uses ofi. Is that possible? Thx. . . . John On 6/11/26 10: 06 AM, Zhou, Hui wrote: [External email - use caution] HI John, Try using the UCX instead of ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thanks, Hui! I am trying to have one build that works for our cluster and also for AWS, which uses ofi. Is that possible? Thx....John On 6/11/26 10:06 AM, Zhou, Hui wrote: [External email - use caution] HI John, Try using the UCX instead of libfabric. You can configure MPICH with ./configure --with-device=ch4:ucx to use UCX. With libfabric, could you try set environment variable FI_PROVIDER=verbs ? If you still have issue, try run a dummy MPI program setting `MPIR_CVAR_DEBUG_SUMMARY=1`. That will provide more logging details on which libfabric provider is being selected. -- Hui Zhou ________________________________ From: John Cary via discuss <discuss(a)mpich.org><mailto:[email protected]> Sent: Thursday, June 11, 2026 8:50 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org><mailto:[email protected]> Cc: John Cary <cary(a)colorado.edu><mailto:[email protected]> Subject: [mpich-discuss] How to get srun/mpich to use the right interface? How to get mpich to use the right interface? mpich configured and built with libfabric as shown below. It is run using slurm (srun). The result is [1781142435. 559314543] ne07: rank64. vorpal: Failed to modify UD QP to INIT on mlx5_0: Operation ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd How to get mpich to use the right interface? mpich configured and built with libfabric as shown below. It is run using slurm (srun). The result is [1781142435.559314543] ne07:rank64.vorpal: Failed to modify UD QP to INIT on mlx5_0: Operation not permitted [1781142435.563376291] ne07:rank66.vorpal: Failed to modify UD QP to INIT on mlx5_0: Operation not permitted Abort(203572367): Fatal error in internal_Init: Other MPI error, error stack ... which I think means that mpich is txrying to run over the mlx5_0 interface, which does not exist. The interfaces are eno1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 eno2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 and we want to use ib0, infiniband. We tried export HYDRA_IFACE=ib0 srun ... and got the same error. How can srun/mpich be instructed to use the ib0 interface by default? Also, how can one see which interface mpich is choosing? Thx... '/user/builds-linux-centos8-zen2/gvxsimall/mpich-5.0.1/configure' \ --prefix=/home/research/user/installs/linux-centos8-zen2/contrib-gcc1140/mpich-5.0.1-shared \ --enable-shared \ --disable-static \ CC='/home/common/gcc-11.4.0/bin/gcc' \ CXX='/home/common/gcc-11.4.0/bin/g++' \ FC='/home/common/gcc-11.4.0/bin/gfortran' \ F77='/home/common/gcc-11.4.0/bin/gfortran' \ CFLAGS='-pthread -pipe -fPIC' \ CXXFLAGS='-pthread -Wno-deprecated-declarations -pipe -fPIC' \ FFLAGS='-fallow-argument-mismatch -pipe -fPIC' \ FCFLAGS='-fallow-argument-mismatch -pipe -fPIC' \ LDFLAGS='-L/home/common/gcc-11.4.0/lib64 -Wl,-rpath,/home/common/gcc-11.4.0/lib64' \ LDSHARED='-L/home/common/gcc-11.4.0/lib64 -Wl,-rpath,/home/common/gcc-11.4.0/lib64' \ --with-libfabric=install \ --with-device=ch4:ofi \ --without-cuda \ --disable-gl

1 0

Re: [mpich-discuss] How to get srun/mpich to use the right interface?
by John Cary 11 Jun '26

11 Jun '26

Thanks, Hui! I am trying to have one build that works for our cluster and also for AWS, which uses ofi. Is that possible? Thx....John On 6/11/26 10:06 AM, Zhou, Hui wrote: > [External email - use caution] > > HI John, > > Try using the UCX instead of libfabric. You can configure MPICH with > |./configure --with-device=ch4:ucx| to use UCX. With libfabric, could > you try set environment variable |FI_PROVIDER=verbs ?| > > If you still have issue, try run a dummy MPI program setting > `MPIR_CVAR_DEBUG_SUMMARY=1`. That will provide more logging details on > which libfabric provider is being selected. > > -- > Hui Zhou > ------------------------------------------------------------------------ > *From:* John Cary via discuss <discuss(a)mpich.org> > *Sent:* Thursday, June 11, 2026 8:50 AM > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* John Cary <cary(a)colorado.edu> > *Subject:* [mpich-discuss] How to get srun/mpich to use the right > interface? > How to get mpich to use the right interface? mpich configured and > built with libfabric as shown below. It is run using slurm (srun). > The result is [1781142435. 559314543] ne07: rank64. vorpal: Failed to > modify UD QP to INIT on mlx5_0: Operation > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > ZjQcmQRYFpfptBannerEnd > How to get mpich to use the right interface? > > mpich configured and built with libfabric as shown below. It is run > using slurm (srun). The result is > > [1781142435.559314543] ne07:rank64.vorpal: Failed to modify UD QP to > INIT on mlx5_0: Operation not permitted > [1781142435.563376291] ne07:rank66.vorpal: Failed to modify UD QP to > INIT on mlx5_0: Operation not permitted > Abort(203572367): Fatal error in internal_Init: Other MPI error, error stack > ... > > which I think means that mpich is txrying to run over the mlx5_0 > interface, which does not exist. The interfaces are > > eno1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 > eno2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044 > lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 > > and we want to use ib0, infiniband. We tried > > export HYDRA_IFACE=ib0 > srun ... > > and got the same error. > > How can srun/mpich be instructed to use the ib0 interface by default? > > Also, how can one see which interface mpich is choosing? > > Thx... > > > > '/user/builds-linux-centos8-zen2/gvxsimall/mpich-5.0.1/configure' \ > --prefix=/home/research/user/installs/linux-centos8-zen2/contrib-gcc1140/mpich-5.0.1-shared > \ > --enable-shared \ > --disable-static \ > CC='/home/common/gcc-11.4.0/bin/gcc' \ > CXX='/home/common/gcc-11.4.0/bin/g++' \ > FC='/home/common/gcc-11.4.0/bin/gfortran' \ > F77='/home/common/gcc-11.4.0/bin/gfortran' \ > CFLAGS='-pthread -pipe -fPIC' \ > CXXFLAGS='-pthread -Wno-deprecated-declarations -pipe -fPIC' \ > FFLAGS='-fallow-argument-mismatch -pipe -fPIC' \ > FCFLAGS='-fallow-argument-mismatch -pipe -fPIC' \ > LDFLAGS='-L/home/common/gcc-11.4.0/lib64 > -Wl,-rpath,/home/common/gcc-11.4.0/lib64' \ > LDSHARED='-L/home/common/gcc-11.4.0/lib64 > -Wl,-rpath,/home/common/gcc-11.4.0/lib64' \ > --with-libfabric=install \ > --with-device=ch4:ofi \ > --without-cuda \ > --disable-gl

1 0

Re: [mpich-discuss] How to get srun/mpich to use the right interface?
by Zhou, Hui 11 Jun '26

11 Jun '26

HI John, Try using the UCX instead of libfabric. You can configure MPICH with ./configure --with-device=ch4:ucx to use UCX. With libfabric, could you try set environment variable FI_PROVIDER=verbs ? If you still have issue, try run a dummy MPI program setting `MPIR_CVAR_DEBUG_SUMMARY=1`. That will provide more logging details on which libfabric provider is being selected. -- Hui Zhou ________________________________ From: John Cary via discuss <discuss(a)mpich.org> Sent: Thursday, June 11, 2026 8:50 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: John Cary <cary(a)colorado.edu> Subject: [mpich-discuss] How to get srun/mpich to use the right interface? How to get mpich to use the right interface? mpich configured and built with libfabric as shown below. It is run using slurm (srun). The result is [1781142435. 559314543] ne07: rank64. vorpal: Failed to modify UD QP to INIT on mlx5_0: Operation ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd How to get mpich to use the right interface? mpich configured and built with libfabric as shown below. It is run using slurm (srun). The result is [1781142435.559314543] ne07:rank64.vorpal: Failed to modify UD QP to INIT on mlx5_0: Operation not permitted [1781142435.563376291] ne07:rank66.vorpal: Failed to modify UD QP to INIT on mlx5_0: Operation not permitted Abort(203572367): Fatal error in internal_Init: Other MPI error, error stack ... which I think means that mpich is txrying to run over the mlx5_0 interface, which does not exist. The interfaces are eno1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 eno2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 and we want to use ib0, infiniband. We tried export HYDRA_IFACE=ib0 srun ... and got the same error. How can srun/mpich be instructed to use the ib0 interface by default? Also, how can one see which interface mpich is choosing? Thx... '/user/builds-linux-centos8-zen2/gvxsimall/mpich-5.0.1/configure' \ --prefix=/home/research/user/installs/linux-centos8-zen2/contrib-gcc1140/mpich-5.0.1-shared \ --enable-shared \ --disable-static \ CC='/home/common/gcc-11.4.0/bin/gcc' \ CXX='/home/common/gcc-11.4.0/bin/g++' \ FC='/home/common/gcc-11.4.0/bin/gfortran' \ F77='/home/common/gcc-11.4.0/bin/gfortran' \ CFLAGS='-pthread -pipe -fPIC' \ CXXFLAGS='-pthread -Wno-deprecated-declarations -pipe -fPIC' \ FFLAGS='-fallow-argument-mismatch -pipe -fPIC' \ FCFLAGS='-fallow-argument-mismatch -pipe -fPIC' \ LDFLAGS='-L/home/common/gcc-11.4.0/lib64 -Wl,-rpath,/home/common/gcc-11.4.0/lib64' \ LDSHARED='-L/home/common/gcc-11.4.0/lib64 -Wl,-rpath,/home/common/gcc-11.4.0/lib64' \ --with-libfabric=install \ --with-device=ch4:ofi \ --without-cuda \ --disable-gl

1 0

How to get srun/mpich to use the right interface?
by John Cary 11 Jun '26

11 Jun '26

1 0