October 2019 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Issue with OrangeFS 2.9.7 direct interface and MPICH 3.3.1 using CH4 device
by Kun Feng 08 Oct '19

08 Oct '19

Hi Rob, The liborangefsposix library works when I use standard file path like /mnt/pfs/data instead of pvfs2://mnt/pfs/data. I want to use that because it gives better performance in metadata operation which I need for my project. I'm not using MPI I/O to access the files on OrangeFS. It is just a standard C interface call (fopen and fclose). CH4 and liborangefsposix is the only combination that produces this error. Here is the result matrix I have tried. MPICH device Linking OrangeFS Result CH3 non direct interface success CH3 direct interface success CH4 non direct interface success CH4 direct interface no output Thanks Kun On Tue, Oct 8, 2019 at 10:10 AM Latham, Robert J. <robl(a)mcs.anl.gov> wrote: > On Sun, 2019-10-06 at 12:20 -0500, Kun Feng via discuss wrote: > > Hi Min, > > > > If that is the case, please ignore this email. Nothing is wrong > > without OrangeFS direct interface. I will try "ch4:ucx". Thank you > > for the info. > > Does the 'pvfs2' driver still work? the liborangefsposix library might > be intercepting system calls MPICH expects to use natively. > > The liborangefsposix library is intended more for non-mpi applications > -- Hadoop workflows, for example. MPICH's pvfs2 driver (the old name > for OrangeFS) speaks directly to the orangefs servers. It also uses a > few optimizations not available if MPICH treats the OrangeFS like a > traditional UNIX-like file system. > > ==rob > > > > > On Sun, Oct 6, 2019 at 10:25 AM Si, Min via discuss < > > discuss(a)mpich.org> wrote: > > > Hi Kun, > > > > > > Can you please try to reproduce the issue in a simple MPI program > > > which does not use OrangeFS ? It is hard for the MPICH community to > > > help when mixing MPI and OrangeFS together, because we are not > > > OrangeFS experts. > > > > > > Besides, for InfiniBand networks, you might want to use `ch4:ucx` > > > instead of `ch4:ofi`. But I do not think it causes the failure in > > > your use case. > > > > > > Best regards, > > > Min > > > > > > On 2019/10/04 12:21, Kun Feng via discuss wrote: > > > > To Whom It May Concern, > > > > > > > > Recently, I switched to CH4 device in MPICH 3.3.1 for better > > > > network performance over the RoCE network we are using. > > > > I realized that my code fails to run when I use direct interface > > > > of OrangeFS 2.9.7. It exits without any error. But even simple > > > > helloworld cannot print anything. It happens only when I enable > > > > direct interface of OrangeFS by linking -lorangefsposix. > > > > Could you please help me on this issue? > > > > Here are some information that might be useful: > > > > Output of ibv_devinfo of 40Gbps Mellanox ConnectX-4 Lx adapter: > > > > hca_id: mlx5_0 > > > > transport: InfiniBand (0) > > > > fw_ver: 14.20.1030 > > > > node_guid: 248a:0703:0015:a800 > > > > sys_image_guid: 248a:0703:0015:a800 > > > > vendor_id: 0x02c9 > > > > vendor_part_id: 4117 > > > > hw_ver: 0x0 > > > > board_id: LNV2430110027 > > > > phys_port_cnt: 1 > > > > port: 1 > > > > state: PORT_ACTIVE (4) > > > > max_mtu: 4096 (5) > > > > active_mtu: 1024 (3) > > > > sm_lid: 0 > > > > port_lid: 0 > > > > port_lmc: 0x00 > > > > link_layer: Ethernet > > > > > > > > hca_id: i40iw0 > > > > transport: iWARP (1) > > > > fw_ver: 0.2 > > > > node_guid: 7cd3:0aef:3da0:0000 > > > > sys_image_guid: 7cd3:0aef:3da0:0000 > > > > vendor_id: 0x8086 > > > > vendor_part_id: 14289 > > > > hw_ver: 0x0 > > > > board_id: I40IW Board ID > > > > phys_port_cnt: 1 > > > > port: 1 > > > > state: PORT_ACTIVE (4) > > > > max_mtu: 4096 (5) > > > > active_mtu: 1024 (3) > > > > sm_lid: 0 > > > > port_lid: 1 > > > > port_lmc: 0x00 > > > > link_layer: Ethernet > > > > MPICH 3.3.1 configuration command: ./configure --with- > > > > device=ch4:ofi --with-pvfs2=/home/kfeng/install --enable-shared > > > > --enable-romio --with-file-system=ufs+pvfs2+zoidfs --enable- > > > > fortran=no --with-libfabric=/home/kfeng/install > > > > OrangeFS 2.9.7 configuration command: ./configure -- > > > > prefix=/home/kfeng/install --enable-shared --enable-jni --with- > > > > jdk=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64 > > > > --with-kernel=/usr/src/kernels/3.10.0-862.el7.x86_64 > > > > Make command: mpicc -o ~/hello ~/hello.c > > > > -L/home/kfeng/install/lib -lorangefsposix > > > > The verbose outputs of mpiexec are attached. > > > > > > > > Thanks > > > > Kun > > > > > > > > > > > > _______________________________________________ > > > > discuss mailing list discuss(a)mpich.org > > > > To manage subscription options or unsubscribe: > > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > > _______________________________________________ > > > discuss mailing list discuss(a)mpich.org > > > To manage subscription options or unsubscribe: > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > >

1 0

Re: [mpich-discuss] Issue with OrangeFS 2.9.7 direct interface and MPICH 3.3.1 using CH4 device
by Latham, Robert J. 08 Oct '19

08 Oct '19

On Sun, 2019-10-06 at 12:20 -0500, Kun Feng via discuss wrote: > Hi Min, > > If that is the case, please ignore this email. Nothing is wrong > without OrangeFS direct interface. I will try "ch4:ucx". Thank you > for the info. Does the 'pvfs2' driver still work? the liborangefsposix library might be intercepting system calls MPICH expects to use natively. The liborangefsposix library is intended more for non-mpi applications -- Hadoop workflows, for example. MPICH's pvfs2 driver (the old name for OrangeFS) speaks directly to the orangefs servers. It also uses a few optimizations not available if MPICH treats the OrangeFS like a traditional UNIX-like file system. ==rob > > On Sun, Oct 6, 2019 at 10:25 AM Si, Min via discuss < > discuss(a)mpich.org> wrote: > > Hi Kun, > > > > Can you please try to reproduce the issue in a simple MPI program > > which does not use OrangeFS ? It is hard for the MPICH community to > > help when mixing MPI and OrangeFS together, because we are not > > OrangeFS experts. > > > > Besides, for InfiniBand networks, you might want to use `ch4:ucx` > > instead of `ch4:ofi`. But I do not think it causes the failure in > > your use case. > > > > Best regards, > > Min > > > > On 2019/10/04 12:21, Kun Feng via discuss wrote: > > > To Whom It May Concern, > > > > > > Recently, I switched to CH4 device in MPICH 3.3.1 for better > > > network performance over the RoCE network we are using. > > > I realized that my code fails to run when I use direct interface > > > of OrangeFS 2.9.7. It exits without any error. But even simple > > > helloworld cannot print anything. It happens only when I enable > > > direct interface of OrangeFS by linking -lorangefsposix. > > > Could you please help me on this issue? > > > Here are some information that might be useful: > > > Output of ibv_devinfo of 40Gbps Mellanox ConnectX-4 Lx adapter: > > > hca_id: mlx5_0 > > > transport: InfiniBand (0) > > > fw_ver: 14.20.1030 > > > node_guid: 248a:0703:0015:a800 > > > sys_image_guid: 248a:0703:0015:a800 > > > vendor_id: 0x02c9 > > > vendor_part_id: 4117 > > > hw_ver: 0x0 > > > board_id: LNV2430110027 > > > phys_port_cnt: 1 > > > port: 1 > > > state: PORT_ACTIVE (4) > > > max_mtu: 4096 (5) > > > active_mtu: 1024 (3) > > > sm_lid: 0 > > > port_lid: 0 > > > port_lmc: 0x00 > > > link_layer: Ethernet > > > > > > hca_id: i40iw0 > > > transport: iWARP (1) > > > fw_ver: 0.2 > > > node_guid: 7cd3:0aef:3da0:0000 > > > sys_image_guid: 7cd3:0aef:3da0:0000 > > > vendor_id: 0x8086 > > > vendor_part_id: 14289 > > > hw_ver: 0x0 > > > board_id: I40IW Board ID > > > phys_port_cnt: 1 > > > port: 1 > > > state: PORT_ACTIVE (4) > > > max_mtu: 4096 (5) > > > active_mtu: 1024 (3) > > > sm_lid: 0 > > > port_lid: 1 > > > port_lmc: 0x00 > > > link_layer: Ethernet > > > MPICH 3.3.1 configuration command: ./configure --with- > > > device=ch4:ofi --with-pvfs2=/home/kfeng/install --enable-shared > > > --enable-romio --with-file-system=ufs+pvfs2+zoidfs --enable- > > > fortran=no --with-libfabric=/home/kfeng/install > > > OrangeFS 2.9.7 configuration command: ./configure -- > > > prefix=/home/kfeng/install --enable-shared --enable-jni --with- > > > jdk=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64 > > > --with-kernel=/usr/src/kernels/3.10.0-862.el7.x86_64 > > > Make command: mpicc -o ~/hello ~/hello.c > > > -L/home/kfeng/install/lib -lorangefsposix > > > The verbose outputs of mpiexec are attached. > > > > > > Thanks > > > Kun > > > > > > > > > _______________________________________________ > > > discuss mailing list discuss(a)mpich.org > > > To manage subscription options or unsubscribe: > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Issue with OrangeFS 2.9.7 direct interface and MPICH 3.3.1 using CH4 device
by Kun Feng 06 Oct '19

06 Oct '19

Hi Min, If that is the case, please ignore this email. Nothing is wrong without OrangeFS direct interface. I will try "ch4:ucx". Thank you for the info. Thanks Kun On Sun, Oct 6, 2019 at 10:25 AM Si, Min via discuss <discuss(a)mpich.org> wrote: > Hi Kun, > > Can you please try to reproduce the issue in a simple MPI program which > does not use OrangeFS ? It is hard for the MPICH community to help when > mixing MPI and OrangeFS together, because we are not OrangeFS experts. > > Besides, for InfiniBand networks, you might want to use `ch4:ucx` instead > of `ch4:ofi`. But I do not think it causes the failure in your use case. > > Best regards, > Min > > On 2019/10/04 12:21, Kun Feng via discuss wrote: > > To Whom It May Concern, > > Recently, I switched to CH4 device in MPICH 3.3.1 for better network > performance over the RoCE network we are using. > I realized that my code fails to run when I use direct interface of > OrangeFS 2.9.7. It exits without any error. But even simple helloworld > cannot print anything. It happens only when I enable direct interface of > OrangeFS by linking -lorangefsposix. > Could you please help me on this issue? > Here are some information that might be useful: > Output of ibv_devinfo of 40Gbps Mellanox ConnectX-4 Lx adapter: > hca_id: mlx5_0 > transport: InfiniBand (0) > fw_ver: 14.20.1030 > node_guid: 248a:0703:0015:a800 > sys_image_guid: 248a:0703:0015:a800 > vendor_id: 0x02c9 > vendor_part_id: 4117 > hw_ver: 0x0 > board_id: LNV2430110027 > phys_port_cnt: 1 > port: 1 > state: PORT_ACTIVE (4) > max_mtu: 4096 (5) > active_mtu: 1024 (3) > sm_lid: 0 > port_lid: 0 > port_lmc: 0x00 > link_layer: Ethernet > > hca_id: i40iw0 > transport: iWARP (1) > fw_ver: 0.2 > node_guid: 7cd3:0aef:3da0:0000 > sys_image_guid: 7cd3:0aef:3da0:0000 > vendor_id: 0x8086 > vendor_part_id: 14289 > hw_ver: 0x0 > board_id: I40IW Board ID > phys_port_cnt: 1 > port: 1 > state: PORT_ACTIVE (4) > max_mtu: 4096 (5) > active_mtu: 1024 (3) > sm_lid: 0 > port_lid: 1 > port_lmc: 0x00 > link_layer: Ethernet > MPICH 3.3.1 configuration command: ./configure --with-device=ch4:ofi > --with-pvfs2=/home/kfeng/install --enable-shared --enable-romio > --with-file-system=ufs+pvfs2+zoidfs --enable-fortran=no > --with-libfabric=/home/kfeng/install > OrangeFS 2.9.7 configuration command: ./configure > --prefix=/home/kfeng/install --enable-shared --enable-jni > --with-jdk=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64 > --with-kernel=/usr/src/kernels/3.10.0-862.el7.x86_64 > Make command: mpicc -o ~/hello ~/hello.c -L/home/kfeng/install/lib > -lorangefsposix > The verbose outputs of mpiexec are attached. > > Thanks > Kun > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe:https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Issue with OrangeFS 2.9.7 direct interface and MPICH 3.3.1 using CH4 device
by Si, Min 06 Oct '19

06 Oct '19

Hi Kun, Can you please try to reproduce the issue in a simple MPI program which does not use OrangeFS ? It is hard for the MPICH community to help when mixing MPI and OrangeFS together, because we are not OrangeFS experts. Besides, for InfiniBand networks, you might want to use `ch4:ucx` instead of `ch4:ofi`. But I do not think it causes the failure in your use case. Best regards, Min On 2019/10/04 12:21, Kun Feng via discuss wrote: To Whom It May Concern, Recently, I switched to CH4 device in MPICH 3.3.1 for better network performance over the RoCE network we are using. I realized that my code fails to run when I use direct interface of OrangeFS 2.9.7. It exits without any error. But even simple helloworld cannot print anything. It happens only when I enable direct interface of OrangeFS by linking -lorangefsposix. Could you please help me on this issue? Here are some information that might be useful: Output of ibv_devinfo of 40Gbps Mellanox ConnectX-4 Lx adapter: hca_id: mlx5_0 transport: InfiniBand (0) fw_ver: 14.20.1030 node_guid: 248a:0703:0015:a800 sys_image_guid: 248a:0703:0015:a800 vendor_id: 0x02c9 vendor_part_id: 4117 hw_ver: 0x0 board_id: LNV2430110027 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: i40iw0 transport: iWARP (1) fw_ver: 0.2 node_guid: 7cd3:0aef:3da0:0000 sys_image_guid: 7cd3:0aef:3da0:0000 vendor_id: 0x8086 vendor_part_id: 14289 hw_ver: 0x0 board_id: I40IW Board ID phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 1 port_lmc: 0x00 link_layer: Ethernet MPICH 3.3.1 configuration command: ./configure --with-device=ch4:ofi --with-pvfs2=/home/kfeng/install --enable-shared --enable-romio --with-file-system=ufs+pvfs2+zoidfs --enable-fortran=no --with-libfabric=/home/kfeng/install OrangeFS 2.9.7 configuration command: ./configure --prefix=/home/kfeng/install --enable-shared --enable-jni --with-jdk=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64 --with-kernel=/usr/src/kernels/3.10.0-862.el7.x86_64 Make command: mpicc -o ~/hello ~/hello.c -L/home/kfeng/install/lib -lorangefsposix The verbose outputs of mpiexec are attached. Thanks Kun _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Issue with OrangeFS 2.9.7 direct interface and MPICH 3.3.1 using CH4 device
by Kun Feng 04 Oct '19

04 Oct '19

To Whom It May Concern, Recently, I switched to CH4 device in MPICH 3.3.1 for better network performance over the RoCE network we are using. I realized that my code fails to run when I use direct interface of OrangeFS 2.9.7. It exits without any error. But even simple helloworld cannot print anything. It happens only when I enable direct interface of OrangeFS by linking -lorangefsposix. Could you please help me on this issue? Here are some information that might be useful: Output of ibv_devinfo of 40Gbps Mellanox ConnectX-4 Lx adapter: hca_id: mlx5_0 transport: InfiniBand (0) fw_ver: 14.20.1030 node_guid: 248a:0703:0015:a800 sys_image_guid: 248a:0703:0015:a800 vendor_id: 0x02c9 vendor_part_id: 4117 hw_ver: 0x0 board_id: LNV2430110027 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: i40iw0 transport: iWARP (1) fw_ver: 0.2 node_guid: 7cd3:0aef:3da0:0000 sys_image_guid: 7cd3:0aef:3da0:0000 vendor_id: 0x8086 vendor_part_id: 14289 hw_ver: 0x0 board_id: I40IW Board ID phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 1 port_lmc: 0x00 link_layer: Ethernet MPICH 3.3.1 configuration command: ./configure --with-device=ch4:ofi --with-pvfs2=/home/kfeng/install --enable-shared --enable-romio --with-file-system=ufs+pvfs2+zoidfs --enable-fortran=no --with-libfabric=/home/kfeng/install OrangeFS 2.9.7 configuration command: ./configure --prefix=/home/kfeng/install --enable-shared --enable-jni --with-jdk=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64 --with-kernel=/usr/src/kernels/3.10.0-862.el7.x86_64 Make command: mpicc -o ~/hello ~/hello.c -L/home/kfeng/install/lib -lorangefsposix The verbose outputs of mpiexec are attached. Thanks Kun

1 0

Re: [mpich-discuss] MPICH 3.3.1 tests fail
by Zhou, Hui 04 Oct '19

04 Oct '19

Hi Mark, Some of the tests will fail due to resource limit or issues that haven’t been addressed yet. I took a look at your test log and the failed tests are all expected. In particular, the `getfence|putfence` tests are with very large non-contiguous data type that results in timeout due to yet to be optimized data loop code. They designed as stress tests that doesn’t affect most applications. The persistent tests are known failures. The bcastlength test is merely a test fail to produce expected error message. So they are all within expectations. — Hui Zhou On Oct 4, 2019, at 7:35 AM, Mark Cunningham via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: I have built MPICH 3.3.1 on Ubuntu 18.04 with the gcc-7 compiler suite. (Build script is provided: build_mpich.) I have built with both --with-device=ch3:nemesis and --with-device=ch3:nemesis:ofi with the same outcome in the make test. A dozen of the tests fail with timeouts. I ran the test on an 8 processor Thinkstation but what would be beneficial is some sort of sense of what the test results should be. I've browsed the test results that you've posted but these appear to be summaries that conclude All Tests Passed, without individual timings for each test. In any case, I would appreciate any guidance that you might be able to provide. I have provided the config files and the summary.junit.xml file. (On my machine the summary.xml file just reports that there is an error on line 9562.) The result of mpiexec -info is: ~/build/mpich-3.3.1/src/pm/hydra % ./mpiexec.hydra -info HYDRA build details: Version: 3.3.1 Release Date: Wed Jun 5 14:57:33 CDT 2019 CC: gcc CXX: g++ F77: gfortran F90: gfortran Configure options: '--disable-option-checking' '--prefix=/home/share/mpich/mpich-3.3.1' '--with-devce=ch3:nemesis' '--cache-file=/dev/null' '--srcdir=.' 'CC=gcc' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS= -I/home/mark/buildmpich-3.3.1/src/mpl/include -I/home/mark/build/mpich-3.3.1/src/mpl/include -I/home/mark/build/mpich-3.3.1/src/openpa/src -I/hme/mark/build/mpich-3.3.1/src/openpa/src -D_REENTRANT -I/home/mark/build/mpich-3.3.1/src/mpi/romio/include' 'MPLLIBNAME=mpl' Process Manager: pmi Launchers available: ssh rsh fork slurm ll lsf sge manual persist Topology libraries available: hwloc Resource management kernels available: user slurm ll lsf sge pbs cobalt Checkpointing libraries available: Demux engines available: poll select --------------------------------------------------------- Thanks for your help, Mark Mark A Cunningham, PhD Senior Scientist Applied Research in Acoustics LLC Culpeper VA 22701 _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Fwd: MPICH 3.3.1 tests fail
by Benson Muite 04 Oct '19

04 Oct '19

Hi, You would want some of the drivers for the other communication stacks, but TCP/IP should already be available on your system. You may want to check the hardware available on your server farm to decide best communication software stack to use. Benson On 10/4/19 4:13 PM, Mark Cunningham via discuss wrote: > I was unaware that there was a repository in Ubuntu. So, if there are > binaries available, that solves my immediate problem. > > But, I was under the impression that the default build would do > ch3:nemesis anyway, so isn't the --with-device really a no-op? > We also have a bigger server farm, which is why I was fiddling with > the ofi bit. > Thanks, > Mark > > On Fri, Oct 4, 2019 at 8:51 AM Benson Muite via discuss > <discuss(a)mpich.org <mailto:[email protected]>> wrote: > > Ok. Does MPICH from Ubuntu repositories work? > > Have you tried a basic build: > > ./configure --prefix=/home/share/mpich/mpich-3.3.1 > > > On 10/4/19 3:42 PM, Mark Cunningham via discuss wrote: >> This is just running in shared memory for the test. >> >> >> On Fri, Oct 4, 2019 at 8:41 AM Benson Muite via discuss >> <discuss(a)mpich.org <mailto:[email protected]>> wrote: >> >> Hi Mark, >> >> What interconnect is on your machine? >> >> Regards, >> >> Benson >> >> On 10/4/19 3:37 PM, Mark Cunningham via discuss wrote: >>> Oops, sorry I forgot the attachment. >>> >>> ---------- Forwarded message --------- >>> From: *Mark Cunningham* <mark.cunningham(a)ariacoustics.com >>> <mailto:[email protected]>> >>> Date: Fri, Oct 4, 2019 at 8:35 AM >>> Subject: MPICH 3.3.1 tests fail >>> To: <discuss(a)mpich.org <mailto:[email protected]>> >>> >>> >>> I have built MPICH 3.3.1 on Ubuntu 18.04 with the gcc-7 >>> compiler suite. (Build script is provided: build_mpich.) I >>> have built with both --with-device=ch3:nemesis and >>> --with-device=ch3:nemesis:ofi >>> with the same outcome in the make test. A dozen of the >>> tests fail with timeouts. >>> I ran the test on an 8 processor Thinkstation but what would >>> be beneficial is some sort of sense of what the test results >>> should be. I've browsed the test results that you've posted >>> but these appear to be summaries that conclude All Tests >>> Passed, without individual timings for each test. >>> >>> In any case, I would appreciate any guidance that you might >>> be able to provide. I have provided the config files and >>> the summary.junit.xml file. (On my machine the summary.xml >>> file just reports that there is an error on line 9562.) The >>> result of mpiexec -info is: >>> ~/build/mpich-3.3.1/src/pm/hydra % ./mpiexec.hydra -info >>> HYDRA build details: >>> Version: 3.3.1 >>> Release Date: Wed Jun 5 14:57:33 CDT 2019 >>> CC: gcc >>> CXX: g++ >>> F77: gfortran >>> F90: gfortran >>> Configure options: '--disable-option-checking' >>> '--prefix=/home/share/mpich/mpich-3.3.1' >>> '--with-devce=ch3:nemesis' '--cache-file=/dev/null' >>> '--srcdir=.' 'CC=gcc' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=' >>> 'CPPFLAGS= -I/home/mark/buildmpich-3.3.1/src/mpl/include >>> -I/home/mark/build/mpich-3.3.1/src/mpl/include >>> -I/home/mark/build/mpich-3.3.1/src/openpa/src >>> -I/hme/mark/build/mpich-3.3.1/src/openpa/src -D_REENTRANT >>> -I/home/mark/build/mpich-3.3.1/src/mpi/romio/include' >>> 'MPLLIBNAME=mpl' >>> Process Manager: pmi >>> Launchers available: ssh rsh fork slurm ll lsf >>> sge manual persist >>> Topology libraries available: hwloc >>> Resource management kernels available: user slurm ll >>> lsf sge pbs cobalt >>> Checkpointing libraries available: >>> Demux engines available: poll select >>> --------------------------------------------------------- >>> Thanks for your help, >>> Mark >>> >>> Mark A Cunningham, PhD >>> Senior Scientist >>> Applied Research in Acoustics LLC >>> Culpeper VA 22701 >>> >>> >>> _______________________________________________ >>> discuss mailing listdiscuss(a)mpich.org <mailto:[email protected]> >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> _______________________________________________ >> discuss mailing listdiscuss(a)mpich.org <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Fwd: MPICH 3.3.1 tests fail
by Mark Cunningham 04 Oct '19

04 Oct '19

I was unaware that there was a repository in Ubuntu. So, if there are binaries available, that solves my immediate problem. But, I was under the impression that the default build would do ch3:nemesis anyway, so isn't the --with-device really a no-op? We also have a bigger server farm, which is why I was fiddling with the ofi bit. Thanks, Mark On Fri, Oct 4, 2019 at 8:51 AM Benson Muite via discuss <discuss(a)mpich.org> wrote: > Ok. Does MPICH from Ubuntu repositories work? > > Have you tried a basic build: > > ./configure --prefix=/home/share/mpich/mpich-3.3.1 > > > On 10/4/19 3:42 PM, Mark Cunningham via discuss wrote: > > This is just running in shared memory for the test. > > > On Fri, Oct 4, 2019 at 8:41 AM Benson Muite via discuss <discuss(a)mpich.org> > wrote: > >> Hi Mark, >> >> What interconnect is on your machine? >> >> Regards, >> >> Benson >> On 10/4/19 3:37 PM, Mark Cunningham via discuss wrote: >> >> Oops, sorry I forgot the attachment. >> >> ---------- Forwarded message --------- >> From: Mark Cunningham <mark.cunningham(a)ariacoustics.com> >> Date: Fri, Oct 4, 2019 at 8:35 AM >> Subject: MPICH 3.3.1 tests fail >> To: <discuss(a)mpich.org> >> >> >> I have built MPICH 3.3.1 on Ubuntu 18.04 with the gcc-7 compiler suite. >> (Build script is provided: build_mpich.) I have built with both >> --with-device=ch3:nemesis and --with-device=ch3:nemesis:ofi >> with the same outcome in the make test. A dozen of the tests fail with >> timeouts. >> I ran the test on an 8 processor Thinkstation but what would be >> beneficial is some sort of sense of what the test results should be. I've >> browsed the test results that you've posted but these appear to be >> summaries that conclude All Tests Passed, without individual timings for >> each test. >> >> In any case, I would appreciate any guidance that you might be able to >> provide. I have provided the config files and the summary.junit.xml file. >> (On my machine the summary.xml file just reports that there is an error on >> line 9562.) The result of mpiexec -info is: >> ~/build/mpich-3.3.1/src/pm/hydra % ./mpiexec.hydra -info >> HYDRA build details: >> Version: 3.3.1 >> Release Date: Wed Jun 5 14:57:33 CDT 2019 >> CC: gcc >> CXX: g++ >> F77: gfortran >> F90: gfortran >> Configure options: '--disable-option-checking' >> '--prefix=/home/share/mpich/mpich-3.3.1' '--with-devce=ch3:nemesis' >> '--cache-file=/dev/null' '--srcdir=.' 'CC=gcc' 'CFLAGS= -O2' 'LDFLAGS=' >> 'LIBS=' 'CPPFLAGS= -I/home/mark/buildmpich-3.3.1/src/mpl/include >> -I/home/mark/build/mpich-3.3.1/src/mpl/include >> -I/home/mark/build/mpich-3.3.1/src/openpa/src >> -I/hme/mark/build/mpich-3.3.1/src/openpa/src -D_REENTRANT >> -I/home/mark/build/mpich-3.3.1/src/mpi/romio/include' 'MPLLIBNAME=mpl' >> Process Manager: pmi >> Launchers available: ssh rsh fork slurm ll lsf >> sge manual persist >> Topology libraries available: hwloc >> Resource management kernels available: user slurm ll lsf sge pbs >> cobalt >> Checkpointing libraries available: >> Demux engines available: poll select >> --------------------------------------------------------- >> Thanks for your help, >> Mark >> >> Mark A Cunningham, PhD >> Senior Scientist >> Applied Research in Acoustics LLC >> Culpeper VA 22701 >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe:https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe:https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Fwd: MPICH 3.3.1 tests fail
by Benson Muite 04 Oct '19

04 Oct '19

May also wish to try ./configure --prefix=/home/share/mpich/mpich-3.3.1 --with-device=ch3:sock On 10/4/19 3:51 PM, Benson Muite via discuss wrote: > > Ok. Does MPICH from Ubuntu repositories work? > > Have you tried a basic build: > > ./configure --prefix=/home/share/mpich/mpich-3.3.1 > > > On 10/4/19 3:42 PM, Mark Cunningham via discuss wrote: >> This is just running in shared memory for the test. >> >> >> On Fri, Oct 4, 2019 at 8:41 AM Benson Muite via discuss >> <discuss(a)mpich.org <mailto:[email protected]>> wrote: >> >> Hi Mark, >> >> What interconnect is on your machine? >> >> Regards, >> >> Benson >> >> On 10/4/19 3:37 PM, Mark Cunningham via discuss wrote: >>> Oops, sorry I forgot the attachment. >>> >>> ---------- Forwarded message --------- >>> From: *Mark Cunningham* <mark.cunningham(a)ariacoustics.com >>> <mailto:[email protected]>> >>> Date: Fri, Oct 4, 2019 at 8:35 AM >>> Subject: MPICH 3.3.1 tests fail >>> To: <discuss(a)mpich.org <mailto:[email protected]>> >>> >>> >>> I have built MPICH 3.3.1 on Ubuntu 18.04 with the gcc-7 compiler >>> suite. (Build script is provided: build_mpich.) I have built >>> with both --with-device=ch3:nemesis and >>> --with-device=ch3:nemesis:ofi >>> with the same outcome in the make test. A dozen of the tests >>> fail with timeouts. >>> I ran the test on an 8 processor Thinkstation but what would be >>> beneficial is some sort of sense of what the test results should >>> be. I've browsed the test results that you've posted but these >>> appear to be summaries that conclude All Tests Passed, without >>> individual timings for each test. >>> >>> In any case, I would appreciate any guidance that you might be >>> able to provide. I have provided the config files and the >>> summary.junit.xml file. (On my machine the summary.xml file just >>> reports that there is an error on line 9562.) The result of >>> mpiexec -info is: >>> ~/build/mpich-3.3.1/src/pm/hydra % ./mpiexec.hydra -info >>> HYDRA build details: >>> Version: 3.3.1 >>> Release Date: Wed Jun 5 14:57:33 CDT 2019 >>> CC: gcc >>> CXX: g++ >>> F77: gfortran >>> F90: gfortran >>> Configure options: '--disable-option-checking' >>> '--prefix=/home/share/mpich/mpich-3.3.1' >>> '--with-devce=ch3:nemesis' '--cache-file=/dev/null' '--srcdir=.' >>> 'CC=gcc' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS= >>> -I/home/mark/buildmpich-3.3.1/src/mpl/include >>> -I/home/mark/build/mpich-3.3.1/src/mpl/include >>> -I/home/mark/build/mpich-3.3.1/src/openpa/src >>> -I/hme/mark/build/mpich-3.3.1/src/openpa/src -D_REENTRANT >>> -I/home/mark/build/mpich-3.3.1/src/mpi/romio/include' >>> 'MPLLIBNAME=mpl' >>> Process Manager: pmi >>> Launchers available: ssh rsh fork slurm ll lsf sge manual >>> persist >>> Topology libraries available: hwloc >>> Resource management kernels available: user slurm ll lsf >>> sge pbs cobalt >>> Checkpointing libraries available: >>> Demux engines available: poll select >>> --------------------------------------------------------- >>> Thanks for your help, >>> Mark >>> >>> Mark A Cunningham, PhD >>> Senior Scientist >>> Applied Research in Acoustics LLC >>> Culpeper VA 22701 >>> >>> >>> _______________________________________________ >>> discuss mailing listdiscuss(a)mpich.org <mailto:[email protected]> >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> _______________________________________________ >> discuss mailing listdiscuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Fwd: MPICH 3.3.1 tests fail
by Benson Muite 04 Oct '19

04 Oct '19

Ok. Does MPICH from Ubuntu repositories work? Have you tried a basic build: ./configure --prefix=/home/share/mpich/mpich-3.3.1 On 10/4/19 3:42 PM, Mark Cunningham via discuss wrote: > This is just running in shared memory for the test. > > > On Fri, Oct 4, 2019 at 8:41 AM Benson Muite via discuss > <discuss(a)mpich.org <mailto:[email protected]>> wrote: > > Hi Mark, > > What interconnect is on your machine? > > Regards, > > Benson > > On 10/4/19 3:37 PM, Mark Cunningham via discuss wrote: >> Oops, sorry I forgot the attachment. >> >> ---------- Forwarded message --------- >> From: *Mark Cunningham* <mark.cunningham(a)ariacoustics.com >> <mailto:[email protected]>> >> Date: Fri, Oct 4, 2019 at 8:35 AM >> Subject: MPICH 3.3.1 tests fail >> To: <discuss(a)mpich.org <mailto:[email protected]>> >> >> >> I have built MPICH 3.3.1 on Ubuntu 18.04 with the gcc-7 compiler >> suite. (Build script is provided: build_mpich.) I have built >> with both --with-device=ch3:nemesis and --with-device=ch3:nemesis:ofi >> with the same outcome in the make test. A dozen of the tests fail >> with timeouts. >> I ran the test on an 8 processor Thinkstation but what would be >> beneficial is some sort of sense of what the test results should >> be. I've browsed the test results that you've posted but these >> appear to be summaries that conclude All Tests Passed, without >> individual timings for each test. >> >> In any case, I would appreciate any guidance that you might be >> able to provide. I have provided the config files and the >> summary.junit.xml file. (On my machine the summary.xml file just >> reports that there is an error on line 9562.) The result of >> mpiexec -info is: >> ~/build/mpich-3.3.1/src/pm/hydra % ./mpiexec.hydra -info >> HYDRA build details: >> Version: 3.3.1 >> Release Date: Wed Jun 5 14:57:33 CDT 2019 >> CC: gcc >> CXX: g++ >> F77: gfortran >> F90: gfortran >> Configure options: '--disable-option-checking' >> '--prefix=/home/share/mpich/mpich-3.3.1' >> '--with-devce=ch3:nemesis' '--cache-file=/dev/null' '--srcdir=.' >> 'CC=gcc' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS= >> -I/home/mark/buildmpich-3.3.1/src/mpl/include >> -I/home/mark/build/mpich-3.3.1/src/mpl/include >> -I/home/mark/build/mpich-3.3.1/src/openpa/src >> -I/hme/mark/build/mpich-3.3.1/src/openpa/src -D_REENTRANT >> -I/home/mark/build/mpich-3.3.1/src/mpi/romio/include' >> 'MPLLIBNAME=mpl' >> Process Manager: pmi >> Launchers available: ssh rsh fork slurm ll lsf sge manual persist >> Topology libraries available: hwloc >> Resource management kernels available: user slurm ll lsf sge >> pbs cobalt >> Checkpointing libraries available: >> Demux engines available: poll select >> --------------------------------------------------------- >> Thanks for your help, >> Mark >> >> Mark A Cunningham, PhD >> Senior Scientist >> Applied Research in Acoustics LLC >> Culpeper VA 22701 >> >> >> _______________________________________________ >> discuss mailing listdiscuss(a)mpich.org <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0