January 2018 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Segmentation fault with MXM
by Min Si 30 Jan '18

30 Jan '18

MVAPICH does not rely on the mxm layer. It uses the InfiniBand verbs directly. Thus, you do not need Mellanox enhancements (e.g., mxm) for running MVAPICH. I would suggest that you contact the Mellanox tech support for the Mellanox OFED installation issue. It looks like the hostname issue is also because of that. We usually expect that the Mellanox OFED can provide the best performance on Mellanox InfiniBand. All ongoing development for InfiniBand in MPICH are relying on the Mellanox features (e.g., hcoll, ucx). Min On 2018/01/29 18:56, admin(a)genome.arizona.edu wrote: > Min Si wrote on 01/29/2018 03:50 PM: >> I confirmed this segfault issue with >> hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm. However, mpich-3.3a3 >> works fine with the MLNX_OFED version. > > Thanks Min. The "inbox" refers to the software provided by the OS, > while the OFED is the Mellanox version. We had tried using OFED on > our cluster but it turns out that RDMA support for NFS was removed for > unknown reason (no response from Mellanox). The Redhat 'inbox' > software supports NFS/RDMA just fine as it comes directly from NFS > team, and we get much better performance with our disks. > > Since the MXM software is separate, it may work. I did try compiling > MPICH-3.3a3 with the OFED MXM as you suggested, and the bandwidth test > completes without error. However, i noticed we could not use standard > hostfile any longer which uses hostnames like this: > > n001:1 > n002:1 > > With the osu_bibw (bi-directional bandwidth) test, the speed was about > 234 MB/s and was using 1Gb ethernet. Even though "n001" does not > define infiniband IP address in /etc/hosts, the MVAPICH was able to > figure it out. So I had to use modified hostfile like this: > > 10.10.11.1:1 > 10.10.11.2:1 > > Still with this definition, the results from osu_bibw were slower than > with MVAPICH. As well, there are errors when using MPICH-3.2.1 with > the OFED MXM, perhaps due to mixture of inbox/OFED, see below for test > results. It seems we should rely on MPICH-3.3a3 or MVAPICH2 if we > want to use infiniband. > Thanks > > > $ which mpirun > /opt/mpich-3.3a3-install/bin/mpirun > > $ mpirun -n 2 -hostfile /tmp/machinelist ./osu_bibw > # OSU MPI Bi-Directional Bandwidth Test v5.4.0 > # Size Bandwidth (MB/s) > 1 0.26 > 2 0.50 > 4 0.57 > 8 2.02 > 16 4.14 > 32 7.68 > 64 15.60 > 128 26.25 > 256 47.55 > 512 72.63 > 1024 105.92 > 2048 194.08 > 4096 361.07 > 8192 696.26 > 16384 1088.92 > 32768 1496.74 > 65536 1709.40 > 131072 1733.74 > 262144 1918.49 > 524288 1945.67 > 1048576 1963.35 > 2097152 1639.91 > 4194304 1752.34 > > $ > > <change to MVAPICH2 and re-login> > > $ which mpirun > /opt/mvapich2-install/bin/mpirun > > $ mpirun -n 2 -hostfile /tmp/machinelist ./osu_bibw > # OSU MPI Bi-Directional Bandwidth Test v5.4.0 > # Size Bandwidth (MB/s) > 1 0.63 > 2 1.41 > 4 2.80 > 8 5.58 > 16 11.19 > 32 22.29 > 64 43.83 > 128 185.59 > 256 408.23 > 512 770.38 > 1024 1399.30 > 2048 2441.95 > 4096 4001.28 > 8192 4545.17 > 16384 5979.06 > 32768 8097.11 > 65536 8781.06 > 131072 8830.59 > 262144 7966.66 > 524288 7761.30 > 1048576 8099.95 > 2097152 8851.12 > 4194304 9107.43 > $ > > <change to MPICH-3.2.1 and re-login> > > $ which mpirun > /opt/mpich-install/bin/mpirun > > $ mpirun -n 2 -hostfile /tmp/machinelist ./osu_bibw > [1517272925.686782] [n002:21714:0] sys.c:744 MXM WARN > Conflicting CPU frequencies detected, using: 2101.00 > [1517272925.750534] [n001:6646 :0] sys.c:744 MXM WARN > Conflicting CPU frequencies detected, using: 2101.00 > [1517272925.759946] [n002:21714:0] ib_dev.c:533 MXM WARN failed > call to ibv_exp_use_priv_env(): Function not implemented > [1517272925.759991] [n002:21714:0] ib_dev.c:544 MXM ERROR > ibv_query_device() returned 38: Function not implemented > [1517272925.755605] [n001:6646 :0] ib_dev.c:533 MXM WARN failed > call to ibv_exp_use_priv_env(): Function not implemented > [1517272925.755624] [n001:6646 :0] ib_dev.c:544 MXM ERROR > ibv_query_device() returned 38: Function not implemented > Fatal error in MPI_Init: Other MPI error, error stack: > MPIR_Init_thread(474).........: > MPID_Init(190)................: channel initialization failed > MPIDI_CH3_Init(89)............: > MPID_nem_init(320)............: > MPID_nem_mxm_init(163)........: > MPID_nem_mxm_get_ordering(469): mxm_init failed (Input/output error) > Fatal error in MPI_Init: Other MPI error, error stack: > MPIR_Init_thread(474).........: > MPID_Init(190)................: channel initialization failed > MPIDI_CH3_Init(89)............: > MPID_nem_init(320)............: > MPID_nem_mxm_init(163)........: > MPID_nem_mxm_get_ordering(469): mxm_init failed (Input/output error) > $ > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Segmentation fault with MXM
by admin＠genome.arizona.edu 29 Jan '18

29 Jan '18

Min Si wrote on 01/29/2018 03:50 PM: > I confirmed this segfault issue with > hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm. However, mpich-3.3a3 works > fine with the MLNX_OFED version. Thanks Min. The "inbox" refers to the software provided by the OS, while the OFED is the Mellanox version. We had tried using OFED on our cluster but it turns out that RDMA support for NFS was removed for unknown reason (no response from Mellanox). The Redhat 'inbox' software supports NFS/RDMA just fine as it comes directly from NFS team, and we get much better performance with our disks. Since the MXM software is separate, it may work. I did try compiling MPICH-3.3a3 with the OFED MXM as you suggested, and the bandwidth test completes without error. However, i noticed we could not use standard hostfile any longer which uses hostnames like this: n001:1 n002:1 With the osu_bibw (bi-directional bandwidth) test, the speed was about 234 MB/s and was using 1Gb ethernet. Even though "n001" does not define infiniband IP address in /etc/hosts, the MVAPICH was able to figure it out. So I had to use modified hostfile like this: 10.10.11.1:1 10.10.11.2:1 Still with this definition, the results from osu_bibw were slower than with MVAPICH. As well, there are errors when using MPICH-3.2.1 with the OFED MXM, perhaps due to mixture of inbox/OFED, see below for test results. It seems we should rely on MPICH-3.3a3 or MVAPICH2 if we want to use infiniband. Thanks $ which mpirun /opt/mpich-3.3a3-install/bin/mpirun $ mpirun -n 2 -hostfile /tmp/machinelist ./osu_bibw # OSU MPI Bi-Directional Bandwidth Test v5.4.0 # Size Bandwidth (MB/s) 1 0.26 2 0.50 4 0.57 8 2.02 16 4.14 32 7.68 64 15.60 128 26.25 256 47.55 512 72.63 1024 105.92 2048 194.08 4096 361.07 8192 696.26 16384 1088.92 32768 1496.74 65536 1709.40 131072 1733.74 262144 1918.49 524288 1945.67 1048576 1963.35 2097152 1639.91 4194304 1752.34 $ <change to MVAPICH2 and re-login> $ which mpirun /opt/mvapich2-install/bin/mpirun $ mpirun -n 2 -hostfile /tmp/machinelist ./osu_bibw # OSU MPI Bi-Directional Bandwidth Test v5.4.0 # Size Bandwidth (MB/s) 1 0.63 2 1.41 4 2.80 8 5.58 16 11.19 32 22.29 64 43.83 128 185.59 256 408.23 512 770.38 1024 1399.30 2048 2441.95 4096 4001.28 8192 4545.17 16384 5979.06 32768 8097.11 65536 8781.06 131072 8830.59 262144 7966.66 524288 7761.30 1048576 8099.95 2097152 8851.12 4194304 9107.43 $ <change to MPICH-3.2.1 and re-login> $ which mpirun /opt/mpich-install/bin/mpirun $ mpirun -n 2 -hostfile /tmp/machinelist ./osu_bibw [1517272925.686782] [n002:21714:0] sys.c:744 MXM WARN Conflicting CPU frequencies detected, using: 2101.00 [1517272925.750534] [n001:6646 :0] sys.c:744 MXM WARN Conflicting CPU frequencies detected, using: 2101.00 [1517272925.759946] [n002:21714:0] ib_dev.c:533 MXM WARN failed call to ibv_exp_use_priv_env(): Function not implemented [1517272925.759991] [n002:21714:0] ib_dev.c:544 MXM ERROR ibv_query_device() returned 38: Function not implemented [1517272925.755605] [n001:6646 :0] ib_dev.c:533 MXM WARN failed call to ibv_exp_use_priv_env(): Function not implemented [1517272925.755624] [n001:6646 :0] ib_dev.c:544 MXM ERROR ibv_query_device() returned 38: Function not implemented Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(474).........: MPID_Init(190)................: channel initialization failed MPIDI_CH3_Init(89)............: MPID_nem_init(320)............: MPID_nem_mxm_init(163)........: MPID_nem_mxm_get_ordering(469): mxm_init failed (Input/output error) Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(474).........: MPID_Init(190)................: channel initialization failed MPIDI_CH3_Init(89)............: MPID_nem_init(320)............: MPID_nem_mxm_init(163)........: MPID_nem_mxm_get_ordering(469): mxm_init failed (Input/output error) $ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Segmentation fault with MXM
by Min Si 29 Jan '18

29 Jan '18

Hi, I confirmed this segfault issue with hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm. However, mpich-3.3a3 works fine with the MLNX_OFED version. You can download it at: http://www.mellanox.com/downloads/hpc/hpc-x/v2.0/hpcx-v2.0.0-gcc-MLNX_OFED_… I am not sure what is the difference of the mxm libraries (close-source) in the *MLNX_OFED* and *inbox* versions. Perhaps you could get more information from the mxm tech support. Min On 2018/01/26 17:27, admin(a)genome.arizona.edu wrote: > Min Si wrote on 01/26/2018 04:04 PM: >> Could you please confirm if this is your configure ? >> MPICH version: 3.2.1 >> ./configure --prefix=/opt/mpich-install --with-device=ch3:nemesis:mxm >> --with-mxm=/opt/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm > > Yes this is correct. > >> Meanwhile, could you please also try mpich-3.3a3 (see >> http://www.mpich.org/downloads/) ? It includes a few bug fixes. > > I tried with mpich-3.3a3 and there was still a segmentation fault: > > $ which mpirun > /opt/mpich-3.3a3-install/bin/mpirun > > $ mpirun -np 2 -hostfile /tmp/machinelist ./osu_bibw > [1517009019.605233] [n001:18235:0] sys.c:744 MXM WARN > Conflicting CPU frequencies detected, using: 2101.00 > [1517009019.650442] [n001:18235:0] proto_ep.c:179 MXM WARN tl dc > is requested but not supported > [1517009019.651182] [n002:32467:0] proto_ep.c:179 MXM WARN tl dc > is requested but not supported > # OSU MPI Bi-Directional Bandwidth Test v5.4.0 > # Size Bandwidth (MB/s) > 1 1.57 > 2 4.91 > 4 9.80 > 8 20.10 > 16 40.09 > 32 77.33 > 64 149.54 > [n001:18235:0] Caught signal 11 (Segmentation fault) > ==== backtrace ==== > 2 0x000000000005767c mxm_handle_error() > /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/util/debug/debug.c:641 > 3 0x00000000000577ec mxm_error_signal_handler() > /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/util/debug/debug.c:616 > 4 0x0000003c80832510 killpg() ??:0 > 5 0x0000000000056258 mxm_mpool_put() > /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/util/datatype/mpool.c:210 > 6 0x00000000000689ce mxm_cib_ep_poll_tx() > /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/tl/cib/cib_progress.c:527 > 7 0x000000000006913d mxm_cib_ep_progress() > /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/tl/cib/cib_progress.c:552 > 8 0x000000000004268a mxm_notifier_chain_call() > /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/./mxm/util/datatype/callback.h:74 > 9 0x000000000004268a mxm_progress_internal() > /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/core/mxm.c:64 > 10 0x000000000004268a mxm_progress() > /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/core/mxm.c:346 > 11 0x0000000000177a49 MPID_nem_mxm_poll() ??:0 > 12 0x0000000000169be8 MPIDI_CH3I_Progress() ??:0 > 13 0x00000000000d0ba7 MPIR_Waitall_impl() ??:0 > 14 0x00000000000d1308 PMPI_Waitall() ??:0 > 15 0x00000000004016f5 main() > /opt/downloads/osu-micro-benchmarks-5.4/mpi/pt2pt/osu_bibw.c:124 > 16 0x0000003c8081ed1d __libc_start_main() ??:0 > 17 0x0000000000401269 _start() ??:0 > =================== > > =================================================================================== > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 18235 RUNNING AT n001 > = EXIT CODE: 139 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > > [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb > (pm/pmiserv/pmip_cb.c:892): assert (!closed) failed > [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event > (tools/demux/demux_poll.c:77): callback returned error status > [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:202): > demux engine error waiting for event > [mpiexec(a)pac.genome.arizona.edu] HYDT_bscu_wait_for_completion > (tools/bootstrap/utils/bscu_wait.c:76): one of the processes > terminated badly; aborting > [mpiexec(a)pac.genome.arizona.edu] HYDT_bsci_wait_for_completion > (tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting > for completion > [mpiexec(a)pac.genome.arizona.edu] HYD_pmci_wait_for_completion > (pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for > completion > [mpiexec(a)pac.genome.arizona.edu] main (ui/mpich/mpiexec.c:340): > process manager error waiting for completion > > > Thanks Min > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Segmentation fault with MXM
by admin＠genome.arizona.edu 26 Jan '18

26 Jan '18

Min Si wrote on 01/26/2018 04:04 PM: > Could you please confirm if this is your configure ? > MPICH version: 3.2.1 > ./configure --prefix=/opt/mpich-install --with-device=ch3:nemesis:mxm > --with-mxm=/opt/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm Yes this is correct. > Meanwhile, could you please also try mpich-3.3a3 (see > http://www.mpich.org/downloads/) ? It includes a few bug fixes. I tried with mpich-3.3a3 and there was still a segmentation fault: $ which mpirun /opt/mpich-3.3a3-install/bin/mpirun $ mpirun -np 2 -hostfile /tmp/machinelist ./osu_bibw [1517009019.605233] [n001:18235:0] sys.c:744 MXM WARN Conflicting CPU frequencies detected, using: 2101.00 [1517009019.650442] [n001:18235:0] proto_ep.c:179 MXM WARN tl dc is requested but not supported [1517009019.651182] [n002:32467:0] proto_ep.c:179 MXM WARN tl dc is requested but not supported # OSU MPI Bi-Directional Bandwidth Test v5.4.0 # Size Bandwidth (MB/s) 1 1.57 2 4.91 4 9.80 8 20.10 16 40.09 32 77.33 64 149.54 [n001:18235:0] Caught signal 11 (Segmentation fault) ==== backtrace ==== 2 0x000000000005767c mxm_handle_error() /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/util/debug/debug.c:641 3 0x00000000000577ec mxm_error_signal_handler() /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/util/debug/debug.c:616 4 0x0000003c80832510 killpg() ??:0 5 0x0000000000056258 mxm_mpool_put() /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/util/datatype/mpool.c:210 6 0x00000000000689ce mxm_cib_ep_poll_tx() /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/tl/cib/cib_progress.c:527 7 0x000000000006913d mxm_cib_ep_progress() /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/tl/cib/cib_progress.c:552 8 0x000000000004268a mxm_notifier_chain_call() /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/./mxm/util/datatype/callback.h:74 9 0x000000000004268a mxm_progress_internal() /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/core/mxm.c:64 10 0x000000000004268a mxm_progress() /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/core/mxm.c:346 11 0x0000000000177a49 MPID_nem_mxm_poll() ??:0 12 0x0000000000169be8 MPIDI_CH3I_Progress() ??:0 13 0x00000000000d0ba7 MPIR_Waitall_impl() ??:0 14 0x00000000000d1308 PMPI_Waitall() ??:0 15 0x00000000004016f5 main() /opt/downloads/osu-micro-benchmarks-5.4/mpi/pt2pt/osu_bibw.c:124 16 0x0000003c8081ed1d __libc_start_main() ??:0 17 0x0000000000401269 _start() ??:0 =================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 18235 RUNNING AT n001 = EXIT CODE: 139 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:892): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): callback returned error status [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:202): demux engine error waiting for event [mpiexec(a)pac.genome.arizona.edu] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec(a)pac.genome.arizona.edu] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec(a)pac.genome.arizona.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion [mpiexec(a)pac.genome.arizona.edu] main (ui/mpich/mpiexec.c:340): process manager error waiting for completion Thanks Min _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Segmentation fault with MXM
by Min Si 26 Jan '18

26 Jan '18

It looks like the segfault was reported from the MXM library. But as you mentioned, MVAPICH works fine. Thus, we should verify whether this is an MPICH issue or a MXM issue. I will try to reproduce the issue on our test platform and keep you updated. Could you please confirm if this is your configure ? MPICH version: 3.2.1 ./configure --prefix=/opt/mpich-install --with-device=ch3:nemesis:mxm --with-mxm=/opt/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm Meanwhile, could you please also try mpich-3.3a3 (see http://www.mpich.org/downloads/) ? It includes a few bug fixes. Thanks, Min On 2018/01/26 16:47, admin(a)genome.arizona.edu wrote: > I tried using OSU benchmark with OSU MVAPICH2 and that worked fine, > was able to verify infiniband is being used. However, there is a > segmentation fault when using MPICH, see below. When re-compiled > MPICH without MXM, the OSU benchmark works as expected as well as > reports slower speed of 1Gb ethernet network... > > It seems to be related to MXM, so perhaps I need to contact Mellanox > regarding this? > > Thanks > > > > $ which mpirun > /opt/mpich-install/bin/mpirun > > $ mpirun -np 2 -hostfile /opt/machinelist ./osu_bibw > [1517005194.381407] [n001:18086:0] sys.c:744 MXM WARN > Conflicting CPU frequencies detected, using: 2101.00 > [1517005194.513782] [n002:32344:0] sys.c:744 MXM WARN > Conflicting CPU frequencies detected, using: 2101.00 > [1517005194.599025] [n002:32344:0] proto_ep.c:179 MXM WARN tl dc > is requested but not supported > [1517005194.655493] [n001:18086:0] proto_ep.c:179 MXM WARN tl dc > is requested but not supported > # OSU MPI Bi-Directional Bandwidth Test v5.4.0 > # Size Bandwidth (MB/s) > 1 1.64 > 2 4.88 > 4 9.84 > 8 19.83 > 16 39.48 > 32 76.25 > 64 150.78 > [n002:32344:0] Caught signal 11 (Segmentation fault) > ==== backtrace ==== > 2 0x000000000005767c mxm_handle_error() > /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/util/debug/debug.c:641 > 3 0x00000000000577ec mxm_error_signal_handler() > /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/util/debug/debug.c:616 > 4 0x0000003550432510 killpg() ??:0 > 5 0x0000000000056258 mxm_mpool_put() > /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/util/datatype/mpool.c:210 > 6 0x00000000000689ce mxm_cib_ep_poll_tx() > /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/tl/cib/cib_progress.c:527 > 7 0x000000000006913d mxm_cib_ep_progress() > /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/tl/cib/cib_progress.c:552 > 8 0x000000000004268a mxm_notifier_chain_call() > /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/./mxm/util/datatype/callback.h:74 > 9 0x000000000004268a mxm_progress_internal() > /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/core/mxm.c:64 > 10 0x000000000004268a mxm_progress() > /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/core/mxm.c:346 > 11 0x0000000000177a49 MPID_nem_mxm_poll() ??:0 > 12 0x0000000000169be8 MPIDI_CH3I_Progress() ??:0 > 13 0x00000000000d0ba7 MPIR_Waitall_impl() ??:0 > 14 0x00000000000d1308 PMPI_Waitall() ??:0 > 15 0x0000000000401904 main() > /opt/downloads/osu-micro-benchmarks-5.4/mpi/pt2pt/osu_bibw.c:146 > 16 0x000000355041ed1d __libc_start_main() ??:0 > 17 0x0000000000401269 _start() ??:0 > =================== > > =================================================================================== > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 32344 RUNNING AT n002 > = EXIT CODE: 139 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > > [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb > (pm/pmiserv/pmip_cb.c:887): assert (!closed) failed > [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event > (tools/demux/demux_poll.c:76): callback returned error status > [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:202): > demux engine error waiting for event > [mpiexec(a)pac.genome.arizona.edu] HYDT_bscu_wait_for_completion > (tools/bootstrap/utils/bscu_wait.c:76): one of the processes > terminated badly; aborting > [mpiexec(a)pac.genome.arizona.edu] HYDT_bsci_wait_for_completion > (tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting > for completion > [mpiexec(a)pac.genome.arizona.edu] HYD_pmci_wait_for_completion > (pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for > completion > [mpiexec(a)pac.genome.arizona.edu] main (ui/mpich/mpiexec.c:340): > process manager error waiting for completion > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Segmentation fault with MXM
by admin＠genome.arizona.edu 26 Jan '18

26 Jan '18

I tried using OSU benchmark with OSU MVAPICH2 and that worked fine, was able to verify infiniband is being used. However, there is a segmentation fault when using MPICH, see below. When re-compiled MPICH without MXM, the OSU benchmark works as expected as well as reports slower speed of 1Gb ethernet network... It seems to be related to MXM, so perhaps I need to contact Mellanox regarding this? Thanks $ which mpirun /opt/mpich-install/bin/mpirun $ mpirun -np 2 -hostfile /opt/machinelist ./osu_bibw [1517005194.381407] [n001:18086:0] sys.c:744 MXM WARN Conflicting CPU frequencies detected, using: 2101.00 [1517005194.513782] [n002:32344:0] sys.c:744 MXM WARN Conflicting CPU frequencies detected, using: 2101.00 [1517005194.599025] [n002:32344:0] proto_ep.c:179 MXM WARN tl dc is requested but not supported [1517005194.655493] [n001:18086:0] proto_ep.c:179 MXM WARN tl dc is requested but not supported # OSU MPI Bi-Directional Bandwidth Test v5.4.0 # Size Bandwidth (MB/s) 1 1.64 2 4.88 4 9.84 8 19.83 16 39.48 32 76.25 64 150.78 [n002:32344:0] Caught signal 11 (Segmentation fault) ==== backtrace ==== 2 0x000000000005767c mxm_handle_error() /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/util/debug/debug.c:641 3 0x00000000000577ec mxm_error_signal_handler() /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/util/debug/debug.c:616 4 0x0000003550432510 killpg() ??:0 5 0x0000000000056258 mxm_mpool_put() /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/util/datatype/mpool.c:210 6 0x00000000000689ce mxm_cib_ep_poll_tx() /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/tl/cib/cib_progress.c:527 7 0x000000000006913d mxm_cib_ep_progress() /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/tl/cib/cib_progress.c:552 8 0x000000000004268a mxm_notifier_chain_call() /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/./mxm/util/datatype/callback.h:74 9 0x000000000004268a mxm_progress_internal() /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/core/mxm.c:64 10 0x000000000004268a mxm_progress() /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel6-u9-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm-v3.6/src/mxm/core/mxm.c:346 11 0x0000000000177a49 MPID_nem_mxm_poll() ??:0 12 0x0000000000169be8 MPIDI_CH3I_Progress() ??:0 13 0x00000000000d0ba7 MPIR_Waitall_impl() ??:0 14 0x00000000000d1308 PMPI_Waitall() ??:0 15 0x0000000000401904 main() /opt/downloads/osu-micro-benchmarks-5.4/mpi/pt2pt/osu_bibw.c:146 16 0x000000355041ed1d __libc_start_main() ??:0 17 0x0000000000401269 _start() ??:0 =================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 32344 RUNNING AT n002 = EXIT CODE: 139 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:887): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:202): demux engine error waiting for event [mpiexec(a)pac.genome.arizona.edu] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec(a)pac.genome.arizona.edu] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec(a)pac.genome.arizona.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion [mpiexec(a)pac.genome.arizona.edu] main (ui/mpich/mpiexec.c:340): process manager error waiting for completion _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Help on ticket #2889
by Shuwei Zhao 24 Jan '18

24 Jan '18

Hi, all Sorry to disturb you on the forum. I’m writing email to seek help on ticket #2889 for ipv6 https://github.com/pmodels/mpich/issues/2889 My team has an IPV6 project due in the middle of Feb and all we lack is the mpich library with the update. I'll appreciate any help in advance！ Thanks, Shuwei _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Verify infiniband communication
by Min Si 24 Jan '18

24 Jan '18

If you specified the mxm device at installation, it should use the InfiniBand network. MPICH does not have a mechanism to check the layer under mxm (i.e., InfinBand or Ethernet). A simple way to verify is to test the bandwidth. You can use OSU benchmark or Intel MPI benchmark to test the MPI pt2pt bandwidth. If the InfiniBand network is used, it is usually several GB/s which is much faster than that on Ethernet. Min On 2018/01/24 4:14, Min Si wrote: > Hi, > > You can run /opt/mpich-install/examples/cpi to test the basic > functionality of MPICH/mxm. > For example: > mpiexec -np 2 -ppn 1 -f <hostfile> ./cpi > > For more testing options, you can find information at > http://www.mpich.org/documentation/guides/ > > Thanks, > Min > > On 2018/01/23 19:09, Chandler wrote: >> Hi all, >> I configured and installed mpich-3.2.1 on our cluster with Mellanox >> infiniband adapters. I used this command: >> >> ./configure --prefix=/opt/mpich-install --with-device=ch3:nemesis:mxm >> --with-mxm=/opt/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm >> >> which worked fine without errors, then "make install". Now would >> like to verify that MPICH will use our infiniband network by default >> (the cluster also has 1Gbps ethernet network). Can you let me know >> some test commands to run to verify the infiniband devices are used? >> >> Thanks, > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Verify infiniband communication
by Min Si 24 Jan '18

24 Jan '18

Hi, You can run /opt/mpich-install/examples/cpi to test the basic functionality of MPICH/mxm. For example: mpiexec -np 2 -ppn 1 -f <hostfile> ./cpi For more testing options, you can find information at http://www.mpich.org/documentation/guides/ Thanks, Min On 2018/01/23 19:09, Chandler wrote: > Hi all, > I configured and installed mpich-3.2.1 on our cluster with Mellanox > infiniband adapters. I used this command: > > ./configure --prefix=/opt/mpich-install --with-device=ch3:nemesis:mxm > --with-mxm=/opt/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm > > which worked fine without errors, then "make install". Now would like > to verify that MPICH will use our infiniband network by default (the > cluster also has 1Gbps ethernet network). Can you let me know some > test commands to run to verify the infiniband devices are used? > > Thanks, _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Verify infiniband communication
by Chandler 23 Jan '18

23 Jan '18

Hi all, I configured and installed mpich-3.2.1 on our cluster with Mellanox infiniband adapters. I used this command: ./configure --prefix=/opt/mpich-install --with-device=ch3:nemesis:mxm --with-mxm=/opt/hpcx-v2.0.0-gcc-inbox-redhat6.9-x86_64/mxm which worked fine without errors, then "make install". Now would like to verify that MPICH will use our infiniband network by default (the cluster also has 1Gbps ethernet network). Can you let me know some test commands to run to verify the infiniband devices are used? Thanks, -- Chandler / Systems Administrator Arizona Genomics Institute www.genome.arizona.edu _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0