Dear MPIch users and developers, first of all many thanks for all the great work you have done for MPIch! I'd like to have 4 MPI jobs scheduled by SGE starting 1 OpenMP job each with 10 threads, running on 2 nodes, each having 2 sockets, with 10 cores & 10 hwthreads. Only 10 cores (no hwthreads) should be used on each socket. 4 MPI: 1 OpenMP with 10 thread (i.e. 4x10 threads) 2 nodes, 2 sockets each, 10 cores & 10 hwthreads each lscpu -a -e CPU NODE SOCKET CORE L1d:L1i:L2:L3 0 0 0 0 0:0:0:0 1 1 1 1 1:1:1:1 2 0 0 2 2:2:2:0 3 1 1 3 3:3:3:1 4 0 0 4 4:4:4:0 5 1 1 5 5:5:5:1 6 0 0 6 6:6:6:0 7 1 1 7 7:7:7:1 8 0 0 8 8:8:8:0 9 1 1 9 9:9:9:1 10 0 0 10 10:10:10:0 11 1 1 11 11:11:11:1 12 0 0 12 12:12:12:0 13 1 1 13 13:13:13:1 14 0 0 14 14:14:14:0 15 1 1 15 15:15:15:1 16 0 0 16 16:16:16:0 17 1 1 17 17:17:17:1 18 0 0 18 18:18:18:0 19 1 1 19 19:19:19:1 20 0 0 0 0:0:0:0 21 1 1 1 1:1:1:1 22 0 0 2 2:2:2:0 23 1 1 3 3:3:3:1 24 0 0 4 4:4:4:0 25 1 1 5 5:5:5:1 26 0 0 6 6:6:6:0 27 1 1 7 7:7:7:1 28 0 0 8 8:8:8:0 29 1 1 9 9:9:9:1 30 0 0 10 10:10:10:0 31 1 1 11 11:11:11:1 32 0 0 12 12:12:12:0 33 1 1 13 13:13:13:1 34 0 0 14 14:14:14:0 35 1 1 15 15:15:15:1 36 0 0 16 16:16:16:0 37 1 1 17 17:17:17:1 38 0 0 18 18:18:18:0 39 1 1 19 19:19:19:1 When I try to submit the job by using HYDRA_TOPO_DEBUG=1 mpirun -np 4 -bind-to socket:1 -map-by socket ./myid the distribution is done to 2 sockets on 2 nodes correctly, but all 10 cores + 10 hwthreads are used on each node: process 0 binding: 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 process 1 binding: 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 process 2 binding: 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 process 3 binding: 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 Additionally it seems, that the CPU masks & Cpus_allowed_list are not set for the called processes: MPI Instance 0001 of 0004 is on pascal-1-04, 0x000000ff,0xffffffff, Cpus_allowed_list: 0-39 MPI Instance 0002 of 0004 is on pascal-1-04, 0x000000ff,0xffffffff, Cpus_allowed_list: 0-39 MPI Instance 0003 of 0004 is on pascal-3-06, 0x000000ff,0xffffffff, Cpus_allowed_list: 0-39 MPI Instance 0004 of 0004 is on pascal-3-06, 0x000000ff,0xffffffff, Cpus_allowed_list: 0-39 Another way to achive my task would be to use "-bind-to user" HYDRA_TOPO_DEBUG=1 mpirun -np $nmpi -ppn $nmpipn -bind-to user:0+2+4+6+8+10+12+14+16+18,1+3+5+7+9+11+13+15+17+19 ./myid This works great up to specifying cores 9 cores on each socket ("0+2+4+6+8+10+12+14+16,1+3+5+7+9+11+13+15+17"). As soon as I add ...+18,...+19, the job crashes with these messages: *** Error in `/usr/bin/hydra_pmi_proxy': free(): invalid next size (fast): 0x0000000000f9f0c0 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x704fb)[0x2b3f62f5c4fb] /lib64/libc.so.6(+0x76976)[0x2b3f62f62976] /lib64/libc.so.6(+0x7716e)[0x2b3f62f6316e] /usr/bin/hydra_pmi_proxy[0x41aa27] /usr/bin/hydra_pmi_proxy[0x418aea] /usr/bin/hydra_pmi_proxy[0x407c0c] /usr/bin/hydra_pmi_proxy[0x420bb0] /usr/bin/hydra_pmi_proxy[0x403e49] /lib64/libc.so.6(__libc_start_main+0xf1)[0x2b3f62f0c4e1] /usr/bin/hydra_pmi_proxy[0x404cca] ======= Memory map: ======== 00400000-0045c000 r-xp 00000000 00:10 2142949266 /afs/mpa/@sys/system/MPA-8.03/usr/bin/hydra_pmi_proxy 0065c000-0065d000 r--p 0005c000 00:10 2142949266 /afs/mpa/@sys/system/MPA-8.03/usr/bin/hydra_pmi_proxy 0065d000-0065f000 rw-p 0005d000 00:10 2142949266 /afs/mpa/@sys/system/MPA-8.03/usr/bin/hydra_pmi_proxy 0065f000-006ff000 rw-p 00000000 00:00 0 00f98000-00fda000 rw-p 00000000 00:00 0 [heap] 2b3f619c5000-2b3f619e8000 r-xp 00000000 00:02 1229 /lib64/ld-2.25.so 2b3f619e8000-2b3f619eb000 rw-p 00000000 00:00 0 2b3f61a09000-2b3f61a0d000 rw-p 00000000 00:00 0 2b3f61be7000-2b3f61be8000 r--p 00022000 00:02 1229 /lib64/ld-2.25.so 2b3f61be8000-2b3f61be9000 rw-p 00023000 00:02 1229 /lib64/ld-2.25.so 2b3f61be9000-2b3f61bea000 rw-p 00000000 00:00 0 2b3f61bea000-2b3f61c03000 r-xp 00000000 00:02 1417 /lib64/libpthread-2.25.so 2b3f61c03000-2b3f61e02000 ---p 00019000 00:02 1417 /lib64/libpthread-2.25.so 2b3f61e02000-2b3f61e03000 r--p 00018000 00:02 1417 /lib64/libpthread-2.25.so 2b3f61e03000-2b3f61e04000 rw-p 00019000 00:02 1417 /lib64/libpthread-2.25.so 2b3f61e04000-2b3f61e08000 rw-p 00000000 00:00 0 2b3f61e08000-2b3f61e2c000 r-xp 00000000 00:02 1227 /lib64/libudev.so.1.6.3 2b3f61e2c000-2b3f6202b000 ---p 00024000 00:02 1227 /lib64/libudev.so.1.6.3 2b3f6202b000-2b3f6202c000 r--p 00023000 00:02 1227 /lib64/libudev.so.1.6.3 2b3f6202c000-2b3f6202d000 rw-p 00024000 00:02 1227 /lib64/libudev.so.1.6.3 2b3f6202d000-2b3f62035000 r-xp 00000000 00:10 2142711818 /afs/mpa/@sys/system/MPA-8.03/usr/X11/lib64/libpciaccess.so.0.11.1 2b3f62035000-2b3f62234000 ---p 00008000 00:10 2142711818 /afs/mpa/@sys/system/MPA-8.03/usr/X11/lib64/libpciaccess.so.0.11.1 2b3f62234000-2b3f62235000 r--p 00007000 00:10 2142711818 /afs/mpa/@sys/system/MPA-8.03/usr/X11/lib64/libpciaccess.so.0.11.1 2b3f62235000-2b3f62236000 rw-p 00008000 00:10 2142711818 /afs/mpa/@sys/system/MPA-8.03/usr/X11/lib64/libpciaccess.so.0.11.1 2b3f62236000-2b3f62390000 r-xp 00000000 00:10 2142836572 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libxml2.so.2.9.4 2b3f62390000-2b3f62590000 ---p 0015a000 00:10 2142836572 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libxml2.so.2.9.4 2b3f62590000-2b3f62598000 r--p 0015a000 00:10 2142836572 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libxml2.so.2.9.4 2b3f62598000-2b3f6259a000 rw-p 00162000 00:10 2142836572 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libxml2.so.2.9.4 2b3f6259a000-2b3f6259b000 rw-p 00000000 00:00 0 2b3f6259b000-2b3f6259e000 r-xp 00000000 00:02 1433 /lib64/libdl-2.25.so 2b3f6259e000-2b3f6279d000 ---p 00003000 00:02 1433 /lib64/libdl-2.25.so 2b3f6279d000-2b3f6279e000 r--p 00002000 00:02 1433 /lib64/libdl-2.25.so 2b3f6279e000-2b3f6279f000 rw-p 00003000 00:02 1433 /lib64/libdl-2.25.so 2b3f6279f000-2b3f627b4000 r-xp 00000000 00:10 2142836590 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libz.so.1.2.6 2b3f627b4000-2b3f629b3000 ---p 00015000 00:10 2142836590 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libz.so.1.2.6 2b3f629b3000-2b3f629b4000 r--p 00014000 00:10 2142836590 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libz.so.1.2.6 2b3f629b4000-2b3f629b5000 rw-p 00015000 00:10 2142836590 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libz.so.1.2.6 2b3f629b5000-2b3f629d9000 r-xp 00000000 00:10 2142835680 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/liblzma.so.5.2.3 2b3f629d9000-2b3f62bd9000 ---p 00024000 00:10 2142835680 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/liblzma.so.5.2.3 2b3f62bd9000-2b3f62bda000 r--p 00024000 00:10 2142835680 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/liblzma.so.5.2.3 2b3f62bda000-2b3f62bdb000 rw-p 00025000 00:10 2142835680 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/liblzma.so.5.2.3 2b3f62bdb000-2b3f62ceb000 r-xp 00000000 00:02 1401 /lib64/libm-2.25.so 2b3f62ceb000-2b3f62eea000 ---p 00110000 00:02 1401 /lib64/libm-2.25.so 2b3f62eea000-2b3f62eeb000 r--p 0010f000 00:02 1401 /lib64/libm-2.25.so 2b3f62eeb000-2b3f62eec000 rw-p 00110000 00:02 1401 /lib64/libm-2.25.so 2b3f62eec000-2b3f63081000 r-xp 00000000 00:02 1233 /lib64/libc-2.25.so 2b3f63081000-2b3f63280000 ---p 00195000 00:02 1233 /lib64/libc-2.25.so 2b3f63280000-2b3f63284000 r--p 00194000 00:02 1233 /lib64/libc-2.25.so 2b3f63284000-2b3f63286000 rw-p 00198000 00:02 1233 /lib64/libc-2.25.so 2b3f63286000-2b3f6328a000 rw-p 00000000 00:00 0 2b3f6328a000-2b3f632a0000 r-xp 00000000 00:10 2142835216 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libgcc_s.so.1 2b3f632a0000-2b3f6349f000 ---p 00016000 00:10 2142835216 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libgcc_s.so.1 2b3f6349f000-2b3f634a0000 r--p 00015000 00:10 2142835216 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libgcc_s.so.1 2b3f634a0000-2b3f634a1000 rw-p 00016000 00:10 2142835216 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libgcc_s.so.1 2b3f64000000-2b3f64021000 rw-p 00000000 00:00 0 2b3f64021000-2b3f68000000 ---p 00000000 00:00 0 7ffc37219000-7ffc3723a000 rw-p 00000000 00:00 0 [stack] 7ffc37355000-7ffc37357000 r--p 00000000 00:00 0 [vvar] 7ffc37357000-7ffc37359000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] *** Error in `/usr/bin/hydra_pmi_proxy': free(): invalid next size (fast): 0x0000000001c734c0 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x704fb)[0x2b4dc54914fb] /lib64/libc.so.6(+0x76976)[0x2b4dc5497976] /lib64/libc.so.6(+0x7716e)[0x2b4dc549816e] /usr/bin/hydra_pmi_proxy[0x41aa27] /usr/bin/hydra_pmi_proxy[0x418aea] /usr/bin/hydra_pmi_proxy[0x407c0c] /usr/bin/hydra_pmi_proxy[0x420bb0] /usr/bin/hydra_pmi_proxy[0x403e49] /lib64/libc.so.6(__libc_start_main+0xf1)[0x2b4dc54414e1] /usr/bin/hydra_pmi_proxy[0x404cca] ======= Memory map: ======== 00400000-0045c000 r-xp 00000000 00:10 2142949266 /afs/mpa/@sys/system/MPA-8.03/usr/bin/hydra_pmi_proxy 0065c000-0065d000 r--p 0005c000 00:10 2142949266 /afs/mpa/@sys/system/MPA-8.03/usr/bin/hydra_pmi_proxy 0065d000-0065f000 rw-p 0005d000 00:10 2142949266 /afs/mpa/@sys/system/MPA-8.03/usr/bin/hydra_pmi_proxy 0065f000-006ff000 rw-p 00000000 00:00 0 01c6b000-01cad000 rw-p 00000000 00:00 0 [heap] 2b4dc3efa000-2b4dc3f1d000 r-xp 00000000 00:02 4297 /lib64/ld-2.25.so 2b4dc3f1d000-2b4dc3f20000 rw-p 00000000 00:00 0 2b4dc3f3e000-2b4dc3f42000 rw-p 00000000 00:00 0 2b4dc411c000-2b4dc411d000 r--p 00022000 00:02 4297 /lib64/ld-2.25.so 2b4dc411d000-2b4dc411e000 rw-p 00023000 00:02 4297 /lib64/ld-2.25.so 2b4dc411e000-2b4dc411f000 rw-p 00000000 00:00 0 2b4dc411f000-2b4dc4138000 r-xp 00000000 00:02 5136 /lib64/libpthread-2.25.so 2b4dc4138000-2b4dc4337000 ---p 00019000 00:02 5136 /lib64/libpthread-2.25.so 2b4dc4337000-2b4dc4338000 r--p 00018000 00:02 5136 /lib64/libpthread-2.25.so 2b4dc4338000-2b4dc4339000 rw-p 00019000 00:02 5136 /lib64/libpthread-2.25.so 2b4dc4339000-2b4dc433d000 rw-p 00000000 00:00 0 2b4dc433d000-2b4dc4361000 r-xp 00000000 00:02 4295 /lib64/libudev.so.1.6.3 2b4dc4361000-2b4dc4560000 ---p 00024000 00:02 4295 /lib64/libudev.so.1.6.3 2b4dc4560000-2b4dc4561000 r--p 00023000 00:02 4295 /lib64/libudev.so.1.6.3 2b4dc4561000-2b4dc4562000 rw-p 00024000 00:02 4295 /lib64/libudev.so.1.6.3 2b4dc4562000-2b4dc456a000 r-xp 00000000 00:10 2142711818 /afs/mpa/@sys/system/MPA-8.03/usr/X11/lib64/libpciaccess.so.0.11.1 2b4dc456a000-2b4dc4769000 ---p 00008000 00:10 2142711818 /afs/mpa/@sys/system/MPA-8.03/usr/X11/lib64/libpciaccess.so.0.11.1 2b4dc4769000-2b4dc476a000 r--p 00007000 00:10 2142711818 /afs/mpa/@sys/system/MPA-8.03/usr/X11/lib64/libpciaccess.so.0.11.1 2b4dc476a000-2b4dc476b000 rw-p 00008000 00:10 2142711818 /afs/mpa/@sys/system/MPA-8.03/usr/X11/lib64/libpciaccess.so.0.11.1 2b4dc476b000-2b4dc48c5000 r-xp 00000000 00:10 2142836572 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libxml2.so.2.9.4 2b4dc48c5000-2b4dc4ac5000 ---p 0015a000 00:10 2142836572 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libxml2.so.2.9.4 2b4dc4ac5000-2b4dc4acd000 r--p 0015a000 00:10 2142836572 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libxml2.so.2.9.4 2b4dc4acd000-2b4dc4acf000 rw-p 00162000 00:10 2142836572 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libxml2.so.2.9.4 2b4dc4acf000-2b4dc4ad0000 rw-p 00000000 00:00 0 2b4dc4ad0000-2b4dc4ad3000 r-xp 00000000 00:02 5152 /lib64/libdl-2.25.so 2b4dc4ad3000-2b4dc4cd2000 ---p 00003000 00:02 5152 /lib64/libdl-2.25.so 2b4dc4cd2000-2b4dc4cd3000 r--p 00002000 00:02 5152 /lib64/libdl-2.25.so 2b4dc4cd3000-2b4dc4cd4000 rw-p 00003000 00:02 5152 /lib64/libdl-2.25.so 2b4dc4cd4000-2b4dc4ce9000 r-xp 00000000 00:10 2142836590 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libz.so.1.2.6 2b4dc4ce9000-2b4dc4ee8000 ---p 00015000 00:10 2142836590 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libz.so.1.2.6 2b4dc4ee8000-2b4dc4ee9000 r--p 00014000 00:10 2142836590 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libz.so.1.2.6 2b4dc4ee9000-2b4dc4eea000 rw-p 00015000 00:10 2142836590 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libz.so.1.2.6 2b4dc4eea000-2b4dc4f0e000 r-xp 00000000 00:10 2142835680 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/liblzma.so.5.2.3 2b4dc4f0e000-2b4dc510e000 ---p 00024000 00:10 2142835680 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/liblzma.so.5.2.3 2b4dc510e000-2b4dc510f000 r--p 00024000 00:10 2142835680 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/liblzma.so.5.2.3 2b4dc510f000-2b4dc5110000 rw-p 00025000 00:10 2142835680 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/liblzma.so.5.2.3 2b4dc5110000-2b4dc5220000 r-xp 00000000 00:02 4469 /lib64/libm-2.25.so 2b4dc5220000-2b4dc541f000 ---p 00110000 00:02 4469 /lib64/libm-2.25.so 2b4dc541f000-2b4dc5420000 r--p 0010f000 00:02 4469 /lib64/libm-2.25.so 2b4dc5420000-2b4dc5421000 rw-p 00110000 00:02 4469 /lib64/libm-2.25.so 2b4dc5421000-2b4dc55b6000 r-xp 00000000 00:02 4301 /lib64/libc-2.25.so 2b4dc55b6000-2b4dc57b5000 ---p 00195000 00:02 4301 /lib64/libc-2.25.so 2b4dc57b5000-2b4dc57b9000 r--p 00194000 00:02 4301 /lib64/libc-2.25.so 2b4dc57b9000-2b4dc57bb000 rw-p 00198000 00:02 4301 /lib64/libc-2.25.so 2b4dc57bb000-2b4dc57bf000 rw-p 00000000 00:00 0 2b4dc57bf000-2b4dc57c4000 r-xp 00000000 00:02 4321 /lib64/libnss_dns-2.25.so 2b4dc57c4000-2b4dc59c3000 ---p 00005000 00:02 4321 /lib64/libnss_dns-2.25.so 2b4dc59c3000-2b4dc59c4000 r--p 00004000 00:02 4321 /lib64/libnss_dns-2.25.so 2b4dc59c4000-2b4dc59c5000 rw-p 00005000 00:02 4321 /lib64/libnss_dns-2.25.so 2b4dc59c5000-2b4dc59d7000 r-xp 00000000 00:02 4457 /lib64/libresolv-2.25.so 2b4dc59d7000-2b4dc5bd7000 ---p 00012000 00:02 4457 /lib64/libresolv-2.25.so 2b4dc5bd7000-2b4dc5bd8000 r--p 00012000 00:02 4457 /lib64/libresolv-2.25.so 2b4dc5bd8000-2b4dc5bd9000 rw-p 00013000 00:02 4457 /lib64/libresolv-2.25.so 2b4dc5bd9000-2b4dc5bdb000 rw-p 00000000 00:00 0 2b4dc5bdb000-2b4dc5bf1000 r-xp 00000000 00:10 2142835216 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libgcc_s.so.1 2b4dc5bf1000-2b4dc5df0000 ---p 00016000 00:10 2142835216 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libgcc_s.so.1 2b4dc5df0000-2b4dc5df1000 r--p 00015000 00:10 2142835216 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libgcc_s.so.1 2b4dc5df1000-2b4dc5df2000 rw-p 00016000 00:10 2142835216 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libgcc_s.so.1 2b4dc8000000-2b4dc8021000 rw-p 00000000 00:00 0 2b4dc8021000-2b4dcc000000 ---p 00000000 00:00 0 7fff6a554000-7fff6a575000 rw-p 00000000 00:00 0 [stack] 7fff6a58e000-7fff6a590000 r--p 00000000 00:00 0 [vvar] 7fff6a590000-7fff6a592000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] [mpiexec@pascal-3-06] control_cb (/tmp/S/mpich-3.2/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:200): assert (!closed) failed [mpiexec@pascal-3-06] HYDT_dmxu_poll_wait_for_event (/tmp/S/mpich-3.2/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [mpiexec@pascal-3-06] HYD_pmci_wait_for_completion (/tmp/S/mpich-3.2/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec@pascal-3-06] main (/tmp/S/mpich-3.2/src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion Do you have any hint which option & parameters of mpirun I have to choose to achieve my task? Kind regards, Ado Arnolds _______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss