September 2013 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] affinity problems with mpiexec.hydra 3.0.4
by Bill Ryder 21 Sep '13

21 Sep '13

That is cleared up now, Thanks Ken. --- Bill On 21/09/2013 3:11 a.m., Ken Raffenetti wrote: > Bill, > > I have a correction on the interpretation of -bind-to. > >> For two socket machines: >> >> If I use --bind-to socket I expect one rank on one socket, and the >> other on the other socket. > This is correct. > >> If I use --bind-to socket:1 I expect both ranks on the same socket. > This is incorrect. "--bind-to socket:1" is equivalent to "--bind-to socket" > >> If I use --bind-to socket:2 I expect one rank on each socket (ie the >> same as --bind-to socket for a two socket machine) > Also incorrect. This would bind processes to the group of 2 sockets together. > > I hope that is cleared up. Now there does seem to be a bug in the "--bind-to socket" case when ppn=2 on some hardware. I am able to replicate this behavior and will file a report. ppn>2 and things work as expected on the same hardware. > > Ken > > ----- Original Message ----- >> From: "Bill Ryder" <bryder(a)wetafx.co.nz> >> To: discuss(a)mpich.org >> Sent: Wednesday, September 18, 2013 4:10:49 PM >> Subject: Re: [mpich-discuss] affinity problems with mpiexec.hydra 3.0.4 >> >> Hi Ken, >> >> Same result using hwloc-bind (which is a relief because if that >> didn't agree with /proc/pid/status I would have been unpleasantly >> surprised!) >> >> The most curious thing is that if I set socket:3 I get the exact >> binding I want! >> >> Perhaps my interpretation of --ppn 2 bind-to socket is incorrect. >> >> For two socket machines: >> >> If I use --bind-to socket I expect one rank on one socket, and the >> other on the other socket. >> >> If I use --bind-to socket:1 I expect both ranks on the same socket. >> >> If I use --bind-to socket:2 I expect one rank on each socket (ie the >> same as --bind-to socket for a two socket machine) >> >> Please let me know if that's incorrect. >> >> >> >> Here's trying various socket counts - using hwloc and looking at >> /proc/$$/status >> >> >> --bind-to socket - I expect each rank to get it's own socket - but >> that doesn't happen >> >> hydra-3.0.4/mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 >> --launcher=ssh --host=jericho101 --bind-to socket utils/get_affinity >> 2> /dev/null >> jericho101.0 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 >> jericho101.1 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 >> jericho101.1 Affinity from hwloc-bind --get : 0x00ff00ff >> jericho101.0 Affinity from hwloc-bind --get : 0x00ff00ff >> >> socket:1 - both processes bound to the same socket. This is what I >> expect socket:1 to do. >> >> hydra-3.0.4/mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 >> --launcher=ssh --host=jericho101 --bind-to socket:1 >> utils/get_affinity >> jericho101.0 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 >> jericho101.1 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 >> jericho101.0 Affinity from hwloc-bind --get : 0x00ff00ff >> jericho101.1 Affinity from hwloc-bind --get : 0x00ff00ff >> >> socket:2 - I end up with the same binding as socket and socket:1 >> >> hydra-3.0.4/mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 >> --launcher=ssh --host=jericho101 --bind-to socket:2 >> utils/get_affinity >> jericho101.0 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 >> jericho101.1 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 >> jericho101.0 Affinity from hwloc-bind --get : 0x00ff00ff >> jericho101.1 Affinity from hwloc-bind --get : 0x00ff00ff >> >> socket:3 - this gets strange - this is the affinity I want! >> >> hydra-3.0.4/mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 >> --launcher=ssh --host=jericho101 --bind-to socket:3 >> utils/get_affinity >> jericho101.0 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 >> jericho101.1 Affinity from /proc/$$/status : ff00ff00 8-15,24-31 >> jericho101.0 Affinity from hwloc-bind --get : 0x00ff00ff >> jericho101.1 Affinity from hwloc-bind --get : 0xff00ff00 >> >> I was originally was asked to look at this because of bad runtimes on >> faster hardware - fixing the affinity fixed >> the performance problem. For the test case I was using binding each >> process to a socket gave me 8% better performance so it's >> definitely worth working on. >> >> Let me know what I can do to help - I don't mind gdbing into mpiexec >> or something or shoving prints into the code. >> >> Also a probably silly question - Does hydra_pmi_proxy figure out >> and set the affinity? That seems the logical place. >> >> I should also note that the other machine where --bind-to socket, >> socket:1 does the right thing seems to do the wrong thing at >> socket:2 >> >> Of course this all assumes I'm interpreting that socket option >> correctly. >> >> >> hydra-3.0.4/mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 >> --launcher=ssh --host=abrams211a --bind-to socket utils/get_affinity >> abrams211a.0 Affinity from /proc/$$/status : 00555555 >> 0,2,4,6,8,10,12,14,16,18,20,22 >> abrams211a.1 Affinity from /proc/$$/status : 00aaaaaa >> 1,3,5,7,9,11,13,15,17,19,21,23 >> abrams211a.0 Affinity from hwloc-bind --get : 0x00555555 >> abrams211a.1 Affinity from hwloc-bind --get : 0x00aaaaaa >> >> mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 --launcher=ssh >> --host=abrams211a --bind-to socket:1 utils/get_affinity >> abrams211a.0 Affinity from /proc/$$/status : 00555555 >> 0,2,4,6,8,10,12,14,16,18,20,22 >> abrams211a.1 Affinity from /proc/$$/status : 00aaaaaa >> 1,3,5,7,9,11,13,15,17,19,21,23 >> abrams211a.0 Affinity from hwloc-bind --get : 0x00555555 >> abrams211a.1 Affinity from hwloc-bind --get : 0x00aaaaaa >> >> >> hydra-3.0.4/mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 >> --launcher=ssh --host=abrams211a --bind-to socket:2 >> utils/get_affinity >> abrams211a.0 Affinity from /proc/$$/status : 00555555 >> 0,2,4,6,8,10,12,14,16,18,20,22 >> abrams211a.1 Affinity from /proc/$$/status : 00555555 >> 0,2,4,6,8,10,12,14,16,18,20,22 >> abrams211a.0 Affinity from hwloc-bind --get : 0x00555555 >> abrams211a.1 Affinity from hwloc-bind --get : 0x00555555 >> >> >> >> On 09/18/2013 01:50 AM, Ken Raffenetti wrote: >>> Hi Bill, >>> >>> I'm not seeing any obvious problems. However, in my own testing, I >>> found a situation where the grep method you used was unreliable. >>> Can you try using "hwloc-bind --get" instead and see if the >>> results differ? >>> >>> Ken >>> >>> ----- Original Message ----- >>>> From: "Bill Ryder" <bryder(a)wetafx.co.nz> >>>> To: discuss(a)mpich.org >>>> Sent: Monday, September 16, 2013 6:04:45 PM >>>> Subject: [mpich-discuss] affinity problems with mpiexec.hydra >>>> 3.0.4 >>>> >>>> Greetings all, >>>> >>>> I'm trying set affinity for mybrid MPI/OpenMP tasks. >>>> >>>> I want to run two processes on a host, and give one socket to one >>>> rank, and the other socket to the other rank. >>>> >>>> I have two types of hardware - one works perfectly - the other >>>> doesn't. >>>> >>>> I first saw the problem using mpiexec.hydra with slurm but I've >>>> moved >>>> to using ssh to remove some possible variables >>>> >>>> >>>> I have a trivial script which just greps out /proc/$$/status for >>>> the >>>> Cpus_allowed mask and Cpus_allowed_list >>>> >>>> It's just: echo "`hostname` $PMI_RANK `grep Cpus_allowed >>>> /proc/$$/status`" >>>> >>>> >>>> On the machine that is doing what I want I get: >>>> >>>> mpiexec.hydra -ppn 2 -hosts abrams201a --bind-to socket -launcher >>>> ssh >>>> ./get_mapping >>>> abrams201a 0 Cpus_allowed: 00555555 >>>> Cpus_allowed_list: 0,2,4,6,8,10,12,14,16,18,20,22 >>>> abrams201a 1 Cpus_allowed: 00aaaaaa >>>> Cpus_allowed_list: 1,3,5,7,9,11,13,15,17,19,21,23 >>>> >>>> rank 0 gets one socket, rank 1 gets the other socket. This is what >>>> I >>>> want >>>> >>>> But on my other machine with a different topology I get this: >>>> >>>> mpiexec.hydra -ppn 2 -hosts jericho101 --bind-to socket -launcher >>>> ssh ./get_mapping >>>> >>>> jericho101 0 Cpus_allowed: 00000000,00ff00ff >>>> Cpus_allowed_list: 0-7,16-23 >>>> jericho101 1 Cpus_allowed: 00000000,00ff00ff >>>> Cpus_allowed_list: 0-7,16-23 >>>> >>>> So each rank is trying to use the same socket. >>>> >>>> Similarly if I try to bind to a numanode >>>> >>>> mpiexec.hydra -ppn 2 -hosts jericho101 --bind-to numa -launcher >>>> ssh >>>> ./get_mapping >>>> jericho101 1 Cpus_allowed: 00000000,00ff00ff >>>> Cpus_allowed_list: 0-7,16-23 >>>> jericho101 0 Cpus_allowed: 00000000,00ff00ff >>>> Cpus_allowed_list: 0-7,16-23 >>>> >>>> >>>> Or even if I send numa:2 >>>> >>>> mpiexec.hydra -ppn 2 -hosts jericho101 --bind-to numa:2 -launcher >>>> ssh ./get_mapping >>>> >>>> jericho101 0 Cpus_allowed: 00000000,00ff00ff >>>> Cpus_allowed_list: 0-7,16-23 >>>> jericho101 1 Cpus_allowed: 00000000,00ff00ff >>>> Cpus_allowed_list: 0-7,16-23 >>>> >>>> So once again instead of handing a numa node to each process - >>>> it's >>>> handing the same node to both. >>>> >>>> >>>> How would I start debugging this? >>>> >>>> Or am I missing something really obvious >>>> >>>> >>>> >>>> Thanks! >>>> --------- >>>> >>>> Bill Ryder >>>> Weta Digital >>>> >>>> >>>> >>>> >>>> >>>> A bit more data: >>>> >>>> mpiexec.hydra --info >>>> HYDRA build details: >>>> Version: 3.0.4 >>>> Release Date: Wed Apr 24 10:08:10 >>>> CDT >>>> 2013 >>>> CC: cc >>>> CXX: >>>> F77: >>>> F90: >>>> Configure options: '--prefix=/tech/apps/mpich/hydra' >>>> Process Manager: pmi >>>> Launchers available: ssh rsh fork slurm >>>> ll >>>> lsf sge manual persist >>>> Topology libraries available: hwloc >>>> Resource management kernels available: user slurm ll lsf >>>> sge >>>> pbs cobalt >>>> Checkpointing libraries available: >>>> Demux engines available: poll select >>>> >>>> >>>> >>>> I have hwloc 1.3.1 installed locally on each machine >>>> >>>> abrams201a looks like: >>>> >>>> Machine (48GB) >>>> NUMANode L#0 (P#1 24GB) + Socket L#0 + L3 L#0 (12MB) >>>> L2 L#0 (256KB) + L1 L#0 (32KB) + Core L#0 >>>> PU L#0 (P#0) >>>> PU L#1 (P#12) >>>> L2 L#1 (256KB) + L1 L#1 (32KB) + Core L#1 >>>> PU L#2 (P#2) >>>> PU L#3 (P#14) >>>> L2 L#2 (256KB) + L1 L#2 (32KB) + Core L#2 >>>> PU L#4 (P#4) >>>> PU L#5 (P#16) >>>> L2 L#3 (256KB) + L1 L#3 (32KB) + Core L#3 >>>> PU L#6 (P#6) >>>> PU L#7 (P#18) >>>> L2 L#4 (256KB) + L1 L#4 (32KB) + Core L#4 >>>> PU L#8 (P#8) >>>> PU L#9 (P#20) >>>> L2 L#5 (256KB) + L1 L#5 (32KB) + Core L#5 >>>> PU L#10 (P#10) >>>> PU L#11 (P#22) >>>> NUMANode L#1 (P#0 24GB) + Socket L#1 + L3 L#1 (12MB) >>>> L2 L#6 (256KB) + L1 L#6 (32KB) + Core L#6 >>>> PU L#12 (P#1) >>>> PU L#13 (P#13) >>>> L2 L#7 (256KB) + L1 L#7 (32KB) + Core L#7 >>>> PU L#14 (P#3) >>>> PU L#15 (P#15) >>>> L2 L#8 (256KB) + L1 L#8 (32KB) + Core L#8 >>>> PU L#16 (P#5) >>>> PU L#17 (P#17) >>>> L2 L#9 (256KB) + L1 L#9 (32KB) + Core L#9 >>>> PU L#18 (P#7) >>>> PU L#19 (P#19) >>>> L2 L#10 (256KB) + L1 L#10 (32KB) + Core L#10 >>>> PU L#20 (P#9) >>>> PU L#21 (P#21) >>>> L2 L#11 (256KB) + L1 L#11 (32KB) + Core L#11 >>>> PU L#22 (P#11) >>>> PU L#23 (P#23) >>>> HostBridge L#0 >>>> PCIBridge >>>> PCI 8086:10e7 >>>> Net L#0 "eth0" >>>> PCI 8086:10e7 >>>> Net L#1 "eth1" >>>> PCIBridge >>>> PCI 15b3:6746 >>>> Net L#2 "eth2" >>>> OpenFabrics L#3 "mlx4_0" >>>> PCI 15b3:6746 >>>> PCI 15b3:6746 >>>> PCI 15b3:6746 >>>> PCIBridge >>>> PCI 102b:0533 >>>> PCI 8086:3a20 >>>> Block L#4 "sda" >>>> >>>> >>>> And jericho101 looks like: >>>> >>>> achine (96GB) >>>> NUMANode L#0 (P#0 48GB) >>>> Socket L#0 + L3 L#0 (20MB) >>>> L2 L#0 (256KB) + L1 L#0 (32KB) + Core L#0 >>>> PU L#0 (P#0) >>>> PU L#1 (P#16) >>>> L2 L#1 (256KB) + L1 L#1 (32KB) + Core L#1 >>>> PU L#2 (P#1) >>>> PU L#3 (P#17) >>>> L2 L#2 (256KB) + L1 L#2 (32KB) + Core L#2 >>>> PU L#4 (P#2) >>>> PU L#5 (P#18) >>>> L2 L#3 (256KB) + L1 L#3 (32KB) + Core L#3 >>>> PU L#6 (P#3) >>>> PU L#7 (P#19) >>>> L2 L#4 (256KB) + L1 L#4 (32KB) + Core L#4 >>>> PU L#8 (P#4) >>>> PU L#9 (P#20) >>>> L2 L#5 (256KB) + L1 L#5 (32KB) + Core L#5 >>>> PU L#10 (P#5) >>>> PU L#11 (P#21) >>>> L2 L#6 (256KB) + L1 L#6 (32KB) + Core L#6 >>>> PU L#12 (P#6) >>>> PU L#13 (P#22) >>>> L2 L#7 (256KB) + L1 L#7 (32KB) + Core L#7 >>>> PU L#14 (P#7) >>>> PU L#15 (P#23) >>>> HostBridge L#0 >>>> PCIBridge >>>> PCI 14e4:168e >>>> Net L#0 "eth0" >>>> PCI 14e4:168e >>>> Net L#1 "eth1" >>>> PCI 14e4:168e >>>> Net L#2 "eth2" >>>> PCI 14e4:168e >>>> Net L#3 "eth3" >>>> PCI 14e4:168e >>>> Net L#4 "eth4" >>>> PCI 14e4:168e >>>> Net L#5 "eth5" >>>> PCI 14e4:168e >>>> Net L#6 "eth6" >>>> PCI 14e4:168e >>>> Net L#7 "eth7" >>>> PCIBridge >>>> PCI 103c:323b >>>> Block L#8 "sda" >>>> PCIBridge >>>> PCI 102b:0533 >>>> NUMANode L#1 (P#1 48GB) + Socket L#1 + L3 L#1 (20MB) >>>> L2 L#8 (256KB) + L1 L#8 (32KB) + Core L#8 >>>> PU L#16 (P#8) >>>> PU L#17 (P#24) >>>> L2 L#9 (256KB) + L1 L#9 (32KB) + Core L#9 >>>> PU L#18 (P#9) >>>> PU L#19 (P#25) >>>> L2 L#10 (256KB) + L1 L#10 (32KB) + Core L#10 >>>> PU L#20 (P#10) >>>> PU L#21 (P#26) >>>> L2 L#11 (256KB) + L1 L#11 (32KB) + Core L#11 >>>> PU L#22 (P#11) >>>> PU L#23 (P#27) >>>> L2 L#12 (256KB) + L1 L#12 (32KB) + Core L#12 >>>> PU L#24 (P#12) >>>> PU L#25 (P#28) >>>> L2 L#13 (256KB) + L1 L#13 (32KB) + Core L#13 >>>> PU L#26 (P#13) >>>> PU L#27 (P#29) >>>> L2 L#14 (256KB) + L1 L#14 (32KB) + Core L#14 >>>> PU L#28 (P#14) >>>> PU L#29 (P#30) >>>> L2 L#15 (256KB) + L1 L#15 (32KB) + Core L#15 >>>> PU L#30 (P#15) >>>> PU L#31 (P#31) >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Order of completion in MPI_Waitany
by Matthieu Dorier 20 Sep '13

20 Sep '13

Thanks Thomas! ----- Mail original ----- > De: "Thomas Ropars" <thomas.ropars(a)epfl.ch> > À: discuss(a)mpich.org > Envoyé: Vendredi 20 Septembre 2013 09:57:27 > Objet: Re: [mpich-discuss] Order of completion in MPI_Waitany > Hi Matthieu, > MPI does not provide any guarantee on the completion order of the > reception requests in this case. More generally, regarding waitAny, > the MPI-3.0 specification says that "If more than one operation is > enabled and can terminate, one is arbitrarily > chosen". > Thomas > On 09/19/2013 04:31 PM, Matthieu Dorier wrote: > > Hi, > > > I would like a precision on the semantics of MPI_Waitany in this > > situation: > > > - Process 1 does 2 MPI_Irecv with different tags A and B (the order > > doesn't matter); it then waits on the two requests with > > MPI_Waitany. > > > - In the mean time, process 2 does an MPI_Send with tag A, then > > MPI_Send with tag B. > > > Can I ensure that MPI_Waitany will always give me the index of the > > message corresponding to the first MPI_Send, i.e. the one with the > > tag A, even in situations where process 1 called MPI_Irecv on a tag > > B first? > > > Thanks > > > Matthieu Dorier > > > PhD student at ENS Cachan Brittany and IRISA > > > http://people.irisa.fr/Matthieu.Dorier > > > _______________________________________________ > > > discuss mailing list discuss(a)mpich.org To manage subscription > > options > > or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] affinity problems with mpiexec.hydra 3.0.4
by Ken Raffenetti 20 Sep '13

20 Sep '13

Bill, I have a correction on the interpretation of -bind-to. > For two socket machines: > > If I use --bind-to socket I expect one rank on one socket, and the > other on the other socket. This is correct. > > If I use --bind-to socket:1 I expect both ranks on the same socket. This is incorrect. "--bind-to socket:1" is equivalent to "--bind-to socket" > > If I use --bind-to socket:2 I expect one rank on each socket (ie the > same as --bind-to socket for a two socket machine) Also incorrect. This would bind processes to the group of 2 sockets together. I hope that is cleared up. Now there does seem to be a bug in the "--bind-to socket" case when ppn=2 on some hardware. I am able to replicate this behavior and will file a report. ppn>2 and things work as expected on the same hardware. Ken ----- Original Message ----- > From: "Bill Ryder" <bryder(a)wetafx.co.nz> > To: discuss(a)mpich.org > Sent: Wednesday, September 18, 2013 4:10:49 PM > Subject: Re: [mpich-discuss] affinity problems with mpiexec.hydra 3.0.4 > > Hi Ken, > > Same result using hwloc-bind (which is a relief because if that > didn't agree with /proc/pid/status I would have been unpleasantly > surprised!) > > The most curious thing is that if I set socket:3 I get the exact > binding I want! > > Perhaps my interpretation of --ppn 2 bind-to socket is incorrect. > > For two socket machines: > > If I use --bind-to socket I expect one rank on one socket, and the > other on the other socket. > > If I use --bind-to socket:1 I expect both ranks on the same socket. > > If I use --bind-to socket:2 I expect one rank on each socket (ie the > same as --bind-to socket for a two socket machine) > > Please let me know if that's incorrect. > > > > Here's trying various socket counts - using hwloc and looking at > /proc/$$/status > > > --bind-to socket - I expect each rank to get it's own socket - but > that doesn't happen > > hydra-3.0.4/mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 > --launcher=ssh --host=jericho101 --bind-to socket utils/get_affinity > 2> /dev/null > jericho101.0 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 > jericho101.1 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 > jericho101.1 Affinity from hwloc-bind --get : 0x00ff00ff > jericho101.0 Affinity from hwloc-bind --get : 0x00ff00ff > > socket:1 - both processes bound to the same socket. This is what I > expect socket:1 to do. > > hydra-3.0.4/mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 > --launcher=ssh --host=jericho101 --bind-to socket:1 > utils/get_affinity > jericho101.0 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 > jericho101.1 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 > jericho101.0 Affinity from hwloc-bind --get : 0x00ff00ff > jericho101.1 Affinity from hwloc-bind --get : 0x00ff00ff > > socket:2 - I end up with the same binding as socket and socket:1 > > hydra-3.0.4/mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 > --launcher=ssh --host=jericho101 --bind-to socket:2 > utils/get_affinity > jericho101.0 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 > jericho101.1 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 > jericho101.0 Affinity from hwloc-bind --get : 0x00ff00ff > jericho101.1 Affinity from hwloc-bind --get : 0x00ff00ff > > socket:3 - this gets strange - this is the affinity I want! > > hydra-3.0.4/mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 > --launcher=ssh --host=jericho101 --bind-to socket:3 > utils/get_affinity > jericho101.0 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 > jericho101.1 Affinity from /proc/$$/status : ff00ff00 8-15,24-31 > jericho101.0 Affinity from hwloc-bind --get : 0x00ff00ff > jericho101.1 Affinity from hwloc-bind --get : 0xff00ff00 > > I was originally was asked to look at this because of bad runtimes on > faster hardware - fixing the affinity fixed > the performance problem. For the test case I was using binding each > process to a socket gave me 8% better performance so it's > definitely worth working on. > > Let me know what I can do to help - I don't mind gdbing into mpiexec > or something or shoving prints into the code. > > Also a probably silly question - Does hydra_pmi_proxy figure out > and set the affinity? That seems the logical place. > > I should also note that the other machine where --bind-to socket, > socket:1 does the right thing seems to do the wrong thing at > socket:2 > > Of course this all assumes I'm interpreting that socket option > correctly. > > > hydra-3.0.4/mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 > --launcher=ssh --host=abrams211a --bind-to socket utils/get_affinity > abrams211a.0 Affinity from /proc/$$/status : 00555555 > 0,2,4,6,8,10,12,14,16,18,20,22 > abrams211a.1 Affinity from /proc/$$/status : 00aaaaaa > 1,3,5,7,9,11,13,15,17,19,21,23 > abrams211a.0 Affinity from hwloc-bind --get : 0x00555555 > abrams211a.1 Affinity from hwloc-bind --get : 0x00aaaaaa > > mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 --launcher=ssh > --host=abrams211a --bind-to socket:1 utils/get_affinity > abrams211a.0 Affinity from /proc/$$/status : 00555555 > 0,2,4,6,8,10,12,14,16,18,20,22 > abrams211a.1 Affinity from /proc/$$/status : 00aaaaaa > 1,3,5,7,9,11,13,15,17,19,21,23 > abrams211a.0 Affinity from hwloc-bind --get : 0x00555555 > abrams211a.1 Affinity from hwloc-bind --get : 0x00aaaaaa > > > hydra-3.0.4/mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 > --launcher=ssh --host=abrams211a --bind-to socket:2 > utils/get_affinity > abrams211a.0 Affinity from /proc/$$/status : 00555555 > 0,2,4,6,8,10,12,14,16,18,20,22 > abrams211a.1 Affinity from /proc/$$/status : 00555555 > 0,2,4,6,8,10,12,14,16,18,20,22 > abrams211a.0 Affinity from hwloc-bind --get : 0x00555555 > abrams211a.1 Affinity from hwloc-bind --get : 0x00555555 > > > > On 09/18/2013 01:50 AM, Ken Raffenetti wrote: > > Hi Bill, > > > > I'm not seeing any obvious problems. However, in my own testing, I > > found a situation where the grep method you used was unreliable. > > Can you try using "hwloc-bind --get" instead and see if the > > results differ? > > > > Ken > > > > ----- Original Message ----- > >> From: "Bill Ryder" <bryder(a)wetafx.co.nz> > >> To: discuss(a)mpich.org > >> Sent: Monday, September 16, 2013 6:04:45 PM > >> Subject: [mpich-discuss] affinity problems with mpiexec.hydra > >> 3.0.4 > >> > >> Greetings all, > >> > >> I'm trying set affinity for mybrid MPI/OpenMP tasks. > >> > >> I want to run two processes on a host, and give one socket to one > >> rank, and the other socket to the other rank. > >> > >> I have two types of hardware - one works perfectly - the other > >> doesn't. > >> > >> I first saw the problem using mpiexec.hydra with slurm but I've > >> moved > >> to using ssh to remove some possible variables > >> > >> > >> I have a trivial script which just greps out /proc/$$/status for > >> the > >> Cpus_allowed mask and Cpus_allowed_list > >> > >> It's just: echo "`hostname` $PMI_RANK `grep Cpus_allowed > >> /proc/$$/status`" > >> > >> > >> On the machine that is doing what I want I get: > >> > >> mpiexec.hydra -ppn 2 -hosts abrams201a --bind-to socket -launcher > >> ssh > >> ./get_mapping > >> abrams201a 0 Cpus_allowed: 00555555 > >> Cpus_allowed_list: 0,2,4,6,8,10,12,14,16,18,20,22 > >> abrams201a 1 Cpus_allowed: 00aaaaaa > >> Cpus_allowed_list: 1,3,5,7,9,11,13,15,17,19,21,23 > >> > >> rank 0 gets one socket, rank 1 gets the other socket. This is what > >> I > >> want > >> > >> But on my other machine with a different topology I get this: > >> > >> mpiexec.hydra -ppn 2 -hosts jericho101 --bind-to socket -launcher > >> ssh ./get_mapping > >> > >> jericho101 0 Cpus_allowed: 00000000,00ff00ff > >> Cpus_allowed_list: 0-7,16-23 > >> jericho101 1 Cpus_allowed: 00000000,00ff00ff > >> Cpus_allowed_list: 0-7,16-23 > >> > >> So each rank is trying to use the same socket. > >> > >> Similarly if I try to bind to a numanode > >> > >> mpiexec.hydra -ppn 2 -hosts jericho101 --bind-to numa -launcher > >> ssh > >> ./get_mapping > >> jericho101 1 Cpus_allowed: 00000000,00ff00ff > >> Cpus_allowed_list: 0-7,16-23 > >> jericho101 0 Cpus_allowed: 00000000,00ff00ff > >> Cpus_allowed_list: 0-7,16-23 > >> > >> > >> Or even if I send numa:2 > >> > >> mpiexec.hydra -ppn 2 -hosts jericho101 --bind-to numa:2 -launcher > >> ssh ./get_mapping > >> > >> jericho101 0 Cpus_allowed: 00000000,00ff00ff > >> Cpus_allowed_list: 0-7,16-23 > >> jericho101 1 Cpus_allowed: 00000000,00ff00ff > >> Cpus_allowed_list: 0-7,16-23 > >> > >> So once again instead of handing a numa node to each process - > >> it's > >> handing the same node to both. > >> > >> > >> How would I start debugging this? > >> > >> Or am I missing something really obvious > >> > >> > >> > >> Thanks! > >> --------- > >> > >> Bill Ryder > >> Weta Digital > >> > >> > >> > >> > >> > >> A bit more data: > >> > >> mpiexec.hydra --info > >> HYDRA build details: > >> Version: 3.0.4 > >> Release Date: Wed Apr 24 10:08:10 > >> CDT > >> 2013 > >> CC: cc > >> CXX: > >> F77: > >> F90: > >> Configure options: '--prefix=/tech/apps/mpich/hydra' > >> Process Manager: pmi > >> Launchers available: ssh rsh fork slurm > >> ll > >> lsf sge manual persist > >> Topology libraries available: hwloc > >> Resource management kernels available: user slurm ll lsf > >> sge > >> pbs cobalt > >> Checkpointing libraries available: > >> Demux engines available: poll select > >> > >> > >> > >> I have hwloc 1.3.1 installed locally on each machine > >> > >> abrams201a looks like: > >> > >> Machine (48GB) > >> NUMANode L#0 (P#1 24GB) + Socket L#0 + L3 L#0 (12MB) > >> L2 L#0 (256KB) + L1 L#0 (32KB) + Core L#0 > >> PU L#0 (P#0) > >> PU L#1 (P#12) > >> L2 L#1 (256KB) + L1 L#1 (32KB) + Core L#1 > >> PU L#2 (P#2) > >> PU L#3 (P#14) > >> L2 L#2 (256KB) + L1 L#2 (32KB) + Core L#2 > >> PU L#4 (P#4) > >> PU L#5 (P#16) > >> L2 L#3 (256KB) + L1 L#3 (32KB) + Core L#3 > >> PU L#6 (P#6) > >> PU L#7 (P#18) > >> L2 L#4 (256KB) + L1 L#4 (32KB) + Core L#4 > >> PU L#8 (P#8) > >> PU L#9 (P#20) > >> L2 L#5 (256KB) + L1 L#5 (32KB) + Core L#5 > >> PU L#10 (P#10) > >> PU L#11 (P#22) > >> NUMANode L#1 (P#0 24GB) + Socket L#1 + L3 L#1 (12MB) > >> L2 L#6 (256KB) + L1 L#6 (32KB) + Core L#6 > >> PU L#12 (P#1) > >> PU L#13 (P#13) > >> L2 L#7 (256KB) + L1 L#7 (32KB) + Core L#7 > >> PU L#14 (P#3) > >> PU L#15 (P#15) > >> L2 L#8 (256KB) + L1 L#8 (32KB) + Core L#8 > >> PU L#16 (P#5) > >> PU L#17 (P#17) > >> L2 L#9 (256KB) + L1 L#9 (32KB) + Core L#9 > >> PU L#18 (P#7) > >> PU L#19 (P#19) > >> L2 L#10 (256KB) + L1 L#10 (32KB) + Core L#10 > >> PU L#20 (P#9) > >> PU L#21 (P#21) > >> L2 L#11 (256KB) + L1 L#11 (32KB) + Core L#11 > >> PU L#22 (P#11) > >> PU L#23 (P#23) > >> HostBridge L#0 > >> PCIBridge > >> PCI 8086:10e7 > >> Net L#0 "eth0" > >> PCI 8086:10e7 > >> Net L#1 "eth1" > >> PCIBridge > >> PCI 15b3:6746 > >> Net L#2 "eth2" > >> OpenFabrics L#3 "mlx4_0" > >> PCI 15b3:6746 > >> PCI 15b3:6746 > >> PCI 15b3:6746 > >> PCIBridge > >> PCI 102b:0533 > >> PCI 8086:3a20 > >> Block L#4 "sda" > >> > >> > >> And jericho101 looks like: > >> > >> achine (96GB) > >> NUMANode L#0 (P#0 48GB) > >> Socket L#0 + L3 L#0 (20MB) > >> L2 L#0 (256KB) + L1 L#0 (32KB) + Core L#0 > >> PU L#0 (P#0) > >> PU L#1 (P#16) > >> L2 L#1 (256KB) + L1 L#1 (32KB) + Core L#1 > >> PU L#2 (P#1) > >> PU L#3 (P#17) > >> L2 L#2 (256KB) + L1 L#2 (32KB) + Core L#2 > >> PU L#4 (P#2) > >> PU L#5 (P#18) > >> L2 L#3 (256KB) + L1 L#3 (32KB) + Core L#3 > >> PU L#6 (P#3) > >> PU L#7 (P#19) > >> L2 L#4 (256KB) + L1 L#4 (32KB) + Core L#4 > >> PU L#8 (P#4) > >> PU L#9 (P#20) > >> L2 L#5 (256KB) + L1 L#5 (32KB) + Core L#5 > >> PU L#10 (P#5) > >> PU L#11 (P#21) > >> L2 L#6 (256KB) + L1 L#6 (32KB) + Core L#6 > >> PU L#12 (P#6) > >> PU L#13 (P#22) > >> L2 L#7 (256KB) + L1 L#7 (32KB) + Core L#7 > >> PU L#14 (P#7) > >> PU L#15 (P#23) > >> HostBridge L#0 > >> PCIBridge > >> PCI 14e4:168e > >> Net L#0 "eth0" > >> PCI 14e4:168e > >> Net L#1 "eth1" > >> PCI 14e4:168e > >> Net L#2 "eth2" > >> PCI 14e4:168e > >> Net L#3 "eth3" > >> PCI 14e4:168e > >> Net L#4 "eth4" > >> PCI 14e4:168e > >> Net L#5 "eth5" > >> PCI 14e4:168e > >> Net L#6 "eth6" > >> PCI 14e4:168e > >> Net L#7 "eth7" > >> PCIBridge > >> PCI 103c:323b > >> Block L#8 "sda" > >> PCIBridge > >> PCI 102b:0533 > >> NUMANode L#1 (P#1 48GB) + Socket L#1 + L3 L#1 (20MB) > >> L2 L#8 (256KB) + L1 L#8 (32KB) + Core L#8 > >> PU L#16 (P#8) > >> PU L#17 (P#24) > >> L2 L#9 (256KB) + L1 L#9 (32KB) + Core L#9 > >> PU L#18 (P#9) > >> PU L#19 (P#25) > >> L2 L#10 (256KB) + L1 L#10 (32KB) + Core L#10 > >> PU L#20 (P#10) > >> PU L#21 (P#26) > >> L2 L#11 (256KB) + L1 L#11 (32KB) + Core L#11 > >> PU L#22 (P#11) > >> PU L#23 (P#27) > >> L2 L#12 (256KB) + L1 L#12 (32KB) + Core L#12 > >> PU L#24 (P#12) > >> PU L#25 (P#28) > >> L2 L#13 (256KB) + L1 L#13 (32KB) + Core L#13 > >> PU L#26 (P#13) > >> PU L#27 (P#29) > >> L2 L#14 (256KB) + L1 L#14 (32KB) + Core L#14 > >> PU L#28 (P#14) > >> PU L#29 (P#30) > >> L2 L#15 (256KB) + L1 L#15 (32KB) + Core L#15 > >> PU L#30 (P#15) > >> PU L#31 (P#31) > >> > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > >> > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Order of completion in MPI_Waitany
by Thomas Ropars 20 Sep '13

20 Sep '13

Hi Matthieu, MPI does not provide any guarantee on the completion order of the reception requests in this case. More generally, regarding waitAny, the MPI-3.0 specification says that "If more than one operation is enabled and can terminate, one is arbitrarily chosen". Thomas On 09/19/2013 04:31 PM, Matthieu Dorier wrote: > Hi, > > I would like a precision on the semantics of MPI_Waitany in this > situation: > > - Process 1 does 2 MPI_Irecv with different tags A and B (the order > doesn't matter); it then waits on the two requests with MPI_Waitany. > - In the mean time, process 2 does an MPI_Send with tag A, then > MPI_Send with tag B. > Can I ensure that MPI_Waitany will always give me the index of the > message corresponding to the first MPI_Send, i.e. the one with the tag > A, even in situations where process 1 called MPI_Irecv on a tag B first? > > Thanks > > Matthieu Dorier > PhD student at ENS Cachan Brittany and IRISA > http://people.irisa.fr/Matthieu.Dorier > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

segfault in MPIR_Type_get_envelope_impl
by Ryan Crocker 19 Sep '13

19 Sep '13

Hi all, So i have a program that makes it through 1 iteration then crashes in the second at an call MPI_FILE_SET_VIEW. The GDB output give: Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: 13 at address: 0x0000000000000000 0x0000000100ac4b19 in MPIR_Type_get_envelope_impl () (gdb) backtrace #0 0x0000000100ac4b19 in MPIR_Type_get_envelope_impl () #1 0x0000000100ac4d6a in PMPI_Type_get_envelope () the subroutine snippet is: integer :: fileview_hexa integer :: iunit,ierr,size,ibuffer,i,j,k,count_wedge_,count_hexa_ character(len=80) :: buffer character(len=str_short) :: name character(len=str_medium) :: file real(WP),dimension(imino_:imaxo_,jmino_:jmaxo_,kmino_:kmaxo_) :: scalar integer, dimension(MPI_STATUS_SIZE) :: status integer(kind=MPI_OFFSET_KIND) :: disp logical :: file_is_there call MPI_FILE_SET_VIEW(iunit,disp,MPI_REAL_SP,fileview_hexa,"native",mpi_info,ierr) it's from a piece of code that worked fine until i changed some things in the output now i get this segfault. Ryan University of Vermont

1 0

Re: [mpich-discuss] affinity problems with mpiexec.hydra 3.0.4
by Ken Raffenetti 19 Sep '13

19 Sep '13

Bill, Your understanding is correct. This is looking like a bug. I will make a report and add you to the CC list once I confirm. Thanks for pointing it out. Ken ----- Original Message ----- > From: "Bill Ryder" <bryder(a)wetafx.co.nz> > To: discuss(a)mpich.org > Sent: Wednesday, September 18, 2013 4:10:49 PM > Subject: Re: [mpich-discuss] affinity problems with mpiexec.hydra 3.0.4 > > Hi Ken, > > Same result using hwloc-bind (which is a relief because if that > didn't agree with /proc/pid/status I would have been unpleasantly > surprised!) > > The most curious thing is that if I set socket:3 I get the exact > binding I want! > > Perhaps my interpretation of --ppn 2 bind-to socket is incorrect. > > For two socket machines: > > If I use --bind-to socket I expect one rank on one socket, and the > other on the other socket. > > If I use --bind-to socket:1 I expect both ranks on the same socket. > > If I use --bind-to socket:2 I expect one rank on each socket (ie the > same as --bind-to socket for a two socket machine) > > Please let me know if that's incorrect. > > > > Here's trying various socket counts - using hwloc and looking at > /proc/$$/status > > > --bind-to socket - I expect each rank to get it's own socket - but > that doesn't happen > > hydra-3.0.4/mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 > --launcher=ssh --host=jericho101 --bind-to socket utils/get_affinity > 2> /dev/null > jericho101.0 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 > jericho101.1 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 > jericho101.1 Affinity from hwloc-bind --get : 0x00ff00ff > jericho101.0 Affinity from hwloc-bind --get : 0x00ff00ff > > socket:1 - both processes bound to the same socket. This is what I > expect socket:1 to do. > > hydra-3.0.4/mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 > --launcher=ssh --host=jericho101 --bind-to socket:1 > utils/get_affinity > jericho101.0 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 > jericho101.1 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 > jericho101.0 Affinity from hwloc-bind --get : 0x00ff00ff > jericho101.1 Affinity from hwloc-bind --get : 0x00ff00ff > > socket:2 - I end up with the same binding as socket and socket:1 > > hydra-3.0.4/mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 > --launcher=ssh --host=jericho101 --bind-to socket:2 > utils/get_affinity > jericho101.0 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 > jericho101.1 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 > jericho101.0 Affinity from hwloc-bind --get : 0x00ff00ff > jericho101.1 Affinity from hwloc-bind --get : 0x00ff00ff > > socket:3 - this gets strange - this is the affinity I want! > > hydra-3.0.4/mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 > --launcher=ssh --host=jericho101 --bind-to socket:3 > utils/get_affinity > jericho101.0 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 > jericho101.1 Affinity from /proc/$$/status : ff00ff00 8-15,24-31 > jericho101.0 Affinity from hwloc-bind --get : 0x00ff00ff > jericho101.1 Affinity from hwloc-bind --get : 0xff00ff00 > > I was originally was asked to look at this because of bad runtimes on > faster hardware - fixing the affinity fixed > the performance problem. For the test case I was using binding each > process to a socket gave me 8% better performance so it's > definitely worth working on. > > Let me know what I can do to help - I don't mind gdbing into mpiexec > or something or shoving prints into the code. > > Also a probably silly question - Does hydra_pmi_proxy figure out > and set the affinity? That seems the logical place. > > I should also note that the other machine where --bind-to socket, > socket:1 does the right thing seems to do the wrong thing at > socket:2 > > Of course this all assumes I'm interpreting that socket option > correctly. > > > hydra-3.0.4/mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 > --launcher=ssh --host=abrams211a --bind-to socket utils/get_affinity > abrams211a.0 Affinity from /proc/$$/status : 00555555 > 0,2,4,6,8,10,12,14,16,18,20,22 > abrams211a.1 Affinity from /proc/$$/status : 00aaaaaa > 1,3,5,7,9,11,13,15,17,19,21,23 > abrams211a.0 Affinity from hwloc-bind --get : 0x00555555 > abrams211a.1 Affinity from hwloc-bind --get : 0x00aaaaaa > > mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 --launcher=ssh > --host=abrams211a --bind-to socket:1 utils/get_affinity > abrams211a.0 Affinity from /proc/$$/status : 00555555 > 0,2,4,6,8,10,12,14,16,18,20,22 > abrams211a.1 Affinity from /proc/$$/status : 00aaaaaa > 1,3,5,7,9,11,13,15,17,19,21,23 > abrams211a.0 Affinity from hwloc-bind --get : 0x00555555 > abrams211a.1 Affinity from hwloc-bind --get : 0x00aaaaaa > > > hydra-3.0.4/mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 > --launcher=ssh --host=abrams211a --bind-to socket:2 > utils/get_affinity > abrams211a.0 Affinity from /proc/$$/status : 00555555 > 0,2,4,6,8,10,12,14,16,18,20,22 > abrams211a.1 Affinity from /proc/$$/status : 00555555 > 0,2,4,6,8,10,12,14,16,18,20,22 > abrams211a.0 Affinity from hwloc-bind --get : 0x00555555 > abrams211a.1 Affinity from hwloc-bind --get : 0x00555555 > > > > On 09/18/2013 01:50 AM, Ken Raffenetti wrote: > > Hi Bill, > > > > I'm not seeing any obvious problems. However, in my own testing, I > > found a situation where the grep method you used was unreliable. > > Can you try using "hwloc-bind --get" instead and see if the > > results differ? > > > > Ken > > > > ----- Original Message ----- > >> From: "Bill Ryder" <bryder(a)wetafx.co.nz> > >> To: discuss(a)mpich.org > >> Sent: Monday, September 16, 2013 6:04:45 PM > >> Subject: [mpich-discuss] affinity problems with mpiexec.hydra > >> 3.0.4 > >> > >> Greetings all, > >> > >> I'm trying set affinity for mybrid MPI/OpenMP tasks. > >> > >> I want to run two processes on a host, and give one socket to one > >> rank, and the other socket to the other rank. > >> > >> I have two types of hardware - one works perfectly - the other > >> doesn't. > >> > >> I first saw the problem using mpiexec.hydra with slurm but I've > >> moved > >> to using ssh to remove some possible variables > >> > >> > >> I have a trivial script which just greps out /proc/$$/status for > >> the > >> Cpus_allowed mask and Cpus_allowed_list > >> > >> It's just: echo "`hostname` $PMI_RANK `grep Cpus_allowed > >> /proc/$$/status`" > >> > >> > >> On the machine that is doing what I want I get: > >> > >> mpiexec.hydra -ppn 2 -hosts abrams201a --bind-to socket -launcher > >> ssh > >> ./get_mapping > >> abrams201a 0 Cpus_allowed: 00555555 > >> Cpus_allowed_list: 0,2,4,6,8,10,12,14,16,18,20,22 > >> abrams201a 1 Cpus_allowed: 00aaaaaa > >> Cpus_allowed_list: 1,3,5,7,9,11,13,15,17,19,21,23 > >> > >> rank 0 gets one socket, rank 1 gets the other socket. This is what > >> I > >> want > >> > >> But on my other machine with a different topology I get this: > >> > >> mpiexec.hydra -ppn 2 -hosts jericho101 --bind-to socket -launcher > >> ssh ./get_mapping > >> > >> jericho101 0 Cpus_allowed: 00000000,00ff00ff > >> Cpus_allowed_list: 0-7,16-23 > >> jericho101 1 Cpus_allowed: 00000000,00ff00ff > >> Cpus_allowed_list: 0-7,16-23 > >> > >> So each rank is trying to use the same socket. > >> > >> Similarly if I try to bind to a numanode > >> > >> mpiexec.hydra -ppn 2 -hosts jericho101 --bind-to numa -launcher > >> ssh > >> ./get_mapping > >> jericho101 1 Cpus_allowed: 00000000,00ff00ff > >> Cpus_allowed_list: 0-7,16-23 > >> jericho101 0 Cpus_allowed: 00000000,00ff00ff > >> Cpus_allowed_list: 0-7,16-23 > >> > >> > >> Or even if I send numa:2 > >> > >> mpiexec.hydra -ppn 2 -hosts jericho101 --bind-to numa:2 -launcher > >> ssh ./get_mapping > >> > >> jericho101 0 Cpus_allowed: 00000000,00ff00ff > >> Cpus_allowed_list: 0-7,16-23 > >> jericho101 1 Cpus_allowed: 00000000,00ff00ff > >> Cpus_allowed_list: 0-7,16-23 > >> > >> So once again instead of handing a numa node to each process - > >> it's > >> handing the same node to both. > >> > >> > >> How would I start debugging this? > >> > >> Or am I missing something really obvious > >> > >> > >> > >> Thanks! > >> --------- > >> > >> Bill Ryder > >> Weta Digital > >> > >> > >> > >> > >> > >> A bit more data: > >> > >> mpiexec.hydra --info > >> HYDRA build details: > >> Version: 3.0.4 > >> Release Date: Wed Apr 24 10:08:10 > >> CDT > >> 2013 > >> CC: cc > >> CXX: > >> F77: > >> F90: > >> Configure options: '--prefix=/tech/apps/mpich/hydra' > >> Process Manager: pmi > >> Launchers available: ssh rsh fork slurm > >> ll > >> lsf sge manual persist > >> Topology libraries available: hwloc > >> Resource management kernels available: user slurm ll lsf > >> sge > >> pbs cobalt > >> Checkpointing libraries available: > >> Demux engines available: poll select > >> > >> > >> > >> I have hwloc 1.3.1 installed locally on each machine > >> > >> abrams201a looks like: > >> > >> Machine (48GB) > >> NUMANode L#0 (P#1 24GB) + Socket L#0 + L3 L#0 (12MB) > >> L2 L#0 (256KB) + L1 L#0 (32KB) + Core L#0 > >> PU L#0 (P#0) > >> PU L#1 (P#12) > >> L2 L#1 (256KB) + L1 L#1 (32KB) + Core L#1 > >> PU L#2 (P#2) > >> PU L#3 (P#14) > >> L2 L#2 (256KB) + L1 L#2 (32KB) + Core L#2 > >> PU L#4 (P#4) > >> PU L#5 (P#16) > >> L2 L#3 (256KB) + L1 L#3 (32KB) + Core L#3 > >> PU L#6 (P#6) > >> PU L#7 (P#18) > >> L2 L#4 (256KB) + L1 L#4 (32KB) + Core L#4 > >> PU L#8 (P#8) > >> PU L#9 (P#20) > >> L2 L#5 (256KB) + L1 L#5 (32KB) + Core L#5 > >> PU L#10 (P#10) > >> PU L#11 (P#22) > >> NUMANode L#1 (P#0 24GB) + Socket L#1 + L3 L#1 (12MB) > >> L2 L#6 (256KB) + L1 L#6 (32KB) + Core L#6 > >> PU L#12 (P#1) > >> PU L#13 (P#13) > >> L2 L#7 (256KB) + L1 L#7 (32KB) + Core L#7 > >> PU L#14 (P#3) > >> PU L#15 (P#15) > >> L2 L#8 (256KB) + L1 L#8 (32KB) + Core L#8 > >> PU L#16 (P#5) > >> PU L#17 (P#17) > >> L2 L#9 (256KB) + L1 L#9 (32KB) + Core L#9 > >> PU L#18 (P#7) > >> PU L#19 (P#19) > >> L2 L#10 (256KB) + L1 L#10 (32KB) + Core L#10 > >> PU L#20 (P#9) > >> PU L#21 (P#21) > >> L2 L#11 (256KB) + L1 L#11 (32KB) + Core L#11 > >> PU L#22 (P#11) > >> PU L#23 (P#23) > >> HostBridge L#0 > >> PCIBridge > >> PCI 8086:10e7 > >> Net L#0 "eth0" > >> PCI 8086:10e7 > >> Net L#1 "eth1" > >> PCIBridge > >> PCI 15b3:6746 > >> Net L#2 "eth2" > >> OpenFabrics L#3 "mlx4_0" > >> PCI 15b3:6746 > >> PCI 15b3:6746 > >> PCI 15b3:6746 > >> PCIBridge > >> PCI 102b:0533 > >> PCI 8086:3a20 > >> Block L#4 "sda" > >> > >> > >> And jericho101 looks like: > >> > >> achine (96GB) > >> NUMANode L#0 (P#0 48GB) > >> Socket L#0 + L3 L#0 (20MB) > >> L2 L#0 (256KB) + L1 L#0 (32KB) + Core L#0 > >> PU L#0 (P#0) > >> PU L#1 (P#16) > >> L2 L#1 (256KB) + L1 L#1 (32KB) + Core L#1 > >> PU L#2 (P#1) > >> PU L#3 (P#17) > >> L2 L#2 (256KB) + L1 L#2 (32KB) + Core L#2 > >> PU L#4 (P#2) > >> PU L#5 (P#18) > >> L2 L#3 (256KB) + L1 L#3 (32KB) + Core L#3 > >> PU L#6 (P#3) > >> PU L#7 (P#19) > >> L2 L#4 (256KB) + L1 L#4 (32KB) + Core L#4 > >> PU L#8 (P#4) > >> PU L#9 (P#20) > >> L2 L#5 (256KB) + L1 L#5 (32KB) + Core L#5 > >> PU L#10 (P#5) > >> PU L#11 (P#21) > >> L2 L#6 (256KB) + L1 L#6 (32KB) + Core L#6 > >> PU L#12 (P#6) > >> PU L#13 (P#22) > >> L2 L#7 (256KB) + L1 L#7 (32KB) + Core L#7 > >> PU L#14 (P#7) > >> PU L#15 (P#23) > >> HostBridge L#0 > >> PCIBridge > >> PCI 14e4:168e > >> Net L#0 "eth0" > >> PCI 14e4:168e > >> Net L#1 "eth1" > >> PCI 14e4:168e > >> Net L#2 "eth2" > >> PCI 14e4:168e > >> Net L#3 "eth3" > >> PCI 14e4:168e > >> Net L#4 "eth4" > >> PCI 14e4:168e > >> Net L#5 "eth5" > >> PCI 14e4:168e > >> Net L#6 "eth6" > >> PCI 14e4:168e > >> Net L#7 "eth7" > >> PCIBridge > >> PCI 103c:323b > >> Block L#8 "sda" > >> PCIBridge > >> PCI 102b:0533 > >> NUMANode L#1 (P#1 48GB) + Socket L#1 + L3 L#1 (20MB) > >> L2 L#8 (256KB) + L1 L#8 (32KB) + Core L#8 > >> PU L#16 (P#8) > >> PU L#17 (P#24) > >> L2 L#9 (256KB) + L1 L#9 (32KB) + Core L#9 > >> PU L#18 (P#9) > >> PU L#19 (P#25) > >> L2 L#10 (256KB) + L1 L#10 (32KB) + Core L#10 > >> PU L#20 (P#10) > >> PU L#21 (P#26) > >> L2 L#11 (256KB) + L1 L#11 (32KB) + Core L#11 > >> PU L#22 (P#11) > >> PU L#23 (P#27) > >> L2 L#12 (256KB) + L1 L#12 (32KB) + Core L#12 > >> PU L#24 (P#12) > >> PU L#25 (P#28) > >> L2 L#13 (256KB) + L1 L#13 (32KB) + Core L#13 > >> PU L#26 (P#13) > >> PU L#27 (P#29) > >> L2 L#14 (256KB) + L1 L#14 (32KB) + Core L#14 > >> PU L#28 (P#14) > >> PU L#29 (P#30) > >> L2 L#15 (256KB) + L1 L#15 (32KB) + Core L#15 > >> PU L#30 (P#15) > >> PU L#31 (P#31) > >> > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > >> > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Order of completion in MPI_Waitany
by Matthieu Dorier 19 Sep '13

19 Sep '13

Hi, I would like a precision on the semantics of MPI_Waitany in this situation: - Process 1 does 2 MPI_Irecv with different tags A and B (the order doesn't matter); it then waits on the two requests with MPI_Waitany. - In the mean time, process 2 does an MPI_Send with tag A, then MPI_Send with tag B. Can I ensure that MPI_Waitany will always give me the index of the message corresponding to the first MPI_Send, i.e. the one with the tag A, even in situations where process 1 called MPI_Irecv on a tag B first? Thanks Matthieu Dorier PhD student at ENS Cachan Brittany and IRISA http://people.irisa.fr/Matthieu.Dorier

1 0

Re: [mpich-discuss] affinity problems with mpiexec.hydra 3.0.4
by Bill Ryder 18 Sep '13

18 Sep '13

Hi Ken, Same result using hwloc-bind (which is a relief because if that didn't agree with /proc/pid/status I would have been unpleasantly surprised!) The most curious thing is that if I set socket:3 I get the exact binding I want! Perhaps my interpretation of --ppn 2 bind-to socket is incorrect. For two socket machines: If I use --bind-to socket I expect one rank on one socket, and the other on the other socket. If I use --bind-to socket:1 I expect both ranks on the same socket. If I use --bind-to socket:2 I expect one rank on each socket (ie the same as --bind-to socket for a two socket machine) Please let me know if that's incorrect. Here's trying various socket counts - using hwloc and looking at /proc/$$/status --bind-to socket - I expect each rank to get it's own socket - but that doesn't happen hydra-3.0.4/mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 --launcher=ssh --host=jericho101 --bind-to socket utils/get_affinity 2> /dev/null jericho101.0 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 jericho101.1 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 jericho101.1 Affinity from hwloc-bind --get : 0x00ff00ff jericho101.0 Affinity from hwloc-bind --get : 0x00ff00ff socket:1 - both processes bound to the same socket. This is what I expect socket:1 to do. hydra-3.0.4/mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 --launcher=ssh --host=jericho101 --bind-to socket:1 utils/get_affinity jericho101.0 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 jericho101.1 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 jericho101.0 Affinity from hwloc-bind --get : 0x00ff00ff jericho101.1 Affinity from hwloc-bind --get : 0x00ff00ff socket:2 - I end up with the same binding as socket and socket:1 hydra-3.0.4/mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 --launcher=ssh --host=jericho101 --bind-to socket:2 utils/get_affinity jericho101.0 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 jericho101.1 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 jericho101.0 Affinity from hwloc-bind --get : 0x00ff00ff jericho101.1 Affinity from hwloc-bind --get : 0x00ff00ff socket:3 - this gets strange - this is the affinity I want! hydra-3.0.4/mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 --launcher=ssh --host=jericho101 --bind-to socket:3 utils/get_affinity jericho101.0 Affinity from /proc/$$/status : 00ff00ff 0-7,16-23 jericho101.1 Affinity from /proc/$$/status : ff00ff00 8-15,24-31 jericho101.0 Affinity from hwloc-bind --get : 0x00ff00ff jericho101.1 Affinity from hwloc-bind --get : 0xff00ff00 I was originally was asked to look at this because of bad runtimes on faster hardware - fixing the affinity fixed the performance problem. For the test case I was using binding each process to a socket gave me 8% better performance so it's definitely worth working on. Let me know what I can do to help - I don't mind gdbing into mpiexec or something or shoving prints into the code. Also a probably silly question - Does hydra_pmi_proxy figure out and set the affinity? That seems the logical place. I should also note that the other machine where --bind-to socket, socket:1 does the right thing seems to do the wrong thing at socket:2 Of course this all assumes I'm interpreting that socket option correctly. hydra-3.0.4/mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 --launcher=ssh --host=abrams211a --bind-to socket utils/get_affinity abrams211a.0 Affinity from /proc/$$/status : 00555555 0,2,4,6,8,10,12,14,16,18,20,22 abrams211a.1 Affinity from /proc/$$/status : 00aaaaaa 1,3,5,7,9,11,13,15,17,19,21,23 abrams211a.0 Affinity from hwloc-bind --get : 0x00555555 abrams211a.1 Affinity from hwloc-bind --get : 0x00aaaaaa mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 --launcher=ssh --host=abrams211a --bind-to socket:1 utils/get_affinity abrams211a.0 Affinity from /proc/$$/status : 00555555 0,2,4,6,8,10,12,14,16,18,20,22 abrams211a.1 Affinity from /proc/$$/status : 00aaaaaa 1,3,5,7,9,11,13,15,17,19,21,23 abrams211a.0 Affinity from hwloc-bind --get : 0x00555555 abrams211a.1 Affinity from hwloc-bind --get : 0x00aaaaaa hydra-3.0.4/mpiexec.hydra --prepend-pattern "%h.%r " --ppn=2 --launcher=ssh --host=abrams211a --bind-to socket:2 utils/get_affinity abrams211a.0 Affinity from /proc/$$/status : 00555555 0,2,4,6,8,10,12,14,16,18,20,22 abrams211a.1 Affinity from /proc/$$/status : 00555555 0,2,4,6,8,10,12,14,16,18,20,22 abrams211a.0 Affinity from hwloc-bind --get : 0x00555555 abrams211a.1 Affinity from hwloc-bind --get : 0x00555555 On 09/18/2013 01:50 AM, Ken Raffenetti wrote: > Hi Bill, > > I'm not seeing any obvious problems. However, in my own testing, I found a situation where the grep method you used was unreliable. Can you try using "hwloc-bind --get" instead and see if the results differ? > > Ken > > ----- Original Message ----- >> From: "Bill Ryder" <bryder(a)wetafx.co.nz> >> To: discuss(a)mpich.org >> Sent: Monday, September 16, 2013 6:04:45 PM >> Subject: [mpich-discuss] affinity problems with mpiexec.hydra 3.0.4 >> >> Greetings all, >> >> I'm trying set affinity for mybrid MPI/OpenMP tasks. >> >> I want to run two processes on a host, and give one socket to one >> rank, and the other socket to the other rank. >> >> I have two types of hardware - one works perfectly - the other >> doesn't. >> >> I first saw the problem using mpiexec.hydra with slurm but I've moved >> to using ssh to remove some possible variables >> >> >> I have a trivial script which just greps out /proc/$$/status for the >> Cpus_allowed mask and Cpus_allowed_list >> >> It's just: echo "`hostname` $PMI_RANK `grep Cpus_allowed >> /proc/$$/status`" >> >> >> On the machine that is doing what I want I get: >> >> mpiexec.hydra -ppn 2 -hosts abrams201a --bind-to socket -launcher ssh >> ./get_mapping >> abrams201a 0 Cpus_allowed: 00555555 >> Cpus_allowed_list: 0,2,4,6,8,10,12,14,16,18,20,22 >> abrams201a 1 Cpus_allowed: 00aaaaaa >> Cpus_allowed_list: 1,3,5,7,9,11,13,15,17,19,21,23 >> >> rank 0 gets one socket, rank 1 gets the other socket. This is what I >> want >> >> But on my other machine with a different topology I get this: >> >> mpiexec.hydra -ppn 2 -hosts jericho101 --bind-to socket -launcher >> ssh ./get_mapping >> >> jericho101 0 Cpus_allowed: 00000000,00ff00ff >> Cpus_allowed_list: 0-7,16-23 >> jericho101 1 Cpus_allowed: 00000000,00ff00ff >> Cpus_allowed_list: 0-7,16-23 >> >> So each rank is trying to use the same socket. >> >> Similarly if I try to bind to a numanode >> >> mpiexec.hydra -ppn 2 -hosts jericho101 --bind-to numa -launcher ssh >> ./get_mapping >> jericho101 1 Cpus_allowed: 00000000,00ff00ff >> Cpus_allowed_list: 0-7,16-23 >> jericho101 0 Cpus_allowed: 00000000,00ff00ff >> Cpus_allowed_list: 0-7,16-23 >> >> >> Or even if I send numa:2 >> >> mpiexec.hydra -ppn 2 -hosts jericho101 --bind-to numa:2 -launcher >> ssh ./get_mapping >> >> jericho101 0 Cpus_allowed: 00000000,00ff00ff >> Cpus_allowed_list: 0-7,16-23 >> jericho101 1 Cpus_allowed: 00000000,00ff00ff >> Cpus_allowed_list: 0-7,16-23 >> >> So once again instead of handing a numa node to each process - it's >> handing the same node to both. >> >> >> How would I start debugging this? >> >> Or am I missing something really obvious >> >> >> >> Thanks! >> --------- >> >> Bill Ryder >> Weta Digital >> >> >> >> >> >> A bit more data: >> >> mpiexec.hydra --info >> HYDRA build details: >> Version: 3.0.4 >> Release Date: Wed Apr 24 10:08:10 CDT >> 2013 >> CC: cc >> CXX: >> F77: >> F90: >> Configure options: '--prefix=/tech/apps/mpich/hydra' >> Process Manager: pmi >> Launchers available: ssh rsh fork slurm ll >> lsf sge manual persist >> Topology libraries available: hwloc >> Resource management kernels available: user slurm ll lsf sge >> pbs cobalt >> Checkpointing libraries available: >> Demux engines available: poll select >> >> >> >> I have hwloc 1.3.1 installed locally on each machine >> >> abrams201a looks like: >> >> Machine (48GB) >> NUMANode L#0 (P#1 24GB) + Socket L#0 + L3 L#0 (12MB) >> L2 L#0 (256KB) + L1 L#0 (32KB) + Core L#0 >> PU L#0 (P#0) >> PU L#1 (P#12) >> L2 L#1 (256KB) + L1 L#1 (32KB) + Core L#1 >> PU L#2 (P#2) >> PU L#3 (P#14) >> L2 L#2 (256KB) + L1 L#2 (32KB) + Core L#2 >> PU L#4 (P#4) >> PU L#5 (P#16) >> L2 L#3 (256KB) + L1 L#3 (32KB) + Core L#3 >> PU L#6 (P#6) >> PU L#7 (P#18) >> L2 L#4 (256KB) + L1 L#4 (32KB) + Core L#4 >> PU L#8 (P#8) >> PU L#9 (P#20) >> L2 L#5 (256KB) + L1 L#5 (32KB) + Core L#5 >> PU L#10 (P#10) >> PU L#11 (P#22) >> NUMANode L#1 (P#0 24GB) + Socket L#1 + L3 L#1 (12MB) >> L2 L#6 (256KB) + L1 L#6 (32KB) + Core L#6 >> PU L#12 (P#1) >> PU L#13 (P#13) >> L2 L#7 (256KB) + L1 L#7 (32KB) + Core L#7 >> PU L#14 (P#3) >> PU L#15 (P#15) >> L2 L#8 (256KB) + L1 L#8 (32KB) + Core L#8 >> PU L#16 (P#5) >> PU L#17 (P#17) >> L2 L#9 (256KB) + L1 L#9 (32KB) + Core L#9 >> PU L#18 (P#7) >> PU L#19 (P#19) >> L2 L#10 (256KB) + L1 L#10 (32KB) + Core L#10 >> PU L#20 (P#9) >> PU L#21 (P#21) >> L2 L#11 (256KB) + L1 L#11 (32KB) + Core L#11 >> PU L#22 (P#11) >> PU L#23 (P#23) >> HostBridge L#0 >> PCIBridge >> PCI 8086:10e7 >> Net L#0 "eth0" >> PCI 8086:10e7 >> Net L#1 "eth1" >> PCIBridge >> PCI 15b3:6746 >> Net L#2 "eth2" >> OpenFabrics L#3 "mlx4_0" >> PCI 15b3:6746 >> PCI 15b3:6746 >> PCI 15b3:6746 >> PCIBridge >> PCI 102b:0533 >> PCI 8086:3a20 >> Block L#4 "sda" >> >> >> And jericho101 looks like: >> >> achine (96GB) >> NUMANode L#0 (P#0 48GB) >> Socket L#0 + L3 L#0 (20MB) >> L2 L#0 (256KB) + L1 L#0 (32KB) + Core L#0 >> PU L#0 (P#0) >> PU L#1 (P#16) >> L2 L#1 (256KB) + L1 L#1 (32KB) + Core L#1 >> PU L#2 (P#1) >> PU L#3 (P#17) >> L2 L#2 (256KB) + L1 L#2 (32KB) + Core L#2 >> PU L#4 (P#2) >> PU L#5 (P#18) >> L2 L#3 (256KB) + L1 L#3 (32KB) + Core L#3 >> PU L#6 (P#3) >> PU L#7 (P#19) >> L2 L#4 (256KB) + L1 L#4 (32KB) + Core L#4 >> PU L#8 (P#4) >> PU L#9 (P#20) >> L2 L#5 (256KB) + L1 L#5 (32KB) + Core L#5 >> PU L#10 (P#5) >> PU L#11 (P#21) >> L2 L#6 (256KB) + L1 L#6 (32KB) + Core L#6 >> PU L#12 (P#6) >> PU L#13 (P#22) >> L2 L#7 (256KB) + L1 L#7 (32KB) + Core L#7 >> PU L#14 (P#7) >> PU L#15 (P#23) >> HostBridge L#0 >> PCIBridge >> PCI 14e4:168e >> Net L#0 "eth0" >> PCI 14e4:168e >> Net L#1 "eth1" >> PCI 14e4:168e >> Net L#2 "eth2" >> PCI 14e4:168e >> Net L#3 "eth3" >> PCI 14e4:168e >> Net L#4 "eth4" >> PCI 14e4:168e >> Net L#5 "eth5" >> PCI 14e4:168e >> Net L#6 "eth6" >> PCI 14e4:168e >> Net L#7 "eth7" >> PCIBridge >> PCI 103c:323b >> Block L#8 "sda" >> PCIBridge >> PCI 102b:0533 >> NUMANode L#1 (P#1 48GB) + Socket L#1 + L3 L#1 (20MB) >> L2 L#8 (256KB) + L1 L#8 (32KB) + Core L#8 >> PU L#16 (P#8) >> PU L#17 (P#24) >> L2 L#9 (256KB) + L1 L#9 (32KB) + Core L#9 >> PU L#18 (P#9) >> PU L#19 (P#25) >> L2 L#10 (256KB) + L1 L#10 (32KB) + Core L#10 >> PU L#20 (P#10) >> PU L#21 (P#26) >> L2 L#11 (256KB) + L1 L#11 (32KB) + Core L#11 >> PU L#22 (P#11) >> PU L#23 (P#27) >> L2 L#12 (256KB) + L1 L#12 (32KB) + Core L#12 >> PU L#24 (P#12) >> PU L#25 (P#28) >> L2 L#13 (256KB) + L1 L#13 (32KB) + Core L#13 >> PU L#26 (P#13) >> PU L#27 (P#29) >> L2 L#14 (256KB) + L1 L#14 (32KB) + Core L#14 >> PU L#28 (P#14) >> PU L#29 (P#30) >> L2 L#15 (256KB) + L1 L#15 (32KB) + Core L#15 >> PU L#30 (P#15) >> PU L#31 (P#31) >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Better SGE integration of MPICH3 (spawning several queues)
by Reuti 17 Sep '13

17 Sep '13

Hi, the $PE_HOSTFILE is parsed in src/pm/hydra/tools/bootstrap/external/sge_query_node_list.c after MPICH detects that it's running inside an SGE job. Although it's not used often, it's allowed to create a PE (parallel environment) in SGE which spawns several queues. As a result, you get more than one entry in the $PE_HOSTFILE for a host and each one will be added to the list separately. This results in the effect, that remote hosts might get two or more `qrsh -inherit ...`, or even the local machine (where the master task of an MPI application is running) faces several bunches of `forks`, each bunch gets the number of forks MPICH detected in the $PE_HOSTFILE in a single line. The correct handling would be, to sum up all the slots which are granted for a particular machine, independent from the queue it is targeted for. It turned out (AFAICS), that in src/pm/hydra/utils/others/others.c the procedure "HYDU_add_to_node_list" is responsible for it. A double host entry is already covered, but only if it's the last element of the already read lines, i.e. when the list of nodes is sorted. Please find attached other.c (from 3.1b1), which should cover duplicate entries at any position. It's tested with SGE only though. -- Reuti PS: I don't know Torque in all details, but somehow I remember that it depends on its setup whether you get machines (as node numbers) in 1 1 1 2 2 3 or 1 2 3 1 2 1 order. In the latter case, it might have failed also there in the past for Torque (or "did it in a suboptimal process startup").

1 0

Re: [mpich-discuss] affinity problems with mpiexec.hydra 3.0.4
by Ken Raffenetti 17 Sep '13

17 Sep '13

Hi Bill, I'm not seeing any obvious problems. However, in my own testing, I found a situation where the grep method you used was unreliable. Can you try using "hwloc-bind --get" instead and see if the results differ? Ken ----- Original Message ----- > From: "Bill Ryder" <bryder(a)wetafx.co.nz> > To: discuss(a)mpich.org > Sent: Monday, September 16, 2013 6:04:45 PM > Subject: [mpich-discuss] affinity problems with mpiexec.hydra 3.0.4 > > Greetings all, > > I'm trying set affinity for mybrid MPI/OpenMP tasks. > > I want to run two processes on a host, and give one socket to one > rank, and the other socket to the other rank. > > I have two types of hardware - one works perfectly - the other > doesn't. > > I first saw the problem using mpiexec.hydra with slurm but I've moved > to using ssh to remove some possible variables > > > I have a trivial script which just greps out /proc/$$/status for the > Cpus_allowed mask and Cpus_allowed_list > > It's just: echo "`hostname` $PMI_RANK `grep Cpus_allowed > /proc/$$/status`" > > > On the machine that is doing what I want I get: > > mpiexec.hydra -ppn 2 -hosts abrams201a --bind-to socket -launcher ssh > ./get_mapping > abrams201a 0 Cpus_allowed: 00555555 > Cpus_allowed_list: 0,2,4,6,8,10,12,14,16,18,20,22 > abrams201a 1 Cpus_allowed: 00aaaaaa > Cpus_allowed_list: 1,3,5,7,9,11,13,15,17,19,21,23 > > rank 0 gets one socket, rank 1 gets the other socket. This is what I > want > > But on my other machine with a different topology I get this: > > mpiexec.hydra -ppn 2 -hosts jericho101 --bind-to socket -launcher > ssh ./get_mapping > > jericho101 0 Cpus_allowed: 00000000,00ff00ff > Cpus_allowed_list: 0-7,16-23 > jericho101 1 Cpus_allowed: 00000000,00ff00ff > Cpus_allowed_list: 0-7,16-23 > > So each rank is trying to use the same socket. > > Similarly if I try to bind to a numanode > > mpiexec.hydra -ppn 2 -hosts jericho101 --bind-to numa -launcher ssh > ./get_mapping > jericho101 1 Cpus_allowed: 00000000,00ff00ff > Cpus_allowed_list: 0-7,16-23 > jericho101 0 Cpus_allowed: 00000000,00ff00ff > Cpus_allowed_list: 0-7,16-23 > > > Or even if I send numa:2 > > mpiexec.hydra -ppn 2 -hosts jericho101 --bind-to numa:2 -launcher > ssh ./get_mapping > > jericho101 0 Cpus_allowed: 00000000,00ff00ff > Cpus_allowed_list: 0-7,16-23 > jericho101 1 Cpus_allowed: 00000000,00ff00ff > Cpus_allowed_list: 0-7,16-23 > > So once again instead of handing a numa node to each process - it's > handing the same node to both. > > > How would I start debugging this? > > Or am I missing something really obvious > > > > Thanks! > --------- > > Bill Ryder > Weta Digital > > > > > > A bit more data: > > mpiexec.hydra --info > HYDRA build details: > Version: 3.0.4 > Release Date: Wed Apr 24 10:08:10 CDT > 2013 > CC: cc > CXX: > F77: > F90: > Configure options: '--prefix=/tech/apps/mpich/hydra' > Process Manager: pmi > Launchers available: ssh rsh fork slurm ll > lsf sge manual persist > Topology libraries available: hwloc > Resource management kernels available: user slurm ll lsf sge > pbs cobalt > Checkpointing libraries available: > Demux engines available: poll select > > > > I have hwloc 1.3.1 installed locally on each machine > > abrams201a looks like: > > Machine (48GB) > NUMANode L#0 (P#1 24GB) + Socket L#0 + L3 L#0 (12MB) > L2 L#0 (256KB) + L1 L#0 (32KB) + Core L#0 > PU L#0 (P#0) > PU L#1 (P#12) > L2 L#1 (256KB) + L1 L#1 (32KB) + Core L#1 > PU L#2 (P#2) > PU L#3 (P#14) > L2 L#2 (256KB) + L1 L#2 (32KB) + Core L#2 > PU L#4 (P#4) > PU L#5 (P#16) > L2 L#3 (256KB) + L1 L#3 (32KB) + Core L#3 > PU L#6 (P#6) > PU L#7 (P#18) > L2 L#4 (256KB) + L1 L#4 (32KB) + Core L#4 > PU L#8 (P#8) > PU L#9 (P#20) > L2 L#5 (256KB) + L1 L#5 (32KB) + Core L#5 > PU L#10 (P#10) > PU L#11 (P#22) > NUMANode L#1 (P#0 24GB) + Socket L#1 + L3 L#1 (12MB) > L2 L#6 (256KB) + L1 L#6 (32KB) + Core L#6 > PU L#12 (P#1) > PU L#13 (P#13) > L2 L#7 (256KB) + L1 L#7 (32KB) + Core L#7 > PU L#14 (P#3) > PU L#15 (P#15) > L2 L#8 (256KB) + L1 L#8 (32KB) + Core L#8 > PU L#16 (P#5) > PU L#17 (P#17) > L2 L#9 (256KB) + L1 L#9 (32KB) + Core L#9 > PU L#18 (P#7) > PU L#19 (P#19) > L2 L#10 (256KB) + L1 L#10 (32KB) + Core L#10 > PU L#20 (P#9) > PU L#21 (P#21) > L2 L#11 (256KB) + L1 L#11 (32KB) + Core L#11 > PU L#22 (P#11) > PU L#23 (P#23) > HostBridge L#0 > PCIBridge > PCI 8086:10e7 > Net L#0 "eth0" > PCI 8086:10e7 > Net L#1 "eth1" > PCIBridge > PCI 15b3:6746 > Net L#2 "eth2" > OpenFabrics L#3 "mlx4_0" > PCI 15b3:6746 > PCI 15b3:6746 > PCI 15b3:6746 > PCIBridge > PCI 102b:0533 > PCI 8086:3a20 > Block L#4 "sda" > > > And jericho101 looks like: > > achine (96GB) > NUMANode L#0 (P#0 48GB) > Socket L#0 + L3 L#0 (20MB) > L2 L#0 (256KB) + L1 L#0 (32KB) + Core L#0 > PU L#0 (P#0) > PU L#1 (P#16) > L2 L#1 (256KB) + L1 L#1 (32KB) + Core L#1 > PU L#2 (P#1) > PU L#3 (P#17) > L2 L#2 (256KB) + L1 L#2 (32KB) + Core L#2 > PU L#4 (P#2) > PU L#5 (P#18) > L2 L#3 (256KB) + L1 L#3 (32KB) + Core L#3 > PU L#6 (P#3) > PU L#7 (P#19) > L2 L#4 (256KB) + L1 L#4 (32KB) + Core L#4 > PU L#8 (P#4) > PU L#9 (P#20) > L2 L#5 (256KB) + L1 L#5 (32KB) + Core L#5 > PU L#10 (P#5) > PU L#11 (P#21) > L2 L#6 (256KB) + L1 L#6 (32KB) + Core L#6 > PU L#12 (P#6) > PU L#13 (P#22) > L2 L#7 (256KB) + L1 L#7 (32KB) + Core L#7 > PU L#14 (P#7) > PU L#15 (P#23) > HostBridge L#0 > PCIBridge > PCI 14e4:168e > Net L#0 "eth0" > PCI 14e4:168e > Net L#1 "eth1" > PCI 14e4:168e > Net L#2 "eth2" > PCI 14e4:168e > Net L#3 "eth3" > PCI 14e4:168e > Net L#4 "eth4" > PCI 14e4:168e > Net L#5 "eth5" > PCI 14e4:168e > Net L#6 "eth6" > PCI 14e4:168e > Net L#7 "eth7" > PCIBridge > PCI 103c:323b > Block L#8 "sda" > PCIBridge > PCI 102b:0533 > NUMANode L#1 (P#1 48GB) + Socket L#1 + L3 L#1 (20MB) > L2 L#8 (256KB) + L1 L#8 (32KB) + Core L#8 > PU L#16 (P#8) > PU L#17 (P#24) > L2 L#9 (256KB) + L1 L#9 (32KB) + Core L#9 > PU L#18 (P#9) > PU L#19 (P#25) > L2 L#10 (256KB) + L1 L#10 (32KB) + Core L#10 > PU L#20 (P#10) > PU L#21 (P#26) > L2 L#11 (256KB) + L1 L#11 (32KB) + Core L#11 > PU L#22 (P#11) > PU L#23 (P#27) > L2 L#12 (256KB) + L1 L#12 (32KB) + Core L#12 > PU L#24 (P#12) > PU L#25 (P#28) > L2 L#13 (256KB) + L1 L#13 (32KB) + Core L#13 > PU L#26 (P#13) > PU L#27 (P#29) > L2 L#14 (256KB) + L1 L#14 (32KB) + Core L#14 > PU L#28 (P#14) > PU L#29 (P#30) > L2 L#15 (256KB) + L1 L#15 (32KB) + Core L#15 > PU L#30 (P#15) > PU L#31 (P#31) > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0