May 2015 - discuss - mailman3.cels.anl.gov

Error while using MPI platform with CPP program
by Umesh Shisode 18 May '15

18 May '15

Hello Sir / Madam, This is Umesh Sharad Shisode, pursuing Masters in Modeling and Simulation, at Centre for Modeling and Simulation, Pune University. I am in the final stage of my course. My Masters Thesis title is : Development of Conjugate Gradient solver in OpenFOAM / CPP using MPI I am now doing MPI part of my project and facing some problems with MPI functions syntax. I am trying to use CPP class object with MPI functions but it is giving some error. Which I don't understand. Operating system : LINUX Programming Language : CPP Commands I am using are : > mpic++ main.cpp -o result > mpirun -np 2 result Please help me to sort out this issue . I am attaching herewith zip file containing the code which I want to parallelize with the png file containing error. Waiting for your reply. Thanks in advance. Regards, Umesh Sharad Shisode, M.Tech in Modeling and Simulation(2013-15), Centre for Modeling and Simulation, University of Pune. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpiexec.hydra binding for multiple compute nodes
by Justin Chang 13 May '15

13 May '15

Oh okay. I see that one of the binding options within hydra is to bind to the motherboard. Would that allow one to assign ranks to certain nodes? (Assuming that my compute node only has one motherboard). If I had #SBATCH -N 2 and #SBATCH -n 32 y guess is to have something like '-bind-to board' sequentially bind 0-15 to node 0 and 16-31 to node 1, or would this option still result in striped even/odd assignments? But yes otherwise I would very much like to be included in that CC if a ticket is to be made. Thanks On Wednesday, May 13, 2015, Kenneth Raffenetti <raffenet(a)mcs.anl.gov> wrote: > Ah, you are right. They will be striped odd/even across the nodes. Not > sequentially. I don't see a way to serve that binding with the current > options. It's something we can look at adding in a future release. I'll > open up a ticket and add you to the CC list, if you are interested. > > Ken > > On 05/12/2015 02:16 PM, Justin Chang wrote: > >> So just to be clear, if I have the following script: >> >> #SBATCH -N 2 >> #SBATCH -n 32 >> >> mpiexec.hydra -bind-to user:0,1,2,3,4,5,6,7,10,11,12,13,14,15,16,17 -n >> 32 ./my_program <args> >> >> ranks 0-15 will be binded as such on the first node and ranks 16-31 will >> be the same for the second node? Or will all the even ranks be on one >> node and the odd on the other? >> >> Thanks, >> >> On Mon, May 11, 2015 at 9:27 PM, Kenneth Raffenetti >> <raffenet(a)mcs.anl.gov <mailto:[email protected]>> wrote: >> >> Ah, I see now the problem. I misread the first email. Your original >> line should work fine! The user bindings are listings of hw >> elements, not processes, so your binding will be applied identically >> on each node. >> >> Ken >> >> >> On 05/11/2015 05:03 PM, Justin Chang wrote: >> >> Ken, >> >> "-bind-to core" gives me the following topology: >> >> process 0 binding: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 1 binding: 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 2 binding: 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 3 binding: 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 4 binding: 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 5 binding: 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 6 binding: 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 7 binding: 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 >> process 8 binding: 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 >> process 9 binding: 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 >> process 10 binding: 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 >> process 11 binding: 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 >> process 12 binding: 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 >> process 13 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 >> process 14 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 >> process 15 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 >> >> but I want this: >> >> process 0 binding: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 1 binding: 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 2 binding: 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 3 binding: 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 4 binding: 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 5 binding: 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 6 binding: 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 7 binding: 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 >> process 8 binding: 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 >> process 9 binding: 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 >> process 10 binding: 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 >> process 11 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 >> process 12 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 >> process 13 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 >> process 14 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 >> process 15 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 >> >> The latter gives me better performance for my application, and I >> am >> guessing it's because I have evenly distribute the processes >> among the >> two sockets (sequentially). Which is why I resorted to what I had >> originally with the custom binding. >> >> Thanks, >> >> On Mon, May 11, 2015 at 4:53 PM, Kenneth Raffenetti >> <raffenet(a)mcs.anl.gov <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>>> >> wrote: >> >> Justin, >> >> Try using the "-bind-to core" option instead. It should do >> exactly >> what you are wanting. See this page with examples for more >> details >> >> https://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager#Proc… >> >> Ken >> >> >> On 05/11/2015 04:48 PM, Justin Chang wrote: >> >> Hello everyone, >> >> I am working with an HPC machine that has this >> configuring for a >> single >> compute node: >> >> Machine (64GB total) >> NUMANode L#0 (P#0 32GB) >> Socket L#0 + L3 L#0 (25MB) >> L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 >> (32KB) + Core >> L#0 + PU >> L#0 (P#0) >> L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 >> (32KB) + Core >> L#1 + PU >> L#1 (P#1) >> L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 >> (32KB) + Core >> L#2 + PU >> L#2 (P#2) >> L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 >> (32KB) + Core >> L#3 + PU >> L#3 (P#3) >> L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 >> (32KB) + Core >> L#4 + PU >> L#4 (P#4) >> L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 >> (32KB) + Core >> L#5 + PU >> L#5 (P#5) >> L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 >> (32KB) + Core >> L#6 + PU >> L#6 (P#6) >> L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 >> (32KB) + Core >> L#7 + PU >> L#7 (P#7) >> L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 >> (32KB) + Core >> L#8 + PU >> L#8 (P#8) >> L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 >> (32KB) + Core >> L#9 + PU >> L#9 (P#9) >> HostBridge L#0 >> PCIBridge >> PCI 1000:0087 >> Block L#0 "sda" >> PCIBridge >> PCI 15b3:1003 >> Net L#1 "eth0" >> Net L#2 "ib0" >> OpenFabrics L#3 "mlx4_0" >> PCIBridge >> PCI 8086:1521 >> Net L#4 "eth1" >> PCI 8086:1521 >> Net L#5 "eth2" >> PCIBridge >> PCI 102b:0533 >> PCI 8086:1d02 >> NUMANode L#1 (P#1 32GB) + Socket L#1 + L3 L#1 (25MB) >> L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 >> (32KB) + Core >> L#10 + >> PU L#10 (P#10) >> L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 >> (32KB) + Core >> L#11 + >> PU L#11 (P#11) >> L2 L#12 (256KB) + L1d L#12 (32KB) + L1i L#12 >> (32KB) + Core >> L#12 + >> PU L#12 (P#12) >> L2 L#13 (256KB) + L1d L#13 (32KB) + L1i L#13 >> (32KB) + Core >> L#13 + >> PU L#13 (P#13) >> L2 L#14 (256KB) + L1d L#14 (32KB) + L1i L#14 >> (32KB) + Core >> L#14 + >> PU L#14 (P#14) >> L2 L#15 (256KB) + L1d L#15 (32KB) + L1i L#15 >> (32KB) + Core >> L#15 + >> PU L#15 (P#15) >> L2 L#16 (256KB) + L1d L#16 (32KB) + L1i L#16 >> (32KB) + Core >> L#16 + >> PU L#16 (P#16) >> L2 L#17 (256KB) + L1d L#17 (32KB) + L1i L#17 >> (32KB) + Core >> L#17 + >> PU L#17 (P#17) >> L2 L#18 (256KB) + L1d L#18 (32KB) + L1i L#18 >> (32KB) + Core >> L#18 + >> PU L#18 (P#18) >> L2 L#19 (256KB) + L1d L#19 (32KB) + L1i L#19 >> (32KB) + Core >> L#19 + >> PU L#19 (P#19) >> >> If I ran my program with 16 processes, I would have the >> follow >> batch script: >> >> #!/bin/bash >> #SBATCH -N 1 >> #SBATCH -n 20 >> #SBATCH -t 0-09:00 >> #SBATCH -o output.txt >> >> mpiexec.hydra -bind-to >> user:0,1,2,3,4,5,6,7,10,11,12,13,14,15,16,17 -n >> 16 ./my_program <args> >> >> This would give me decent speedup. However, what if I >> want to use 32 >> processes? Since each node only has 20 cores I would need >> #SBATCH -N 2 >> and #SBATCH -n 40. However, I want ranks 0-15 and 16-31 >> to have >> the same >> mapping as above but on different compute nodes, so how >> would I >> do this? >> Or would the above line work so long as I have a >> multiple of 16 >> processes? >> >> Thanks, >> >> -- >> Justin Chang >> PhD Candidate, Civil Engineering - Computational Sciences >> University of Houston, Department of Civil and >> Environmental >> Engineering >> Houston, TX 77004 >> (512) 963-3262 <tel:%28512%29%20963-3262> >> <tel:%28512%29%20963-3262> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> <mailto:[email protected]> <mailto:[email protected] >> <mailto:[email protected]>> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> <mailto:[email protected]> <mailto:[email protected] >> <mailto:[email protected]>> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> >> -- >> Justin Chang >> PhD Candidate, Civil Engineering - Computational Sciences >> University of Houston, Department of Civil and Environmental >> Engineering >> Houston, TX 77004 >> (512) 963-3262 <tel:%28512%29%20963-3262> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> >> -- >> Justin Chang >> PhD Candidate, Civil Engineering - Computational Sciences >> University of Houston, Department of Civil and Environmental Engineering >> Houston, TX 77004 >> (512) 963-3262 >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpiexec.hydra binding for multiple compute nodes
by Kenneth Raffenetti 13 May '15

13 May '15

Ah, you are right. They will be striped odd/even across the nodes. Not sequentially. I don't see a way to serve that binding with the current options. It's something we can look at adding in a future release. I'll open up a ticket and add you to the CC list, if you are interested. Ken On 05/12/2015 02:16 PM, Justin Chang wrote: > So just to be clear, if I have the following script: > > #SBATCH -N 2 > #SBATCH -n 32 > > mpiexec.hydra -bind-to user:0,1,2,3,4,5,6,7,10,11,12,13,14,15,16,17 -n > 32 ./my_program <args> > > ranks 0-15 will be binded as such on the first node and ranks 16-31 will > be the same for the second node? Or will all the even ranks be on one > node and the odd on the other? > > Thanks, > > On Mon, May 11, 2015 at 9:27 PM, Kenneth Raffenetti > <raffenet(a)mcs.anl.gov <mailto:[email protected]>> wrote: > > Ah, I see now the problem. I misread the first email. Your original > line should work fine! The user bindings are listings of hw > elements, not processes, so your binding will be applied identically > on each node. > > Ken > > > On 05/11/2015 05:03 PM, Justin Chang wrote: > > Ken, > > "-bind-to core" gives me the following topology: > > process 0 binding: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 1 binding: 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 2 binding: 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 3 binding: 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 4 binding: 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 5 binding: 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 6 binding: 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 7 binding: 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 > process 8 binding: 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 > process 9 binding: 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 > process 10 binding: 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 > process 11 binding: 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 > process 12 binding: 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 > process 13 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 > process 14 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 > process 15 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 > > but I want this: > > process 0 binding: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 1 binding: 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 2 binding: 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 3 binding: 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 4 binding: 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 5 binding: 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 6 binding: 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 7 binding: 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 > process 8 binding: 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 > process 9 binding: 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 > process 10 binding: 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 > process 11 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 > process 12 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 > process 13 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 > process 14 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 > process 15 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 > > The latter gives me better performance for my application, and I am > guessing it's because I have evenly distribute the processes > among the > two sockets (sequentially). Which is why I resorted to what I had > originally with the custom binding. > > Thanks, > > On Mon, May 11, 2015 at 4:53 PM, Kenneth Raffenetti > <raffenet(a)mcs.anl.gov <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>>> wrote: > > Justin, > > Try using the "-bind-to core" option instead. It should do > exactly > what you are wanting. See this page with examples for more > details > https://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager#Proc… > > Ken > > > On 05/11/2015 04:48 PM, Justin Chang wrote: > > Hello everyone, > > I am working with an HPC machine that has this > configuring for a > single > compute node: > > Machine (64GB total) > NUMANode L#0 (P#0 32GB) > Socket L#0 + L3 L#0 (25MB) > L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 > (32KB) + Core > L#0 + PU > L#0 (P#0) > L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 > (32KB) + Core > L#1 + PU > L#1 (P#1) > L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 > (32KB) + Core > L#2 + PU > L#2 (P#2) > L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 > (32KB) + Core > L#3 + PU > L#3 (P#3) > L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 > (32KB) + Core > L#4 + PU > L#4 (P#4) > L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 > (32KB) + Core > L#5 + PU > L#5 (P#5) > L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 > (32KB) + Core > L#6 + PU > L#6 (P#6) > L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 > (32KB) + Core > L#7 + PU > L#7 (P#7) > L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 > (32KB) + Core > L#8 + PU > L#8 (P#8) > L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 > (32KB) + Core > L#9 + PU > L#9 (P#9) > HostBridge L#0 > PCIBridge > PCI 1000:0087 > Block L#0 "sda" > PCIBridge > PCI 15b3:1003 > Net L#1 "eth0" > Net L#2 "ib0" > OpenFabrics L#3 "mlx4_0" > PCIBridge > PCI 8086:1521 > Net L#4 "eth1" > PCI 8086:1521 > Net L#5 "eth2" > PCIBridge > PCI 102b:0533 > PCI 8086:1d02 > NUMANode L#1 (P#1 32GB) + Socket L#1 + L3 L#1 (25MB) > L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 > (32KB) + Core > L#10 + > PU L#10 (P#10) > L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 > (32KB) + Core > L#11 + > PU L#11 (P#11) > L2 L#12 (256KB) + L1d L#12 (32KB) + L1i L#12 > (32KB) + Core > L#12 + > PU L#12 (P#12) > L2 L#13 (256KB) + L1d L#13 (32KB) + L1i L#13 > (32KB) + Core > L#13 + > PU L#13 (P#13) > L2 L#14 (256KB) + L1d L#14 (32KB) + L1i L#14 > (32KB) + Core > L#14 + > PU L#14 (P#14) > L2 L#15 (256KB) + L1d L#15 (32KB) + L1i L#15 > (32KB) + Core > L#15 + > PU L#15 (P#15) > L2 L#16 (256KB) + L1d L#16 (32KB) + L1i L#16 > (32KB) + Core > L#16 + > PU L#16 (P#16) > L2 L#17 (256KB) + L1d L#17 (32KB) + L1i L#17 > (32KB) + Core > L#17 + > PU L#17 (P#17) > L2 L#18 (256KB) + L1d L#18 (32KB) + L1i L#18 > (32KB) + Core > L#18 + > PU L#18 (P#18) > L2 L#19 (256KB) + L1d L#19 (32KB) + L1i L#19 > (32KB) + Core > L#19 + > PU L#19 (P#19) > > If I ran my program with 16 processes, I would have the > follow > batch script: > > #!/bin/bash > #SBATCH -N 1 > #SBATCH -n 20 > #SBATCH -t 0-09:00 > #SBATCH -o output.txt > > mpiexec.hydra -bind-to > user:0,1,2,3,4,5,6,7,10,11,12,13,14,15,16,17 -n > 16 ./my_program <args> > > This would give me decent speedup. However, what if I > want to use 32 > processes? Since each node only has 20 cores I would need > #SBATCH -N 2 > and #SBATCH -n 40. However, I want ranks 0-15 and 16-31 > to have > the same > mapping as above but on different compute nodes, so how > would I > do this? > Or would the above line work so long as I have a > multiple of 16 > processes? > > Thanks, > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and > Environmental > Engineering > Houston, TX 77004 > (512) 963-3262 <tel:%28512%29%20963-3262> <tel:%28512%29%20963-3262> > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > <mailto:[email protected]> <mailto:[email protected] > <mailto:[email protected]>> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > <mailto:[email protected]> <mailto:[email protected] > <mailto:[email protected]>> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental > Engineering > Houston, TX 77004 > (512) 963-3262 <tel:%28512%29%20963-3262> > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpiexec.hydra binding for multiple compute nodes
by Justin Chang 12 May '15

12 May '15

So just to be clear, if I have the following script: #SBATCH -N 2 #SBATCH -n 32 mpiexec.hydra -bind-to user:0,1,2,3,4,5,6,7,10,11,12,13,14,15,16,17 -n 32 ./my_program <args> ranks 0-15 will be binded as such on the first node and ranks 16-31 will be the same for the second node? Or will all the even ranks be on one node and the odd on the other? Thanks, On Mon, May 11, 2015 at 9:27 PM, Kenneth Raffenetti <raffenet(a)mcs.anl.gov> wrote: > Ah, I see now the problem. I misread the first email. Your original line > should work fine! The user bindings are listings of hw elements, not > processes, so your binding will be applied identically on each node. > > Ken > > > On 05/11/2015 05:03 PM, Justin Chang wrote: > >> Ken, >> >> "-bind-to core" gives me the following topology: >> >> process 0 binding: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 1 binding: 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 2 binding: 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 3 binding: 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 4 binding: 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 5 binding: 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 6 binding: 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 7 binding: 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 >> process 8 binding: 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 >> process 9 binding: 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 >> process 10 binding: 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 >> process 11 binding: 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 >> process 12 binding: 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 >> process 13 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 >> process 14 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 >> process 15 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 >> >> but I want this: >> >> process 0 binding: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 1 binding: 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 2 binding: 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 3 binding: 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 4 binding: 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 5 binding: 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 6 binding: 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 >> process 7 binding: 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 >> process 8 binding: 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 >> process 9 binding: 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 >> process 10 binding: 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 >> process 11 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 >> process 12 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 >> process 13 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 >> process 14 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 >> process 15 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 >> >> The latter gives me better performance for my application, and I am >> guessing it's because I have evenly distribute the processes among the >> two sockets (sequentially). Which is why I resorted to what I had >> originally with the custom binding. >> >> Thanks, >> >> On Mon, May 11, 2015 at 4:53 PM, Kenneth Raffenetti >> <raffenet(a)mcs.anl.gov <mailto:[email protected]>> wrote: >> >> Justin, >> >> Try using the "-bind-to core" option instead. It should do exactly >> what you are wanting. See this page with examples for more details >> >> https://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager#Proc… >> >> Ken >> >> >> On 05/11/2015 04:48 PM, Justin Chang wrote: >> >> Hello everyone, >> >> I am working with an HPC machine that has this configuring for a >> single >> compute node: >> >> Machine (64GB total) >> NUMANode L#0 (P#0 32GB) >> Socket L#0 + L3 L#0 (25MB) >> L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core >> L#0 + PU >> L#0 (P#0) >> L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core >> L#1 + PU >> L#1 (P#1) >> L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core >> L#2 + PU >> L#2 (P#2) >> L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core >> L#3 + PU >> L#3 (P#3) >> L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core >> L#4 + PU >> L#4 (P#4) >> L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core >> L#5 + PU >> L#5 (P#5) >> L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core >> L#6 + PU >> L#6 (P#6) >> L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core >> L#7 + PU >> L#7 (P#7) >> L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core >> L#8 + PU >> L#8 (P#8) >> L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core >> L#9 + PU >> L#9 (P#9) >> HostBridge L#0 >> PCIBridge >> PCI 1000:0087 >> Block L#0 "sda" >> PCIBridge >> PCI 15b3:1003 >> Net L#1 "eth0" >> Net L#2 "ib0" >> OpenFabrics L#3 "mlx4_0" >> PCIBridge >> PCI 8086:1521 >> Net L#4 "eth1" >> PCI 8086:1521 >> Net L#5 "eth2" >> PCIBridge >> PCI 102b:0533 >> PCI 8086:1d02 >> NUMANode L#1 (P#1 32GB) + Socket L#1 + L3 L#1 (25MB) >> L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core >> L#10 + >> PU L#10 (P#10) >> L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core >> L#11 + >> PU L#11 (P#11) >> L2 L#12 (256KB) + L1d L#12 (32KB) + L1i L#12 (32KB) + Core >> L#12 + >> PU L#12 (P#12) >> L2 L#13 (256KB) + L1d L#13 (32KB) + L1i L#13 (32KB) + Core >> L#13 + >> PU L#13 (P#13) >> L2 L#14 (256KB) + L1d L#14 (32KB) + L1i L#14 (32KB) + Core >> L#14 + >> PU L#14 (P#14) >> L2 L#15 (256KB) + L1d L#15 (32KB) + L1i L#15 (32KB) + Core >> L#15 + >> PU L#15 (P#15) >> L2 L#16 (256KB) + L1d L#16 (32KB) + L1i L#16 (32KB) + Core >> L#16 + >> PU L#16 (P#16) >> L2 L#17 (256KB) + L1d L#17 (32KB) + L1i L#17 (32KB) + Core >> L#17 + >> PU L#17 (P#17) >> L2 L#18 (256KB) + L1d L#18 (32KB) + L1i L#18 (32KB) + Core >> L#18 + >> PU L#18 (P#18) >> L2 L#19 (256KB) + L1d L#19 (32KB) + L1i L#19 (32KB) + Core >> L#19 + >> PU L#19 (P#19) >> >> If I ran my program with 16 processes, I would have the follow >> batch script: >> >> #!/bin/bash >> #SBATCH -N 1 >> #SBATCH -n 20 >> #SBATCH -t 0-09:00 >> #SBATCH -o output.txt >> >> mpiexec.hydra -bind-to >> user:0,1,2,3,4,5,6,7,10,11,12,13,14,15,16,17 -n >> 16 ./my_program <args> >> >> This would give me decent speedup. However, what if I want to use >> 32 >> processes? Since each node only has 20 cores I would need >> #SBATCH -N 2 >> and #SBATCH -n 40. However, I want ranks 0-15 and 16-31 to have >> the same >> mapping as above but on different compute nodes, so how would I >> do this? >> Or would the above line work so long as I have a multiple of 16 >> processes? >> >> Thanks, >> >> -- >> Justin Chang >> PhD Candidate, Civil Engineering - Computational Sciences >> University of Houston, Department of Civil and Environmental >> Engineering >> Houston, TX 77004 >> (512) 963-3262 <tel:%28512%29%20963-3262> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> >> -- >> Justin Chang >> PhD Candidate, Civil Engineering - Computational Sciences >> University of Houston, Department of Civil and Environmental Engineering >> Houston, TX 77004 >> (512) 963-3262 >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpiexec.hydra binding for multiple compute nodes
by Kenneth Raffenetti 11 May '15

11 May '15

Ah, I see now the problem. I misread the first email. Your original line should work fine! The user bindings are listings of hw elements, not processes, so your binding will be applied identically on each node. Ken On 05/11/2015 05:03 PM, Justin Chang wrote: > Ken, > > "-bind-to core" gives me the following topology: > > process 0 binding: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 1 binding: 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 2 binding: 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 3 binding: 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 4 binding: 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 5 binding: 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 6 binding: 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 7 binding: 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 > process 8 binding: 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 > process 9 binding: 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 > process 10 binding: 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 > process 11 binding: 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 > process 12 binding: 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 > process 13 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 > process 14 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 > process 15 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 > > but I want this: > > process 0 binding: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 1 binding: 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 2 binding: 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 3 binding: 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 4 binding: 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 5 binding: 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 6 binding: 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 > process 7 binding: 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 > process 8 binding: 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 > process 9 binding: 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 > process 10 binding: 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 > process 11 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 > process 12 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 > process 13 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 > process 14 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 > process 15 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 > > The latter gives me better performance for my application, and I am > guessing it's because I have evenly distribute the processes among the > two sockets (sequentially). Which is why I resorted to what I had > originally with the custom binding. > > Thanks, > > On Mon, May 11, 2015 at 4:53 PM, Kenneth Raffenetti > <raffenet(a)mcs.anl.gov <mailto:[email protected]>> wrote: > > Justin, > > Try using the "-bind-to core" option instead. It should do exactly > what you are wanting. See this page with examples for more details > https://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager#Proc… > > Ken > > > On 05/11/2015 04:48 PM, Justin Chang wrote: > > Hello everyone, > > I am working with an HPC machine that has this configuring for a > single > compute node: > > Machine (64GB total) > NUMANode L#0 (P#0 32GB) > Socket L#0 + L3 L#0 (25MB) > L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core > L#0 + PU > L#0 (P#0) > L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core > L#1 + PU > L#1 (P#1) > L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core > L#2 + PU > L#2 (P#2) > L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core > L#3 + PU > L#3 (P#3) > L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core > L#4 + PU > L#4 (P#4) > L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core > L#5 + PU > L#5 (P#5) > L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core > L#6 + PU > L#6 (P#6) > L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core > L#7 + PU > L#7 (P#7) > L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core > L#8 + PU > L#8 (P#8) > L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core > L#9 + PU > L#9 (P#9) > HostBridge L#0 > PCIBridge > PCI 1000:0087 > Block L#0 "sda" > PCIBridge > PCI 15b3:1003 > Net L#1 "eth0" > Net L#2 "ib0" > OpenFabrics L#3 "mlx4_0" > PCIBridge > PCI 8086:1521 > Net L#4 "eth1" > PCI 8086:1521 > Net L#5 "eth2" > PCIBridge > PCI 102b:0533 > PCI 8086:1d02 > NUMANode L#1 (P#1 32GB) + Socket L#1 + L3 L#1 (25MB) > L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core > L#10 + > PU L#10 (P#10) > L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core > L#11 + > PU L#11 (P#11) > L2 L#12 (256KB) + L1d L#12 (32KB) + L1i L#12 (32KB) + Core > L#12 + > PU L#12 (P#12) > L2 L#13 (256KB) + L1d L#13 (32KB) + L1i L#13 (32KB) + Core > L#13 + > PU L#13 (P#13) > L2 L#14 (256KB) + L1d L#14 (32KB) + L1i L#14 (32KB) + Core > L#14 + > PU L#14 (P#14) > L2 L#15 (256KB) + L1d L#15 (32KB) + L1i L#15 (32KB) + Core > L#15 + > PU L#15 (P#15) > L2 L#16 (256KB) + L1d L#16 (32KB) + L1i L#16 (32KB) + Core > L#16 + > PU L#16 (P#16) > L2 L#17 (256KB) + L1d L#17 (32KB) + L1i L#17 (32KB) + Core > L#17 + > PU L#17 (P#17) > L2 L#18 (256KB) + L1d L#18 (32KB) + L1i L#18 (32KB) + Core > L#18 + > PU L#18 (P#18) > L2 L#19 (256KB) + L1d L#19 (32KB) + L1i L#19 (32KB) + Core > L#19 + > PU L#19 (P#19) > > If I ran my program with 16 processes, I would have the follow > batch script: > > #!/bin/bash > #SBATCH -N 1 > #SBATCH -n 20 > #SBATCH -t 0-09:00 > #SBATCH -o output.txt > > mpiexec.hydra -bind-to > user:0,1,2,3,4,5,6,7,10,11,12,13,14,15,16,17 -n > 16 ./my_program <args> > > This would give me decent speedup. However, what if I want to use 32 > processes? Since each node only has 20 cores I would need > #SBATCH -N 2 > and #SBATCH -n 40. However, I want ranks 0-15 and 16-31 to have > the same > mapping as above but on different compute nodes, so how would I > do this? > Or would the above line work so long as I have a multiple of 16 > processes? > > Thanks, > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental > Engineering > Houston, TX 77004 > (512) 963-3262 <tel:%28512%29%20963-3262> > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Shared Memory Segfaults
by Jeff Hammond 11 May '15

11 May '15

Thanks for the clarification. The smallest possible reproducer is always the best, from a bug report perspective :-) Jeff On Mon, May 4, 2015 at 8:20 AM, Brian Cornille <bcornille(a)wisc.edu> wrote: > Jeff, > > I agree that the code I supplied has no interesting parallelism. That code was simply an attempt to diagnose or replicated an error I was getting with a more complicated code which also had shared memory structures pointing to other shared memory structures. The linked list was just a quick example of a similar structure to test what the issue could be. It ended up having very similar error behavior as the original program so I thought I would share it instead of a more complicated project. For the project I was working on I wanted to create large shared memory structures that would be read-only once the initial set-up was done. Some of these shared memory structures were to have shared memory structures allocated within them, which appeared to generate the errors I was encountering. If you are interested in the original code that produced these errors it is posted at: > https://github.com/bcornille/mpi-shared-ne506/tree/develop/src (mostly in geoms.c) > There are likely ways that I could change the structure of the code to work around this problem, but I was hoping to try to understand what was causing the very strange errors that I produced. > I encountered errors when I tried to make assignments to shared memory regions that were pointed to from other shared memory regions. I am hoping that the error in the code I provided is the mirror image of the error in my original code. > > Thanks and best, > Brian Cornille > > On Sat, May 2 at 5:46 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> >>I do not understand what this program is supposed to do. >>MPI_Win_allocate_shared is collective, as is MPI_Win_fence, so I don't >>see how any interesting parallelism emerges from this type of >>implementation of a linked-list. >> >>Are you just using the linked-list to manage a collection of shared >>memory windows (GMR does this in >>http://git.mpich.org/armci-mpi.git/blob/HEAD:/src/gmr.c, for example)? >> >>If you want to do a distributed linked-list using RMA, I recall there >>is a linked-list example in the MPI-3 spec or perhaps somewhere else >>(somebody on this list will remember, or I can look it up). And this >>example probably uses MPI_Win_create_dynamic and MPI_Win_attach, which >>means no interprocess load-store. >> >>Jeff >> >>On Sat, May 2, 2015 at 4:06 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: >>> I can reproduce the segfault with the latest mpich. In Clean_List(), I think >>> there is a data race, since all shmem members update the head node >>> head->next_win = temp_win; >>> head->next = temp_next; >>> I tried to simplify Clean_List() further as follows, >>> >>> void Clean_List() >>> { >>> Node *cur_node = head; >>> MPI_Win cur_win = head_win; >>> Node *next_node; >>> MPI_Win next_win; >>> >>> while (cur_node) { >>> next_node = cur_node->next; >>> next_win = cur_node->next_win; >>> MPI_Win_free(&cur_win); >>> cur_node = next_node; >>> cur_win = next_win; >>> } >>> >>> head = tail = NULL; >>> } >>> >>> But I still met segfault. With gdb, the segfault disappears. If I comment >>> out the call to Clean_List in main(), the error also disappear. >>> I Cc'ed our local RMA expert Xin to see if she has new findings. >>> >>> --Junchao Zhang >>> >>> On Sat, May 2, 2015 at 10:26 AM, Brian Cornille <bcornille(a)wisc.edu> wrote: >>>> >>>> Hello, >>>> >>>> >>>> In working on a project that is attempting to use MPI shared memory (from >>>> MPI_Win_allocate_shared) I began getting inconsistent segfaults in portions >>>> of the code that appeared to have no memory errors when investigated with >>>> gdb. I believe I have somewhat reproduced this error in a small code >>>> (attached) that creates a linked list of MPI shared memory allocated >>>> elements. >>>> >>>> >>>> The attached program segfaults for me when run with more than one process. >>>> However, will not segfault if run in gdb (e.g. mpirun -n 2 xterm -e gdb >>>> ./mpi_shmem_ll). I have done what I can to eliminate any apparent race >>>> conditions. Any help in this matter would be much appreciated. >>>> >>>> >>>> Thanks and best, >>>> >>>> Brian Cornille >>>> >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >>-- >>Jeff Hammond >>jeff.science(a)gmail.com >>http://jeffhammond.github.io/ > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpiexec.hydra binding for multiple compute nodes
by Martin Cuma 11 May '15

11 May '15

Justin, I would also do #SBATCH -N 2 #SBATCH -n 32 since if you do #SBATCH -n 40 and then run mpiexec -np 32, Slurm will put 20 tasks on node 0 and 12 on node 1. With the #SBATCH -n 32, Slurm will give you a node with 16 or more cores, so, 20 core node will be used if core count <20 is not available.. MC -- Martin Cuma Center for High Performance Computing Department of Geology and Geophysics University of Utah _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpiexec.hydra binding for multiple compute nodes
by Justin Chang 11 May '15

11 May '15

Ken, "-bind-to core" gives me the following topology: process 0 binding: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 process 1 binding: 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 process 2 binding: 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 process 3 binding: 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 process 4 binding: 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 process 5 binding: 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 process 6 binding: 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 process 7 binding: 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 process 8 binding: 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 process 9 binding: 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 process 10 binding: 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 process 11 binding: 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 process 12 binding: 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 process 13 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 process 14 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 process 15 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 but I want this: process 0 binding: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 process 1 binding: 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 process 2 binding: 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 process 3 binding: 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 process 4 binding: 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 process 5 binding: 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 process 6 binding: 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 process 7 binding: 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 process 8 binding: 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 process 9 binding: 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 process 10 binding: 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 process 11 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 process 12 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 process 13 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 process 14 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 process 15 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 The latter gives me better performance for my application, and I am guessing it's because I have evenly distribute the processes among the two sockets (sequentially). Which is why I resorted to what I had originally with the custom binding. Thanks, On Mon, May 11, 2015 at 4:53 PM, Kenneth Raffenetti <raffenet(a)mcs.anl.gov> wrote: > Justin, > > Try using the "-bind-to core" option instead. It should do exactly what > you are wanting. See this page with examples for more details > https://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager#Proc… > > Ken > > > On 05/11/2015 04:48 PM, Justin Chang wrote: > >> Hello everyone, >> >> I am working with an HPC machine that has this configuring for a single >> compute node: >> >> Machine (64GB total) >> NUMANode L#0 (P#0 32GB) >> Socket L#0 + L3 L#0 (25MB) >> L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU >> L#0 (P#0) >> L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 + PU >> L#1 (P#1) >> L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2 + PU >> L#2 (P#2) >> L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3 + PU >> L#3 (P#3) >> L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4 + PU >> L#4 (P#4) >> L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5 + PU >> L#5 (P#5) >> L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6 + PU >> L#6 (P#6) >> L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7 + PU >> L#7 (P#7) >> L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8 + PU >> L#8 (P#8) >> L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9 + PU >> L#9 (P#9) >> HostBridge L#0 >> PCIBridge >> PCI 1000:0087 >> Block L#0 "sda" >> PCIBridge >> PCI 15b3:1003 >> Net L#1 "eth0" >> Net L#2 "ib0" >> OpenFabrics L#3 "mlx4_0" >> PCIBridge >> PCI 8086:1521 >> Net L#4 "eth1" >> PCI 8086:1521 >> Net L#5 "eth2" >> PCIBridge >> PCI 102b:0533 >> PCI 8086:1d02 >> NUMANode L#1 (P#1 32GB) + Socket L#1 + L3 L#1 (25MB) >> L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10 + >> PU L#10 (P#10) >> L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11 + >> PU L#11 (P#11) >> L2 L#12 (256KB) + L1d L#12 (32KB) + L1i L#12 (32KB) + Core L#12 + >> PU L#12 (P#12) >> L2 L#13 (256KB) + L1d L#13 (32KB) + L1i L#13 (32KB) + Core L#13 + >> PU L#13 (P#13) >> L2 L#14 (256KB) + L1d L#14 (32KB) + L1i L#14 (32KB) + Core L#14 + >> PU L#14 (P#14) >> L2 L#15 (256KB) + L1d L#15 (32KB) + L1i L#15 (32KB) + Core L#15 + >> PU L#15 (P#15) >> L2 L#16 (256KB) + L1d L#16 (32KB) + L1i L#16 (32KB) + Core L#16 + >> PU L#16 (P#16) >> L2 L#17 (256KB) + L1d L#17 (32KB) + L1i L#17 (32KB) + Core L#17 + >> PU L#17 (P#17) >> L2 L#18 (256KB) + L1d L#18 (32KB) + L1i L#18 (32KB) + Core L#18 + >> PU L#18 (P#18) >> L2 L#19 (256KB) + L1d L#19 (32KB) + L1i L#19 (32KB) + Core L#19 + >> PU L#19 (P#19) >> >> If I ran my program with 16 processes, I would have the follow batch >> script: >> >> #!/bin/bash >> #SBATCH -N 1 >> #SBATCH -n 20 >> #SBATCH -t 0-09:00 >> #SBATCH -o output.txt >> >> mpiexec.hydra -bind-to user:0,1,2,3,4,5,6,7,10,11,12,13,14,15,16,17 -n >> 16 ./my_program <args> >> >> This would give me decent speedup. However, what if I want to use 32 >> processes? Since each node only has 20 cores I would need #SBATCH -N 2 >> and #SBATCH -n 40. However, I want ranks 0-15 and 16-31 to have the same >> mapping as above but on different compute nodes, so how would I do this? >> Or would the above line work so long as I have a multiple of 16 processes? >> >> Thanks, >> >> -- >> Justin Chang >> PhD Candidate, Civil Engineering - Computational Sciences >> University of Houston, Department of Civil and Environmental Engineering >> Houston, TX 77004 >> (512) 963-3262 >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpiexec.hydra binding for multiple compute nodes
by Kenneth Raffenetti 11 May '15

11 May '15

Justin, Try using the "-bind-to core" option instead. It should do exactly what you are wanting. See this page with examples for more details https://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager#Proc… Ken On 05/11/2015 04:48 PM, Justin Chang wrote: > Hello everyone, > > I am working with an HPC machine that has this configuring for a single > compute node: > > Machine (64GB total) > NUMANode L#0 (P#0 32GB) > Socket L#0 + L3 L#0 (25MB) > L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU > L#0 (P#0) > L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 + PU > L#1 (P#1) > L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2 + PU > L#2 (P#2) > L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3 + PU > L#3 (P#3) > L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4 + PU > L#4 (P#4) > L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5 + PU > L#5 (P#5) > L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6 + PU > L#6 (P#6) > L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7 + PU > L#7 (P#7) > L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8 + PU > L#8 (P#8) > L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9 + PU > L#9 (P#9) > HostBridge L#0 > PCIBridge > PCI 1000:0087 > Block L#0 "sda" > PCIBridge > PCI 15b3:1003 > Net L#1 "eth0" > Net L#2 "ib0" > OpenFabrics L#3 "mlx4_0" > PCIBridge > PCI 8086:1521 > Net L#4 "eth1" > PCI 8086:1521 > Net L#5 "eth2" > PCIBridge > PCI 102b:0533 > PCI 8086:1d02 > NUMANode L#1 (P#1 32GB) + Socket L#1 + L3 L#1 (25MB) > L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10 + > PU L#10 (P#10) > L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11 + > PU L#11 (P#11) > L2 L#12 (256KB) + L1d L#12 (32KB) + L1i L#12 (32KB) + Core L#12 + > PU L#12 (P#12) > L2 L#13 (256KB) + L1d L#13 (32KB) + L1i L#13 (32KB) + Core L#13 + > PU L#13 (P#13) > L2 L#14 (256KB) + L1d L#14 (32KB) + L1i L#14 (32KB) + Core L#14 + > PU L#14 (P#14) > L2 L#15 (256KB) + L1d L#15 (32KB) + L1i L#15 (32KB) + Core L#15 + > PU L#15 (P#15) > L2 L#16 (256KB) + L1d L#16 (32KB) + L1i L#16 (32KB) + Core L#16 + > PU L#16 (P#16) > L2 L#17 (256KB) + L1d L#17 (32KB) + L1i L#17 (32KB) + Core L#17 + > PU L#17 (P#17) > L2 L#18 (256KB) + L1d L#18 (32KB) + L1i L#18 (32KB) + Core L#18 + > PU L#18 (P#18) > L2 L#19 (256KB) + L1d L#19 (32KB) + L1i L#19 (32KB) + Core L#19 + > PU L#19 (P#19) > > If I ran my program with 16 processes, I would have the follow batch script: > > #!/bin/bash > #SBATCH -N 1 > #SBATCH -n 20 > #SBATCH -t 0-09:00 > #SBATCH -o output.txt > > mpiexec.hydra -bind-to user:0,1,2,3,4,5,6,7,10,11,12,13,14,15,16,17 -n > 16 ./my_program <args> > > This would give me decent speedup. However, what if I want to use 32 > processes? Since each node only has 20 cores I would need #SBATCH -N 2 > and #SBATCH -n 40. However, I want ranks 0-15 and 16-31 to have the same > mapping as above but on different compute nodes, so how would I do this? > Or would the above line work so long as I have a multiple of 16 processes? > > Thanks, > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

mpiexec.hydra binding for multiple compute nodes
by Justin Chang 11 May '15

11 May '15

Hello everyone, I am working with an HPC machine that has this configuring for a single compute node: Machine (64GB total) NUMANode L#0 (P#0 32GB) Socket L#0 + L3 L#0 (25MB) L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU L#0 (P#0) L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 + PU L#1 (P#1) L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2 + PU L#2 (P#2) L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3 + PU L#3 (P#3) L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4 + PU L#4 (P#4) L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5 + PU L#5 (P#5) L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6 + PU L#6 (P#6) L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7 + PU L#7 (P#7) L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8 + PU L#8 (P#8) L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9 + PU L#9 (P#9) HostBridge L#0 PCIBridge PCI 1000:0087 Block L#0 "sda" PCIBridge PCI 15b3:1003 Net L#1 "eth0" Net L#2 "ib0" OpenFabrics L#3 "mlx4_0" PCIBridge PCI 8086:1521 Net L#4 "eth1" PCI 8086:1521 Net L#5 "eth2" PCIBridge PCI 102b:0533 PCI 8086:1d02 NUMANode L#1 (P#1 32GB) + Socket L#1 + L3 L#1 (25MB) L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10 + PU L#10 (P#10) L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11 + PU L#11 (P#11) L2 L#12 (256KB) + L1d L#12 (32KB) + L1i L#12 (32KB) + Core L#12 + PU L#12 (P#12) L2 L#13 (256KB) + L1d L#13 (32KB) + L1i L#13 (32KB) + Core L#13 + PU L#13 (P#13) L2 L#14 (256KB) + L1d L#14 (32KB) + L1i L#14 (32KB) + Core L#14 + PU L#14 (P#14) L2 L#15 (256KB) + L1d L#15 (32KB) + L1i L#15 (32KB) + Core L#15 + PU L#15 (P#15) L2 L#16 (256KB) + L1d L#16 (32KB) + L1i L#16 (32KB) + Core L#16 + PU L#16 (P#16) L2 L#17 (256KB) + L1d L#17 (32KB) + L1i L#17 (32KB) + Core L#17 + PU L#17 (P#17) L2 L#18 (256KB) + L1d L#18 (32KB) + L1i L#18 (32KB) + Core L#18 + PU L#18 (P#18) L2 L#19 (256KB) + L1d L#19 (32KB) + L1i L#19 (32KB) + Core L#19 + PU L#19 (P#19) If I ran my program with 16 processes, I would have the follow batch script: #!/bin/bash #SBATCH -N 1 #SBATCH -n 20 #SBATCH -t 0-09:00 #SBATCH -o output.txt mpiexec.hydra -bind-to user:0,1,2,3,4,5,6,7,10,11,12,13,14,15,16,17 -n 16 ./my_program <args> This would give me decent speedup. However, what if I want to use 32 processes? Since each node only has 20 cores I would need #SBATCH -N 2 and #SBATCH -n 40. However, I want ranks 0-15 and 16-31 to have the same mapping as above but on different compute nodes, so how would I do this? Or would the above line work so long as I have a multiple of 16 processes? Thanks, -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0