May 2020 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Understanding process bindings in MPICH
by Benson Muite 16 May '20

16 May '20

On Fri, May 15, 2020, at 8:52 PM, hritikesh semwal via discuss wrote: > Hello, > > I am working on a parallel CFD solver with MPI and I am using an account on a cluster to run my executable. The hardware structure of my account is as follows; > > Architecture: x86_64 > CPU op-mode(s): 32-bit, 64-bit > Byte Order: Little Endian > CPU(s): 32 > On-line CPU(s) list: 0-31 > Thread(s) per core: 2 > Core(s) per socket: 8 > CPU socket(s): 2 > NUMA node(s): 2 > Vendor ID: GenuineIntel > CPU family: 6 > Model: 62 > Stepping: 4 > CPU MHz: 2600.079 > BogoMIPS: 5199.25 > Virtualization: VT-x > L1d cache: 32K > L1i cache: 32K > L2 cache: 256K > L3 cache: 20480K > NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30 > NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31 > > Initially, I was running my executable with any binding options and in that case, whenever I was switching from 2 to 4 processors my computation time was also increasing along with communication time inside some iterative loop. > > Today, somewhere I read about binding options in MPI through which I can manage the allocation of processors. Initially, I used the "-bind-to core" option and the results were different and I got time reduction up to 16 processors and after that with 24 and 32 processors, it has started increasing. Results of timing are as follows; > 2 procs- 160 seconds, 4 procs- 84 seconds, 8 procs- 45 seconds, 16 procs- 28 seconds, 24 procs- 38 seconds, 32 procs- 34 seconds. > > After that, I used some other combinations of binding option but did not get better timing results compared to -bind-to core option. So, I back edited the bind to option to core but now I am getting different timing results with the same executable which are as follows, > 2 procs- 164 seconds, 4 procs- 85 seconds, 8 procs- 45 seconds, 16 procs- 48 seconds, 24 procs- 52 seconds, 32 seconds- 98 seconds. Hitesh, You might find the following online workshop useful: http://www.hlrs.de/training/2020-05-25-VI-HPS/ Regards, Benson

1 0

Re: [mpich-discuss] Understanding process bindings in MPICH
by Benson Muite 15 May '20

15 May '20

On Fri, May 15, 2020, at 8:52 PM, hritikesh semwal via discuss wrote: > Hello, > > I am working on a parallel CFD solver with MPI and I am using an account on a cluster to run my executable. The hardware structure of my account is as follows; > > Architecture: x86_64 > CPU op-mode(s): 32-bit, 64-bit > Byte Order: Little Endian > CPU(s): 32 > On-line CPU(s) list: 0-31 > Thread(s) per core: 2 > Core(s) per socket: 8 > CPU socket(s): 2 > NUMA node(s): 2 > Vendor ID: GenuineIntel > CPU family: 6 > Model: 62 > Stepping: 4 > CPU MHz: 2600.079 > BogoMIPS: 5199.25 > Virtualization: VT-x > L1d cache: 32K > L1i cache: 32K > L2 cache: 256K > L3 cache: 20480K > NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30 > NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31 > > Initially, I was running my executable with any binding options and in that case, whenever I was switching from 2 to 4 processors my computation time was also increasing along with communication time inside some iterative loop. > > Today, somewhere I read about binding options in MPI through which I can manage the allocation of processors. Initially, I used the "-bind-to core" option and the results were different and I got time reduction up to 16 processors and after that with 24 and 32 processors, it has started increasing. Results of timing are as follows; > 2 procs- 160 seconds, 4 procs- 84 seconds, 8 procs- 45 seconds, 16 procs- 28 seconds, 24 procs- 38 seconds, 32 procs- 34 seconds. This seems reasonable. Are you able to turn of hyperthreading? For most numerical codes this is not useful as they are typically bandwidth limited. Thus for more than 16 processors will not see much speed up.

1 0

Understanding process bindings in MPICH
by hritikesh semwal 15 May '20

15 May '20

Hello, I am working on a parallel CFD solver with MPI and I am using an account on a cluster to run my executable. The hardware structure of my account is as follows; Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 32 On-line CPU(s) list: 0-31 Thread(s) per core: 2 Core(s) per socket: 8 CPU socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 62 Stepping: 4 CPU MHz: 2600.079 BogoMIPS: 5199.25 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 20480K NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31 Initially, I was running my executable with any binding options and in that case, whenever I was switching from 2 to 4 processors my computation time was also increasing along with communication time inside some iterative loop. Today, somewhere I read about binding options in MPI through which I can manage the allocation of processors. Initially, I used the "-bind-to core" option and the results were different and I got time reduction up to 16 processors and after that with 24 and 32 processors, it has started increasing. Results of timing are as follows; 2 procs- 160 seconds, 4 procs- 84 seconds, 8 procs- 45 seconds, 16 procs- 28 seconds, 24 procs- 38 seconds, 32 procs- 34 seconds. After that, I used some other combinations of binding option but did not get better timing results compared to -bind-to core option. So, I back edited the bind to option to core but now I am getting different timing results with the same executable which are as follows, 2 procs- 164 seconds, 4 procs- 85 seconds, 8 procs- 45 seconds, 16 procs- 48 seconds, 24 procs- 52 seconds, 32 seconds- 98 seconds. I have following two questions for which I am seeking your help, 1. Can anyone please suggest me is it possible an optimum binding and mapping options based on my cluster account hardware topology?? If yes then please tell me. 2. Why I am getting such an irregular pattern of jump in timing without binding option and why with a binding option, my timings are varying for each run? Is it my cluster network problem or my MPI code problem? If you need further details about my iterative loop then please tell me. As this message got too long I can share it later if you think above data is not sufficient. Thank you.

1 0

Re: [mpich-discuss] MPICH 3.3.2 and ch4:ucx launcher
by Tony Curtis 14 May '20

14 May '20

> On May 14, 2020, at 10:54 AM, Raffenetti, Kenneth J. via discuss <discuss(a)mpich.org> wrote: > > Sorry, I just read the configure line more closely. Do you have a strict dependency on PMIx? It is not required to run with ch4:ucx. The default process manager (Hydra) supports launching on Slurm clusters out of the box. > I was just playing with / looking for potential PMIx-aware launchers. This isn’t stopping anything from happening. Tony

1 0

Re: [mpich-discuss] MPICH 3.3.2 and ch4:ucx launcher
by Raffenetti, Kenneth J. 14 May '20

14 May '20

Sorry, I just read the configure line more closely. Do you have a strict dependency on PMIx? It is not required to run with ch4:ucx. The default process manager (Hydra) supports launching on Slurm clusters out of the box. Ken On 5/14/20, 9:52 AM, "Raffenetti, Kenneth J. via discuss" <discuss(a)mpich.org> wrote: Sorry for the delayed response. Why are you adding --with-pm=no to your configuration? Ken On 5/1/20, 3:20 PM, "Tony Curtis via discuss" <discuss(a)mpich.org> wrote: On May 1, 2020, at 4:04 PM, Sajid Ali <sajidsyed2021(a)u.northwestern.edu> wrote: Hi Tony, Yes, and be sure to set the correct mpi launcher at runtime via the --mpi argument to srun (for more info check this page https://slurm.schedmd.com/mpi_guide.html ). Thanks - thought so. I’m just a j. Random user so I can’t do much about this. Tony _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH 3.3.2 and ch4:ucx launcher
by Raffenetti, Kenneth J. 14 May '20

14 May '20

Sorry for the delayed response. Why are you adding --with-pm=no to your configuration? Ken On 5/1/20, 3:20 PM, "Tony Curtis via discuss" <discuss(a)mpich.org> wrote: On May 1, 2020, at 4:04 PM, Sajid Ali <sajidsyed2021(a)u.northwestern.edu> wrote: Hi Tony, Yes, and be sure to set the correct mpi launcher at runtime via the --mpi argument to srun (for more info check this page https://slurm.schedmd.com/mpi_guide.html ). Thanks - thought so. I’m just a j. Random user so I can’t do much about this. Tony

1 0

Why does hydra_mpi_proxy kill client processes?
by Yuri 12 May '20

12 May '20

I am running NWchem (a quantum chemistry program) in parallel on a single host using mpich-3.2.1_6 on FreeBSD 12.1 It runs, but every once in a while hydra_mpi_proxy sends SIGKILL to a client process nwchem. The same task usually succeeds when rerun again, and some other task fails with SIGKILL next time, so the problem is sporadic. Why does hydra_mpi_proxy kill client processes? My original NWchem bug report is https://github.com/nwchemgit/nwchem/issues/198 Thank you, Yuri

1 0

Re: [mpich-discuss] Run mpich on localhost
by Martin Ivanov 10 May '20

10 May '20

Hello Tony, Thank you very much for your help. Interestingly, there are both mpich and hwloc packages for dfly. They should be working ok. For now, with 5 cores is OK for me, 50 % of the time it starts successfully. If I manage to compile WRF I may experiment updating mpich. But for now I prefer to stick to the library versions recommended by the WRF compilation tutorial. Best regards, Martin Von meinem iPhone gesendet > Am 10.05.2020 um 22:17 schrieb Tony Curtis <anthony.curtis(a)stonybrook.edu>: > > > >> On May 10, 2020, at 3:16 AM, Martin Ivanov <marto1980(a)gmail.com> wrote: >> >> Hello Tony, >> Thank you very much for your reply. I am posting you the output of 'ps x', when 'mpirun -n 5 mpich-3.0.4/examples/hellow' freezes: >> >> After that freeze, I killed mpirun with Ctrl + C: >> " >> marto@dragonfly% mpirun -n 5 mpich-3.0.4/examples/hellow >> ^C[mpiexec@dragonfly] Sending Ctrl-C to processes as requested >> [mpiexec@dragonfly] Press Ctrl-C again to force abort >> [proxy:0:0@dragonfly] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:71): assert (!(pollfds[i].revents & ~POLLIN & ~POLLOUT & ~POLLHUP & ~POLLERR)) failed >> [proxy:0:0@dragonfly] main (./pm/pmiserv/pmip.c:206): demux engine error waiting for event >> [mpiexec@dragonfly] control_cb (./pm/pmiserv/pmiserv_cb.c:202): assert (!closed) failed >> [mpiexec@dragonfly] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status >> [mpiexec@dragonfly] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:197): error waiting for event >> [mpiexec@dragonfly] main (./ui/mpich/mpiexec.c:331): process manager error waiting for completion >> " >> >> The next relaunch of mpirun with 5 cores was successful. For completeness, I am attaching the output of 'ps x' after mpirun with 2 cores freezes, which it actually with 2 cores always does. >> >> I hope this was helpful. I am looking forward to your reply. >> > > Hi, > > Well, I took this as an opportunity to play with dfly again, so I replicated mpich 3.0.4 and see much the same behavior. Mpirun -n 2 hangs consistently, -n 4 works reliably (= # system cores), -n 5 sometimes. > > I’m not an mpich developer so I will leave further prognostication to their capable hands, but your suspicions about hwloc from the warning messages seem to be well-founded. > > Tony >

1 0

Re: [mpich-discuss] Run mpich on localhost
by Tony Curtis 10 May '20

10 May '20

> On May 10, 2020, at 3:16 AM, Martin Ivanov <marto1980(a)gmail.com> wrote: > > Hello Tony, > Thank you very much for your reply. I am posting you the output of 'ps x', when 'mpirun -n 5 mpich-3.0.4/examples/hellow' freezes: > > After that freeze, I killed mpirun with Ctrl + C: > " > marto@dragonfly% mpirun -n 5 mpich-3.0.4/examples/hellow > ^C[mpiexec@dragonfly] Sending Ctrl-C to processes as requested > [mpiexec@dragonfly] Press Ctrl-C again to force abort > [proxy:0:0@dragonfly] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:71): assert (!(pollfds[i].revents & ~POLLIN & ~POLLOUT & ~POLLHUP & ~POLLERR)) failed > [proxy:0:0@dragonfly] main (./pm/pmiserv/pmip.c:206): demux engine error waiting for event > [mpiexec@dragonfly] control_cb (./pm/pmiserv/pmiserv_cb.c:202): assert (!closed) failed > [mpiexec@dragonfly] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status > [mpiexec@dragonfly] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:197): error waiting for event > [mpiexec@dragonfly] main (./ui/mpich/mpiexec.c:331): process manager error waiting for completion > " > > The next relaunch of mpirun with 5 cores was successful. For completeness, I am attaching the output of 'ps x' after mpirun with 2 cores freezes, which it actually with 2 cores always does. > > I hope this was helpful. I am looking forward to your reply. > Hi, Well, I took this as an opportunity to play with dfly again, so I replicated mpich 3.0.4 and see much the same behavior. Mpirun -n 2 hangs consistently, -n 4 works reliably (= # system cores), -n 5 sometimes. I’m not an mpich developer so I will leave further prognostication to their capable hands, but your suspicions about hwloc from the warning messages seem to be well-founded. Tony

1 0

Re: [mpich-discuss] 1. Single file reading by all processors in MPI. 2.Difference between blocking and non-blocking call of MPI_neighbor_alltoallw
by Joseph Schuchart 10 May '20

10 May '20

On 5/10/20 7:30 PM, Jeff Hammond via discuss wrote: > > >> On May 9, 2020, at 8:34 PM, hritikesh semwal >> <hritikesh.semwal(a)gmail.com> wrote: >> >> >> >> >> On Sun, May 10, 2020 at 4:18 AM Jeff Hammond <jeff.science(a)gmail.com >> <mailto:[email protected]>> wrote: >> >> >> >> On Sat, May 9, 2020 at 10:10 AM hritikesh semwal via discuss >> <discuss(a)mpich.org <mailto:[email protected]>> wrote: >> >> Hello, >> >> I have following two questions with MPI, >> >> 1. Do I need to give a separate input file to each processor >> in a group or all of them can read the input from a single >> file? All data inside the file should be read by every processor. >> >> >> You can read the input file from every process but it isn't a good >> idea. Read the input file from rank 0 and broadcast the >> contents. This assumes that your input file is small. If your >> input file is huge, then you should consider designing it for >> parallel reading with MPI_File_* functions. >> >> >> Thanks for your response. I am reading two input files in my parallel >> CFD solver at different places of the code. The first input file has 7 >> integer values and second input file has 15 integer values. Is it >> large to read it through conventional C functions for file reading >> i.e. fscanf() and is broadcast good for this amount of data? > > It depends on your filesystem details but I’d expect the cutoff where > parallel reading of input to be necessary is greater than 1 MB and > potentially much more than that. Anything less than 4K (or whatever the > filesystem page size is if not 4K) is going to be dominated by block > read latency, which can’t be reduced with parallelism. The latency/bandwidth for reading is only one aspect though, the filesystem overhead is another. Your site administrators will be grateful for you *not* trying to open the same file from a couple thousand processes at once... Reading it once in a single process and distributing it in a broadcast is imo the best option in this case. Cheers Joseph, who brought down Lustre installations in the past... > > Jeff > >> 2. Can you please tell me what is the difference between >> MPI_Neighbor_alltoallw and MPI_Ineighbor_alltoallw? As I read >> in MPI 3.0 document that MPI_Neighbor_alltoallw also used >> non-blocking send and receive inside its execution. >> >> >> MPI_Neighbor_alltoallw is a blocking function while >> MPI_Ineighbor_alltoallw is not. Using non-blocking send and >> receive inside its execution does not mean >> that MPI_Neighbor_alltoallw doesn't block - if that is how it's >> implemented, then there will be the equivalent of a MPI_Waitall >> before the function returns. But in any case, just because the >> MPI document suggests a possible implementation does not mean that >> is how all implementations work. >> >> >> Yes, there is MPI_Waitall inside the MPI_Neighbor_alltoallw before it >> returns. >> >> >> Jeff >> >> Thank you. >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com <mailto:[email protected]> >> http://jeffhammond.github.io/ >> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0