December 2018 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Weird performance of locally concurrent MPI_Put
by Kun Feng 31 Dec '18

31 Dec '18

Good advice. Now I find out that the performance drops to sequential again when I move all the sources of MPI_Put into threads in one MPI rank even using MPI_Win_allocate. Do you have similar experience before? Thanks Kun On Sat, Dec 29, 2018 at 9:36 PM Jeff Hammond <jeff.science(a)gmail.com> wrote: > I don’t know why you are timing win_allocate. I’d only time > lock-put-unlock or put-flush. > > Jeff > > On Sat, Dec 29, 2018 at 9:11 AM Kun Feng <kfeng1(a)hawk.iit.edu> wrote: > >> Thank you for the replies. >> MPI_Win_allocate gives me much better performance. It is even faster than >> what I got from pure memory bandwidth test. >> I'm putting the same memory block from the source rank to the same memory >> address on the destination rank followed by MPI_Win_flush to synchronize. >> Do I do it correctly? The source code is attached. >> >> Thanks >> Kun >> >> >> On Fri, Dec 21, 2018 at 11:15 AM Jeff Hammond <jeff.science(a)gmail.com> >> wrote: >> >>> Use MPI_Win_allocate instead of MPI_Win_create. MPI_Win_create cannot >>> allocate shared memory so you will not get good performance within a node. >>> >>> Jeff >>> >>> On Fri, Dec 21, 2018 at 8:18 AM Kun Feng via discuss <discuss(a)mpich.org> >>> wrote: >>> >>>> Hi all, >>>> >>>> I'm working on a project in which one half of the processes need to >>>> send data to the other half in each node. >>>> I'm using passive target mode of one-sided communication in which the >>>> receivers expose memory using MPI_Win_create, wait on MPI_Win_free and the >>>> senders send the data using MPI_Put. >>>> The code works. However, I get weird performance using this concurrent >>>> MPI_Put communication. The peak aggregate bandwidth is only around 5GB/s. >>>> It does not make sense as an aggregate performance in one single node. >>>> I thought the node-local communication is implemented as local memcpy. >>>> But concurrent memcpy on the same testbed has 4x to 5x higher aggregate >>>> bandwidth. >>>> Even concurrent memcpy using Linux shared memory across processes is 3x >>>> faster than my code. >>>> I'm using CH3 in MPICH 3.2.1. CH4 in MPICH 3.3 is even 2x slower. >>>> Does the performance make sense? Does MPICH has some queue for all >>>> one-sided communication in one node? Or do I understand it incorrectly? >>>> >>>> Thanks >>>> Kun >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>> >>> >>> -- >>> Jeff Hammond >>> jeff.science(a)gmail.com >>> http://jeffhammond.github.io/ >>> >> -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ >

1 0

Re: [mpich-discuss] Weird performance of locally concurrent MPI_Put
by Jeff Hammond 29 Dec '18

29 Dec '18

I don’t know why you are timing win_allocate. I’d only time lock-put-unlock or put-flush. Jeff On Sat, Dec 29, 2018 at 9:11 AM Kun Feng <kfeng1(a)hawk.iit.edu> wrote: > Thank you for the replies. > MPI_Win_allocate gives me much better performance. It is even faster than > what I got from pure memory bandwidth test. > I'm putting the same memory block from the source rank to the same memory > address on the destination rank followed by MPI_Win_flush to synchronize. > Do I do it correctly? The source code is attached. > > Thanks > Kun > > > On Fri, Dec 21, 2018 at 11:15 AM Jeff Hammond <jeff.science(a)gmail.com> > wrote: > >> Use MPI_Win_allocate instead of MPI_Win_create. MPI_Win_create cannot >> allocate shared memory so you will not get good performance within a node. >> >> Jeff >> >> On Fri, Dec 21, 2018 at 8:18 AM Kun Feng via discuss <discuss(a)mpich.org> >> wrote: >> >>> Hi all, >>> >>> I'm working on a project in which one half of the processes need to send >>> data to the other half in each node. >>> I'm using passive target mode of one-sided communication in which the >>> receivers expose memory using MPI_Win_create, wait on MPI_Win_free and the >>> senders send the data using MPI_Put. >>> The code works. However, I get weird performance using this concurrent >>> MPI_Put communication. The peak aggregate bandwidth is only around 5GB/s. >>> It does not make sense as an aggregate performance in one single node. >>> I thought the node-local communication is implemented as local memcpy. >>> But concurrent memcpy on the same testbed has 4x to 5x higher aggregate >>> bandwidth. >>> Even concurrent memcpy using Linux shared memory across processes is 3x >>> faster than my code. >>> I'm using CH3 in MPICH 3.2.1. CH4 in MPICH 3.3 is even 2x slower. >>> Does the performance make sense? Does MPICH has some queue for all >>> one-sided communication in one node? Or do I understand it incorrectly? >>> >>> Thanks >>> Kun >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> http://jeffhammond.github.io/ >> > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-discuss] Weird performance of locally concurrent MPI_Put
by Kun Feng 29 Dec '18

29 Dec '18

Thank you for the replies. MPI_Win_allocate gives me much better performance. It is even faster than what I got from pure memory bandwidth test. I'm putting the same memory block from the source rank to the same memory address on the destination rank followed by MPI_Win_flush to synchronize. Do I do it correctly? The source code is attached. Thanks Kun On Fri, Dec 21, 2018 at 11:15 AM Jeff Hammond <jeff.science(a)gmail.com> wrote: > Use MPI_Win_allocate instead of MPI_Win_create. MPI_Win_create cannot > allocate shared memory so you will not get good performance within a node. > > Jeff > > On Fri, Dec 21, 2018 at 8:18 AM Kun Feng via discuss <discuss(a)mpich.org> > wrote: > >> Hi all, >> >> I'm working on a project in which one half of the processes need to send >> data to the other half in each node. >> I'm using passive target mode of one-sided communication in which the >> receivers expose memory using MPI_Win_create, wait on MPI_Win_free and the >> senders send the data using MPI_Put. >> The code works. However, I get weird performance using this concurrent >> MPI_Put communication. The peak aggregate bandwidth is only around 5GB/s. >> It does not make sense as an aggregate performance in one single node. >> I thought the node-local communication is implemented as local memcpy. >> But concurrent memcpy on the same testbed has 4x to 5x higher aggregate >> bandwidth. >> Even concurrent memcpy using Linux shared memory across processes is 3x >> faster than my code. >> I'm using CH3 in MPICH 3.2.1. CH4 in MPICH 3.3 is even 2x slower. >> Does the performance make sense? Does MPICH has some queue for all >> one-sided communication in one node? Or do I understand it incorrectly? >> >> Thanks >> Kun >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ >

1 0

Re: [mpich-discuss] Weird performance of locally concurrent MPI_Put
by Jeff Hammond 21 Dec '18

21 Dec '18

Use MPI_Win_allocate instead of MPI_Win_create. MPI_Win_create cannot allocate shared memory so you will not get good performance within a node. Jeff On Fri, Dec 21, 2018 at 8:18 AM Kun Feng via discuss <discuss(a)mpich.org> wrote: > Hi all, > > I'm working on a project in which one half of the processes need to send > data to the other half in each node. > I'm using passive target mode of one-sided communication in which the > receivers expose memory using MPI_Win_create, wait on MPI_Win_free and the > senders send the data using MPI_Put. > The code works. However, I get weird performance using this concurrent > MPI_Put communication. The peak aggregate bandwidth is only around 5GB/s. > It does not make sense as an aggregate performance in one single node. > I thought the node-local communication is implemented as local memcpy. > But concurrent memcpy on the same testbed has 4x to 5x higher aggregate > bandwidth. > Even concurrent memcpy using Linux shared memory across processes is 3x > faster than my code. > I'm using CH3 in MPICH 3.2.1. CH4 in MPICH 3.3 is even 2x slower. > Does the performance make sense? Does MPICH has some queue for all > one-sided communication in one node? Or do I understand it incorrectly? > > Thanks > Kun > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-discuss] Weird performance of locally concurrent MPI_Put
by Si, Min 21 Dec '18

21 Dec '18

Hi Kun, Can you please try MPI_Win_allocate instead of MPI_Win_create in your program ? Single memory copy based MPI_Put is enabled only for MPI_Win_allocate in current MPICH. Best regards, Min On 2018/12/21 10:15, Kun Feng via discuss wrote: Hi all, I'm working on a project in which one half of the processes need to send data to the other half in each node. I'm using passive target mode of one-sided communication in which the receivers expose memory using MPI_Win_create, wait on MPI_Win_free and the senders send the data using MPI_Put. The code works. However, I get weird performance using this concurrent MPI_Put communication. The peak aggregate bandwidth is only around 5GB/s. It does not make sense as an aggregate performance in one single node. I thought the node-local communication is implemented as local memcpy. But concurrent memcpy on the same testbed has 4x to 5x higher aggregate bandwidth. Even concurrent memcpy using Linux shared memory across processes is 3x faster than my code. I'm using CH3 in MPICH 3.2.1. CH4 in MPICH 3.3 is even 2x slower. Does the performance make sense? Does MPICH has some queue for all one-sided communication in one node? Or do I understand it incorrectly? Thanks Kun _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Weird performance of locally concurrent MPI_Put
by Kun Feng 21 Dec '18

21 Dec '18

Hi all, I'm working on a project in which one half of the processes need to send data to the other half in each node. I'm using passive target mode of one-sided communication in which the receivers expose memory using MPI_Win_create, wait on MPI_Win_free and the senders send the data using MPI_Put. The code works. However, I get weird performance using this concurrent MPI_Put communication. The peak aggregate bandwidth is only around 5GB/s. It does not make sense as an aggregate performance in one single node. I thought the node-local communication is implemented as local memcpy. But concurrent memcpy on the same testbed has 4x to 5x higher aggregate bandwidth. Even concurrent memcpy using Linux shared memory across processes is 3x faster than my code. I'm using CH3 in MPICH 3.2.1. CH4 in MPICH 3.3 is even 2x slower. Does the performance make sense? Does MPICH has some queue for all one-sided communication in one node? Or do I understand it incorrectly? Thanks Kun

1 0

Re: [mpich-discuss] how to dynamically schedule process in mpich
by Shuwei Zhao 17 Dec '18

17 Dec '18

Hi, Zhou, Sorry for the late reply, I was busy with some customer requests. Yes my system is UGE similar to SGE. Your explanation makes a lot senses. I haven't had a chance to verify it. I'll try it later. Thank you a lot for your explanations. I have great learning experiences in the discussion forum. Wish you and your team member have a nice holiday! Thanks, Shuwei On Thu, Nov 29, 2018 at 5:17 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > What is your system? PBS? You have to prescribe a pool of nodes with qsub > and run mpiexec -n 1 (for example). Then when you launch additional mpi > process dynamically, hydra will spawn new processes within the prescribed > pool. If your original pool is a single host, then all new dynamic process > will be on that single host. If you describe multiple hosts, I believe > hydra will do round-robin style allocation, i.e. the next processes will go > to host 2, host 3, …., host N, host 1 (circle back). > > This is my current understanding. Let me know whether you can confirm. > > — > Hui Zhou > > > > On Nov 29, 2018, at 4:41 PM, Shuwei Zhao <shuweizhao1991(a)gmail.com> wrote: > > Hi Hui > > I cannot run mpiexec directly on my hosts because both rsh and ssh > connections are disabled. I have to use qsub -P bnormal -pe dp 4 mpiexec > ./my_exe to run my binary on remote hosts. > > Thanks > Shuwei > >> >> >

1 0

mpiexec tries to connect to one of my Ethernet connections
by Payman Shamlu 12 Dec '18

12 Dec '18

I use OpenVPN system on my pc, and while I want to use the mpiexec it run too slowly ( e.g. takes 1min to run hello world), but when I disable the ethernet connection that corresponds to the Openvpn, mpiexec works promptly. The problem is that I cant use both my OpenVPN and mpiexec at the same time every time I have to disable it. How can I be able to stop the mpiexec to connect to the OpenVpn?

1 0

Re: [mpich-discuss] MPI_Comm_spawn zombies have risen from the dead.
by Thomas Pak 11 Dec '18

11 Dec '18

Hi Hui, Sorry the late response. No problem! The application that I am developing is a tool that automates the parallelisation of scientific simulations as part of an advanced parameter sweeping algorithm. One of the key goals is to do this in a modular manner. I have decided to let the simulation be performed by an executable provided by the user, a so-called "simulator". This provides the most flexibility as the user can then write their simulator in any programming or scripting language. Moreover, this also implies that the user can make use of any software package or library. For regular non-MPI simulators, my application invokes the simulator with the system calls fork() and exec(). These types of simulator work perfectly fine with my application as it currently stands. However, when the simulator itself uses MPI, the fork-exec fails because an MPI application forking another MPI application is undefined and causes all kinds of troubles. Thus, I have resorted to using the native MPI_Comm_spawn function for dynamic process management for those cases. A possible counterpoint to all of this is that there is no use in running parallel instances of an MPI simulator because the simulator itself parallelised. However, this is not always the case because some simulation software packages are written using external libraries that use MPI internally, such as PETSc, despite the application itself not being parallelised. Even though this case might be somewhat niche, it is exactly the situation that I am in. In summary, my MPI application calls external executables provided by the user and runs many parallel instances of them. The external executable itself may be an MPI application, in which case MPI_Comm_spawn is necessary. In a way, the reason I need MPI_Comm_spawn to work is because MPI has become so ubiquitous in scientific computing that even non-parallelised simulators might be using MPI under the hood. I hope this helps. Best wishes, Thomas Pak On Dec 6 2018, at 4:00 pm, Zhou, Hui <zhouh(a)anl.gov> wrote: > Hi Thomas, > > Sure. While I am finding time looking into this, could you talk a bit more about your application? The dynamic MPI processes is still a new territory despite its presence for many years. One of the reasons is due to lack of exposure for such applications. > > — > Hui Zhou > > > > > On Dec 6, 2018, at 6:11 AM, Thomas Pak <thomas.pak(a)maths.ox.ac.uk (mailto:[email protected])> wrote: > > Hi Hui, > > That is exactly right, thanks so much for looking into it! Please let me know if you make any progress on this issue, as the application I am developing critically depends on MPI_Comm_spawn. > > Best wishes, > > Thomas Pak > > > > > >

1 0

Re: [mpich-discuss] symbol lookup error
by Congiu, Giuseppe 06 Dec '18

06 Dec '18

Hi Haider, I would suggest, if you are not already doing that, to build and install mpich yourself in your home (which hopefully resides on a network file system mounted on all the machines of the cluster). Please use the latest stable tarball distribution (you can find it at: https://www.mpich.org/downloads/) In the README you can find instructions on how to configure and build the library. Don’t forget to add your installation path to the PATH environment or else you will still be running with the system installation instead of your local one. Best, Giuseppe On Dec 5, 2018, at 11:47 PM, Haider Abbas via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Dear Giuseppe, You are right, node 4 and node 6 are not working properly. when I am running pw.x on these node, it gives ./pw.x -i relax2.in<http://relax2.in/> >out.out & [1] 8475 mpiuser@node4:~/qe-29-11/bin$ ./pw.x: symbol lookup error: ./pw.x: undefined symbol: mpi_initialized_ I have reinstalled gfortran and mpich but still facing the same problem. Could you please further guide me how to overcome this problem. regards Haider Abbas On Wed, Dec 5, 2018 at 9:17 PM Congiu, Giuseppe via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hello Haider, Are you compiling mpich and installing it in a global path accessible to all machines or you are using local installations? The missing symbol is a fortran symbol. The fact it is not there can mean mpich was not compiled with fortran support while your application is a fortran code. The fact that it works for a certain number of processes might mean some machine do have fortran support while other machines don’t. Can you provide some additional information about your cluster configuration? The MPICH version you are running and how you are configuring it? --Giuseppe On Dec 5, 2018, at 1:53 AM, Haider Abbas via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: mpi_initialized_ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0