- discuss - mailman3.cels.anl.gov

Another integer overflow
by Joachim Protze 31 Jan '19

31 Jan '19

Hi all, we found another integer overflow when using MPI_Sendrecv_replace, see attached file. When the buffer size exceeds MAX_INT, a segmentation fault occurs. I could reproduce this with the MPICH 3.3a2 installation on my Ubuntu desktop. @Jeff: I could reproduce the segfault as well as with IntelMPI 2018 IntelMPI 2019 detects a message truncation Best Joachim -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpich-discuss] Possible integer-overflow for MPI_COMM_WORLD in MPI_Iprobe
by Jeff Hammond 21 Jan '19

21 Jan '19

I was able to reproduce with my own test ( https://github.com/jeffhammond/HPCInfo/blob/master/mpi/bugs/iprobe-overflow…) with Intel MPI 2019, so I will report that bug to the Intel MPI team. It should be easy enough for them to figure out if this bug is from MPICH or not. 2139000000 iterations, 627.143272 seconds 2140000000 iterations, 627.436206 seconds 2141000000 iterations, 627.729135 seconds 2142000000 iterations, 628.022049 seconds 2143000000 iterations, 628.315015 seconds 2144000000 iterations, 628.608066 seconds 2145000000 iterations, 628.901065 seconds 2146000000 iterations, 629.193992 seconds 2147000000 iterations, 629.488107 seconds Abort(738833413) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Iprobe: Invalid communicator, error stack: PMPI_Iprobe(123): MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, MPI_COMM_WORLD, flag=0x7ffdf75a396c, status=0x7ffdf75a3970) failed PMPI_Iprobe(90).: Invalid communicator Jeff, who works for Intel but knows more about MPICH than Intel MPI On Mon, Jan 21, 2019 at 11:19 AM Joachim Protze via discuss < discuss(a)mpich.org> wrote: > Hi all, > > we detected the behavior with IntelMPI 2019 (which is based on MPICH > 3.3). Reproducing it with MPICH-3.3 was not yet successful. But I fear, > that our built of MPICH just not uses the necessary code path / build > flags. > > When calling MPI_Iprobe with the same communicator for ~2^31 times > (which can take 10-30 minutes), the execution stops with: > > Abort(201962501) on node 0 (rank 0 in comm 0): Fatal error in > PMPI_Iprobe: Invalid communicator, error stack: > PMPI_Iprobe(123): MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, > MPI_COMM_WORLD, flag=0x7ffd925056c0, status=0x7ffd92505694) failed > PMPI_Iprobe(90).: Invalid communicator > > From my understanding of the referenced MPICH code lines, I guess, that > the ref-count for MPI_COMM_WORLD overflows, which triggers this error > message. > > Best > Joachim > > -- > Dipl.-Inf. Joachim Protze > > IT Center > Group: High Performance Computing > Division: Computational Science and Engineering > RWTH Aachen University > Seffenter Weg 23 > D 52074 Aachen (Germany) > Tel: +49 241 80- 24765 > Fax: +49 241 80-624765 > protze(a)itc.rwth-aachen.de > www.itc.rwth-aachen.de > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Possible integer-overflow for MPI_COMM_WORLD in MPI_Iprobe
by Joachim Protze 21 Jan '19

21 Jan '19

Hi all, we detected the behavior with IntelMPI 2019 (which is based on MPICH 3.3). Reproducing it with MPICH-3.3 was not yet successful. But I fear, that our built of MPICH just not uses the necessary code path / build flags. When calling MPI_Iprobe with the same communicator for ~2^31 times (which can take 10-30 minutes), the execution stops with: Abort(201962501) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Iprobe: Invalid communicator, error stack: PMPI_Iprobe(123): MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, MPI_COMM_WORLD, flag=0x7ffd925056c0, status=0x7ffd92505694) failed PMPI_Iprobe(90).: Invalid communicator From my understanding of the referenced MPICH code lines, I guess, that the ref-count for MPI_COMM_WORLD overflows, which triggers this error message. Best Joachim -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpich-discuss] Configuration options request
by Raffenetti, Kenneth J. 11 Jan '19

11 Jan '19

These options have to do with how MPI processes communicate with the process manager and its proxies. Typically, MPI processes communicate over a Unix socket with the process manager or proxy. --enable-allowport makes it so TCP/IP communication can also be used for this feature. This option only applies to the gforker and remshell process managers (--with-pm=gforker,remshell). --enable-pmiport applies to the Hydra process manager, but is conceptually the same. Hope this helps. Ken On 1/10/19 2:20 PM, Michael L. Stokes via discuss wrote: > Was looking through the configure help output for mpich-3.3 and ran > across the configure flags --enable-allowport and --enable-pmiport. Can > someone tell me the significance of using these options during the build? > > --Mike > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Configuration options request
by Michael.Stokes＠uah.edu 10 Jan '19

10 Jan '19

Was looking through the configure help output for mpich-3.3 and ran across the configure flags --enable-allowport and --enable-pmiport. Can someone tell me the significance of using these options during the build? --Mike

1 0

Re: [mpich-discuss] Weird performance of locally concurrent MPI_Put
by Jeff Hammond 03 Jan '19

03 Jan '19

I don't use RMA from threads much but I do not expect MPICH to have internal concurrency for bandwidth-limited RMA operations in shared-memory, so it doesn't surprise me that the bandwidth is not improved by multiple threads. Jeff On Mon, Dec 31, 2018 at 2:30 PM Kun Feng <kfeng1(a)hawk.iit.edu> wrote: > Good advice. > Now I find out that the performance drops to sequential again when I move > all the sources of MPI_Put into threads in one MPI rank even using > MPI_Win_allocate. > Do you have similar experience before? > > Thanks > Kun > > > On Sat, Dec 29, 2018 at 9:36 PM Jeff Hammond <jeff.science(a)gmail.com> > wrote: > >> I don’t know why you are timing win_allocate. I’d only time >> lock-put-unlock or put-flush. >> >> Jeff >> >> On Sat, Dec 29, 2018 at 9:11 AM Kun Feng <kfeng1(a)hawk.iit.edu> wrote: >> >>> Thank you for the replies. >>> MPI_Win_allocate gives me much better performance. It is even faster >>> than what I got from pure memory bandwidth test. >>> I'm putting the same memory block from the source rank to the same >>> memory address on the destination rank followed by MPI_Win_flush to >>> synchronize. >>> Do I do it correctly? The source code is attached. >>> >>> Thanks >>> Kun >>> >>> >>> On Fri, Dec 21, 2018 at 11:15 AM Jeff Hammond <jeff.science(a)gmail.com> >>> wrote: >>> >>>> Use MPI_Win_allocate instead of MPI_Win_create. MPI_Win_create cannot >>>> allocate shared memory so you will not get good performance within a node. >>>> >>>> Jeff >>>> >>>> On Fri, Dec 21, 2018 at 8:18 AM Kun Feng via discuss <discuss(a)mpich.org> >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I'm working on a project in which one half of the processes need to >>>>> send data to the other half in each node. >>>>> I'm using passive target mode of one-sided communication in which the >>>>> receivers expose memory using MPI_Win_create, wait on MPI_Win_free and the >>>>> senders send the data using MPI_Put. >>>>> The code works. However, I get weird performance using this concurrent >>>>> MPI_Put communication. The peak aggregate bandwidth is only around 5GB/s. >>>>> It does not make sense as an aggregate performance in one single node. >>>>> I thought the node-local communication is implemented as local memcpy. >>>>> But concurrent memcpy on the same testbed has 4x to 5x higher >>>>> aggregate bandwidth. >>>>> Even concurrent memcpy using Linux shared memory across processes is >>>>> 3x faster than my code. >>>>> I'm using CH3 in MPICH 3.2.1. CH4 in MPICH 3.3 is even 2x slower. >>>>> Does the performance make sense? Does MPICH has some queue for all >>>>> one-sided communication in one node? Or do I understand it incorrectly? >>>>> >>>>> Thanks >>>>> Kun >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> >>>> >>>> >>>> -- >>>> Jeff Hammond >>>> jeff.science(a)gmail.com >>>> http://jeffhammond.github.io/ >>>> >>> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> http://jeffhammond.github.io/ >> > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-discuss] Weird performance of locally concurrent MPI_Put
by Kun Feng 31 Dec '18

31 Dec '18

Good advice. Now I find out that the performance drops to sequential again when I move all the sources of MPI_Put into threads in one MPI rank even using MPI_Win_allocate. Do you have similar experience before? Thanks Kun On Sat, Dec 29, 2018 at 9:36 PM Jeff Hammond <jeff.science(a)gmail.com> wrote: > I don’t know why you are timing win_allocate. I’d only time > lock-put-unlock or put-flush. > > Jeff > > On Sat, Dec 29, 2018 at 9:11 AM Kun Feng <kfeng1(a)hawk.iit.edu> wrote: > >> Thank you for the replies. >> MPI_Win_allocate gives me much better performance. It is even faster than >> what I got from pure memory bandwidth test. >> I'm putting the same memory block from the source rank to the same memory >> address on the destination rank followed by MPI_Win_flush to synchronize. >> Do I do it correctly? The source code is attached. >> >> Thanks >> Kun >> >> >> On Fri, Dec 21, 2018 at 11:15 AM Jeff Hammond <jeff.science(a)gmail.com> >> wrote: >> >>> Use MPI_Win_allocate instead of MPI_Win_create. MPI_Win_create cannot >>> allocate shared memory so you will not get good performance within a node. >>> >>> Jeff >>> >>> On Fri, Dec 21, 2018 at 8:18 AM Kun Feng via discuss <discuss(a)mpich.org> >>> wrote: >>> >>>> Hi all, >>>> >>>> I'm working on a project in which one half of the processes need to >>>> send data to the other half in each node. >>>> I'm using passive target mode of one-sided communication in which the >>>> receivers expose memory using MPI_Win_create, wait on MPI_Win_free and the >>>> senders send the data using MPI_Put. >>>> The code works. However, I get weird performance using this concurrent >>>> MPI_Put communication. The peak aggregate bandwidth is only around 5GB/s. >>>> It does not make sense as an aggregate performance in one single node. >>>> I thought the node-local communication is implemented as local memcpy. >>>> But concurrent memcpy on the same testbed has 4x to 5x higher aggregate >>>> bandwidth. >>>> Even concurrent memcpy using Linux shared memory across processes is 3x >>>> faster than my code. >>>> I'm using CH3 in MPICH 3.2.1. CH4 in MPICH 3.3 is even 2x slower. >>>> Does the performance make sense? Does MPICH has some queue for all >>>> one-sided communication in one node? Or do I understand it incorrectly? >>>> >>>> Thanks >>>> Kun >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>> >>> >>> -- >>> Jeff Hammond >>> jeff.science(a)gmail.com >>> http://jeffhammond.github.io/ >>> >> -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ >

1 0

Re: [mpich-discuss] Weird performance of locally concurrent MPI_Put
by Jeff Hammond 29 Dec '18

29 Dec '18

I don’t know why you are timing win_allocate. I’d only time lock-put-unlock or put-flush. Jeff On Sat, Dec 29, 2018 at 9:11 AM Kun Feng <kfeng1(a)hawk.iit.edu> wrote: > Thank you for the replies. > MPI_Win_allocate gives me much better performance. It is even faster than > what I got from pure memory bandwidth test. > I'm putting the same memory block from the source rank to the same memory > address on the destination rank followed by MPI_Win_flush to synchronize. > Do I do it correctly? The source code is attached. > > Thanks > Kun > > > On Fri, Dec 21, 2018 at 11:15 AM Jeff Hammond <jeff.science(a)gmail.com> > wrote: > >> Use MPI_Win_allocate instead of MPI_Win_create. MPI_Win_create cannot >> allocate shared memory so you will not get good performance within a node. >> >> Jeff >> >> On Fri, Dec 21, 2018 at 8:18 AM Kun Feng via discuss <discuss(a)mpich.org> >> wrote: >> >>> Hi all, >>> >>> I'm working on a project in which one half of the processes need to send >>> data to the other half in each node. >>> I'm using passive target mode of one-sided communication in which the >>> receivers expose memory using MPI_Win_create, wait on MPI_Win_free and the >>> senders send the data using MPI_Put. >>> The code works. However, I get weird performance using this concurrent >>> MPI_Put communication. The peak aggregate bandwidth is only around 5GB/s. >>> It does not make sense as an aggregate performance in one single node. >>> I thought the node-local communication is implemented as local memcpy. >>> But concurrent memcpy on the same testbed has 4x to 5x higher aggregate >>> bandwidth. >>> Even concurrent memcpy using Linux shared memory across processes is 3x >>> faster than my code. >>> I'm using CH3 in MPICH 3.2.1. CH4 in MPICH 3.3 is even 2x slower. >>> Does the performance make sense? Does MPICH has some queue for all >>> one-sided communication in one node? Or do I understand it incorrectly? >>> >>> Thanks >>> Kun >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> http://jeffhammond.github.io/ >> > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-discuss] Weird performance of locally concurrent MPI_Put
by Kun Feng 29 Dec '18

29 Dec '18

Thank you for the replies. MPI_Win_allocate gives me much better performance. It is even faster than what I got from pure memory bandwidth test. I'm putting the same memory block from the source rank to the same memory address on the destination rank followed by MPI_Win_flush to synchronize. Do I do it correctly? The source code is attached. Thanks Kun On Fri, Dec 21, 2018 at 11:15 AM Jeff Hammond <jeff.science(a)gmail.com> wrote: > Use MPI_Win_allocate instead of MPI_Win_create. MPI_Win_create cannot > allocate shared memory so you will not get good performance within a node. > > Jeff > > On Fri, Dec 21, 2018 at 8:18 AM Kun Feng via discuss <discuss(a)mpich.org> > wrote: > >> Hi all, >> >> I'm working on a project in which one half of the processes need to send >> data to the other half in each node. >> I'm using passive target mode of one-sided communication in which the >> receivers expose memory using MPI_Win_create, wait on MPI_Win_free and the >> senders send the data using MPI_Put. >> The code works. However, I get weird performance using this concurrent >> MPI_Put communication. The peak aggregate bandwidth is only around 5GB/s. >> It does not make sense as an aggregate performance in one single node. >> I thought the node-local communication is implemented as local memcpy. >> But concurrent memcpy on the same testbed has 4x to 5x higher aggregate >> bandwidth. >> Even concurrent memcpy using Linux shared memory across processes is 3x >> faster than my code. >> I'm using CH3 in MPICH 3.2.1. CH4 in MPICH 3.3 is even 2x slower. >> Does the performance make sense? Does MPICH has some queue for all >> one-sided communication in one node? Or do I understand it incorrectly? >> >> Thanks >> Kun >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ >

1 0

Re: [mpich-discuss] Weird performance of locally concurrent MPI_Put
by Jeff Hammond 21 Dec '18

21 Dec '18

Use MPI_Win_allocate instead of MPI_Win_create. MPI_Win_create cannot allocate shared memory so you will not get good performance within a node. Jeff On Fri, Dec 21, 2018 at 8:18 AM Kun Feng via discuss <discuss(a)mpich.org> wrote: > Hi all, > > I'm working on a project in which one half of the processes need to send > data to the other half in each node. > I'm using passive target mode of one-sided communication in which the > receivers expose memory using MPI_Win_create, wait on MPI_Win_free and the > senders send the data using MPI_Put. > The code works. However, I get weird performance using this concurrent > MPI_Put communication. The peak aggregate bandwidth is only around 5GB/s. > It does not make sense as an aggregate performance in one single node. > I thought the node-local communication is implemented as local memcpy. > But concurrent memcpy on the same testbed has 4x to 5x higher aggregate > bandwidth. > Even concurrent memcpy using Linux shared memory across processes is 3x > faster than my code. > I'm using CH3 in MPICH 3.2.1. CH4 in MPICH 3.3 is even 2x slower. > Does the performance make sense? Does MPICH has some queue for all > one-sided communication in one node? Or do I understand it incorrectly? > > Thanks > Kun > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0