Send discuss mailing list submissions to
        discuss@mpich.org

To subscribe or unsubscribe via the World Wide Web, visit
        https://lists.mpich.org/mailman/listinfo/discuss
or, via email, send a message with subject or body 'help' to
        discuss-request@mpich.org

You can reach the person managing the list at
        discuss-owner@mpich.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of discuss digest..."


Today's Topics:

   1.  Better alternatives of MPI_Allreduce() (hritikesh semwal)
   2. Re:  Better alternatives of MPI_Allreduce() (Benson Muite)
   3. Re:  Better alternatives of MPI_Allreduce() (Benson Muite)
   4. Re:  Better alternatives of MPI_Allreduce() (Joachim Protze)


----------------------------------------------------------------------

Message: 1
Date: Tue, 5 May 2020 11:08:20 +0530
From: hritikesh semwal <hritikesh.semwal@gmail.com>
To: discuss@mpich.org
Subject: [mpich-discuss] Better alternatives of MPI_Allreduce()
Message-ID:
        <CAA+35d1zgxVjcfH=jpio5rLm1FkjxMC3Bis-qB9D43qjfEogRw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hello all,

I am working on the development of a parallel CFD solver and I am using
MPI_Allreduce for the global summation of the local errors calculated on
all processes of a group and the summation is to be used by all the
processes. My concern is that MPI_Allreduce is taking almost 27-30% of the
total time used, which is a significant amount. So, I want to ask if anyone
can suggest me better alternative/s to replace MPI_Allreduce which can
reduce the time consumption.

Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mpich.org/pipermail/discuss/attachments/20200505/b2c8224a/attachment-0001.html>

------------------------------

Message: 2
Date: Tue, 05 May 2020 09:57:43 +0300
From: "Benson Muite" <benson_muite@emailplus.org>
To: discuss@mpich.org
Subject: Re: [mpich-discuss] Better alternatives of MPI_Allreduce()
Message-ID: <d3cfd641-1849-4ed8-a036-7db371ebb225@www.fastmail.com>
Content-Type: text/plain; charset="us-ascii"




On Tue, May 5, 2020, at 8:38 AM, hritikesh semwal via discuss wrote:
> Hello all,
>
> I am working on the development of a parallel CFD solver and I am using MPI_Allreduce for the global summation of the local errors calculated on all processes of a group and the summation is to be used by all the processes. My concern is that MPI_Allreduce is taking almost 27-30% of the total time used, which is a significant amount. So, I want to ask if anyone can suggest me better alternative/s to replace MPI_Allreduce which can reduce the time consumption.
>
> Thank you.
> _______________________________________________
> discuss mailing list discuss@mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss
>

Hi Hitesh,

What hardware are you running on and what is the interconnect?
Have you tried changing any of the MPI settings?
Can the reduction be done asynchronously?

Regards,
Benson
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mpich.org/pipermail/discuss/attachments/20200505/d81885c3/attachment-0001.html>

------------------------------

Message: 3
Date: Tue, 05 May 2020 09:59:37 +0300
From: "Benson Muite" <benson_muite@emailplus.org>
To: discuss@mpich.org
Subject: Re: [mpich-discuss] Better alternatives of MPI_Allreduce()
Message-ID: <5ed0cd5f-c68c-404b-9041-a5189dd5e0e2@www.fastmail.com>
Content-Type: text/plain; charset="us-ascii"




On Tue, May 5, 2020, at 9:57 AM, Benson Muite wrote:
>
>
>
> On Tue, May 5, 2020, at 8:38 AM, hritikesh semwal via discuss wrote:
>> Hello all,
>>
>> I am working on the development of a parallel CFD solver and I am using MPI_Allreduce for the global summation of the local errors calculated on all processes of a group and the summation is to be used by all the processes. My concern is that MPI_Allreduce is taking almost 27-30% of the total time used, which is a significant amount. So, I want to ask if anyone can suggest me better alternative/s to replace MPI_Allreduce which can reduce the time consumption.
>>
>> Thank you.
>> _______________________________________________
>> discuss mailing list discuss@mpich.org
>> To manage subscription options or unsubscribe:
>> https://lists.mpich.org/mailman/listinfo/discuss
>>
>
> Hi Hitesh,
>
> What hardware are you running on and what is the interconnect?

> Have you tried changing any of the MPI settings?

> Can the reduction be done asynchronously?

>
> Regards,
> Benson

Also, is your work load balanced? One way to check this might be to place a barrier just before the all-reduce call. If the barrier ends up taking most of your time, then it is likely you will need to determine a better way to distribute the computational work.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mpich.org/pipermail/discuss/attachments/20200505/f773f670/attachment-0001.html>

------------------------------

Message: 4
Date: Tue, 5 May 2020 09:00:20 +0200
From: Joachim Protze <protze@itc.rwth-aachen.de>
To: <discuss@mpich.org>
Cc: hritikesh semwal <hritikesh.semwal@gmail.com>
Subject: Re: [mpich-discuss] Better alternatives of MPI_Allreduce()
Message-ID: <1189df11-a53f-059d-8fb3-16887890a606@itc.rwth-aachen.de>
Content-Type: text/plain; charset="utf-8"; Format="flowed"

Hello,

it is important to understand, that most of the time you see is not the
cost of the allreduce, but the cost of synchronization (caused by load
imbalance).

You can do an easy experiment and add a barrier before the allreduce.
Then you will see the actual cost of the allreduce, while the cost of
synchronization will go into the barrier.

Now, think about dependencies in your algorithm: do you need the output
value immediately? Is this the same time, where you have the input value
ready?
-> otherwise use non-blocking communication and perform independent work
in between

In any case: fix your load imbalance (the root cause of synchronization
cost).

Best
Joachim

Am 05.05.20 um 07:38 schrieb hritikesh semwal via discuss:
> Hello all,
>
> I am working on the development of a parallel CFD solver and I am using
> MPI_Allreduce for the global summation of the local errors calculated on
> all processes of a group and the summation is to be used by all the
> processes. My concern is that MPI_Allreduce is taking almost 27-30% of
> the total time used, which is a significant amount. So, I want to ask if
> anyone can suggest me better alternative/s to replace MPI_Allreduce
> which can reduce the time consumption.
>
> Thank you.
>
> _______________________________________________
> discuss mailing list     discuss@mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss
>


--
Dipl.-Inf. Joachim Protze

IT Center
Group: High Performance Computing
Division: Computational Science and Engineering
RWTH Aachen University
Seffenter Weg 23
D 52074  Aachen (Germany)
Tel: +49 241 80- 24765
Fax: +49 241 80-624765
protze@itc.rwth-aachen.de
www.itc.rwth-aachen.de

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5327 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.mpich.org/pipermail/discuss/attachments/20200505/f09fd7f5/attachment.p7s>

------------------------------

Subject: Digest Footer

_______________________________________________
discuss mailing list
discuss@mpich.org
https://lists.mpich.org/mailman/listinfo/discuss


------------------------------

End of discuss Digest, Vol 91, Issue 3
**************************************