discuss
Threads by month
- ----- 2026 -----
- July
- June
- May
- April
- March
- February
- January
- ----- 2025 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2024 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2023 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2022 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2021 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2020 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2019 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2018 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2017 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2016 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2015 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2014 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2013 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2012 -----
- December
- November
June 2026
- 3 participants
- 30 discussions
loads and stores in the MPI_Win_lock_all epochs using MPI_Fetch_and_op (see
attached files).
This version behaves very similar to the original code and also fails from
time to time. Putting a sleep into the acquire busy loop (usleep(100)) will
make the code "much more robust" (I hack, I know, but indicating some
underlying race condition?!). Let me know if you see any problems in the
way I am using MPI_Fetch_and_op in a busy loop. Flushing or syncing is not
necessary in this case, right?
All work is done with export MPIR_CVAR_ASYNC_PROGRESS=1 on mpich-3.2 and
mpich-3.3a2
On Wed, Mar 8, 2017 at 4: 21 PM, Halim Amer <aamer(a)anl.gov> wrote:
> I cannot claim that I thoroughly verified the correctness of that code, so
> take it with a grain of salt. Please keep in mind that it is a test code
> from a tutorial book; those codes are meant for learning purposes not for
> deployment.
>
> If your goal is to have a high performance RMA lock, I suggest you to look
> into the recent HPDC'16 paper: "High-Performance Distributed RMA Locks".
>
> Halim
> www.mcs.anl.gov/~aamer
>
> On 3/8/17 3: 06 AM, Ask Jakobsen wrote:
>
>> You are absolutely correct, Halim. Removing the test lmem[nextRank] == -1
>> in release fixes the problem. Great work. Now I will try to understand why
>> you are right. I hope the authors of the book will credit you for
>> discovering the bug.
>>
>> So in conclusion you need to remove the above mentioned test AND enable
>> asynchronous progression using the environment variable
>> MPIR_CVAR_ASYNC_PROGRESS=1 in MPICH (BTW I still can't get the code to
>> work
>> in openmpi).
>>
>> On Tue, Mar 7, 2017 at 5: 37 PM, Halim Amer <aamer(a)anl.gov> wrote:
>>
>> detect that another process is being or already enqueued in the MCS
>>>>
>>> queue.
>>>
>>> Actually the problem occurs only when the waiting process already
>>> enqueued
>>> itself, i.e., the accumulate operation on the nextRank field succeeded.
>>>
>>> Halim
>>> www.mcs.anl.gov/~aamer <http: //www.mcs.anl.gov/%7Eaamer>
>>>
>>>
>>> On 3/7/17 10: 29 AM, Halim Amer wrote:
>>>
>>> In the Release protocol, try removing this test:
>>>>
>>>> if (lmem[nextRank] == -1) {
>>>> If-Block;
>>>> }
>>>>
>>>> but keep the If-Block.
>>>>
>>>> The hang occurs because the process releasing the MCS lock fails to
>>>> detect that another process is being or already enqueued in the MCS
>>>> queue.
>>>>
>>>> Halim
>>>> www.mcs.anl.gov/~aamer <http: //www.mcs.anl.gov/%7Eaamer>
>>>>
>>>>
>>>> On 3/7/17 6: 43 AM, Ask Jakobsen wrote:
>>>>
>>>> Thanks, Halim. I have now enabled asynchronous progress in MPICH (can't
>>>>> find something similar in openmpi) and now all ranks acquire the lock
>>>>> and
>>>>> the program finish as expected. However if I put a while(1) loop
>>>>> around the
>>>>> acquire-release code in main.c it will fail again at random and go
>>>>> into an
>>>>> infinite loop. The simple unfair lock does not have this problem.
>>>>>
>>>>> On Tue, Mar 7, 2017 at 12: 44 AM, Halim Amer <aamer(a)anl.gov> wrote:
>>>>>
>>>>> My understanding is that this code assumes asynchronous progress.
>>>>>
>>>>>> An example of when the processes hang is as follows:
>>>>>>
>>>>>> 1) P0 Finishes MCSLockAcquire()
>>>>>> 2) P1 is busy waiting in MCSLockAcquire() at
>>>>>> do {
>>>>>> MPI_Win_sync(win);
>>>>>> } while (lmem[blocked] == 1);
>>>>>> 3) P0 executes MCSLockRelease()
>>>>>> 4) P0 waits on MPI_Win_lock_all() inside MCSLockRlease()
>>>>>>
>>>>>> Hang!
>>>>>>
>>>>>> For P1 to get out of the loop, P0 has to get out of
>>>>>> MPI_Win_lock_all() and
>>>>>> executes its Compare_and_swap().
>>>>>>
>>>>>> For P0 to get out MPI_Win_lock_all(), it needs an ACK from P1 that it
>>>>>> got
>>>>>> the lock.
>>>>>>
>>>>>> P1 does not make communication progress because MPI_Win_sync is not
>>>>>> required to do so. It only synchronizes private and public copies.
>>>>>>
>>>>>> For this hang to disappear, one can either trigger progress manually
>>>>>> by
>>>>>> using heavy-duty synchronization calls instead of Win_sync (e.g.,
>>>>>> Win_unlock_all + Win_lock_all), or enable asynchronous progress.
>>>>>>
>>>>>> To enable asynchronous progress in MPICH, set the
>>>>>> MPIR_CVAR_ASYNC_PROGRESS
>>>>>> env var to 1.
>>>>>>
>>>>>> Halim
>>>>>> www.mcs.anl.gov/~aamer <http: //www.mcs.anl.gov/%7Eaamer> <
>>>>>> http: //www.mcs.anl.gov/%7Eaamer>
>>>>>>
>>>>>>
>>>>>> On 3/6/17 1: 11 PM, Ask Jakobsen wrote:
>>>>>>
>>>>>> I am testing on x86_64 platform.
>>>>>>
>>>>>>>
>>>>>>> I have tried to built both the mpich and the mcs lock code with -O0
>>>>>>> to
>>>>>>> avoid agressive optimization. After your suggestion I have also
>>>>>>> tried to
>>>>>>> make volatile int *pblocked pointing to lmem[blocked] in the
>>>>>>> MCSLockAcquire
>>>>>>> function and volatile int *pnextrank pointing to lmem[nextRank] in
>>>>>>> MCSLockRelease, but it does not appear to make a difference.
>>>>>>>
>>>>>>> On suggestion from Richard Warren I have also tried building the code
>>>>>>> using
>>>>>>> openmpi-2.0.2 without any luck (however it appears to acquire the
>>>>>>> lock a
>>>>>>> couple of extra times before failing) which I find troubling.
>>>>>>>
>>>>>>> I think I will give up using local load/stores and will see if I can
>>>>>>> figure
>>>>>>> out if rewrite using MPI calls like MPI_Fetch_and_op as you suggest.
>>>>>>> Thanks for your help.
>>>>>>>
>>>>>>> On Mon, Mar 6, 2017 at 7: 20 PM, Jeff Hammond <jeff.science(a)gmail.com
>>>>>>> >
>>>>>>> wrote:
>>>>>>>
>>>>>>> What processor architecture are you testing?
>>>>>>>
>>>>>>>
>>>>>>>> Maybe set lmem to volatile or read it with MPI_Fetch_and_op rather
>>>>>>>> than a
>>>>>>>> load. MPI_Win_sync cannot prevent the compiler from caching *lmem
>>>>>>>> in a
>>>>>>>> register.
>>>>>>>>
>>>>>>>> Jeff
>>>>>>>>
>>>>>>>> On Sat, Mar 4, 2017 at 12: 30 AM, Ask Jakobsen <afj(a)qeye-labs.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>>
>>>>>>>>> I have downloaded the source code for the MCS lock from the
>>>>>>>>> excellent
>>>>>>>>> book "Using Advanced MPI" from http: //www.mcs.anl.gov/researc
>>>>>>>>> h/projects/mpi/usingmpi/examples-advmpi/rma2/mcs-lock.c
>>>>>>>>>
>>>>>>>>> I have made a very simple piece of test code for testing the MCS
>>>>>>>>> lock
>>>>>>>>> but
>>>>>>>>> it works at random and often never escapes the busy loops in the
>>>>>>>>> acquire
>>>>>>>>> and release functions (see attached source code). The code appears
>>>>>>>>> semantically correct to my eyes.
>>>>>>>>>
>>>>>>>>> #include <stdio.h>
>>>>>>>>> #include <mpi.h>
>>>>>>>>> #include "mcs-lock.h"
>>>>>>>>>
>>>>>>>>> int main(int argc, char *argv[])
>>>>>>>>> {
>>>>>>>>> MPI_Win win;
>>>>>>>>> MPI_Init( &argc, &argv );
>>>>>>>>>
>>>>>>>>> MCSLockInit(MPI_COMM_WORLD, &win);
>>>>>>>>>
>>>>>>>>> int rank, size;
>>>>>>>>> MPI_Comm_rank(MPI_COMM_WORLD, &rank);
>>>>>>>>> MPI_Comm_size(MPI_COMM_WORLD, &size);
>>>>>>>>>
>>>>>>>>> printf("rank: %d, size: %d\n", rank, size);
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> MCSLockAcquire(win);
>>>>>>>>> printf("rank %d aquired lock\n", rank); fflush(stdout);
>>>>>>>>> MCSLockRelease(win);
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> MPI_Win_free(&win);
>>>>>>>>> MPI_Finalize();
>>>>>>>>> return 0;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have tested on several hardware platforms and mpich-3.2 and
>>>>>>>>> mpich-3.3a2
>>>>>>>>> but with no luck.
>>>>>>>>>
>>>>>>>>> It appears that the MPI_Win_Sync are not "refreshing" the local
>>>>>>>>> data or
>>>>>>>>> I
>>>>>>>>> have a bug I can't spot.
>>>>>>>>>
>>>>>>>>> A simple unfair lock like http: //www.mcs.anl.gov/researc
>>>>>>>>> h/projects/mpi/usingmpi/examples-advmpi/rma2/ga_mutex1.c works
>>>>>>>>> perfectly.
>>>>>>>>>
>>>>>>>>> Best regards, Ask Jakobsen
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> discuss mailing list discuss(a)mpich.org
>>>>>>>>> To manage subscription options or unsubscribe:
>>>>>>>>> https: //lists.mpich.org/mailman/listinfo/discuss
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>> Jeff Hammond
>>>>>>>> jeff.science(a)gmail.com
>>>>>>>> http: //jeffhammond.github.io/
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> discuss mailing list discuss(a)mpich.org
>>>>>>>> To manage subscription options or unsubscribe:
>>>>>>>> https: //lists.mpich.org/mailman/listinfo/discuss
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> discuss mailing list discuss(a)mpich.org
>>>>>>> To manage subscription options or unsubscribe:
>>>>>>> https: //lists.mpich.org/mailman/listinfo/discuss
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>>
>>>>>>> discuss mailing list discuss(a)mpich.org
>>>>>> To manage subscription options or unsubscribe:
>>>>>> https: //lists.mpich.org/mailman/listinfo/discuss
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> discuss mailing list discuss(a)mpich.org
>>>>> To manage subscription options or unsubscribe:
>>>>> https: //lists.mpich.org/mailman/listinfo/discuss
>>>>>
>>>>> _______________________________________________
>>>>>
>>>> discuss mailing list discuss(a)mpich.org
>>>> To manage subscription options or unsubscribe:
>>>> https: //lists.mpich.org/mailman/listinfo/discuss
>>>>
>>>> _______________________________________________
>>> discuss mailing list discuss(a)mpich.org
>>> To manage subscription options or unsubscribe:
>>> https: //lists.mpich.org/mailman/listinfo/discuss
>>>
>>>
>>
>>
>> _______________________________________________
>> discuss mailing list discuss(a)mpich.org
>> To manage subscription options or unsubscribe:
>> https: //lists.mpich.org/mailman/listinfo/discuss
>>
>> _______________________________________________
> discuss mailing list discuss(a)mpich.org
> To manage subscription options or unsubscribe:
> https: //lists.mpich.org/mailman/listinfo/discuss
>
--001a1141dbd03846c4054a8dd443
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dutf-8"><d=
iv dir=3D"ltr">Interestingly, according to the paper you suggested it appea=
rs to include a similar test in pseudo code <a href=3D"https: //htor.inf.eth=
z.ch/publications/img/hpclocks.pdf" target=3D"_blank">https: //htor.inf.ethz=
.ch/publi<wbr>cations/img/hpclocks.pdf</a> (see Listing 3 in paper).<div><b=
r></div><div>Unfortunately, removing the test in the release protocol did n=
ot solve the problem. The race condition is much more difficult to provoke,=
but I managed when setting the size of the communicator to 3 (only tested =
even sizes so far).</div><div><br></div><div>From Jeff's suggestion I have =
attempted to rewrite the code removing local loads and stores in the MPI_Wi=
n_lock_all epochs using MPI_Fetch_and_op (see attached files).</div><d=
iv><br></div><div>This version behaves very similar to the original code an=
d also fails from time to time. Putting a sleep into the acquire busy loop =
(usleep(100)) will make the code "much more robust" (I hack, I kn=
ow, but indicating some underlying race condition?!). Let me know if you se=
e any problems in the way I am using MPI_Fetch_and_op in a busy loop. Flush=
ing or syncing is not necessary in this case, right?</div>
<div><br></div>All work is done with export MPIR_CVAR_ASYNC_PROGRESS=3D1 on=
mpich-3.2 and mpich-3.3a2<div><div><div class=3D"gmail_extra"><br></div><d=
iv class=3D"gmail_extra"><div class=3D"gmail_quote">On Wed, Mar 8, 2017 at =
4: 21 PM, Halim Amer <span dir=3D"ltr"><<a href=3D"mailto:[email protected]" =
target=3D"_blank">aamer(a)anl.gov</a>></span> wrote: <br><blockquote class=
=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex;border-left:1px solid rg=
b(204,204,204);padding-left: 1ex">I cannot claim that I thoroughly verified =
the correctness of that code, so take it with a grain of salt. Please keep =
in mind that it is a test code from a tutorial book; those codes are meant =
for learning purposes not for deployment.<br>
<br>
If your goal is to have a high performance RMA lock, I suggest you to look =
into the recent HPDC'16 paper: "High-Performance Distributed RMA Locks=
".<br>
<br>
Halim<br>
<a href=3D"http: //www.mcs.anl.gov/~aamer" rel=3D"noreferrer" target=3D"_bla=
nk">www.mcs.anl.gov/~aamer</a><span class=3D"gmail-m_8000249642949159004gma=
il-m_1322166946455689838gmail-"><br>
<br>
On 3/8/17 3: 06 AM, Ask Jakobsen wrote:<br>
</span><blockquote class=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex;=
border-left: 1px solid rgb(204,204,204);padding-left:1ex"><span class=3D"gma=
il-m_8000249642949159004gmail-m_1322166946455689838gmail-">
You are absolutely correct, Halim. Removing the test lmem[nextRank] =3D=3D =
-1<br>
in release fixes the problem. Great work. Now I will try to understand why<=
br>
you are right. I hope the authors of the book will credit you for<br>
discovering the bug.<br>
<br>
So in conclusion you need to remove the above mentioned test AND enable<br>
asynchronous progression using the environment variable<br>
MPIR_CVAR_ASYNC_PROGRESS=3D1 in MPICH (BTW I still can't get the code to wo=
rk<br>
in openmpi).<br>
<br>
On Tue, Mar 7, 2017 at 5: 37 PM, Halim Amer <<a href=3D"mailto:aamer@anl.=
gov" target=3D"_blank">aamer(a)anl.gov</a>> wrote: <br>
<br>
</span><blockquote class=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex;=
border-left: 1px solid rgb(204,204,204);padding-left:1ex"><span class=3D"gma=
il-m_8000249642949159004gmail-m_1322166946455689838gmail-"><blockquote clas=
s=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex;border-left:1px solid r=
gb(204,204,204);padding-left: 1ex">
detect that another process is being or already enqueued in the MCS<br>
</blockquote>
queue.<br>
<br>
Actually the problem occurs only when the waiting process already enqueued<=
br>
itself, i.e., the accumulate operation on the nextRank field succeeded.<br>
<br>
Halim<br>
</span><a href=3D"http: //www.mcs.anl.gov/~aamer" rel=3D"noreferrer" target=
=3D"_blank">www.mcs.anl.gov/~aamer</a> <<a href=3D"http: //www.mcs.anl.go=
v/%7Eaamer" rel=3D"noreferrer" target=3D"_blank">http: //www.mcs.anl.gov/%7E=
aam<wbr>er</a>><span class=3D"gmail-m_8000249642949159004gmail-m_1322166=
946455689838gmail-"><br>
<br>
<br>
On 3/7/17 10: 29 AM, Halim Amer wrote:<br>
<br>
</span><blockquote class=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex;=
border-left: 1px solid rgb(204,204,204);padding-left:1ex"><span class=3D"gma=
il-m_8000249642949159004gmail-m_1322166946455689838gmail-">
In the Release protocol, try removing this test: <br>
<br>
if (lmem[nextRank] =3D=3D -1) {<br>
If-Block;<br>
}<br>
<br>
but keep the If-Block.<br>
<br>
The hang occurs because the process releasing the MCS lock fails to<br>
detect that another process is being or already enqueued in the MCS queue.<=
br>
<br>
Halim<br>
</span><a href=3D"http: //www.mcs.anl.gov/~aamer" rel=3D"noreferrer" target=
=3D"_blank">www.mcs.anl.gov/~aamer</a> <<a href=3D"http: //www.mcs.anl.go=
v/%7Eaamer" rel=3D"noreferrer" target=3D"_blank">http: //www.mcs.anl.gov/%7E=
aam<wbr>er</a>><div><div class=3D"gmail-m_8000249642949159004gmail-m_132=
2166946455689838gmail-h5"><br>
<br>
On 3/7/17 6: 43 AM, Ask Jakobsen wrote:<br>
<br>
<blockquote class=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex;border-=
left: 1px solid rgb(204,204,204);padding-left:1ex">
Thanks, Halim. I have now enabled asynchronous progress in MPICH (can't<br>
find something similar in openmpi) and now all ranks acquire the lock and<b=
r>
the program finish as expected. However if I put a while(1) loop<br>
around the<br>
acquire-release code in main.c it will fail again at random and go<br>
into an<br>
infinite loop. The simple unfair lock does not have this problem.<br>
<br>
On Tue, Mar 7, 2017 at 12: 44 AM, Halim Amer <<a href=3D"mailto:aamer@anl=
.gov" target=3D"_blank">aamer(a)anl.gov</a>> wrote: <br>
<br>
My understanding is that this code assumes asynchronous progress.<br>
<blockquote class=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex;border-=
left: 1px solid rgb(204,204,204);padding-left:1ex">
An example of when the processes hang is as follows: <br>
<br>
1) P0 Finishes MCSLockAcquire()<br>
2) P1 is busy waiting in MCSLockAcquire() at<br>
do {<br>
MPI_Win_sync(win);<br>
} while (lmem[blocked] =3D=3D 1);<br>
3) P0 executes MCSLockRelease()<br>
4) P0 waits on MPI_Win_lock_all() inside MCSLockRlease()<br>
<br>
Hang!<br>
<br>
For P1 to get out of the loop, P0 has to get out of<br>
MPI_Win_lock_all() and<br>
executes its Compare_and_swap().<br>
<br>
For P0 to get out MPI_Win_lock_all(), it needs an ACK from P1 that it<br>
got<br>
the lock.<br>
<br>
P1 does not make communication progress because MPI_Win_sync is not<br>
required to do so. It only synchronizes private and public copies.<br>
<br>
For this hang to disappear, one can either trigger progress manually by<br>
using heavy-duty synchronization calls instead of Win_sync (e.g.,<br>
Win_unlock_all + Win_lock_all), or enable asynchronous progress.<br>
<br>
To enable asynchronous progress in MPICH, set the<br>
MPIR_CVAR_ASYNC_PROGRESS<br>
env var to 1.<br>
<br>
Halim<br>
<a href=3D"http: //www.mcs.anl.gov/~aamer" rel=3D"noreferrer" target=3D"_bla=
nk">www.mcs.anl.gov/~aamer</a> <<a href=3D"http: //www.mcs.anl.gov/%7Eaam=
er" rel=3D"noreferrer" target=3D"_blank">http: //www.mcs.anl.gov/%7Eaam<wbr>=
er</a>> <<br>
<a href=3D"http: //www.mcs.anl.gov/%7Eaamer" rel=3D"noreferrer" target=3D"_b=
lank">http: //www.mcs.anl.gov/%7Eaame<wbr>r</a>><br>
<br>
<br>
On 3/6/17 1: 11 PM, Ask Jakobsen wrote:<br>
<br>
I am testing on x86_64 platform.<br>
<blockquote class=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex;border-=
left: 1px solid rgb(204,204,204);padding-left:1ex">
<br>
I have tried to built both the mpich and the mcs lock code with -O0 to<br>
avoid agressive optimization. After your suggestion I have also<br>
tried to<br>
make volatile int *pblocked pointing to lmem[blocked] in the<br>
MCSLockAcquire<br>
function and volatile int *pnextrank pointing to lmem[nextRank] in<br>
MCSLockRelease, but it does not appear to make a difference.<br>
<br>
On suggestion from Richard Warren I have also tried building the code<br>
using<br>
openmpi-2.0.2 without any luck (however it appears to acquire the<br>
lock a<br>
couple of extra times before failing) which I find troubling.<br>
<br>
I think I will give up using local load/stores and will see if I can<br>
figure<br>
out if rewrite using MPI calls like MPI_Fetch_and_op as you suggest.<=
br>
Thanks for your help.<br>
<br>
On Mon, Mar 6, 2017 at 7: 20 PM, Jeff Hammond <<a href=3D"mailto:jeff.sci=
ence(a)gmail.com" target=3D"_blank">jeff.science(a)gmail.com</a>><br>
wrote: <br>
<br>
What processor architecture are you testing?<br>
<br>
<blockquote class=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex;border-=
left: 1px solid rgb(204,204,204);padding-left:1ex">
<br>
Maybe set lmem to volatile or read it with MPI_Fetch_and_op rather<br>
than a<br>
load. MPI_Win_sync cannot prevent the compiler from caching *lmem<br>
in a<br>
register.<br>
<br>
Jeff<br>
<br>
On Sat, Mar 4, 2017 at 12: 30 AM, Ask Jakobsen <<a href=3D"mailto:afj@qey=
e-labs.com" target=3D"_blank">afj(a)qeye-labs.com</a>><br>
wrote: <br>
<br>
Hi,<br>
<br>
<blockquote class=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex;border-=
left: 1px solid rgb(204,204,204);padding-left:1ex">
<br>
I have downloaded the source code for the MCS lock from the excellent<br>
book "Using Advanced MPI" from <a href=3D"http: //www.mcs.anl.gov/=
researc" rel=3D"noreferrer" target=3D"_blank">http: //www.mcs.anl.gov/resear=
c</a><br>
h/projects/mpi/usingmpi/exampl<wbr>es-advmpi/rma2/mcs-lock.c<br>
<br>
I have made a very simple piece of test code for testing the MCS lock<br>
but<br>
it works at random and often never escapes the busy loops in the<br>
acquire<br>
and release functions (see attached source code). The code appears<br>
semantically correct to my eyes.<br>
<br>
#include <stdio.h><br>
#include <mpi.h><br>
#include "mcs-lock.h"<br>
<br>
int main(int argc, char *argv[])<br>
{<br>
MPI_Win win;<br>
MPI_Init( &argc, &argv );<br>
<br>
MCSLockInit(MPI_COMM_WORLD, &win);<br>
<br>
int rank, size;<br>
MPI_Comm_rank(MPI_COMM_WORLD, &rank);<br>
MPI_Comm_size(MPI_COMM_WORLD, &size);<br>
<br>
printf("rank: %d, size: %d\n", rank, size);<br>
<br>
<br>
MCSLockAcquire(win);<br>
printf("rank %d aquired lock\n", rank); fflush=
(stdout);<br>
MCSLockRelease(win);<br>
<br>
<br>
MPI_Win_free(&win);<br>
MPI_Finalize();<br>
return 0;<br>
}<br>
<br>
<br>
I have tested on several hardware platforms and mpich-3.2 and<br>
mpich-3.3a2<br>
but with no luck.<br>
<br>
It appears that the MPI_Win_Sync are not "refreshing" the local<b=
r>
data or<br>
I<br>
have a bug I can't spot.<br>
<br>
A simple unfair lock like <a href=3D"http: //www.mcs.anl.gov/researc" rel=3D=
"noreferrer" target=3D"_blank">http: //www.mcs.anl.gov/researc</a><br>
h/projects/mpi/usingmpi/exampl<wbr>es-advmpi/rma2/ga_mutex1.c works<br>
perfectly.<br>
<br>
Best regards, Ask Jakobsen<br>
<br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list <a href=3D"mailto: discuss(a)mpich.org=
" target=3D"_blank">discuss(a)mpich.org</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https: //lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
<br>
<br>
</blockquote>
<br>
--<br>
Jeff Hammond<br>
<a href=3D"mailto: jeff.science(a)gmail.com" target=3D"_blank">jeff.science@gm=
ail.com</a><br>
<a href=3D"http: //jeffhammond.github.io/" rel=3D"noreferrer" target=3D"_bla=
nk">http: //jeffhammond.github.io/</a><br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list <a href=3D"mailto: discuss(a)mpich.org=
" target=3D"_blank">discuss(a)mpich.org</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https: //lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
<br>
<br>
</blockquote>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list <a href=3D"mailto: discuss(a)mpich.org=
" target=3D"_blank">discuss(a)mpich.org</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https: //lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
______________________________<wbr>_________________<br>
<br>
</blockquote>
discuss mailing list <a href=3D"mailto: discuss(a)mpich.org=
" target=3D"_blank">discuss(a)mpich.org</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https: //lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
<br>
</blockquote>
<br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list <a href=3D"mailto: discuss(a)mpich.org=
" target=3D"_blank">discuss(a)mpich.org</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https: //lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
______________________________<wbr>_________________<br>
</blockquote>
discuss mailing list <a href=3D"mailto: discuss(a)mpich.org=
" target=3D"_blank">discuss(a)mpich.org</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https: //lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
</div></div></blockquote><div><div class=3D"gmail-m_8000249642949159004gmai=
l-m_1322166946455689838gmail-h5">
______________________________<wbr>_________________<br>
discuss mailing list <a href=3D"mailto: discuss(a)mpich.org=
" target=3D"_blank">discuss(a)mpich.org</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https: //lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
</div></div></blockquote><div><div class=3D"gmail-m_8000249642949159004gmai=
l-m_1322166946455689838gmail-h5">
<br>
<br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list <a href=3D"mailto: discuss(a)mpich.org=
" target=3D"_blank">discuss(a)mpich.org</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https: //lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
</div></div></blockquote><div class=3D"gmail-m_8000249642949159004gmail-m_1=
322166946455689838gmail-HOEnZb"><div class=3D"gmail-m_8000249642949159004gm=
ail-m_1322166946455689838gmail-h5">
______________________________<wbr>_________________<br>
discuss mailing list <a href=3D"mailto: discuss(a)mpich.org=
" target=3D"_blank">discuss(a)mpich.org</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https: //lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a></div></div></blockquote></div>
</div></div></div></div>
--001a1141dbd03846c4054a8dd443--
--001a1141dbd03846ca054a8dd445
Content-Type: text/x-csrc; charset="US-ASCII"; name="main.c"
Content-Disposition: attachment; filename="main.c"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_j0732una0
I2luY2x1ZGUgPHN0ZGlvLmg+CiNpbmNsdWRlIDxtcGkuaD4KI2luY2x1ZGUgIm1jcy1sb2NrLmgi
CgppbnQgbWFpbihpbnQgYXJnYywgY2hhciAqYXJndltdKQp7CiAgTVBJX1dpbiB3aW47CiAgTVBJ
X0luaXQoICZhcmdjLCAmYXJndiApOwoKICBNQ1NMb2NrSW5pdChNUElfQ09NTV9XT1JMRCwgJndp
bik7CgogIGludCByYW5rLCBzaXplOwogIE1QSV9Db21tX3JhbmsoTVBJX0NPTU1fV09STEQsICZy
YW5rKTsKICBNUElfQ29tbV9zaXplKE1QSV9DT01NX1dPUkxELCAmc2l6ZSk7CgogIHByaW50Zigi
cmFuazogJWQsIHNpemU6ICVkXG4iLCByYW5rLCBzaXplKTsKCiAgaW50IGNvdW50ID0gMDsKICB3
aGlsZSgxKSB7CiAgICBNQ1NMb2NrQWNxdWlyZSh3aW4pOwogICAgLy9wcmludGYoInJhbmsgJWQg
YXF1aXJlZCBsb2NrXG4iLCByYW5rKTsgZmZsdXNoKHN0ZG91dCk7CiAgICBNQ1NMb2NrUmVsZWFz
ZSh3aW4pOwogICAgaWYgKGNvdW50PjEwMDApCiAgICAgIGJyZWFrOwogICAgY291bnQrKzsKICB9
CgogIE1QSV9XaW5fZnJlZSgmd2luKTsgCiAgTVBJX0ZpbmFsaXplKCk7CiAgcmV0dXJuIDA7Cn0K
--001a1141dbd03846ca054a8dd445
Content-Type: text/x-csrc; charset="US-ASCII"; name="mcs-lock-fop.c"
Content-Disposition: attachment; filename="mcs-lock-fop.c"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_j0732unw1
I2luY2x1ZGUgPHVuaXN0ZC5oPgoKI2luY2x1ZGUgIm1waS5oIgoKc3RhdGljIGludCBNQ1NfTE9D
S1JBTksgPSBNUElfS0VZVkFMX0lOVkFMSUQ7CmVudW0geyBuZXh0UmFuaz0wLCBibG9ja2VkPTEs
IGxvY2tUYWlsPTIgfTsKCnZvaWQgTUNTTG9ja0luaXQoTVBJX0NvbW0gY29tbSwgTVBJX1dpbiAq
d2luKQp7CiAgaW50ICAgICAgKmxtZW0sIHJhbms7CiAgTVBJX0FpbnQgd2luc2l6ZTsKICBNUElf
Q29tbV9yYW5rKGNvbW0sJnJhbmspOwoKICBpZiAoTUNTX0xPQ0tSQU5LID09IE1QSV9LRVlWQUxf
SU5WQUxJRCkKICAgIE1QSV9XaW5fY3JlYXRlX2tleXZhbChNUElfV0lOX05VTExfQ09QWV9GTiwK
CQkJICBNUElfV0lOX05VTExfREVMRVRFX0ZOLAoJCQkgICZNQ1NfTE9DS1JBTkssICh2b2lkKikw
KTsKCiAgd2luc2l6ZSA9IDIgKiBzaXplb2YoaW50KTsKICBpZiAocmFuayA9PSAwKSB3aW5zaXpl
ICs9IHNpemVvZihpbnQpOwogIE1QSV9XaW5fYWxsb2NhdGUod2luc2l6ZSwgc2l6ZW9mKGludCks
IE1QSV9JTkZPX05VTEwsIGNvbW0sCiAgICAgICAgICAgICAgICAgICAmbG1lbSwgd2luKTsKICBs
bWVtW25leHRSYW5rXSA9IC0xOwogIGxtZW1bYmxvY2tlZF0gID0gMDsKICBpZiAocmFuayA9PSAw
KSB7CiAgICBsbWVtW2xvY2tUYWlsXSA9IC0xOwogIH0KICBNUElfV2luX3NldF9hdHRyKCp3aW4s
IE1DU19MT0NLUkFOSywgKHZvaWQqKShNUElfQWludClyYW5rKTsKICBNUElfQmFycmllcihjb21t
KTsKfQoKdm9pZCBNQ1NMb2NrQWNxdWlyZShNUElfV2luIHdpbikKewogIGludCAgZmxhZywgbXly
YW5rLCBwcmVkZWNlc3NvciwgKmxtZW07CiAgdm9pZCAqYXR0cnZhbDsKICBpbnQgZmV0Y2hfYmxv
Y2tlZCwgZHVtbXk7CgogIE1QSV9XaW5fZ2V0X2F0dHIod2luLCBNQ1NfTE9DS1JBTkssICZhdHRy
dmFsLCAmZmxhZyk7CiAgbXlyYW5rID0gKGludCkoTVBJX0FpbnQpYXR0cnZhbDsKICBNUElfV2lu
X2dldF9hdHRyKHdpbiwgTVBJX1dJTl9CQVNFLCAmbG1lbSwgJmZsYWcpOwogIGxtZW1bYmxvY2tl
ZF0gPSAxOyAvKiBJbiBjYXNlIHdlIGFyZSBibG9ja2VkICovCiAgTVBJX1dpbl9sb2NrX2FsbCgw
LCB3aW4pOwogIE1QSV9GZXRjaF9hbmRfb3AoJm15cmFuaywgJnByZWRlY2Vzc29yLCBNUElfSU5U
LAogICAgICAgICAgICAgICAgICAgMCwgbG9ja1RhaWwsIE1QSV9SRVBMQUNFLCB3aW4pOwogIE1Q
SV9XaW5fZmx1c2goMCwgd2luKTsKICBpZiAocHJlZGVjZXNzb3IgIT0gLTEpIHsKICAgIC8qIFdl
IGRpZG4ndCBnZXQgdGhlIGxvY2suICBBZGQgdXMgdG8gdGhlIHRhaWwgb2YgdGhlIGxpc3QgKi8K
ICAgIE1QSV9BY2N1bXVsYXRlKCZteXJhbmssIDEsIE1QSV9JTlQsIHByZWRlY2Vzc29yLAogICAg
ICAgICAgICAgICAgICAgbmV4dFJhbmssIDEsIE1QSV9JTlQsIE1QSV9SRVBMQUNFLCB3aW4pOwog
ICAgLyogTm93IHNwaW4gb24gb3VyIGxvY2FsIHZhbHVlICJibG9ja2VkIiB1bnRpbCB3ZSBhcmUK
ICAgICAgIGdpdmVuIHRoZSBsb2NrICovCiAgICBkbyB7CiAgICAgIE1QSV9GZXRjaF9hbmRfb3Ao
JmR1bW15LCAmZmV0Y2hfYmxvY2tlZCwgTVBJX0lOVCwKICAgICAgICAgICAgICAgICAgIG15cmFu
aywgYmxvY2tlZCwgTVBJX05PX09QLCB3aW4pOwogICAgICB1c2xlZXAoMTAwKTsKICAgIH0gd2hp
bGUgKGZldGNoX2Jsb2NrZWQ9PTEpOwoKICB9CiAgLy8gZWxzZSB3ZSBoYXZlIHRoZSBsb2NrCiAg
TVBJX1dpbl91bmxvY2tfYWxsKHdpbik7Cn0Kdm9pZCBNQ1NMb2NrUmVsZWFzZShNUElfV2luIHdp
bikKewogIGludCBudWxscmFuayA9IC0xLCB6ZXJvPTAsIG15cmFuaywgY3VydGFpbCwgZmxhZywg
KmxtZW07CiAgdm9pZCAqYXR0cnZhbDsKICBpbnQgZmV0Y2hfbmV4dHJhbmssIGR1bW15OwoKICBN
UElfV2luX2dldF9hdHRyKHdpbiwgTUNTX0xPQ0tSQU5LLCAmYXR0cnZhbCwgJmZsYWcpOwogIG15
cmFuayA9IChpbnQpKE1QSV9BaW50KWF0dHJ2YWw7CiAgTVBJX1dpbl9nZXRfYXR0cih3aW4sIE1Q
SV9XSU5fQkFTRSwgJmxtZW0sICZmbGFnKTsKICBNUElfV2luX2xvY2tfYWxsKDAsIHdpbik7CiAg
TVBJX0ZldGNoX2FuZF9vcCgmZHVtbXksICZmZXRjaF9uZXh0cmFuaywgTVBJX0lOVCwKICAgICAg
ICAgICAgICAgICAgIG15cmFuaywgbmV4dFJhbmssIE1QSV9OT19PUCwgd2luKTsKCiAgaWYgKGZl
dGNoX25leHRyYW5rID09IC0xKSB7CiAgICAvKiBTZWUgaWYgd2UncmUgd2FpdGluZyBmb3IgdGhl
IG5leHQgdG8gbm90aWZ5IHVzICovCiAgICBNUElfQ29tcGFyZV9hbmRfc3dhcCgmbnVsbHJhbmss
ICZteXJhbmssICZjdXJ0YWlsLCBNUElfSU5ULAogICAgICAgICAgICAgICAgICAgICAgICAgMCwg
bG9ja1RhaWwsIHdpbik7CiAgICBpZiAoY3VydGFpbCA9PSBteXJhbmspIHsKICAgICAgLyogV2Ug
YXJlIHRoZSBvbmx5IHByb2Nlc3MgaW4gdGhlIGxpc3QgKi8KICAgICAgTVBJX1dpbl91bmxvY2tf
YWxsKHdpbik7CiAgICAgIHJldHVybjsKICAgIH0KICAgIC8qIE90aGVyd2lzZSwgc29tZW9uZSBl
bHNlIGhhcyBhZGRlZCB0aGVtc2VsdmVzIHRvIHRoZSBsaXN0LiovCiAgICBkbyB7CiAgICAgIE1Q
SV9GZXRjaF9hbmRfb3AoJmR1bW15LCAmZmV0Y2hfbmV4dHJhbmssIE1QSV9JTlQsCiAgICAgICAg
ICAgICAgICAgICBteXJhbmssIG5leHRSYW5rLCBNUElfTk9fT1AsIHdpbik7CiAgICB9IHdoaWxl
IChmZXRjaF9uZXh0cmFuaz09LTEpOwoKICAgfQogIC8qIE5vdyB3ZSBjYW4gbm90aWZ5IHRoZW0u
ICBVc2UgYWNjdW11bGF0ZSB3aXRoIHJlcGxhY2UgaW5zdGVhZCAKICAgICBvZiBwdXQgc2luY2Ug
d2Ugd2FudCBhbiBhdG9taWMgdXBkYXRlIG9mIHRoZSBsb2NhdGlvbiAqLwogIE1QSV9BY2N1bXVs
YXRlKCZ6ZXJvLCAxLCBNUElfSU5ULCBmZXRjaF9uZXh0cmFuaywgYmxvY2tlZCwKCQkgMSwgTVBJ
X0lOVCwgTVBJX1JFUExBQ0UsIHdpbik7CiAgTVBJX1dpbl91bmxvY2tfYWxsKHdpbik7Cn0K
--001a1141dbd03846ca054a8dd445
Content-Type: text/x-chdr; charset="US-ASCII"; name="mcs-lock.h"
Content-Disposition: attachment; filename="mcs-lock.h"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_j0732uo22
dm9pZCBNQ1NMb2NrSW5pdChNUElfQ29tbSBjb21tLCBNUElfV2luICp3aW4pOwp2b2lkIE1DU0xv
Y2tBY3F1aXJlKE1QSV9XaW4gd2luKTsKdm9pZCBNQ1NMb2NrUmVsZWFzZShNUElfV2luIHdpbik7
Cgo=
--001a1141dbd03846ca054a8dd445
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
discuss mailing list discuss(a)mpich.org
To manage subscription options or unsubscribe:
https: //lists.mpich.org/mailman/listinfo/discuss
--001a1141dbd03846ca054a8dd445--
Message-ID: <sanitized-1986(a)migration.local>
1
0
sparse algorithm. i.e. the problem may not lie with the inefficiency of
your communication between threads but just that your algorithm is not
keeping the processor busy enough for a large number of threads. The only
way that you will know for sure whether this is a comms issue or an
algorithmic one is to use a profiling tool, such as Vampir or Paraver.
With the profiling result you will be able to determine whether you need to
make algorithmic changes in your bulk processing and enhance comms as per
Huiwei's notes.
I would be interested to know what your profiling shows.
Regards, bob
On Thu, Oct 23, 2014 at 12: 02 AM, Qiguo Jing <qjing(a)trinityconsultants.com>
wrote:
> Hi All,
>
>
>
> We have a parallel program running on a cluster. We recently found a
> case, which decreases the CPU usage and increase the run-time when
> increases Nodes. Below is the results table.
>
>
>
> The particular run requires a lot of data communication between nodes.
>
>
>
> Any thoughts about this phenomena? Or is there any way we can improve the
> CPU usage when using higher number of nodes?
>
>
>
> Average CPU Usage (%)
>
> Number of Nodes
>
> Number of Threads/Node
>
> 100
>
> 1
>
> 8
>
> 92
>
> 2
>
> 8
>
> 50
>
> 3
>
> 8
>
> 40
>
> 4
>
> 8
>
> 35
>
> 5
>
> 8
>
> 30
>
> 6
>
> 8
>
> 25
>
> 7
>
> 8
>
> 20
>
> 8
>
> 8
>
> 20
>
> 8
>
> 4
>
>
>
>
>
> Thanks!
>
> _________________________________________________________________________
>
> The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential and/or privileged
> material. Any review, retransmission, dissemination or other use of, or
> taking of any action in reliance upon, this information by persons or
> entities other than the intended recipient is prohibited. If you received
> this in error, please contact the sender and delete the material from any
> computer.
> _________________________________________________________________________
>
> _______________________________________________
> discuss mailing list discuss(a)mpich.org
> To manage subscription options or unsubscribe:
> https: //lists.mpich.org/mailman/listinfo/discuss
>
--20cf30363855c043ab050610eddd
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dutf-8"><d=
iv dir=3D"ltr"><div>Hi Qiguo,</div><div><br></div><div>From the results tab=
le it looks as if you are using a computationally sparse algorithm. i.=
e. the problem may not lie with the inefficiency of your communication betw=
een threads but just that your algorithm is not keeping the processor busy =
enough for a large number of threads. The only way that you will know for s=
ure whether this is a comms issue or an algorithmic one is to use a profili=
ng tool, such as Vampir or Paraver.</div><div><br></div><div>With the profi=
ling result you will be able to determine whether you need to make algorith=
mic changes in your bulk processing and enhance comms as per Huiwei's notes=
.</div><div><br></div><div>I would be interested to know what your profilin=
g shows.</div><div><br></div><div>Regards, bob</div></div><div class=3D"gma=
il_extra"><br><div class=3D"gmail_quote">On Thu, Oct 23, 2014 at 12: 02 AM, =
Qiguo Jing <span dir=3D"ltr"><<a href=3D"mailto: qjing@trinityconsultants=
.com" target=3D"_blank">qjing(a)trinityconsultants.com</a>></span> wrote: <=
br><blockquote class=3D"gmail_quote" style=3D"margin: 0 0 0 .8ex;border-left=
: 1px #ccc solid;padding-left:1ex">
<div lang=3D"EN-US" vlink=3D"#954F72" link=3D"#0563C1">
<div>
<p class=3D"MsoNormal">Hi All,<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u> <u></u></p>
<p class=3D"MsoNormal">We have a parallel program running on a cluster.&nbs=
p; We recently found a case, which decreases the CPU usage and increase the=
run-time when increases Nodes. Below is the results table.<u><=
/u><u></u></p>
<p class=3D"MsoNormal"><u></u> <u></u></p>
<p class=3D"MsoNormal">The particular run requires a lot of data communicat=
ion between nodes.
<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u> <u></u></p>
<p class=3D"MsoNormal">Any thoughts about this phenomena? Or is there=
any way we can improve the CPU usage when using higher number of nodes?
<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u> <u></u></p>
<table width=3D"441" style=3D"width: 331pt;border-collapse:collapse" border=
=3D"0" cellspacing=3D"0" cellpadding=3D"0">
<tbody>
<tr style=3D"height: 15pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"padding: 0in 5.4pt;=
border: 1pt solid windowtext;width:115pt;height:15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">Average CPU Usage (%)<u></u><u></u></span></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"border-width: 1pt 1=
pt 1pt medium;border-style: solid solid solid none;border-color:windowtext w=
indowtext windowtext currentColor;padding: 0in 5.4pt;width:89pt;height:15pt"=
>
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">Number of Nodes<u></u><u></u></span></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"border-width: 1pt 1=
pt 1pt medium;border-style: solid solid solid none;border-color:windowtext w=
indowtext windowtext currentColor;padding: 0in 5.4pt;width:127pt;height:15pt=
">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">Number of Threads/Node<u></u><u></u></span></p>
</td>
</tr>
<tr style=3D"height: 15pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt;border-style: none solid solid;border-color:currentColor windowtex=
t windowtext;padding: 0in 5.4pt;width:115pt;height:15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">100<u></u><u></u></span></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt medium;border-style: none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding: 0in 5.4pt;width:89pt;height:1=
5pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">1<u></u><u></u></span></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt medium;border-style: none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding: 0in 5.4pt;width:127pt;height:=
15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">8<u></u><u></u></span></p>
</td>
</tr>
<tr style=3D"height: 15pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt;border-style: none solid solid;border-color:currentColor windowtex=
t windowtext;padding: 0in 5.4pt;width:115pt;height:15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">92<u></u><u></u></span></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt medium;border-style: none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding: 0in 5.4pt;width:89pt;height:1=
5pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">2<u></u><u></u></span></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt medium;border-style: none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding: 0in 5.4pt;width:127pt;height:=
15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">8<u></u><u></u></span></p>
</td>
</tr>
<tr style=3D"height: 15pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt;border-style: none solid solid;border-color:currentColor windowtex=
t windowtext;padding: 0in 5.4pt;width:115pt;height:15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">50<u></u><u></u></span></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt medium;border-style: none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding: 0in 5.4pt;width:89pt;height:1=
5pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">3<u></u><u></u></span></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt medium;border-style: none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding: 0in 5.4pt;width:127pt;height:=
15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">8<u></u><u></u></span></p>
</td>
</tr>
<tr style=3D"height: 15pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt;border-style: none solid solid;border-color:currentColor windowtex=
t windowtext;padding: 0in 5.4pt;width:115pt;height:15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">40<u></u><u></u></span></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt medium;border-style: none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding: 0in 5.4pt;width:89pt;height:1=
5pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">4<u></u><u></u></span></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt medium;border-style: none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding: 0in 5.4pt;width:127pt;height:=
15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">8<u></u><u></u></span></p>
</td>
</tr>
<tr style=3D"height: 15pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt;border-style: none solid solid;border-color:currentColor windowtex=
t windowtext;padding: 0in 5.4pt;width:115pt;height:15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">35<u></u><u></u></span></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt medium;border-style: none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding: 0in 5.4pt;width:89pt;height:1=
5pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">5<u></u><u></u></span></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt medium;border-style: none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding: 0in 5.4pt;width:127pt;height:=
15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">8<u></u><u></u></span></p>
</td>
</tr>
<tr style=3D"height: 15pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt;border-style: none solid solid;border-color:currentColor windowtex=
t windowtext;padding: 0in 5.4pt;width:115pt;height:15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">30<u></u><u></u></span></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt medium;border-style: none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding: 0in 5.4pt;width:89pt;height:1=
5pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">6<u></u><u></u></span></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt medium;border-style: none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding: 0in 5.4pt;width:127pt;height:=
15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">8<u></u><u></u></span></p>
</td>
</tr>
<tr style=3D"height: 15pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt;border-style: none solid solid;border-color:currentColor windowtex=
t windowtext;padding: 0in 5.4pt;width:115pt;height:15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">25<u></u><u></u></span></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt medium;border-style: none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding: 0in 5.4pt;width:89pt;height:1=
5pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">7<u></u><u></u></span></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt medium;border-style: none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding: 0in 5.4pt;width:127pt;height:=
15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">8<u></u><u></u></span></p>
</td>
</tr>
<tr style=3D"height: 15pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt;border-style: none solid solid;border-color:currentColor windowtex=
t windowtext;padding: 0in 5.4pt;width:115pt;height:15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">20<u></u><u></u></span></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt medium;border-style: none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding: 0in 5.4pt;width:89pt;height:1=
5pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">8<u></u><u></u></span></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt medium;border-style: none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding: 0in 5.4pt;width:127pt;height:=
15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">8<u></u><u></u></span></p>
</td>
</tr>
<tr style=3D"height: 15pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt;border-style: none solid solid;border-color:currentColor windowtex=
t windowtext;padding: 0in 5.4pt;width:115pt;height:15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">20<u></u><u></u></span></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt medium;border-style: none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding: 0in 5.4pt;width:89pt;height:1=
5pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">8<u></u><u></u></span></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"border-width: mediu=
m 1pt 1pt medium;border-style: none solid solid none;border-color:currentCol=
or windowtext windowtext currentColor;padding: 0in 5.4pt;width:127pt;height:=
15pt">
<p align=3D"center" class=3D"MsoNormal" style=3D"text-align: center"><span s=
tyle=3D"color: black">4<u></u><u></u></span></p>
</td>
</tr>
</tbody>
</table>
<p class=3D"MsoNormal"><u></u> <u></u></p>
<p class=3D"MsoNormal"><u></u> <u></u></p>
<p class=3D"MsoNormal">Thanks!<u></u><u></u></p>
</div>
</div>
<br>
______________________________<u></u>______________________________<u></u>_=
____________<br><br>The information transmitted is intended only for the pe=
rson or entity to<br>which it is addressed and may contain confidential and=
/or privileged<br>material. Any review, retransmission, dissemination or o=
ther use of, or<br>taking of any action in reliance upon, this information =
by persons or<br>entities other than the intended recipient is prohibited. =
If you received<br>this in error, please contact the sender and delete th=
e material from any<br>computer.<br>______________________________<u></u>__=
____________________________<u></u>_____________<br><br>___________________=
____________________________<br>
discuss mailing list <a href=3D"mailto: discuss(a)mpich.org=
">discuss(a)mpich.org</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" target=3D"_bla=
nk">https: //lists.mpich.org/mailman/listinfo/discuss</a><br></blockquote></=
div><br></div>
--20cf30363855c043ab050610eddd--
--===============0923015504839801834==
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
discuss mailing list discuss(a)mpich.org
To manage subscription options or unsubscribe:
https: //lists.mpich.org/mailman/listinfo/discuss
--===============0923015504839801834==--
Message-ID: <sanitized-1823(a)migration.local>
1
0
e algorithm. i.e. the problem may not lie with the inefficiency of your com=
munication between threads but just that your algorithm is not keeping the =
processor busy enough for a large number of threads. The only way that you =
will know for sure whether this is a comms issue or an algorithmic one is t=
o use a profiling tool, such as Vampir or Paraver.
With the profiling result you will be able to determine whether you need to=
make algorithmic changes in your bulk processing and enhance comms as per =
Huiwei's notes.
I would be interested to know what your profiling shows.
Regards, bob
On Thu, Oct 23, 2014 at 12: 02 AM, Qiguo Jing <qjing(a)trinityconsultants.com<=
mailto: qjing(a)trinityconsultants.com>> wrote:
Hi All,
We have a parallel program running on a cluster. We recently found a case,=
which decreases the CPU usage and increase the run-time when increases Nod=
es. Below is the results table.
The particular run requires a lot of data communication between nodes.
Any thoughts about this phenomena? Or is there any way we can improve the =
CPU usage when using higher number of nodes?
Average CPU Usage (%)
Number of Nodes
Number of Threads/Node
100
1
8
92
2
8
50
3
8
40
4
8
35
5
8
30
6
8
25
7
8
20
8
8
20
8
4
Thanks!
_________________________________________________________________________
The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.
_________________________________________________________________________
_______________________________________________
discuss mailing list discuss(a)mpich.org<mailto: discuss(a)mpich.org>
To manage subscription options or unsubscribe:
https: //lists.mpich.org/mailman/listinfo/discuss
--=20
_________________________________________________________________________
The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.
_________________________________________________________________________
--_000_9D079B25EA4E3149B260846F07AE80754C594461TCIEXCH03ustrin_
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<html xmlns: v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com: office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns: m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40"><head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dutf-8">
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family: =E5=AE=8B=E4=BD=93;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family: "Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family: Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family: "\@=E5=AE=8B=E4=BD=93";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin: 0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a: link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a: visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type: personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type: export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size: 8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.WordSection1
{page: WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o: shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o: shapelayout v:ext=3D"edit">
<o: idmap v:ext=3D"edit" data=3D"1" />
</o: shapelayout></xml><![endif]-->
</head>
<body lang=3D"EN-US" link=3D"blue" vlink=3D"purple">
<div class=3D"WordSection1">
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D">Hi Bob,<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D">Thanks for your suggestions. &nb=
sp; Here are more tests. We actually have three clusters.<o: p></o:p><=
/span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D">Cluster 1 and 2: 8 nodes, (2 Pr=
ocessors, 4 cores/processor, no HT =E2=80=93 Total 8 Threads)/node<o: p></o:=
p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D">Cluster 3: &nb=
sp; 8 nodes, (1 Processors,=
4 cores /processor, HT =E2=80=93 Total 8 Threads )/node<o: p></o:p></span><=
/p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D">We also have a standalone machine:&nb=
sp; 2 processors, 6 cores/processor, HT =E2=80=93 total 24 threads.<o: p></o=
: p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D">For one particular case:
<o: p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D">Cluster 1 and 2 take 48 min to finish=
with 8 nodes, 8 threads/node, 60% CPU usage; 53 min to finish =
with 3 nodes, 8 threads/node, 90% CPU usage;<o: p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D">Cluster 3 takes 227 min to finish wit=
h 8 nodes, 8 threads/node, 20% CPU usage; 207 min to finish with 3 nodes, 8=
threads/node, 50% CPU usage;<o: p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D">Standalone machine takes 24 min to fi=
nish with 24 threads, 100% CPU usage.<o: p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D">It looks like with 24 threads, they s=
hould be pretty busy? Could the above phenomena be a hardware i=
ssue more than software?<o: p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D">Qiguo<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><b><span style=3D"font-size: 11.0pt;font-family:"=
;Calibri",sans-serif">From: </span></b><span style=3D"font-size:11.0pt;=
font-family: "Calibri",sans-serif"> Bob Ilgner [mailto:bobilgner@g=
mail.com]
<br>
<b>Sent: </b> Thursday, October 23, 2014 1:11 AM<br>
<b>To: </b> discuss(a)mpich.org<br>
<b>Subject: </b> Re: [mpich-discuss] CPU usage versus Nodes, Threads<o:p></o=
: p></span></p>
<p class=3D"MsoNormal"><o: p> </o:p></p>
<div>
<div>
<p class=3D"MsoNormal">Hi Qiguo,<o: p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o: p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal">From the results table it looks as if you are u=
sing a computationally sparse algorithm. i.e. the problem may not lie with =
the inefficiency of your communication between threads but just that your a=
lgorithm is not keeping the processor busy
enough for a large number of threads. The only way that you will know for =
sure whether this is a comms issue or an algorithmic one is to use a profil=
ing tool, such as Vampir or Paraver.<o: p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o: p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal">With the profiling result you will be able to determ=
ine whether you need to make algorithmic changes in your bulk processing an=
d enhance comms as per Huiwei's notes.<o: p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o: p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal">I would be interested to know what your profiling sh=
ows.<o: p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o: p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal">Regards, bob<o: p></o:p></p>
</div>
</div>
<div>
<p class=3D"MsoNormal"><o: p> </o:p></p>
<div>
<p class=3D"MsoNormal">On Thu, Oct 23, 2014 at 12: 02 AM, Qiguo Jing <<a =
href=3D"mailto: qjing(a)trinityconsultants.com" target=3D"_blank">qjing@trinit=
yconsultants.com</a>> wrote: <o:p></o:p></p>
<blockquote style=3D"border: none;border-left:solid #CCCCCC 1.0pt;padding:0i=
n 0in 0in 6.0pt;margin-left: 4.8pt;margin-top:5.0pt;margin-right:0in;margin-=
bottom: 5.0pt">
<div>
<div>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt: auto;mso-margin-bottom-a=
lt: auto">Hi All,<o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt: auto;mso-margin-bottom-a=
lt: auto"> <o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt: auto;mso-margin-bottom-a=
lt: auto">We have a parallel program running on a cluster. We recently=
found a case, which decreases the CPU usage and increase the run-time when=
increases Nodes. Below is the results
table.<o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt: auto;mso-margin-bottom-a=
lt: auto"> <o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt: auto;mso-margin-bottom-a=
lt: auto">The particular run requires a lot of data communication between no=
des.
<o: p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt: auto;mso-margin-bottom-a=
lt: auto"> <o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt: auto;mso-margin-bottom-a=
lt: auto">Any thoughts about this phenomena? Or is there any way we ca=
n improve the CPU usage when using higher number of nodes?
<o: p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt: auto;mso-margin-bottom-a=
lt: auto"> <o:p></o:p></p>
<table class=3D"MsoNormalTable" border=3D"0" cellspacing=3D"0" cellpadding=
=3D"0" width=3D"441" style=3D"width: 331.0pt;border-collapse:collapse">
<tbody>
<tr style=3D"height: 15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width: 115.0pt;bord=
er: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">Average CPU Usage (%)</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width: 89.0pt;borde=
r: solid windowtext 1.0pt;border-left:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t: 15.0pt;border-color:currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">Number of Nodes</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width: 127.0pt;bord=
er: solid windowtext 1.0pt;border-left:none;padding:0in 5.4pt 0in 5.4pt;heig=
ht: 15.0pt;border-color:currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">Number of Threads/Node</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height: 15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width: 115.0pt;bord=
er: solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t: 15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">100</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width: 89.0pt;borde=
r-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">1</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width: 127.0pt;bord=
er-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">8</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height: 15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width: 115.0pt;bord=
er: solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t: 15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">92</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width: 89.0pt;borde=
r-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">2</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width: 127.0pt;bord=
er-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">8</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height: 15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width: 115.0pt;bord=
er: solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t: 15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">50</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width: 89.0pt;borde=
r-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">3</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width: 127.0pt;bord=
er-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">8</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height: 15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width: 115.0pt;bord=
er: solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t: 15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">40</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width: 89.0pt;borde=
r-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">4</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width: 127.0pt;bord=
er-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">8</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height: 15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width: 115.0pt;bord=
er: solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t: 15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">35</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width: 89.0pt;borde=
r-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">5</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width: 127.0pt;bord=
er-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">8</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height: 15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width: 115.0pt;bord=
er: solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t: 15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">30</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width: 89.0pt;borde=
r-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">6</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width: 127.0pt;bord=
er-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">8</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height: 15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width: 115.0pt;bord=
er: solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t: 15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">25</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width: 89.0pt;borde=
r-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">7</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width: 127.0pt;bord=
er-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">8</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height: 15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width: 115.0pt;bord=
er: solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t: 15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">20</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width: 89.0pt;borde=
r-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">8</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width: 127.0pt;bord=
er-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">8</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height: 15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width: 115.0pt;bord=
er: solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t: 15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">20</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width: 89.0pt;borde=
r-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">8</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width: 127.0pt;bord=
er-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">4</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt: auto;mso-margin-bottom-a=
lt: auto"> <o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt: auto;mso-margin-bottom-a=
lt: auto"> <o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt: auto;mso-margin-bottom-a=
lt: auto">Thanks!<o:p></o:p></p>
</div>
</div>
<p class=3D"MsoNormal"><br>
_________________________________________________________________________<b=
r>
<br>
The information transmitted is intended only for the person or entity to<br=
>
which it is addressed and may contain confidential and/or privileged<br>
material. Any review, retransmission, dissemination or other use of, or<br>
taking of any action in reliance upon, this information by persons or<br>
entities other than the intended recipient is prohibited. If you received<b=
r>
this in error, please contact the sender and delete the material from any<b=
r>
computer.<br>
_________________________________________________________________________<b=
r>
<br>
_______________________________________________<br>
discuss mailing list <a href=3D"mailto: discuss(a)mpich.org=
">discuss(a)mpich.org</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" target=3D"_bla=
nk">https: //lists.mpich.org/mailman/listinfo/discuss</a><o:p></o:p></p>
</blockquote>
</div>
<p class=3D"MsoNormal"><o: p> </o:p></p>
</div>
</div>
</body>
</html>
<br>
______________________________<wbr>______________________________<wbr>_____=
________<br><br>The information transmitted is intended only for the person=
or entity to<br>which it is addressed and may contain confidential and/or =
privileged<br>material. Any review, retransmission, dissemination or other=
use of, or<br>taking of any action in reliance upon, this information by p=
ersons or<br>entities other than the intended recipient is prohibited. If=
you received<br>this in error, please contact the sender and delete the ma=
terial from any<br>computer.<br>______________________________<wbr>________=
______________________<wbr>_____________<br>=
--_000_9D079B25EA4E3149B260846F07AE80754C594461TCIEXCH03ustrin_--
--===============5521172785385150317==
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
discuss mailing list discuss(a)mpich.org
To manage subscription options or unsubscribe:
https: //lists.mpich.org/mailman/listinfo/discuss
--===============5521172785385150317==--
Message-ID: <sanitized-1820(a)migration.local>
1
0
e algorithm. i.e. the problem may not lie with the inefficiency of your com=
munication between threads but just that your algorithm is not keeping the =
processor busy enough for a large number of threads. The only way that you =
will know for sure whether this is a comms issue or an algorithmic one is t=
o use a profiling tool, such as Vampir or Paraver.
With the profiling result you will be able to determine whether you need to=
make algorithmic changes in your bulk processing and enhance comms as per =
Huiwei's notes.
I would be interested to know what your profiling shows.
Regards, bob
On Thu, Oct 23, 2014 at 12: 02 AM, Qiguo Jing <qjing(a)trinityconsultants.com<=
mailto: qjing(a)trinityconsultants.com>> wrote:
Hi All,
We have a parallel program running on a cluster. We recently found a case,=
which decreases the CPU usage and increase the run-time when increases Nod=
es. Below is the results table.
The particular run requires a lot of data communication between nodes.
Any thoughts about this phenomena? Or is there any way we can improve the =
CPU usage when using higher number of nodes?
Average CPU Usage (%)
Number of Nodes
Number of Threads/Node
100
1
8
92
2
8
50
3
8
40
4
8
35
5
8
30
6
8
25
7
8
20
8
8
20
8
4
Thanks!
_________________________________________________________________________
The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.
_________________________________________________________________________
_______________________________________________
discuss mailing list discuss(a)mpich.org<mailto: discuss(a)mpich.org>
To manage subscription options or unsubscribe:
https: //lists.mpich.org/mailman/listinfo/discuss
_________________________________________________________________________
The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.
_________________________________________________________________________
--=20
_________________________________________________________________________
The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.
_________________________________________________________________________
--_000_9D079B25EA4E3149B260846F07AE80754C59447FTCIEXCH03ustrin_
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<html xmlns: v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com: office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns: m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40"><head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dutf-8">
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family: =E5=AE=8B=E4=BD=93;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family: "Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family: Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family: "\@=E5=AE=8B=E4=BD=93";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin: 0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a: link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a: visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type: personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle18
{mso-style-type: personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type: export-only;
font-size:10.0pt;}
@page WordSection1
{size: 8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.WordSection1
{page: WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o: shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o: shapelayout v:ext=3D"edit">
<o: idmap v:ext=3D"edit" data=3D"1" />
</o: shapelayout></xml><![endif]-->
</head>
<body lang=3D"EN-US" link=3D"blue" vlink=3D"purple">
<div class=3D"WordSection1">
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D">Hi Bob,<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D">Thanks for your suggestions. &nb=
sp; Here are more tests. We actually have three clusters.<o: p></o:p><=
/span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D">Cluster 1 and 2: 8 nodes, (2 Pr=
ocessors, 4 cores/processor, no HT =E2=80=93 Total 8 Threads)/node<o: p></o:=
p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D">Cluster 3: &nb=
sp; 8 nodes, (1 Processors,=
4 cores /processor, HT =E2=80=93 Total 8 Threads )/node<o: p></o:p></span><=
/p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D">We also have a standalone machine:&nb=
sp; 2 processors, 6 cores/processor, HT =E2=80=93 total 24 threads.<o: p></o=
: p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D">For one particular case:
<o: p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D">Cluster 1 and 2 take 48 min to finish=
with 8 nodes, 8 threads/node, 60% CPU usage; 53 min to finish =
with 3 nodes, 8 threads/node, 90% CPU usage;<o: p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D">Cluster 3 takes 227 min to finish wit=
h 8 nodes, 8 threads/node, 20% CPU usage; 207 min to finish with 3 nodes, 8=
threads/node, 50% CPU usage;<o: p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D">Standalone machine takes 82 min to fi=
nish with 24 threads, 100% CPU usage.<o: p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D">It looks like with 24 threads, they s=
hould be pretty busy? Could the above phenomena be a hardware i=
ssue more than software?<o: p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D">Qiguo<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size: 11.0pt;font-family:"Ca=
libri",sans-serif;color: #1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><b><span style=3D"font-size: 11.0pt;font-family:"=
;Calibri",sans-serif">From: </span></b><span style=3D"font-size:11.0pt;=
font-family: "Calibri",sans-serif"> Bob Ilgner [<a href=3D"mailto:=
bobilgner(a)gmail.com">mailto: bobilgner(a)gmail.com</a>]
<br>
<b>Sent: </b> Thursday, October 23, 2014 1:11 AM<br>
<b>To: </b> <a href=3D"mailto:[email protected]">discuss(a)mpich.org</a><br>
<b>Subject: </b> Re: [mpich-discuss] CPU usage versus Nodes, Threads<o:p></o=
: p></span></p>
<p class=3D"MsoNormal"><o: p> </o:p></p>
<div>
<div>
<p class=3D"MsoNormal">Hi Qiguo,<o: p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o: p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal">From the results table it looks as if you are u=
sing a computationally sparse algorithm. i.e. the problem may not lie with =
the inefficiency of your communication between threads but just that your a=
lgorithm is not keeping the processor busy
enough for a large number of threads. The only way that you will know for =
sure whether this is a comms issue or an algorithmic one is to use a profil=
ing tool, such as Vampir or Paraver.<o: p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o: p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal">With the profiling result you will be able to determ=
ine whether you need to make algorithmic changes in your bulk processing an=
d enhance comms as per Huiwei's notes.<o: p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o: p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal">I would be interested to know what your profiling sh=
ows.<o: p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o: p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal">Regards, bob<o: p></o:p></p>
</div>
</div>
<div>
<p class=3D"MsoNormal"><o: p> </o:p></p>
<div>
<p class=3D"MsoNormal">On Thu, Oct 23, 2014 at 12: 02 AM, Qiguo Jing <<a =
href=3D"mailto: qjing(a)trinityconsultants.com" target=3D"_blank">qjing@trinit=
yconsultants.com</a>> wrote: <o:p></o:p></p>
<blockquote style=3D"border: none;border-left:solid #CCCCCC 1.0pt;padding:0i=
n 0in 0in 6.0pt;margin-left: 4.8pt;margin-top:5.0pt;margin-right:0in;margin-=
bottom: 5.0pt">
<div>
<div>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt: auto;mso-margin-bottom-a=
lt: auto">Hi All,<o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt: auto;mso-margin-bottom-a=
lt: auto"> <o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt: auto;mso-margin-bottom-a=
lt: auto">We have a parallel program running on a cluster. We recently=
found a case, which decreases the CPU usage and increase the run-time when=
increases Nodes. Below is the results
table.<o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt: auto;mso-margin-bottom-a=
lt: auto"> <o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt: auto;mso-margin-bottom-a=
lt: auto">The particular run requires a lot of data communication between no=
des.
<o: p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt: auto;mso-margin-bottom-a=
lt: auto"> <o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt: auto;mso-margin-bottom-a=
lt: auto">Any thoughts about this phenomena? Or is there any way we ca=
n improve the CPU usage when using higher number of nodes?
<o: p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt: auto;mso-margin-bottom-a=
lt: auto"> <o:p></o:p></p>
<table class=3D"MsoNormalTable" border=3D"0" cellspacing=3D"0" cellpadding=
=3D"0" width=3D"441" style=3D"width: 331.0pt;border-collapse:collapse">
<tbody>
<tr style=3D"height: 15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width: 115.0pt;bord=
er: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">Average CPU Usage (%)</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width: 89.0pt;borde=
r: solid windowtext 1.0pt;border-left:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t: 15.0pt;border-color:currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">Number of Nodes</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width: 127.0pt;bord=
er: solid windowtext 1.0pt;border-left:none;padding:0in 5.4pt 0in 5.4pt;heig=
ht: 15.0pt;border-color:currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">Number of Threads/Node</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height: 15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width: 115.0pt;bord=
er: solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t: 15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">100</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width: 89.0pt;borde=
r-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">1</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width: 127.0pt;bord=
er-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">8</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height: 15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width: 115.0pt;bord=
er: solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t: 15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">92</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width: 89.0pt;borde=
r-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">2</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width: 127.0pt;bord=
er-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">8</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height: 15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width: 115.0pt;bord=
er: solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t: 15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">50</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width: 89.0pt;borde=
r-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">3</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width: 127.0pt;bord=
er-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">8</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height: 15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width: 115.0pt;bord=
er: solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t: 15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">40</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width: 89.0pt;borde=
r-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">4</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width: 127.0pt;bord=
er-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">8</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height: 15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width: 115.0pt;bord=
er: solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t: 15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">35</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width: 89.0pt;borde=
r-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">5</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width: 127.0pt;bord=
er-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">8</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height: 15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width: 115.0pt;bord=
er: solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t: 15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">30</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width: 89.0pt;borde=
r-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">6</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width: 127.0pt;bord=
er-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">8</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height: 15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width: 115.0pt;bord=
er: solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t: 15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">25</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width: 89.0pt;borde=
r-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">7</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width: 127.0pt;bord=
er-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">8</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height: 15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width: 115.0pt;bord=
er: solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t: 15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">20</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width: 89.0pt;borde=
r-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">8</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width: 127.0pt;bord=
er-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">8</span><o:p></o:p></p>
</td>
</tr>
<tr style=3D"height: 15.0pt">
<td width=3D"153" nowrap=3D"" valign=3D"bottom" style=3D"width: 115.0pt;bord=
er: solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;heigh=
t: 15.0pt;border-color:currentColor windowtext windowtext">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">20</span><o:p></o:p></p>
</td>
<td width=3D"119" nowrap=3D"" valign=3D"bottom" style=3D"width: 89.0pt;borde=
r-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-rig=
ht: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border-=
color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">8</span><o:p></o:p></p>
</td>
<td width=3D"169" nowrap=3D"" valign=3D"bottom" style=3D"width: 127.0pt;bord=
er-top: none;border-left:none;border-bottom:solid windowtext 1.0pt;border-ri=
ght: solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt;border=
-color: currentColor windowtext windowtext currentColor">
<p class=3D"MsoNormal" align=3D"center" style=3D"mso-margin-top-alt: auto;ms=
o-margin-bottom-alt: auto;text-align:center">
<span style=3D"color: black">4</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt: auto;mso-margin-bottom-a=
lt: auto"> <o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt: auto;mso-margin-bottom-a=
lt: auto"> <o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt: auto;mso-margin-bottom-a=
lt: auto">Thanks!<o:p></o:p></p>
</div>
</div>
<p class=3D"MsoNormal"><br>
_________________________________________________________________________<b=
r>
<br>
The information transmitted is intended only for the person or entity to<br=
>
which it is addressed and may contain confidential and/or privileged<br>
material. Any review, retransmission, dissemination or other use of, or<br>
taking of any action in reliance upon, this information by persons or<br>
entities other than the intended recipient is prohibited. If you received<b=
r>
this in error, please contact the sender and delete the material from any<b=
r>
computer.<br>
_________________________________________________________________________<b=
r>
<br>
_______________________________________________<br>
discuss mailing list <a href=3D"mailto: discuss(a)mpich.org=
">discuss(a)mpich.org</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" target=3D"_bla=
nk">https: //lists.mpich.org/mailman/listinfo/discuss</a><o:p></o:p></p>
</blockquote>
</div>
<p class=3D"MsoNormal"><o: p> </o:p></p>
</div>
<p class=3D"MsoNormal"><br>
_________________________________________________________________________<b=
r>
<br>
The information transmitted is intended only for the person or entity to<br=
>
which it is addressed and may contain confidential and/or privileged<br>
material. Any review, retransmission, dissemination or other use of, or<br>
taking of any action in reliance upon, this information by persons or<br>
entities other than the intended recipient is prohibited. If you received<b=
r>
this in error, please contact the sender and delete the material from any<b=
r>
computer.<br>
_________________________________________________________________________<o=
: p></o:p></p>
</div>
</body>
</html>
<br>
______________________________<wbr>______________________________<wbr>_____=
________<br><br>The information transmitted is intended only for the person=
or entity to<br>which it is addressed and may contain confidential and/or =
privileged<br>material. Any review, retransmission, dissemination or other=
use of, or<br>taking of any action in reliance upon, this information by p=
ersons or<br>entities other than the intended recipient is prohibited. If=
you received<br>this in error, please contact the sender and delete the ma=
terial from any<br>computer.<br>______________________________<wbr>________=
______________________<wbr>_____________<br>=
--_000_9D079B25EA4E3149B260846F07AE80754C59447FTCIEXCH03ustrin_--
--===============5606136153793979325==
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
discuss mailing list discuss(a)mpich.org
To manage subscription options or unsubscribe:
https: //lists.mpich.org/mailman/listinfo/discuss
--===============5606136153793979325==--
Message-ID: <sanitized-1817(a)migration.local>
1
0
--Junchao Zhang
On Wed, Nov 26, 2014 at 4: 25 PM, Amin Hassani <ahassani(a)cis.uab.edu> wrote:
> I disabled the whole firewall in those machines but, still get the same
> problem. connection refuse.
> I run the program in another set of totally different machines that we
> have, but still same problem.
> Any other thought where can be the problem?
>
> Thanks.
>
> Amin Hassani,
> CIS department at UAB,
> Birmingham, AL, USA.
>
> On Wed, Nov 26, 2014 at 9: 25 AM, Kenneth Raffenetti <raffenet(a)mcs.anl.gov=
>
> wrote:
>
>> The connection refused makes me think a firewall is getting in the way.
>> Is TCP communication limited to specific ports on the cluster? If so, yo=
u
>> can use this envvar to enforce a range of ports in MPICH.
>>
>> MPIR_CVAR_CH3_PORT_RANGE
>> Description: The MPIR_CVAR_CH3_PORT_RANGE environment variable allow=
s
>> you to specify the range of TCP ports to be used by the process manager =
and
>> the MPICH library. The format of this variable is <low>: <high>. To spec=
ify
>> any available port, use 0: 0.
>> Default: {0,0}
>>
>>
>> On 11/25/2014 11: 50 PM, Amin Hassani wrote:
>>
>>> Tried with the new configure too. same problem: (
>>>
>>> $ mpirun -hostfile hosts-hydra -np 2 test_dup
>>> Fatal error in MPI_Send: Unknown error class, error stack:
>>> MPI_Send(174)..............: MPI_Send(buf=3D0x7fffd90c76c8, count=3D1,
>>> MPI_INT, dest=3D1, tag=3D0, MPI_COMM_WORLD) failed
>>> MPID_nem_tcp_connpoll(1832): Communication error with rank 1: Connectio=
n
>>> refused
>>>
>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>> =3D BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
>>> =3D PID 5459 RUNNING AT oakmnt-0-a
>>> =3D EXIT CODE: 1
>>> =3D CLEANING UP REMAINING PROCESSES
>>> =3D YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>> [proxy: 0:1@oakmnt-0-b] HYD_pmcd_pmip_control_cmd_cb
>>> (../../../../src/pm/hydra/pm/pmiserv/pmip_cb.c: 885): assert (!closed)
>>> failed
>>> [proxy: 0:1@oakmnt-0-b] HYDT_dmxu_poll_wait_for_event
>>> (../../../../src/pm/hydra/tools/demux/demux_poll.c: 76): callback
>>> returned error status
>>> [proxy: 0:1@oakmnt-0-b] main
>>> (../../../../src/pm/hydra/pm/pmiserv/pmip.c: 206): demux engine error
>>> waiting for event
>>> [mpiexec@oakmnt-0-a] HYDT_bscu_wait_for_completion
>>> (../../../../src/pm/hydra/tools/bootstrap/utils/bscu_wait.c: 76): one of
>>> the processes terminated badly; aborting
>>> [mpiexec@oakmnt-0-a] HYDT_bsci_wait_for_completion
>>> (../../../../src/pm/hydra/tools/bootstrap/src/bsci_wait.c: 23): launcher
>>> returned error waiting for completion
>>> [mpiexec@oakmnt-0-a] HYD_pmci_wait_for_completion
>>> (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c: 218): launcher
>>> returned error waiting for completion
>>> [mpiexec@oakmnt-0-a] main
>>> (../../../../src/pm/hydra/ui/mpich/mpiexec.c: 344): process manager erro=
r
>>> waiting for completion
>>>
>>>
>>> Amin Hassani,
>>> CIS department at UAB,
>>> Birmingham, AL, USA.
>>>
>>> On Tue, Nov 25, 2014 at 11: 44 PM, Lu, Huiwei <huiweilu(a)mcs.anl.gov
>>> <mailto: huiweilu(a)mcs.anl.gov>> wrote:
>>>
>>> So the error only happens when there is communication.
>>>
>>> It may be caused by IB as your guessed before. Could you try to
>>> reconfigure MPICH using "./configure --with-device=3Dch3: nemesis:tc=
p=E2=80=9D
>>> and try again?
>>>
>>> =E2=80=94
>>> Huiwei
>>>
>>> > On Nov 25, 2014, at 11: 23 PM, Amin Hassani <ahassani(a)cis.uab.edu
>>> <mailto: ahassani(a)cis.uab.edu>> wrote:
>>> >
>>> > Yes it works.
>>> > output:
>>> >
>>> > $ mpirun -hostfile hosts-hydra -np 2 test
>>> > rank 1
>>> > rank 0
>>> >
>>> >
>>> > Amin Hassani,
>>> > CIS department at UAB,
>>> > Birmingham, AL, USA.
>>> >
>>> > On Tue, Nov 25, 2014 at 11: 20 PM, Lu, Huiwei
>>> <huiweilu(a)mcs.anl.gov <mailto: huiweilu(a)mcs.anl.gov>> wrote:
>>> > Could you try to run the following simple code to see if it work=
s?
>>> >
>>> > #include <mpi.h>
>>> > #include <stdio.h>
>>> > int main(int argc, char** argv)
>>> > {
>>> > int rank, size;
>>> > MPI_Init(&argc, &argv);
>>> > MPI_Comm_rank(MPI_COMM_WORLD, &rank);
>>> > printf("rank %d\n", rank);
>>> > MPI_Finalize();
>>> > return 0;
>>> > }
>>> >
>>> > =E2=80=94
>>> > Huiwei
>>> >
>>> > > On Nov 25, 2014, at 11: 11 PM, Amin Hassani
>>> <ahassani(a)cis.uab.edu <mailto: ahassani(a)cis.uab.edu>> wrote:
>>> > >
>>> > > No, I checked. Also I always install my MPI's in
>>> /nethome/students/ahassani/usr/mpi. I never install them in
>>> /nethome/students/ahassani/usr. So MPI files will never get there.
>>> Even if put the /usr/mpi/bin in front of /usr/bin, it won't affect
>>> anything. There has never been any mpi installed in /usr/bin.
>>> > >
>>> > > Thank you.
>>> > > _______________________________________________
>>> > > discuss mailing list discuss(a)mpich.org <mailto:
>>> discuss(a)mpich.org>
>>> > > To manage subscription options or unsubscribe:
>>> > > https: //lists.mpich.org/mailman/listinfo/discuss
>>> >
>>> > _______________________________________________
>>> > discuss mailing list discuss(a)mpich.org <mailto: discuss(a)mpich.org=
>
>>> > To manage subscription options or unsubscribe:
>>> > https: //lists.mpich.org/mailman/listinfo/discuss
>>> >
>>> > _______________________________________________
>>> > discuss mailing list discuss(a)mpich.org <mailto: discuss(a)mpich.org=
>
>>> > To manage subscription options or unsubscribe:
>>> > https: //lists.mpich.org/mailman/listinfo/discuss
>>>
>>> _______________________________________________
>>> discuss mailing list discuss(a)mpich.org <mailto: discuss(a)mpich.org>
>>> To manage subscription options or unsubscribe:
>>> https: //lists.mpich.org/mailman/listinfo/discuss
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> discuss mailing list discuss(a)mpich.org
>>> To manage subscription options or unsubscribe:
>>> https: //lists.mpich.org/mailman/listinfo/discuss
>>>
>>> _______________________________________________
>> discuss mailing list discuss(a)mpich.org
>> To manage subscription options or unsubscribe:
>> https: //lists.mpich.org/mailman/listinfo/discuss
>>
>
>
> _______________________________________________
> discuss mailing list discuss(a)mpich.org
> To manage subscription options or unsubscribe:
> https: //lists.mpich.org/mailman/listinfo/discuss
>
--001a11c108e02e57a30508cb605e
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dutf-8"><d=
iv dir=3D"ltr"><div>I have no idea. You may try to trace all events as said=
at <a href=3D"http: //wiki.mpich.org/mpich/index.php/Debug_Event_Logging">h=
ttp: //wiki.mpich.org/mpich/index.php/Debug_Event_Logging</a></div><div>From=
the trace log, one may find out something abnormal. </div></div=
><div class=3D"gmail_extra"><br clear=3D"all"><div><div class=3D"gmail_sign=
ature"><div dir=3D"ltr">--Junchao Zhang</div></div></div>
<br><div class=3D"gmail_quote">On Wed, Nov 26, 2014 at 4: 25 PM, Amin Hassan=
i <span dir=3D"ltr"><<a href=3D"mailto: ahassani(a)cis.uab.edu" target=3D"_=
blank">ahassani(a)cis.uab.edu</a>></span> wrote: <br><blockquote class=3D"g=
mail_quote" style=3D"margin: 0 0 0 .8ex;border-left:1px #ccc solid;padding-l=
eft: 1ex"><div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-family=
: tahoma,sans-serif;font-size:small">I disabled the whole firewall in those =
machines but, still get the same problem. connection refuse.</div><div clas=
s=3D"gmail_default" style=3D"font-family: tahoma,sans-serif;font-size:small"=
>I run the program in another set of totally different machines that we hav=
e, but still same problem.</div><div class=3D"gmail_default" style=3D"font-=
family: tahoma,sans-serif;font-size:small">Any other thought where can be th=
e problem?</div><div class=3D"gmail_default" style=3D"font-family: tahoma,sa=
ns-serif;font-size: small"><br></div><div class=3D"gmail_default" style=3D"f=
ont-family: tahoma,sans-serif;font-size:small">Thanks.</div></div><div class=
=3D"gmail_extra"><span class=3D""><br clear=3D"all"><div><div><div dir=3D"l=
tr">Amin Hassani,<br>CIS department at UAB,<br>
Birmingham, AL, USA.</div></div></div>
<br></span><div><div class=3D"h5"><div class=3D"gmail_quote">On Wed, Nov 26=
, 2014 at 9: 25 AM, Kenneth Raffenetti <span dir=3D"ltr"><<a href=3D"mail=
to: raffenet(a)mcs.anl.gov" target=3D"_blank">raffenet(a)mcs.anl.gov</a>></sp=
an> wrote: <br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;=
border-left: 1px #ccc solid;padding-left:1ex">The connection refused makes m=
e think a firewall is getting in the way. Is TCP communication limited to s=
pecific ports on the cluster? If so, you can use this envvar to enforce a r=
ange of ports in MPICH.<br>
<br>
MPIR_CVAR_CH3_PORT_RANGE<br>
Description: The MPIR_CVAR_CH3_PORT_RANGE environment variabl=
e allows you to specify the range of TCP ports to be used by the process ma=
nager and the MPICH library. The format of this variable is <low>: <=
;high>. To specify any available port, use 0: 0.<br>
Default: {0,0}<div><div><br>
<br>
On 11/25/2014 11: 50 PM, Amin Hassani wrote:<br>
</div></div><blockquote class=3D"gmail_quote" style=3D"margin: 0 0 0 .8ex;bo=
rder-left: 1px #ccc solid;padding-left:1ex"><div><div>
Tried with the new configure too. same problem: (<br>
<br>
$ mpirun -hostfile hosts-hydra -np 2 test_dup<br>
Fatal error in MPI_Send: Unknown error class, error stack:<br>
MPI_Send(174)..............: MPI_Send(buf=3D0x7fffd90c76c8, count=3D1,<br>
MPI_INT, dest=3D1, tag=3D0, MPI_COMM_WORLD) failed<br>
MPID_nem_tcp_connpoll(1832): Communication error with rank 1: Connection<br=
>
refused<br>
<br>
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D<u></u>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<u></u>=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br>
=3D BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES<br>
=3D PID 5459 RUNNING AT oakmnt-0-a<br>
=3D EXIT CODE: 1<br>
=3D CLEANING UP REMAINING PROCESSES<br>
=3D YOU CAN IGNORE THE BELOW CLEANUP MESSAGES<br>
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D<u></u>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<u></u>=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br>
[proxy: 0:1@oakmnt-0-b] HYD_pmcd_pmip_control_cmd_cb<br>
(../../../../src/pm/hydra/pm/<u></u>pmiserv/pmip_cb.c: 885): assert (!closed=
) failed<br>
[proxy: 0:1@oakmnt-0-b] HYDT_dmxu_poll_wait_for_event<br>
(../../../../src/pm/hydra/<u></u>tools/demux/demux_poll.c: 76): callback<br>
returned error status<br>
[proxy: 0:1@oakmnt-0-b] main<br>
(../../../../src/pm/hydra/pm/<u></u>pmiserv/pmip.c: 206): demux engine error=
<br>
waiting for event<br>
[mpiexec@oakmnt-0-a] HYDT_bscu_wait_for_completion<br>
(../../../../src/pm/hydra/<u></u>tools/bootstrap/utils/bscu_<u></u>wait.c: 7=
6): one of<br>
the processes terminated badly; aborting<br>
[mpiexec@oakmnt-0-a] HYDT_bsci_wait_for_completion<br>
(../../../../src/pm/hydra/<u></u>tools/bootstrap/src/bsci_wait.<u></u>c: 23)=
: launcher<br>
returned error waiting for completion<br>
[mpiexec@oakmnt-0-a] HYD_pmci_wait_for_completion<br>
(../../../../src/pm/hydra/pm/<u></u>pmiserv/pmiserv_pmci.c: 218): launcher<b=
r>
returned error waiting for completion<br>
[mpiexec@oakmnt-0-a] main<br>
(../../../../src/pm/hydra/ui/<u></u>mpich/mpiexec.c: 344): process manager e=
rror<br>
waiting for completion<br>
<br>
<br>
Amin Hassani,<br>
CIS department at UAB,<br>
Birmingham, AL, USA.<br>
<br>
On Tue, Nov 25, 2014 at 11: 44 PM, Lu, Huiwei <<a href=3D"mailto:huiweilu=
@mcs.anl.gov" target=3D"_blank">huiweilu(a)mcs.anl.gov</a><br></div></div><sp=
an>
<mailto: <a href=3D"mailto:[email protected]" target=3D"_blank">huiwei=
lu(a)mcs.anl.gov</a>>> wrote: <br>
<br>
So the error only happens when there is communication.<br>
<br>
It may be caused by IB as your guessed before. Could you try =
to<br>
reconfigure MPICH using "./configure --with-device=3Dch3=
: nemesis:tcp=E2=80=9D<br>
and try again?<br>
<br>
=E2=80=94<br>
Huiwei<br>
<br>
> On Nov 25, 2014, at 11: 23 PM, Amin Hassani <<a =
href=3D"mailto: ahassani(a)cis.uab.edu" target=3D"_blank">ahassani(a)cis.uab.edu=
</a><br></span><span>
<mailto: <a href=3D"mailto:[email protected]" target=3D"=
_blank">ahassani(a)cis.uab.edu</a>>> wrote: <br>
><br>
> Yes it works.<br>
> output: <br>
><br>
> $ mpirun -hostfile hosts-hydra -np 2 test<br=
>
> rank 1<br>
> rank 0<br>
><br>
><br>
> Amin Hassani,<br>
> CIS department at UAB,<br>
> Birmingham, AL, USA.<br>
><br>
> On Tue, Nov 25, 2014 at 11: 20 PM, Lu, Huiwei<br></=
span><span>
<<a href=3D"mailto: huiweilu(a)mcs.anl.gov" target=3D"_blank"=
>huiweilu(a)mcs.anl.gov</a> <mailto: <a href=3D"mailto:[email protected]=
" target=3D"_blank">huiweilu(a)mcs.anl.gov</a>>> wrote: <br>
> Could you try to run the following simple code to =
see if it works?<br>
><br>
> #include <mpi.h><br>
> #include <stdio.h><br>
> int main(int argc, char** argv)<br>
> {<br>
> int rank, size;<br>
> MPI_Init(&argc, &argv);=
<br>
> MPI_Comm_rank(MPI_COMM_WORLD, &=
amp;rank);<br>
> printf("rank %d\n", r=
ank);<br>
> MPI_Finalize();<br>
> return 0;<br>
> }<br>
><br>
> =E2=80=94<br>
> Huiwei<br>
><br>
> > On Nov 25, 2014, at 11: 11 PM, Amin Hassani<br=
></span><span>
<<a href=3D"mailto: ahassani(a)cis.uab.edu" target=3D"_blank"=
>ahassani(a)cis.uab.edu</a> <mailto: <a href=3D"mailto:[email protected]=
" target=3D"_blank">ahassani(a)cis.uab.edu</a>>> wrote: <br>
> ><br>
> > No, I checked. Also I always install my MPI's=
in<br>
/nethome/students/ahassani/<u></u>usr/mpi. I never install th=
em in<br>
/nethome/students/ahassani/<u></u>usr. So MPI files will neve=
r get there.<br>
Even if put the /usr/mpi/bin in front of /usr/bin, it won't a=
ffect<br>
anything. There has never been any mpi installed in /usr/bin.=
<br>
> ><br>
> > Thank you.<br>
> > ______________________________<u></u>________=
_________<br></span>
> > discuss mailing list <a href=3D"mailto: discus=
s(a)mpich.org" target=3D"_blank">discuss(a)mpich.org</a> <mailto: <a href=3D"=
mailto: discuss(a)mpich.org" target=3D"_blank">discuss(a)mpich.org</a>><span>=
<br>
> > To manage subscription options or unsubscribe=
: <br>
> > <a href=3D"https: //lists.mpich.org/mailman/li=
stinfo/discuss" target=3D"_blank">https: //lists.mpich.org/<u></u>mailman/li=
stinfo/discuss</a><br>
><br>
> ______________________________<u></u>_____________=
____<br></span>
> discuss mailing list <a href=3D"mailto: discuss@mpi=
ch.org" target=3D"_blank">discuss(a)mpich.org</a> <mailto: <a href=3D"mailt=
o: discuss(a)mpich.org" target=3D"_blank">discuss(a)mpich.org</a>><span><br>
> To manage subscription options or unsubscribe: <br>
> <a href=3D"https: //lists.mpich.org/mailman/listinf=
o/discuss" target=3D"_blank">https: //lists.mpich.org/<u></u>mailman/listinf=
o/discuss</a><br>
><br>
> ______________________________<u></u>_____________=
____<br></span>
> discuss mailing list <a href=3D"mailto: discuss@mpi=
ch.org" target=3D"_blank">discuss(a)mpich.org</a> <mailto: <a href=3D"mailt=
o: discuss(a)mpich.org" target=3D"_blank">discuss(a)mpich.org</a>><span><br>
> To manage subscription options or unsubscribe: <br>
> <a href=3D"https: //lists.mpich.org/mailman/listinf=
o/discuss" target=3D"_blank">https: //lists.mpich.org/<u></u>mailman/listinf=
o/discuss</a><br>
<br>
______________________________<u></u>_________________<br></s=
pan>
discuss mailing list <a href=3D"mailto: discuss(a)mpich.org" tar=
get=3D"_blank">discuss(a)mpich.org</a> <mailto: <a href=3D"mailto:discuss@m=
pich.org" target=3D"_blank">discuss(a)mpich.org</a>><span><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" =
target=3D"_blank">https: //lists.mpich.org/<u></u>mailman/listinfo/discuss</=
a><br>
<br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
discuss mailing list <a href=3D"mailto: discuss(a)mpich.org=
" target=3D"_blank">discuss(a)mpich.org</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" target=3D"_bla=
nk">https: //lists.mpich.org/<u></u>mailman/listinfo/discuss</a><br>
<br>
</span></blockquote><div><div>
______________________________<u></u>_________________<br>
discuss mailing list <a href=3D"mailto: discuss(a)mpich.org=
" target=3D"_blank">discuss(a)mpich.org</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" target=3D"_bla=
nk">https: //lists.mpich.org/<u></u>mailman/listinfo/discuss</a></div></div>=
</blockquote></div><br></div></div></div>
<br>_______________________________________________<br>
discuss mailing list <a href=3D"mailto: discuss(a)mpich.org=
">discuss(a)mpich.org</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" target=3D"_bla=
nk">https: //lists.mpich.org/mailman/listinfo/discuss</a><br></blockquote></=
div><br></div>
--001a11c108e02e57a30508cb605e--
--===============2762240988201403401==
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
discuss mailing list discuss(a)mpich.org
To manage subscription options or unsubscribe:
https: //lists.mpich.org/mailman/listinfo/discuss
--===============2762240988201403401==--
Message-ID: <sanitized-1657(a)migration.local>
1
0
MPI_Barrier before MPI_Allreduce can improve performance
significantly.
This is hardly the full story. It would be useful to know more about
what you are trying to accomplish.
Best,
Jeff
On Wed, Feb 25, 2015 at 3: 07 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote:
> Yes. Many collectives have optimizations for power-of-two processes. In
> MPICH's source code allreduce.c, you can find the following comments.
>
> /* This is the default implementation of allreduce. The algorithm is:
>
> Algorithm: MPI_Allreduce
>
> For the heterogeneous case, we call MPI_Reduce followed by MPI_Bcast
> in order to meet the requirement that all processes must have the
> same result. For the homogeneous case, we use the following algorithms.
>
>
> For long messages and for builtin ops and if count >= pof2 (where
> pof2 is the nearest power-of-two less than or equal to the number
> of processes), we use Rabenseifner's algorithm (see
> http: //www.hlrs.de/mpi/myreduce.html).
> This algorithm implements the allreduce in two steps: first a
> reduce-scatter, followed by an allgather. A recursive-halving
> algorithm (beginning with processes that are distance 1 apart) is
> used for the reduce-scatter, and a recursive doubling
> algorithm is used for the allgather. The non-power-of-two case is
> handled by dropping to the nearest lower power-of-two: the first
> few even-numbered processes send their data to their right neighbors
> (rank+1), and the reduce-scatter and allgather happen among the remaining
> power-of-two processes. At the end, the first few even-numbered
> processes get the result from their right neighbors.
>
> For the power-of-two case, the cost for the reduce-scatter is
> lgp.alpha + n.((p-1)/p).beta + n.((p-1)/p).gamma. The cost for the
> allgather lgp.alpha + n.((p-1)/p).beta. Therefore, the
> total cost is:
> Cost = 2.lgp.alpha + 2.n.((p-1)/p).beta + n.((p-1)/p).gamma
>
> For the non-power-of-two case,
> Cost = (2.floor(lgp)+2).alpha + (2.((p-1)/p) + 2).n.beta +
> n.(1+(p-1)/p).gamma
>
>
> For short messages, for user-defined ops, and for count < pof2
> we use a recursive doubling algorithm (similar to the one in
> MPI_Allgather). We use this algorithm in the case of user-defined ops
> because in this case derived datatypes are allowed, and the user
> could pass basic datatypes on one process and derived on another as
> long as the type maps are the same. Breaking up derived datatypes
> to do the reduce-scatter is tricky.
>
> Cost = lgp.alpha + n.lgp.beta + n.lgp.gamma
>
> Possible improvements:
>
> End Algorithm: MPI_Allreduce
> */
>
> --Junchao Zhang
>
> On Wed, Feb 25, 2015 at 2: 59 PM, Aiman Fang <aimanf(a)cs.uchicago.edu> wrote:
>>
>> Hi,
>>
>> I came across a problem in experiments that makes me wondering if there is
>> any optimization of collective calls, such as MPI_Allreduce, for 2^n number
>> of ranks?
>>
>> We did some experiments on Argonne Vesta system to measure the time of
>> MPI_Allreduce calls using 511, 512 and 513 processes. (one process per
>> node). In each run, the synthetic benchmark first did some computation and
>> then called MPI_Allreduce 30 times, for total 100 loops. We measured the
>> total time spent on communication.
>>
>> We found that 512-process run gives the best performance. The time for
>> 511, 512 and 513 processes are 0.1492, 0.1449 and 0.1547 seconds
>> respectively. 512-proc outperforms 511-proc by 3.7%, and 513-proc by 6.7%.
>>
>> The mpich version we used is as follows.
>>
>> $ mpichversion
>> MPICH Version: 3.1.2
>> MPICH Release date: Mon Jul 21 16:00:21 CDT 2014
>> MPICH Device: pamid
>> MPICH configure: --prefix=/home/fujita/soft/mpich-3.1.2
>> --host=powerpc64-bgq-linux --with-device=pamid --with-file-system=gpfs: BGQ
>> --disable-wrapper-rpath
>> MPICH CC: powerpc64-bgq-linux-gcc -O2
>> MPICH CXX: powerpc64-bgq-linux-g++ -O2
>> MPICH F77: powerpc64-bgq-linux-gfortran -O2
>> MPICH FC: powerpc64-bgq-linux-gfortran -O2
>>
>> Thanks!
>>
>> Best,
>> Aiman
>> _______________________________________________
>> discuss mailing list discuss(a)mpich.org
>> To manage subscription options or unsubscribe:
>> https: //lists.mpich.org/mailman/listinfo/discuss
>
>
>
> _______________________________________________
> discuss mailing list discuss(a)mpich.org
> To manage subscription options or unsubscribe:
> https: //lists.mpich.org/mailman/listinfo/discuss
--
Jeff Hammond
jeff.science(a)gmail.com
http: //jeffhammond.github.io/
_______________________________________________
discuss mailing list discuss(a)mpich.org
To manage subscription options or unsubscribe:
https: //lists.mpich.org/mailman/listinfo/discuss
Message-ID: <sanitized-1438(a)migration.local>
1
0
$ ./mpi-run.sh
Running cpi on machines.u2.mpi
Process 0 of 1 is on UNIT2
pi is approximately 3.1415926544231341, Error is 0.0000000008333410
wall clock time =3D 0.001327
Done!
If I edit the script to change the mpiexec line like this:
sudo -E LD_LIBRARY_PATH=3D${ELIBS} mpiexec --allow-run-as-root -machinefile=
/home/linaro/.machines.u2.mpi -n 1 /home/linaro/myMPI/cpi
Now I get (edited for brevity):
$ ./mpi-run.sh
Running cpi on machines.u2.mpi
linaro@UNIT2's password:
PATH=3D/usr/local/bin[...]: Command not found.
export: Command not found.
LD_LIBRARY_PATH=3D/usr/local/lib[...]: Command not found.
export: Command not found.
DYLD_LIBRARY_PATH: Undefined variable.
And it just stops there. Note that the LD_LIBRARY_PATH being reported is *=
not* the one passed in by the script. I don't think it's managing to reach=
the mpi execution stage itself.
If the machinefile lists more than one host, the password prompts appear tw=
o at a time and interfere with each other such that no login succeeds (alth=
ough all machines have the same password).
Googling around, I've seen this series of error outputs in a wide variety o=
f other contexts, including Open MPI but also some completely unrelated app=
lication suites and SDKs.
My problem is that the mpi binaries I need to run on the hosts absolutely r=
equire sudo elevation. Is sudo mpiexec the way to go? What is going on in=
my example case?
Daniel U. Thibault
RDDC - Centre de recherches de Valcartier | DRDC - Valcartier Research Cent=
re
NAC: 918V QSDJ<http://www.travelgis.com/map.asp?addr=3D918V%20QSDJ> <http:=
//www.travelgis.com/map.asp?addr=3D918V%20QSDJ>
Gouvernement du Canada | Government of Canada
<http: //www.valcartier.drdc-rddc.gc.ca/>
--_000_48CF5AC71E61DB46B70D0F388054EFFD86F70ED5VALE02valcartie_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<html><head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
<meta name=3D"Generator" content=3D"Microsoft Exchange Server">
<!-- converted from rtf -->
<style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left:=
#800000 2px solid; } --></style>
</head>
<body>
<font face=3D"Calibri" size=3D"2"><span style=3D"font-size: 11pt;">
<div> I have a small network of machines all running the same O=
S (Linaro Ubuntu Linux); they were all cloned from the same disk image and =
differ only in their machine names (UNIT1 through UNIT4).</div>
<div> </div>
<div> I can ssh between them at will, trusty has been establish=
ed and I no longer get asked for a password upon connecting. MPICH is=
installed from the Ubuntu repository (not quite the latest version: mpiexe=
c reports version OpenRTE 1.8.1, the mpich package
is 3.0.4-6ubuntu1), and I can run a demo like cpi with no issues, using a l=
ittle mpi-run.sh bash script (the default shell is tcsh, however): </div>
<div> </div>
<div>(begin script)</div>
<div>#!/bin/bash</div>
<div> </div>
<div>set -e</div>
<div> </div>
<div>ESDK=3D${EPIPHANY_HOME}</div>
<div>ELIBS=3D${ESDK}/tools/host/lib: ${LD_LIBRARY_PATH}</div>
<div>EHDF=3D${EPIPHANY_HDF}</div>
<div> </div>
<div>echo "Running cpi on machines.u2.mpi"</div>
<div>LD_LIBRARY_PATH=3D${ELIBS} mpiexec --allow-run-as-root -machinefile /h=
ome/linaro/.machines.u2.mpi -n 1 /home/linaro/myMPI/cpi</div>
<div>echo "Done!"</div>
<div>(end script)</div>
<div> </div>
<div>.machines.u2.mpi consists of the one line: </div>
<div> </div>
<div>linaro@UNIT2</div>
<div> </div>
<div>From UNIT1, if I do: </div>
<div> </div>
<div>$ ./mpi-run.sh</div>
<div>Running cpi on machines.u2.mpi</div>
<div>Process 0 of 1 is on UNIT2</div>
<div>pi is approximately 3.1415926544231341, Error is 0.0000000008333410</d=
iv>
<div>wall clock time =3D 0.001327</div>
<div>Done!</div>
<div> </div>
<div>If I edit the script to change the mpiexec line like this: </div>
<div> </div>
<div>sudo -E LD_LIBRARY_PATH=3D${ELIBS} mpiexec --allow-run-as-root -machin=
efile /home/linaro/.machines.u2.mpi -n 1 /home/linaro/myMPI/cpi</div>
<div> </div>
<div>Now I get (edited for brevity): </div>
<div> </div>
<div>$ ./mpi-run.sh</div>
<div>Running cpi on machines.u2.mpi</div>
<div>linaro@UNIT2’s password: </div>
<div>PATH=3D/usr/local/bin[…]: Command not found.</div>
<div>export: Command not found.</div>
<div>LD_LIBRARY_PATH=3D/usr/local/lib[…]: Command not found.</div>
<div>export: Command not found.</div>
<div>DYLD_LIBRARY_PATH: Undefined variable.</div>
<div> </div>
<div>And it just stops there. Note that the LD_LIBRARY_PATH being rep=
orted is *<b>not</b>* the one passed in by the script. I don’t =
think it’s managing to reach the mpi execution stage itself.</div>
<div> </div>
<div>If the machinefile lists more than one host, the password prompts appe=
ar two at a time and interfere with each other such that no login succeeds =
(although all machines have the same password).</div>
<div> </div>
<div>Googling around, I’ve seen this series of error outputs in a wid=
e variety of other contexts, including Open MPI but also some completely un=
related application suites and SDKs.</div>
<div> </div>
<div>My problem is that the mpi binaries I need to run on the hosts absolut=
ely require sudo elevation. Is sudo mpiexec the way to go? What=
is going on in my example case?</div>
<div> </div>
<div><font face=3D"Consolas">Daniel U. Thibault<br>
RDDC - Centre de recherches de Valcartier | DRDC - Valcartier Research Cent=
re<br>
NAC : <a href=3D"http://www.travelgis.com/map.asp?addr=3D918V%20QSDJ">=
<font face=3D"Courier New" size=3D"2" color=3D"blue"><span style=3D"font-si=
ze: 10pt;"><u>918V QSDJ</u></span></font></a><font face=3D"Calibri" color=3D=
"#1F497D"> </font><font face=3D"Calibri"><</font><font face=3D"Courier N=
ew" size=3D"2" color=3D"blue"><span style=3D"font-size: 10pt;"><u><a href=3D=
"http: //www.travelgis.com/map.asp?addr=3D918V%20QSDJ">http://www.travelgis.=
com/map.asp?addr=3D918V%20QSDJ</a></u></span></font><font face=3D"Calibri">=
><br>
</font>Gouvernement du Canada | Government of Canada<br>
<font face=3D"Calibri"><</font><a href=3D"http: //www.valcartier.drdc-rdd=
c.gc.ca/"><font face=3D"Courier New" size=3D"2" color=3D"blue"><span style=
=3D"font-size: 10pt;"><u>http://www.valcartier.drdc-rddc.gc.ca/</u></span></=
font></a><font face=3D"Calibri">></font></font></div>
<div> </div>
</span></font>
</body>
</html>
--_000_48CF5AC71E61DB46B70D0F388054EFFD86F70ED5VALE02valcartie_--
--===============9162204499992570155==
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
discuss mailing list discuss(a)mpich.org
To manage subscription options or unsubscribe:
https: //lists.mpich.org/mailman/listinfo/discuss
--===============9162204499992570155==--
Message-ID: <sanitized-1170(a)migration.local>
1
0
ld you elaborate on why the dynamic process model is not sufficient for you=
r needs?<br>
<br>
Halim<br>
<a href=3D"http: //www.mcs.anl.gov/~aamer" target=3D"_blank" rel=3D"noreferr=
er">www.mcs.anl.gov/~aamer</a><div><div class=3D"h5"><br>
<br>
On 5/26/17 9: 11 AM, sanjeev s wrote:<br>
</div></div><blockquote class=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0=
.8ex;padding-left: 1ex;border-left-color:rgb(204,204,204);border-left-width:=
1px;border-left-style: solid"><div><div class=3D"h5">
Hi mpich,<br>
<br>
I have a requirement where in we need to add start stop application<br>
instances on the fly before starting a job.Is there any mpich service<br>
available. I looked through dynamic process model, but its not sufficing<br=
>
our need.<br>
<br>
More precisely my requirement is suppose I started 4 instances of my<br>
application. Now I want to add one more instance dynamically to this set<br=
>
<br>
Is there any tool which MPICH supports for fault tolerance behavior?<br>
<br>
Thanks<br>
Sanjeev<br>
<br>
<br>
<br></div></div>
______________________________<wbr>_________________<br>
discuss mailing list <a href=3D"mailto: discuss(a)mpich.org=
" target=3D"_blank">discuss(a)mpich.org</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" target=3D"_bla=
nk" rel=3D"noreferrer">https: //lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
</blockquote>
______________________________<wbr>_________________<br>
discuss mailing list <a href=3D"mailto: discuss(a)mpich.org=
" target=3D"_blank">discuss(a)mpich.org</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" target=3D"_bla=
nk" rel=3D"noreferrer">https: //lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
</blockquote></div><br></div></div>
--94eb2c1309286f930a05506f28f3--
--===============0395035545747262554==
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
discuss mailing list discuss(a)mpich.org
To manage subscription options or unsubscribe:
https: //lists.mpich.org/mailman/listinfo/discuss
--===============0395035545747262554==--
Message-ID: <sanitized-107(a)migration.local>
1
0
ld you elaborate on why the dynamic process model is not sufficient for you=
r needs?<br>
<br>
Halim<br>
</span><a href=3D"http: //www.mcs.anl.gov/~aamer" target=3D"_blank" rel=3D"n=
oreferrer">www.mcs.anl.gov/~aamer</a> <<a href=3D"http: //www.mcs.anl.gov=
/~aamer" target=3D"_blank" rel=3D"noreferrer">http: //www.mcs.anl.gov/~aamer=
</a><wbr>><br>
<div class=3D"gmail-HOEnZb"><div class=3D"gmail-h5"><br>
On 5/26/17 9: 11 AM, sanjeev s wrote:<br>
<br>
<br>
Hi mpich,<br>
<br>
I have a requirement where in we need to add start stop application<br>
instances on the fly before starting a job.Is there any mpich service<br>
available. I looked through dynamic process model, but its not sufficing<br=
>
our need.<br>
<br>
More precisely my requirement is suppose I started 4 instances of my<br>
application. Now I want to add one more instance dynamically to this set<br=
>
<br>
Is there any tool which MPICH supports for fault tolerance behavior?<br>
<br>
Thanks<br>
Sanjeev<br>
<br>
<br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list <a href=3D"mailto: discuss(a)mpich.org=
">discuss(a)mpich.org</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" target=3D"_bla=
nk" rel=3D"noreferrer">https: //lists.mpich.org/<wbr>mailman/listinfo/discus=
s</a><br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list <a href=3D"mailto: discuss(a)mpich.org=
">discuss(a)mpich.org</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" target=3D"_bla=
nk" rel=3D"noreferrer">https: //lists.mpich.org/<wbr>mailman/listinfo/discus=
s</a><br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list <a href=3D"mailto: discuss(a)mpich.org=
">discuss(a)mpich.org</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" target=3D"_bla=
nk" rel=3D"noreferrer">https: //lists.mpich.org/<wbr>mailman/listinfo/discus=
s</a></div></div></blockquote></div><br></div></div>
--001a113d859ad7206a0550a12507--
--===============7312916256483474646==
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
discuss mailing list discuss(a)mpich.org
To manage subscription options or unsubscribe:
https: //lists.mpich.org/mailman/listinfo/discuss
--===============7312916256483474646==--
Message-ID: <sanitized-101(a)migration.local>
1
0
ld you elaborate on why the dynamic process model is not sufficient for you=
r needs?<br>
<br>
Halim<br>
</span><a href=3D"http: //www.mcs.anl.gov/~aamer" target=3D"_blank" rel=3D"n=
oreferrer">www.mcs.anl.gov/~aamer</a> <<a href=3D"http: //www.mcs.anl.gov=
/~aamer" target=3D"_blank" rel=3D"noreferrer">http: //www.mcs.anl.gov/~aamer=
</a><wbr>><br>
<div class=3D"m_-5576725648344379192gmail-HOEnZb"><div class=3D"m_-55767256=
48344379192gmail-h5"><br>
On 5/26/17 9: 11 AM, sanjeev s wrote:<br>
<br>
<br>
Hi mpich,<br>
<br>
I have a requirement where in we need to add start stop application<br>
instances on the fly before starting a job.Is there any mpich service<br>
available. I looked through dynamic process model, but its not sufficing<br=
>
our need.<br>
<br>
More precisely my requirement is suppose I started 4 instances of my<br>
application. Now I want to add one more instance dynamically to this set<br=
>
<br>
Is there any tool which MPICH supports for fault tolerance behavior?<br>
<br>
Thanks<br>
Sanjeev<br>
<br>
<br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list <a href=3D"mailto: discuss(a)mpich.org=
" target=3D"_blank">discuss(a)mpich.org</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" target=3D"_bla=
nk" rel=3D"noreferrer">https: //lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list <a href=3D"mailto: discuss(a)mpich.org=
" target=3D"_blank">discuss(a)mpich.org</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" target=3D"_bla=
nk" rel=3D"noreferrer">https: //lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list <a href=3D"mailto: discuss(a)mpich.org=
" target=3D"_blank">discuss(a)mpich.org</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" target=3D"_bla=
nk" rel=3D"noreferrer">https: //lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a></div></div></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>
--f403045f55784cc1290550c88b83--
--===============4430482482998220797==
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
discuss mailing list discuss(a)mpich.org
To manage subscription options or unsubscribe:
https: //lists.mpich.org/mailman/listinfo/discuss
--===============4430482482998220797==--
Message-ID: <sanitized-99(a)migration.local>
1
0