None

25 Jun 2026

      loads and stores in the MPI_Win_lock_all epochs using MPI_Fetch_and_op (see
attached files).

This version behaves very similar to the original code and also fails from
time to time. Putting a sleep into the acquire busy loop (usleep(100)) will
make the code "much more robust" (I hack, I know, but indicating some
underlying race condition?!). Let me know if you see any problems in the
way I am using MPI_Fetch_and_op in a busy loop. Flushing or syncing is not
necessary in this case, right?

All work is done with export MPIR_CVAR_ASYNC_PROGRESS=1 on mpich-3.2 and
mpich-3.3a2

On Wed, Mar 8, 2017 at 4: 21 PM, Halim Amer <[email protected]> wrote:
...
I cannot claim that I thoroughly verified the correctness of that code, so
take it with a grain of salt. Please keep in mind that it is a test code
from a tutorial book; those codes are meant for learning purposes not for
deployment.
If your goal is to have a high performance RMA lock, I suggest you to look
into the recent HPDC'16 paper: "High-Performance Distributed RMA Locks".
Halim
www.mcs.anl.gov/~aamer
On 3/8/17 3: 06 AM, Ask Jakobsen wrote:
...
You are absolutely correct, Halim. Removing the test lmem[nextRank] == -1
in release fixes the problem. Great work. Now I will try to understand why
you are right. I hope the authors of the book will credit you for
discovering the bug.
So in conclusion you need to remove the above mentioned test AND enable
asynchronous progression using the environment variable
MPIR_CVAR_ASYNC_PROGRESS=1 in MPICH (BTW I still can't get the code to
work
in openmpi).
On Tue, Mar 7, 2017 at 5: 37 PM, Halim Amer <[email protected]> wrote:
detect that another process is being or already enqueued in the MCS
...
...
queue.
Actually the problem occurs only when the waiting process already
enqueued
itself, i.e., the accumulate operation on the nextRank field succeeded.
Halim
www.mcs.anl.gov/~aamer <http: //www.mcs.anl.gov/%7Eaamer>
On 3/7/17 10: 29 AM, Halim Amer wrote:
In the Release protocol, try removing this test:
...
if (lmem[nextRank] == -1) {
   If-Block;
}
but keep the If-Block.
The hang occurs because the process releasing the MCS lock fails to
detect that another process is being or already enqueued in the MCS
queue.
Halim
www.mcs.anl.gov/~aamer <http: //www.mcs.anl.gov/%7Eaamer>
On 3/7/17 6: 43 AM, Ask Jakobsen wrote:
Thanks, Halim. I have now enabled asynchronous progress in MPICH (can't
...
find something similar in openmpi) and now all ranks acquire the lock
and
the program finish as expected. However if I put a while(1) loop
around the
acquire-release code in main.c it will fail again at random and go
into an
infinite loop. The simple unfair lock does not have this problem.
On Tue, Mar 7, 2017 at 12: 44 AM, Halim Amer <[email protected]> wrote:
My understanding is that this code assumes asynchronous progress.
...
An example of when the processes hang is as follows:
1) P0 Finishes MCSLockAcquire()
2) P1 is busy waiting in MCSLockAcquire() at
do {
      MPI_Win_sync(win);
   } while (lmem[blocked] == 1);
3) P0 executes MCSLockRelease()
4) P0 waits on MPI_Win_lock_all() inside MCSLockRlease()
Hang!
For P1 to get out of the loop, P0 has to get out of
MPI_Win_lock_all() and
executes its Compare_and_swap().
For P0 to get out MPI_Win_lock_all(), it needs an ACK from P1 that it
got
the lock.
P1 does not make communication progress because MPI_Win_sync is not
required to do so. It only synchronizes private and public copies.
For this hang to disappear, one can either trigger progress manually
by
using heavy-duty synchronization calls instead of Win_sync (e.g.,
Win_unlock_all + Win_lock_all), or enable asynchronous progress.
To enable asynchronous progress in MPICH, set the
MPIR_CVAR_ASYNC_PROGRESS
env var to 1.
Halim
www.mcs.anl.gov/~aamer <http: //www.mcs.anl.gov/%7Eaamer> <
http: //www.mcs.anl.gov/%7Eaamer>
On 3/6/17 1: 11 PM, Ask Jakobsen wrote:
I am testing on x86_64 platform.
...
I have tried to built both the mpich and the mcs lock code with -O0
to
avoid agressive optimization. After your suggestion I have also
tried to
make volatile int *pblocked pointing to lmem[blocked] in the
MCSLockAcquire
function and volatile int *pnextrank pointing to lmem[nextRank] in
MCSLockRelease, but it does not appear to make a difference.
On suggestion from Richard Warren I have also tried building the code
using
openmpi-2.0.2 without any luck (however it appears to acquire the
lock a
couple of extra times before failing) which I find troubling.
I think I will give up using local load/stores and will see if I can
figure
out if rewrite using MPI calls like MPI_Fetch_and_op  as you suggest.
Thanks for your help.
On Mon, Mar 6, 2017 at 7: 20 PM, Jeff Hammond <[email protected]
...
wrote:
What processor architecture are you testing?
...
Maybe set lmem to volatile or read it with MPI_Fetch_and_op rather
than a
load.  MPI_Win_sync cannot prevent the compiler from caching *lmem
in a
register.
Jeff
On Sat, Mar 4, 2017 at 12: 30 AM, Ask Jakobsen <[email protected]>
wrote:
Hi,
...
I have downloaded the source code for the MCS lock from the
excellent
book "Using Advanced MPI" from http: //www.mcs.anl.gov/researc
h/projects/mpi/usingmpi/examples-advmpi/rma2/mcs-lock.c
I have made a very simple piece of test code for testing the MCS
lock
but
it works at random and often never escapes the busy loops in the
acquire
and release functions (see attached source code). The code appears
semantically correct to my eyes.
#include <stdio.h>
#include <mpi.h>
#include "mcs-lock.h"
int main(int argc, char *argv[])
{
  MPI_Win win;
  MPI_Init( &argc, &argv );
MCSLockInit(MPI_COMM_WORLD, &win);
int rank, size;
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  MPI_Comm_size(MPI_COMM_WORLD, &size);
printf("rank: %d, size: %d\n", rank, size);
MCSLockAcquire(win);
  printf("rank %d aquired lock\n", rank);   fflush(stdout);
  MCSLockRelease(win);
MPI_Win_free(&win);
  MPI_Finalize();
  return 0;
}
I have tested on several hardware platforms and mpich-3.2 and
mpich-3.3a2
but with no luck.
It appears that the MPI_Win_Sync are not "refreshing" the local
data or
I
have a bug I can't spot.
A simple unfair lock like http: //www.mcs.anl.gov/researc
h/projects/mpi/usingmpi/examples-advmpi/rma2/ga_mutex1.c works
perfectly.
Best regards, Ask Jakobsen
_______________________________________________
discuss mailing list     [email protected]
To manage subscription options or unsubscribe: 
https: //lists.mpich.org/mailman/listinfo/discuss
--
Jeff Hammond
[email protected]
http: //jeffhammond.github.io/
_______________________________________________
discuss mailing list     [email protected]
To manage subscription options or unsubscribe: 
https: //lists.mpich.org/mailman/listinfo/discuss
_______________________________________________
discuss mailing list     [email protected]
To manage subscription options or unsubscribe: 
https: //lists.mpich.org/mailman/listinfo/discuss
_______________________________________________
discuss mailing list     [email protected]
To manage subscription options or unsubscribe: 
https: //lists.mpich.org/mailman/listinfo/discuss
_______________________________________________
discuss mailing list     [email protected]
To manage subscription options or unsubscribe: 
https: //lists.mpich.org/mailman/listinfo/discuss
_______________________________________________
discuss mailing list     [email protected]
To manage subscription options or unsubscribe: 
https: //lists.mpich.org/mailman/listinfo/discuss
_______________________________________________
discuss mailing list     [email protected]
To manage subscription options or unsubscribe: 
https: //lists.mpich.org/mailman/listinfo/discuss
_______________________________________________
discuss mailing list     [email protected]
To manage subscription options or unsubscribe: 
https: //lists.mpich.org/mailman/listinfo/discuss
_______________________________________________
discuss mailing list     [email protected]
To manage subscription options or unsubscribe: 
https: //lists.mpich.org/mailman/listinfo/discuss
--001a1141dbd03846c4054a8dd443
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div><br></div>All work is done with export MPIR_CVAR_ASYNC_PROGRESS=3D1 on=
 mpich-3.2 and mpich-3.3a2<div><div><div class=3D"gmail_extra"><br></div><d=
iv class=3D"gmail_extra"><div class=3D"gmail_quote">On Wed, Mar 8, 2017 at =
4: 21 PM, Halim Amer <span dir=3D"ltr"><<a href=3D"mailto:[email protected]" =
target=3D"_blank">[email protected]</a>></span> wrote: <br><blockquote class=
=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex;border-left:1px solid rg=
b(204,204,204);padding-left: 1ex">I cannot claim that I thoroughly verified =
the correctness of that code, so take it with a grain of salt. Please keep =
in mind that it is a test code from a tutorial book; those codes are meant =
for learning purposes not for deployment.<br>
<br>
If your goal is to have a high performance RMA lock, I suggest you to look =
into the recent HPDC'16 paper: "High-Performance Distributed RMA Locks=
".<br>
<br>
Halim<br>
<a href=3D"http: //www.mcs.anl.gov/~aamer" rel=3D"noreferrer" target=3D"_bla=
nk">www.mcs.anl.gov/~aamer</a><span class=3D"gmail-m_8000249642949159004gma=
il-m_1322166946455689838gmail-"><br>
<br>
On 3/8/17 3: 06 AM, Ask Jakobsen wrote:<br>
</span><blockquote class=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex;=
border-left: 1px solid rgb(204,204,204);padding-left:1ex"><span class=3D"gma=
il-m_8000249642949159004gmail-m_1322166946455689838gmail-">
You are absolutely correct, Halim. Removing the test lmem[nextRank] =3D=3D =
-1<br>
in release fixes the problem. Great work. Now I will try to understand why<=
br>
you are right. I hope the authors of the book will credit you for<br>
discovering the bug.<br>
<br>
So in conclusion you need to remove the above mentioned test AND enable<br>
asynchronous progression using the environment variable<br>
MPIR_CVAR_ASYNC_PROGRESS=3D1 in MPICH (BTW I still can't get the code to wo=
rk<br>
in openmpi).<br>
<br>
On Tue, Mar 7, 2017 at 5: 37 PM, Halim Amer <<a href=3D"mailto:aamer@anl.=
gov" target=3D"_blank">[email protected]</a>> wrote: <br>
<br>
</span><blockquote class=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex;=
border-left: 1px solid rgb(204,204,204);padding-left:1ex"><span class=3D"gma=
il-m_8000249642949159004gmail-m_1322166946455689838gmail-"><blockquote clas=
s=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex;border-left:1px solid r=
gb(204,204,204);padding-left: 1ex">
detect that another process is being or already enqueued in the MCS<br>
</blockquote>
queue.<br>
<br>
Actually the problem occurs only when the waiting process already enqueued<=
br>
itself, i.e., the accumulate operation on the nextRank field succeeded.<br>
<br>
Halim<br>
</span><a href=3D"http: //www.mcs.anl.gov/~aamer" rel=3D"noreferrer" target=
=3D"_blank">www.mcs.anl.gov/~aamer</a> <<a href=3D"http: //www.mcs.anl.go=
v/%7Eaamer" rel=3D"noreferrer" target=3D"_blank">http: //www.mcs.anl.gov/%7E=
aam<wbr>er</a>><span class=3D"gmail-m_8000249642949159004gmail-m_1322166=
946455689838gmail-"><br>
<br>
<br>
On 3/7/17 10: 29 AM, Halim Amer wrote:<br>
<br>
</span><blockquote class=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex;=
border-left: 1px solid rgb(204,204,204);padding-left:1ex"><span class=3D"gma=
il-m_8000249642949159004gmail-m_1322166946455689838gmail-">
In the Release protocol, try removing this test: <br>
<br>
if (lmem[nextRank] =3D=3D -1) {<br>
   If-Block;<br>
}<br>
<br>
but keep the If-Block.<br>
<br>
The hang occurs because the process releasing the MCS lock fails to<br>
detect that another process is being or already enqueued in the MCS queue.<=
br>
<br>
Halim<br>
</span><a href=3D"http: //www.mcs.anl.gov/~aamer" rel=3D"noreferrer" target=
=3D"_blank">www.mcs.anl.gov/~aamer</a> <<a href=3D"http: //www.mcs.anl.go=
v/%7Eaamer" rel=3D"noreferrer" target=3D"_blank">http: //www.mcs.anl.gov/%7E=
aam<wbr>er</a>><div><div class=3D"gmail-m_8000249642949159004gmail-m_132=
2166946455689838gmail-h5"><br>
<br>
On 3/7/17 6: 43 AM, Ask Jakobsen wrote:<br>
<br>
<blockquote class=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex;border-=
left: 1px solid rgb(204,204,204);padding-left:1ex">
Thanks, Halim. I have now enabled asynchronous progress in MPICH (can't<br>
find something similar in openmpi) and now all ranks acquire the lock and<b=
r>
the program finish as expected. However if I put a while(1) loop<br>
around the<br>
acquire-release code in main.c it will fail again at random and go<br>
into an<br>
infinite loop. The simple unfair lock does not have this problem.<br>
<br>
On Tue, Mar 7, 2017 at 12: 44 AM, Halim Amer <<a href=3D"mailto:aamer@anl=
.gov" target=3D"_blank">[email protected]</a>> wrote: <br>
<br>
My understanding is that this code assumes asynchronous progress.<br>
<blockquote class=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex;border-=
left: 1px solid rgb(204,204,204);padding-left:1ex">
An example of when the processes hang is as follows: <br>
<br>
1) P0 Finishes MCSLockAcquire()<br>
2) P1 is busy waiting in MCSLockAcquire() at<br>
do {<br>
      MPI_Win_sync(win);<br>
   } while (lmem[blocked] =3D=3D 1);<br>
3) P0 executes MCSLockRelease()<br>
4) P0 waits on MPI_Win_lock_all() inside MCSLockRlease()<br>
<br>
Hang!<br>
<br>
For P1 to get out of the loop, P0 has to get out of<br>
MPI_Win_lock_all() and<br>
executes its Compare_and_swap().<br>
<br>
For P0 to get out MPI_Win_lock_all(), it needs an ACK from P1 that it<br>
got<br>
the lock.<br>
<br>
P1 does not make communication progress because MPI_Win_sync is not<br>
required to do so. It only synchronizes private and public copies.<br>
<br>
For this hang to disappear, one can either trigger progress manually by<br>
using heavy-duty synchronization calls instead of Win_sync (e.g.,<br>
Win_unlock_all + Win_lock_all), or enable asynchronous progress.<br>
<br>
To enable asynchronous progress in MPICH, set the<br>
MPIR_CVAR_ASYNC_PROGRESS<br>
env var to 1.<br>
<br>
Halim<br>
<a href=3D"http: //www.mcs.anl.gov/~aamer" rel=3D"noreferrer" target=3D"_bla=
nk">www.mcs.anl.gov/~aamer</a> <<a href=3D"http: //www.mcs.anl.gov/%7Eaam=
er" rel=3D"noreferrer" target=3D"_blank">http: //www.mcs.anl.gov/%7Eaam<wbr>=
er</a>> <<br>
<a href=3D"http: //www.mcs.anl.gov/%7Eaamer" rel=3D"noreferrer" target=3D"_b=
lank">http: //www.mcs.anl.gov/%7Eaame<wbr>r</a>><br>
<br>
<br>
On 3/6/17 1: 11 PM, Ask Jakobsen wrote:<br>
<br>
 I am testing on x86_64 platform.<br>
<blockquote class=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex;border-=
left: 1px solid rgb(204,204,204);padding-left:1ex">
<br>
I have tried to built both the mpich and the mcs lock code with -O0 to<br>
avoid agressive optimization. After your suggestion I have also<br>
tried to<br>
make volatile int *pblocked pointing to lmem[blocked] in the<br>
MCSLockAcquire<br>
function and volatile int *pnextrank pointing to lmem[nextRank] in<br>
MCSLockRelease, but it does not appear to make a difference.<br>
<br>
On suggestion from Richard Warren I have also tried building the code<br>
using<br>
openmpi-2.0.2 without any luck (however it appears to acquire the<br>
lock a<br>
couple of extra times before failing) which I find troubling.<br>
<br>
I think I will give up using local load/stores and will see if I can<br>
figure<br>
out if rewrite using MPI calls like MPI_Fetch_and_op  as you suggest.<=
br>
Thanks for your help.<br>
<br>
On Mon, Mar 6, 2017 at 7: 20 PM, Jeff Hammond <<a href=3D"mailto:jeff.sci=
[email protected]" target=3D"_blank">[email protected]</a>><br>
wrote: <br>
<br>
What processor architecture are you testing?<br>
<br>
<blockquote class=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex;border-=
left: 1px solid rgb(204,204,204);padding-left:1ex">
<br>
Maybe set lmem to volatile or read it with MPI_Fetch_and_op rather<br>
than a<br>
load.  MPI_Win_sync cannot prevent the compiler from caching *lmem<br>
in a<br>
register.<br>
<br>
Jeff<br>
<br>
On Sat, Mar 4, 2017 at 12: 30 AM, Ask Jakobsen <<a href=3D"mailto:afj@qey=
e-labs.com" target=3D"_blank">[email protected]</a>><br>
wrote: <br>
<br>
Hi,<br>
<br>
<blockquote class=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex;border-=
left: 1px solid rgb(204,204,204);padding-left:1ex">
<br>
I have downloaded the source code for the MCS lock from the excellent<br>
book "Using Advanced MPI" from <a href=3D"http: //www.mcs.anl.gov/=
researc" rel=3D"noreferrer" target=3D"_blank">http: //www.mcs.anl.gov/resear=
c</a><br>
h/projects/mpi/usingmpi/exampl<wbr>es-advmpi/rma2/mcs-lock.c<br>
<br>
I have made a very simple piece of test code for testing the MCS lock<br>
but<br>
it works at random and often never escapes the busy loops in the<br>
acquire<br>
and release functions (see attached source code). The code appears<br>
semantically correct to my eyes.<br>
<br>
#include <stdio.h><br>
#include <mpi.h><br>
#include "mcs-lock.h"<br>
<br>
int main(int argc, char *argv[])<br>
{<br>
  MPI_Win win;<br>
  MPI_Init( &argc, &argv );<br>
<br>
  MCSLockInit(MPI_COMM_WORLD, &win);<br>
<br>
  int rank, size;<br>
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);<br>
  MPI_Comm_size(MPI_COMM_WORLD, &size);<br>
<br>
  printf("rank: %d, size: %d\n", rank, size);<br>
<br>
<br>
  MCSLockAcquire(win);<br>
  printf("rank %d aquired lock\n", rank);   fflush=
(stdout);<br>
  MCSLockRelease(win);<br>
<br>
<br>
  MPI_Win_free(&win);<br>
  MPI_Finalize();<br>
  return 0;<br>
}<br>
<br>
<br>
I have tested on several hardware platforms and mpich-3.2 and<br>
mpich-3.3a2<br>
but with no luck.<br>
<br>
It appears that the MPI_Win_Sync are not "refreshing" the local<b=
r>
data or<br>
I<br>
have a bug I can't spot.<br>
<br>
A simple unfair lock like <a href=3D"http: //www.mcs.anl.gov/researc" rel=3D=
"noreferrer" target=3D"_blank">http: //www.mcs.anl.gov/researc</a><br>
h/projects/mpi/usingmpi/exampl<wbr>es-advmpi/rma2/ga_mutex1.c works<br>
perfectly.<br>
<br>
Best regards, Ask Jakobsen<br>
<br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list     <a href=3D"mailto: [email protected]=
" target=3D"_blank">[email protected]</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https: //lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
<br>
<br>
</blockquote>
<br>
--<br>
Jeff Hammond<br>
<a href=3D"mailto: [email protected]" target=3D"_blank">jeff.science@gm=
ail.com</a><br>
<a href=3D"http: //jeffhammond.github.io/" rel=3D"noreferrer" target=3D"_bla=
nk">http: //jeffhammond.github.io/</a><br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list     <a href=3D"mailto: [email protected]=
" target=3D"_blank">[email protected]</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https: //lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
<br>
<br>
</blockquote>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list     <a href=3D"mailto: [email protected]=
" target=3D"_blank">[email protected]</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https: //lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
______________________________<wbr>_________________<br>
<br>
</blockquote>
discuss mailing list     <a href=3D"mailto: [email protected]=
" target=3D"_blank">[email protected]</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https: //lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
<br>
</blockquote>
<br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list     <a href=3D"mailto: [email protected]=
" target=3D"_blank">[email protected]</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https: //lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
______________________________<wbr>_________________<br>
</blockquote>
discuss mailing list     <a href=3D"mailto: [email protected]=
" target=3D"_blank">[email protected]</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https: //lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
</div></div></blockquote><div><div class=3D"gmail-m_8000249642949159004gmai=
l-m_1322166946455689838gmail-h5">
______________________________<wbr>_________________<br>
discuss mailing list     <a href=3D"mailto: [email protected]=
" target=3D"_blank">[email protected]</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https: //lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
</div></div></blockquote><div><div class=3D"gmail-m_8000249642949159004gmai=
l-m_1322166946455689838gmail-h5">
<br>
<br>
<br>
______________________________<wbr>_________________<br>
discuss mailing list     <a href=3D"mailto: [email protected]=
" target=3D"_blank">[email protected]</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https: //lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a><br>
<br>
</div></div></blockquote><div class=3D"gmail-m_8000249642949159004gmail-m_1=
322166946455689838gmail-HOEnZb"><div class=3D"gmail-m_8000249642949159004gm=
ail-m_1322166946455689838gmail-h5">
______________________________<wbr>_________________<br>
discuss mailing list     <a href=3D"mailto: [email protected]=
" target=3D"_blank">[email protected]</a><br>
To manage subscription options or unsubscribe: <br>
<a href=3D"https: //lists.mpich.org/mailman/listinfo/discuss" rel=3D"norefer=
rer" target=3D"_blank">https: //lists.mpich.org/mailma<wbr>n/listinfo/discus=
s</a></div></div></blockquote></div>
</div></div></div></div>

--001a1141dbd03846c4054a8dd443--

--001a1141dbd03846ca054a8dd445
Content-Type: text/x-csrc; charset="US-ASCII"; name="main.c"
Content-Disposition: attachment; filename="main.c"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_j0732una0

I2luY2x1ZGUgPHN0ZGlvLmg+CiNpbmNsdWRlIDxtcGkuaD4KI2luY2x1ZGUgIm1jcy1sb2NrLmgi
CgppbnQgbWFpbihpbnQgYXJnYywgY2hhciAqYXJndltdKQp7CiAgTVBJX1dpbiB3aW47CiAgTVBJ
X0luaXQoICZhcmdjLCAmYXJndiApOwoKICBNQ1NMb2NrSW5pdChNUElfQ09NTV9XT1JMRCwgJndp
bik7CgogIGludCByYW5rLCBzaXplOwogIE1QSV9Db21tX3JhbmsoTVBJX0NPTU1fV09STEQsICZy
YW5rKTsKICBNUElfQ29tbV9zaXplKE1QSV9DT01NX1dPUkxELCAmc2l6ZSk7CgogIHByaW50Zigi
cmFuazogJWQsIHNpemU6ICVkXG4iLCByYW5rLCBzaXplKTsKCiAgaW50IGNvdW50ID0gMDsKICB3
aGlsZSgxKSB7CiAgICBNQ1NMb2NrQWNxdWlyZSh3aW4pOwogICAgLy9wcmludGYoInJhbmsgJWQg
YXF1aXJlZCBsb2NrXG4iLCByYW5rKTsgZmZsdXNoKHN0ZG91dCk7CiAgICBNQ1NMb2NrUmVsZWFz
ZSh3aW4pOwogICAgaWYgKGNvdW50PjEwMDApCiAgICAgIGJyZWFrOwogICAgY291bnQrKzsKICB9
CgogIE1QSV9XaW5fZnJlZSgmd2luKTsgCiAgTVBJX0ZpbmFsaXplKCk7CiAgcmV0dXJuIDA7Cn0K

--001a1141dbd03846ca054a8dd445
Content-Type: text/x-csrc; charset="US-ASCII"; name="mcs-lock-fop.c"
Content-Disposition: attachment; filename="mcs-lock-fop.c"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_j0732unw1

I2luY2x1ZGUgPHVuaXN0ZC5oPgoKI2luY2x1ZGUgIm1waS5oIgoKc3RhdGljIGludCBNQ1NfTE9D
S1JBTksgPSBNUElfS0VZVkFMX0lOVkFMSUQ7CmVudW0geyBuZXh0UmFuaz0wLCBibG9ja2VkPTEs
IGxvY2tUYWlsPTIgfTsKCnZvaWQgTUNTTG9ja0luaXQoTVBJX0NvbW0gY29tbSwgTVBJX1dpbiAq
d2luKQp7CiAgaW50ICAgICAgKmxtZW0sIHJhbms7CiAgTVBJX0FpbnQgd2luc2l6ZTsKICBNUElf
Q29tbV9yYW5rKGNvbW0sJnJhbmspOwoKICBpZiAoTUNTX0xPQ0tSQU5LID09IE1QSV9LRVlWQUxf
SU5WQUxJRCkKICAgIE1QSV9XaW5fY3JlYXRlX2tleXZhbChNUElfV0lOX05VTExfQ09QWV9GTiwK
CQkJICBNUElfV0lOX05VTExfREVMRVRFX0ZOLAoJCQkgICZNQ1NfTE9DS1JBTkssICh2b2lkKikw
KTsKCiAgd2luc2l6ZSA9IDIgKiBzaXplb2YoaW50KTsKICBpZiAocmFuayA9PSAwKSB3aW5zaXpl
ICs9IHNpemVvZihpbnQpOwogIE1QSV9XaW5fYWxsb2NhdGUod2luc2l6ZSwgc2l6ZW9mKGludCks
IE1QSV9JTkZPX05VTEwsIGNvbW0sCiAgICAgICAgICAgICAgICAgICAmbG1lbSwgd2luKTsKICBs
bWVtW25leHRSYW5rXSA9IC0xOwogIGxtZW1bYmxvY2tlZF0gID0gMDsKICBpZiAocmFuayA9PSAw
KSB7CiAgICBsbWVtW2xvY2tUYWlsXSA9IC0xOwogIH0KICBNUElfV2luX3NldF9hdHRyKCp3aW4s
IE1DU19MT0NLUkFOSywgKHZvaWQqKShNUElfQWludClyYW5rKTsKICBNUElfQmFycmllcihjb21t
KTsKfQoKdm9pZCBNQ1NMb2NrQWNxdWlyZShNUElfV2luIHdpbikKewogIGludCAgZmxhZywgbXly
YW5rLCBwcmVkZWNlc3NvciwgKmxtZW07CiAgdm9pZCAqYXR0cnZhbDsKICBpbnQgZmV0Y2hfYmxv
Y2tlZCwgZHVtbXk7CgogIE1QSV9XaW5fZ2V0X2F0dHIod2luLCBNQ1NfTE9DS1JBTkssICZhdHRy
dmFsLCAmZmxhZyk7CiAgbXlyYW5rID0gKGludCkoTVBJX0FpbnQpYXR0cnZhbDsKICBNUElfV2lu
X2dldF9hdHRyKHdpbiwgTVBJX1dJTl9CQVNFLCAmbG1lbSwgJmZsYWcpOwogIGxtZW1bYmxvY2tl
ZF0gPSAxOyAvKiBJbiBjYXNlIHdlIGFyZSBibG9ja2VkICovCiAgTVBJX1dpbl9sb2NrX2FsbCgw
LCB3aW4pOwogIE1QSV9GZXRjaF9hbmRfb3AoJm15cmFuaywgJnByZWRlY2Vzc29yLCBNUElfSU5U
LAogICAgICAgICAgICAgICAgICAgMCwgbG9ja1RhaWwsIE1QSV9SRVBMQUNFLCB3aW4pOwogIE1Q
SV9XaW5fZmx1c2goMCwgd2luKTsKICBpZiAocHJlZGVjZXNzb3IgIT0gLTEpIHsKICAgIC8qIFdl
IGRpZG4ndCBnZXQgdGhlIGxvY2suICBBZGQgdXMgdG8gdGhlIHRhaWwgb2YgdGhlIGxpc3QgKi8K
ICAgIE1QSV9BY2N1bXVsYXRlKCZteXJhbmssIDEsIE1QSV9JTlQsIHByZWRlY2Vzc29yLAogICAg
ICAgICAgICAgICAgICAgbmV4dFJhbmssIDEsIE1QSV9JTlQsIE1QSV9SRVBMQUNFLCB3aW4pOwog
ICAgLyogTm93IHNwaW4gb24gb3VyIGxvY2FsIHZhbHVlICJibG9ja2VkIiB1bnRpbCB3ZSBhcmUK
ICAgICAgIGdpdmVuIHRoZSBsb2NrICovCiAgICBkbyB7CiAgICAgIE1QSV9GZXRjaF9hbmRfb3Ao
JmR1bW15LCAmZmV0Y2hfYmxvY2tlZCwgTVBJX0lOVCwKICAgICAgICAgICAgICAgICAgIG15cmFu
aywgYmxvY2tlZCwgTVBJX05PX09QLCB3aW4pOwogICAgICB1c2xlZXAoMTAwKTsKICAgIH0gd2hp
bGUgKGZldGNoX2Jsb2NrZWQ9PTEpOwoKICB9CiAgLy8gZWxzZSB3ZSBoYXZlIHRoZSBsb2NrCiAg
TVBJX1dpbl91bmxvY2tfYWxsKHdpbik7Cn0Kdm9pZCBNQ1NMb2NrUmVsZWFzZShNUElfV2luIHdp
bikKewogIGludCBudWxscmFuayA9IC0xLCB6ZXJvPTAsIG15cmFuaywgY3VydGFpbCwgZmxhZywg
KmxtZW07CiAgdm9pZCAqYXR0cnZhbDsKICBpbnQgZmV0Y2hfbmV4dHJhbmssIGR1bW15OwoKICBN
UElfV2luX2dldF9hdHRyKHdpbiwgTUNTX0xPQ0tSQU5LLCAmYXR0cnZhbCwgJmZsYWcpOwogIG15
cmFuayA9IChpbnQpKE1QSV9BaW50KWF0dHJ2YWw7CiAgTVBJX1dpbl9nZXRfYXR0cih3aW4sIE1Q
SV9XSU5fQkFTRSwgJmxtZW0sICZmbGFnKTsKICBNUElfV2luX2xvY2tfYWxsKDAsIHdpbik7CiAg
TVBJX0ZldGNoX2FuZF9vcCgmZHVtbXksICZmZXRjaF9uZXh0cmFuaywgTVBJX0lOVCwKICAgICAg
ICAgICAgICAgICAgIG15cmFuaywgbmV4dFJhbmssIE1QSV9OT19PUCwgd2luKTsKCiAgaWYgKGZl
dGNoX25leHRyYW5rID09IC0xKSB7CiAgICAvKiBTZWUgaWYgd2UncmUgd2FpdGluZyBmb3IgdGhl
IG5leHQgdG8gbm90aWZ5IHVzICovCiAgICBNUElfQ29tcGFyZV9hbmRfc3dhcCgmbnVsbHJhbmss
ICZteXJhbmssICZjdXJ0YWlsLCBNUElfSU5ULAogICAgICAgICAgICAgICAgICAgICAgICAgMCwg
bG9ja1RhaWwsIHdpbik7CiAgICBpZiAoY3VydGFpbCA9PSBteXJhbmspIHsKICAgICAgLyogV2Ug
YXJlIHRoZSBvbmx5IHByb2Nlc3MgaW4gdGhlIGxpc3QgKi8KICAgICAgTVBJX1dpbl91bmxvY2tf
YWxsKHdpbik7CiAgICAgIHJldHVybjsKICAgIH0KICAgIC8qIE90aGVyd2lzZSwgc29tZW9uZSBl
bHNlIGhhcyBhZGRlZCB0aGVtc2VsdmVzIHRvIHRoZSBsaXN0LiovCiAgICBkbyB7CiAgICAgIE1Q
SV9GZXRjaF9hbmRfb3AoJmR1bW15LCAmZmV0Y2hfbmV4dHJhbmssIE1QSV9JTlQsCiAgICAgICAg
ICAgICAgICAgICBteXJhbmssIG5leHRSYW5rLCBNUElfTk9fT1AsIHdpbik7CiAgICB9IHdoaWxl
IChmZXRjaF9uZXh0cmFuaz09LTEpOwoKICAgfQogIC8qIE5vdyB3ZSBjYW4gbm90aWZ5IHRoZW0u
ICBVc2UgYWNjdW11bGF0ZSB3aXRoIHJlcGxhY2UgaW5zdGVhZCAKICAgICBvZiBwdXQgc2luY2Ug
d2Ugd2FudCBhbiBhdG9taWMgdXBkYXRlIG9mIHRoZSBsb2NhdGlvbiAqLwogIE1QSV9BY2N1bXVs
YXRlKCZ6ZXJvLCAxLCBNUElfSU5ULCBmZXRjaF9uZXh0cmFuaywgYmxvY2tlZCwKCQkgMSwgTVBJ
X0lOVCwgTVBJX1JFUExBQ0UsIHdpbik7CiAgTVBJX1dpbl91bmxvY2tfYWxsKHdpbik7Cn0K

--001a1141dbd03846ca054a8dd445
Content-Type: text/x-chdr; charset="US-ASCII"; name="mcs-lock.h"
Content-Disposition: attachment; filename="mcs-lock.h"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_j0732uo22

dm9pZCBNQ1NMb2NrSW5pdChNUElfQ29tbSBjb21tLCBNUElfV2luICp3aW4pOwp2b2lkIE1DU0xv
Y2tBY3F1aXJlKE1QSV9XaW4gd2luKTsKdm9pZCBNQ1NMb2NrUmVsZWFzZShNUElfV2luIHdpbik7
Cgo=

--001a1141dbd03846ca054a8dd445
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
discuss mailing list     [email protected]
To manage subscription options or unsubscribe: 
https: //lists.mpich.org/mailman/listinfo/discuss
--001a1141dbd03846ca054a8dd445--
Message-ID: <[email protected]>