February 2013 - devel - mailman3.cels.anl.gov

Re: [mpich-devel] memory profiling HOWTO?
by Jeff Hammond 23 Feb '13

23 Feb '13

I think I need to override MPIU_Malloc myself for this but I will also try --enable-g=mem and MPIU_trspace to see how the information I get from that lines up with what I get from the BGQ SPI (which can track everything: heap, stack, shm, ...). Thanks for all the pointers. Jeff On Sat, Feb 23, 2013 at 10:16 AM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: > There are two kinds of memory that MPICH with ch3:nemesis allocates: > > 1) regular heap memory via malloc (with interposition possible via MPIU_Malloc and friends) > > 2) shared memory via one of the usual mechanisms (mmap, shm_open, etc.) > > You can determine the current amount of #1 by calling "MPIU_trspace" as long as you configure with "--enable-g=mem" (or some superset thereof). You'll get some additional performance overhead when allocating/freeing memory if you configure this way, since MPICH is doing some additional allocation and bookkeeping in order to track this information for you. If you need a lower overhead method for doing this, you'll need to modify/intercept MPIU_Malloc and friends yourself. You'll need an "extern" declaration for "MPIU_trspace" in order to call it from your test program. > > You can use the attached (untested against HEAD) patch to give you a hook for #2. You'll need to throw down an "extern" declaration for the routine in your test program as well. > > -Dave > > > > > On Feb 22, 2013, at 9:28 AM CST, Jim Dinan wrote: > >> Jeff, >> >> Which memory are you interested in -- memory the application allocates, or memory that MPICH allocates? If the former, you can use the PMPI interface to hook MPI_Alloc/Free_mem. If you are interested in MPICH internal memory use, Dave can probably give the most informed answer. I would probably just modify MPIU_Malloc. >> >> ~Jim. >> >> On 2/22/13 8:44 AM, Jeff Hammond wrote: >>> Is there any documentation of how to properly instrument MPICH with >>> memory profiling? I understand MPIU_MALLOC allows me to do this. >>> >>> My goal is to merely print out every time memory is allocated larger >>> than a particular size. I'd also like the rank to show up in the >>> output. >>> >>> Please RTM me if the M exists. Source comments are also a good M if >>> they are correct and sufficiently complete. >>> >>> Thanks! >>> >>> Jeff >>> > > -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-devel] memory profiling HOWTO?
by Jeff Hammond 23 Feb '13

23 Feb '13

I'm interested in MPICH internal allocation. I'm trying to figure out all the places in BGQ-MPI where O(p) memory is required. Jeff On Fri, Feb 22, 2013 at 9:28 AM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: > Jeff, > > Which memory are you interested in -- memory the application allocates, or > memory that MPICH allocates? If the former, you can use the PMPI interface > to hook MPI_Alloc/Free_mem. If you are interested in MPICH internal memory > use, Dave can probably give the most informed answer. I would probably just > modify MPIU_Malloc. > > ~Jim. > > > On 2/22/13 8:44 AM, Jeff Hammond wrote: >> >> Is there any documentation of how to properly instrument MPICH with >> memory profiling? I understand MPIU_MALLOC allows me to do this. >> >> My goal is to merely print out every time memory is allocated larger >> than a particular size. I'd also like the rank to show up in the >> output. >> >> Please RTM me if the M exists. Source comments are also a good M if >> they are correct and sufficiently complete. >> >> Thanks! >> >> Jeff >> > -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-devel] is sublinear memory usage in MPIR_Group_create (and related ops) possible?
by Dave Goodell 23 Feb '13

23 Feb '13

On Feb 22, 2013, at 5:47 PM CST, Jeff Hammond wrote: > I see this error on BGQ at scale (nproc=524288). […] > MPIR_Group_create(83)...: Unable to allocate 8388608 bytes of memory > for newgroup->lrank_to_lpid (probably out of memory) […] > I haven't thought about it very deeply yet, but is there a way to > implement group operations without using O(p) memory? That depends on the creation/access patterns you want to support and the number of hoops that you want to jump through. Jesper published a very reasonable scheme a few years ago for implementing these efficiently in both time and space: Google Scholar page: http://scholar.google.com/scholar?cluster=13915175648622039304&hl=en&as_sdt… Slides: https://fs.hlrs.de/projects/eurompi2010/TALKS/TUESDAY_MORNING_TRACK1/Compac… I can't quickly find an electronic copy to which I actually have access, though I think I have one in dead-tree form in my office. The MPICH group code is pretty horrible right now. But it's not a heavily used piece of the implementation, so it's usually been hard to justify improving/fixing it. -Dave

1 0

Re: [mpich-devel] memory profiling HOWTO?
by Dave Goodell 23 Feb '13

23 Feb '13

There are two kinds of memory that MPICH with ch3:nemesis allocates: 1) regular heap memory via malloc (with interposition possible via MPIU_Malloc and friends) 2) shared memory via one of the usual mechanisms (mmap, shm_open, etc.) You can determine the current amount of #1 by calling "MPIU_trspace" as long as you configure with "--enable-g=mem" (or some superset thereof). You'll get some additional performance overhead when allocating/freeing memory if you configure this way, since MPICH is doing some additional allocation and bookkeeping in order to track this information for you. If you need a lower overhead method for doing this, you'll need to modify/intercept MPIU_Malloc and friends yourself. You'll need an "extern" declaration for "MPIU_trspace" in order to call it from your test program. You can use the attached (untested against HEAD) patch to give you a hook for #2. You'll need to throw down an "extern" declaration for the routine in your test program as well. -Dave On Feb 22, 2013, at 9:28 AM CST, Jim Dinan wrote: > Jeff, > > Which memory are you interested in -- memory the application allocates, or memory that MPICH allocates? If the former, you can use the PMPI interface to hook MPI_Alloc/Free_mem. If you are interested in MPICH internal memory use, Dave can probably give the most informed answer. I would probably just modify MPIU_Malloc. > > ~Jim. > > On 2/22/13 8:44 AM, Jeff Hammond wrote: >> Is there any documentation of how to properly instrument MPICH with >> memory profiling? I understand MPIU_MALLOC allows me to do this. >> >> My goal is to merely print out every time memory is allocated larger >> than a particular size. I'd also like the rank to show up in the >> output. >> >> Please RTM me if the M exists. Source comments are also a good M if >> they are correct and sufficiently complete. >> >> Thanks! >> >> Jeff >>

1 0

is sublinear memory usage in MPIR_Group_create (and related ops) possible?
by Jeff Hammond 22 Feb '13

22 Feb '13

I see this error on BGQ at scale (nproc=524288). Abort(1) on node 492082 (rank 492082 in comm 1140850688): Fatal error in PMPI_Comm_group: Other MPI error, error stack: PMPI_Comm_group(174)....: MPI_Comm_group(MPI_COMM_WORLD, group=0x1fffffb9b4) failed MPIR_Comm_group_impl(45): MPIR_Group_create(83)...: Unable to allocate 8388608 bytes of memory for newgroup->lrank_to_lpid (probably out of memory) The culprit appears to be (*new_group_ptr)->lrank_to_lpid = (MPID_Group_pmap_t *)MPIU_Malloc( nproc * sizeof(MPID_Group_pmap_t) ); and typedef struct MPID_Group_pmap_t { int lrank; /* Local rank in group (between 0 and size-1) */ int lpid; /* local process id, from VCONN */ int next_lpid; /* Index of next lpid (in lpid order) */ int flag; /* marker, used to implement group operations */ } MPID_Group_pmap_t; I haven't thought about it very deeply yet, but is there a way to implement group operations without using O(p) memory? Thanks, Jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-devel] memory profiling HOWTO?
by Jim Dinan 22 Feb '13

22 Feb '13

Jeff, Which memory are you interested in -- memory the application allocates, or memory that MPICH allocates? If the former, you can use the PMPI interface to hook MPI_Alloc/Free_mem. If you are interested in MPICH internal memory use, Dave can probably give the most informed answer. I would probably just modify MPIU_Malloc. ~Jim. On 2/22/13 8:44 AM, Jeff Hammond wrote: > Is there any documentation of how to properly instrument MPICH with > memory profiling? I understand MPIU_MALLOC allows me to do this. > > My goal is to merely print out every time memory is allocated larger > than a particular size. I'd also like the rank to show up in the > output. > > Please RTM me if the M exists. Source comments are also a good M if > they are correct and sufficiently complete. > > Thanks! > > Jeff >

1 0

memory profiling HOWTO?
by Jeff Hammond 22 Feb '13

22 Feb '13

Is there any documentation of how to properly instrument MPICH with memory profiling? I understand MPIU_MALLOC allows me to do this. My goal is to merely print out every time memory is allocated larger than a particular size. I'd also like the rank to show up in the output. Please RTM me if the M exists. Source comments are also a good M if they are correct and sufficiently complete. Thanks! Jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-devel] patch for missing declarations in test suite
by Jim Dinan 21 Feb '13

21 Feb '13

Hi Jeff, Already done, and committed in 617927fb. Thanks for the patch, ~Jim. On 2/21/13 3:35 PM, Jeff Hammond wrote: > I'll see if I can make the patch Windows-safe then and resubmit. > > Jeff > > On Thu, Feb 21, 2013 at 1:18 PM, William Gropp <wgropp(a)illinois.edu> wrote: >> unistd.h is not part of C - it is for Unix and Unix-like systems. There is >> a test for this file in configure.ac that should be used, and the Unixism in >> the test should be removed or protected. The test programs should not >> assume Unix/POSIX/Linux. The other fixes look good. >> >> Bill >> >> William Gropp >> Director, Parallel Computing Institute >> Deputy Director for Research >> Institute for Advanced Computing Applications and Technologies >> Thomas M. Siebel Chair in Computer Science >> University of Illinois Urbana-Champaign >> >> >> >> >> On Feb 18, 2013, at 6:47 AM, Jeff Hammond wrote: >> >> I'm submitting this via email because Trac is hosed right now (I >> assume it will be fixed in a few hours, but I don't want to enqueue >> this in my head that long). >> >> LLVM complained about use of undeclared functions being invalid in >> C99. I fixed them in the attached patch, which is copied below. >> >> The fix to test/mpi/rma/mutex_bench.c depends upon the patch to >> test/mpi/rma/Makefile.am because otherwise the header will not be >> found. >> >> Best, >> >> Jeff >> >> $ cat test.patch >> diff --git a/test/mpi/coll/nonblocking3.c b/test/mpi/coll/nonblocking3.c >> index b1f66dd..8e896b7 100644 >> --- a/test/mpi/coll/nonblocking3.c >> +++ b/test/mpi/coll/nonblocking3.c >> @@ -16,6 +16,7 @@ >> #include "mpi.h" >> #include <stdlib.h> >> #include <stdio.h> >> +#include <unistd.h> >> #include <string.h> >> #include <assert.h> >> /* USE_STRICT_MPI may be defined in mpitestconf.h */ >> diff --git a/test/mpi/f90/attr/attrlangc.c b/test/mpi/f90/attr/attrlangc.c >> index 6920668..2ee4ef5 100644 >> --- a/test/mpi/f90/attr/attrlangc.c >> +++ b/test/mpi/f90/attr/attrlangc.c >> @@ -124,7 +124,7 @@ void cgetsizes_( MPI_Fint *ptrSize, MPI_Fint >> *intSize, MPI_Fint *aintSize ) >> } >> >> /* ----------------------------------------------------------------------- >> */ >> -/* Copy and delete functions attached to keyvals >> +/* Copy and delete functions attached to keyvals >> */ >> /* ----------------------------------------------------------------------- >> */ >> static int CMPI1_COPY_FN( MPI_Comm comm, int keyval, void *extra, >> void *inval, void *outval, int *flag ) >> diff --git a/test/mpi/manual/mpi_t/nem_fbox_fallback_to_queue_count.c >> b/test/mpi/manual/mpi_t/nem_fbox_fallback_to_queue_count.c >> index f045212..9718bbc 100644 >> --- a/test/mpi/manual/mpi_t/nem_fbox_fallback_to_queue_count.c >> +++ b/test/mpi/manual/mpi_t/nem_fbox_fallback_to_queue_count.c >> @@ -17,6 +17,7 @@ >> #include <stdlib.h> >> #include <stdint.h> >> #include <stdio.h> >> +#include <unistd.h> >> >> #define TRY(func) \ >> do { \ >> diff --git a/test/mpi/rma/Makefile.am b/test/mpi/rma/Makefile.am >> index 579721a..76155c5 100644 >> --- a/test/mpi/rma/Makefile.am >> +++ b/test/mpi/rma/Makefile.am >> @@ -155,5 +155,5 @@ get_accumulate_double_derived_SOURCES = get_accumulate.c >> get_accumulate_int_derived_SOURCES = get_accumulate.c >> >> mutex_bench_SOURCES = mutex_bench.c mcs-mutex.c mcs-mutex.h >> -mutex_bench_shared_CPPFLAGS = -DUSE_WIN_SHARED >> +mutex_bench_shared_CPPFLAGS = -DUSE_WIN_SHARED $(AM_CPPFLAGS) >> mutex_bench_shared_SOURCES = mutex_bench.c mcs-mutex.c mcs-mutex.h >> diff --git a/test/mpi/rma/lockcontention3.c b/test/mpi/rma/lockcontention3.c >> index 6204e94..196dfb5 100644 >> --- a/test/mpi/rma/lockcontention3.c >> +++ b/test/mpi/rma/lockcontention3.c >> @@ -8,6 +8,7 @@ >> #include "stdlib.h" >> #include "mpitest.h" >> #include <assert.h> >> +#include <string.h> >> >> #define LAST_TEST 14 >> #define RMA_SIZE 2048 >> diff --git a/test/mpi/rma/mutex_bench.c b/test/mpi/rma/mutex_bench.c >> index 098382e..2db24e4 100644 >> --- a/test/mpi/rma/mutex_bench.c >> +++ b/test/mpi/rma/mutex_bench.c >> @@ -12,6 +12,7 @@ >> #include <assert.h> >> >> #include <mpi.h> >> +#include "mpitest.h" >> #include "mcs-mutex.h" >> >> #define NUM_ITER 1000 >> >> -- >> Jeff Hammond >> Argonne Leadership Computing Facility >> University of Chicago Computation Institute >> jhammond(a)alcf.anl.gov / (630) 252-5381 >> http://www.linkedin.com/in/jeffhammond >> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >> <test.patch> >> >> > > >

1 0

Re: [mpich-devel] patch for missing declarations in test suite
by Jeff Hammond 21 Feb '13

21 Feb '13

I'll see if I can make the patch Windows-safe then and resubmit. Jeff On Thu, Feb 21, 2013 at 1:18 PM, William Gropp <wgropp(a)illinois.edu> wrote: > unistd.h is not part of C - it is for Unix and Unix-like systems. There is > a test for this file in configure.ac that should be used, and the Unixism in > the test should be removed or protected. The test programs should not > assume Unix/POSIX/Linux. The other fixes look good. > > Bill > > William Gropp > Director, Parallel Computing Institute > Deputy Director for Research > Institute for Advanced Computing Applications and Technologies > Thomas M. Siebel Chair in Computer Science > University of Illinois Urbana-Champaign > > > > > On Feb 18, 2013, at 6:47 AM, Jeff Hammond wrote: > > I'm submitting this via email because Trac is hosed right now (I > assume it will be fixed in a few hours, but I don't want to enqueue > this in my head that long). > > LLVM complained about use of undeclared functions being invalid in > C99. I fixed them in the attached patch, which is copied below. > > The fix to test/mpi/rma/mutex_bench.c depends upon the patch to > test/mpi/rma/Makefile.am because otherwise the header will not be > found. > > Best, > > Jeff > > $ cat test.patch > diff --git a/test/mpi/coll/nonblocking3.c b/test/mpi/coll/nonblocking3.c > index b1f66dd..8e896b7 100644 > --- a/test/mpi/coll/nonblocking3.c > +++ b/test/mpi/coll/nonblocking3.c > @@ -16,6 +16,7 @@ > #include "mpi.h" > #include <stdlib.h> > #include <stdio.h> > +#include <unistd.h> > #include <string.h> > #include <assert.h> > /* USE_STRICT_MPI may be defined in mpitestconf.h */ > diff --git a/test/mpi/f90/attr/attrlangc.c b/test/mpi/f90/attr/attrlangc.c > index 6920668..2ee4ef5 100644 > --- a/test/mpi/f90/attr/attrlangc.c > +++ b/test/mpi/f90/attr/attrlangc.c > @@ -124,7 +124,7 @@ void cgetsizes_( MPI_Fint *ptrSize, MPI_Fint > *intSize, MPI_Fint *aintSize ) > } > > /* ----------------------------------------------------------------------- > */ > -/* Copy and delete functions attached to keyvals > +/* Copy and delete functions attached to keyvals > */ > /* ----------------------------------------------------------------------- > */ > static int CMPI1_COPY_FN( MPI_Comm comm, int keyval, void *extra, > void *inval, void *outval, int *flag ) > diff --git a/test/mpi/manual/mpi_t/nem_fbox_fallback_to_queue_count.c > b/test/mpi/manual/mpi_t/nem_fbox_fallback_to_queue_count.c > index f045212..9718bbc 100644 > --- a/test/mpi/manual/mpi_t/nem_fbox_fallback_to_queue_count.c > +++ b/test/mpi/manual/mpi_t/nem_fbox_fallback_to_queue_count.c > @@ -17,6 +17,7 @@ > #include <stdlib.h> > #include <stdint.h> > #include <stdio.h> > +#include <unistd.h> > > #define TRY(func) \ > do { \ > diff --git a/test/mpi/rma/Makefile.am b/test/mpi/rma/Makefile.am > index 579721a..76155c5 100644 > --- a/test/mpi/rma/Makefile.am > +++ b/test/mpi/rma/Makefile.am > @@ -155,5 +155,5 @@ get_accumulate_double_derived_SOURCES = get_accumulate.c > get_accumulate_int_derived_SOURCES = get_accumulate.c > > mutex_bench_SOURCES = mutex_bench.c mcs-mutex.c mcs-mutex.h > -mutex_bench_shared_CPPFLAGS = -DUSE_WIN_SHARED > +mutex_bench_shared_CPPFLAGS = -DUSE_WIN_SHARED $(AM_CPPFLAGS) > mutex_bench_shared_SOURCES = mutex_bench.c mcs-mutex.c mcs-mutex.h > diff --git a/test/mpi/rma/lockcontention3.c b/test/mpi/rma/lockcontention3.c > index 6204e94..196dfb5 100644 > --- a/test/mpi/rma/lockcontention3.c > +++ b/test/mpi/rma/lockcontention3.c > @@ -8,6 +8,7 @@ > #include "stdlib.h" > #include "mpitest.h" > #include <assert.h> > +#include <string.h> > > #define LAST_TEST 14 > #define RMA_SIZE 2048 > diff --git a/test/mpi/rma/mutex_bench.c b/test/mpi/rma/mutex_bench.c > index 098382e..2db24e4 100644 > --- a/test/mpi/rma/mutex_bench.c > +++ b/test/mpi/rma/mutex_bench.c > @@ -12,6 +12,7 @@ > #include <assert.h> > > #include <mpi.h> > +#include "mpitest.h" > #include "mcs-mutex.h" > > #define NUM_ITER 1000 > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > <test.patch> > > -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-devel] patch for missing declarations in test suite
by William Gropp 21 Feb '13

21 Feb '13

unistd.h is not part of C - it is for Unix and Unix-like systems. There is a test for this file in configure.ac that should be used, and the Unixism in the test should be removed or protected. The test programs should not assume Unix/POSIX/Linux. The other fixes look good. Bill William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Thomas M. Siebel Chair in Computer Science University of Illinois Urbana-Champaign On Feb 18, 2013, at 6:47 AM, Jeff Hammond wrote: > I'm submitting this via email because Trac is hosed right now (I > assume it will be fixed in a few hours, but I don't want to enqueue > this in my head that long). > > LLVM complained about use of undeclared functions being invalid in > C99. I fixed them in the attached patch, which is copied below. > > The fix to test/mpi/rma/mutex_bench.c depends upon the patch to > test/mpi/rma/Makefile.am because otherwise the header will not be > found. > > Best, > > Jeff > > $ cat test.patch > diff --git a/test/mpi/coll/nonblocking3.c b/test/mpi/coll/nonblocking3.c > index b1f66dd..8e896b7 100644 > --- a/test/mpi/coll/nonblocking3.c > +++ b/test/mpi/coll/nonblocking3.c > @@ -16,6 +16,7 @@ > #include "mpi.h" > #include <stdlib.h> > #include <stdio.h> > +#include <unistd.h> > #include <string.h> > #include <assert.h> > /* USE_STRICT_MPI may be defined in mpitestconf.h */ > diff --git a/test/mpi/f90/attr/attrlangc.c b/test/mpi/f90/attr/attrlangc.c > index 6920668..2ee4ef5 100644 > --- a/test/mpi/f90/attr/attrlangc.c > +++ b/test/mpi/f90/attr/attrlangc.c > @@ -124,7 +124,7 @@ void cgetsizes_( MPI_Fint *ptrSize, MPI_Fint > *intSize, MPI_Fint *aintSize ) > } > > /* ----------------------------------------------------------------------- */ > -/* Copy and delete functions attached to keyvals > +/* Copy and delete functions attached to keyvals */ > /* ----------------------------------------------------------------------- */ > static int CMPI1_COPY_FN( MPI_Comm comm, int keyval, void *extra, > void *inval, void *outval, int *flag ) > diff --git a/test/mpi/manual/mpi_t/nem_fbox_fallback_to_queue_count.c > b/test/mpi/manual/mpi_t/nem_fbox_fallback_to_queue_count.c > index f045212..9718bbc 100644 > --- a/test/mpi/manual/mpi_t/nem_fbox_fallback_to_queue_count.c > +++ b/test/mpi/manual/mpi_t/nem_fbox_fallback_to_queue_count.c > @@ -17,6 +17,7 @@ > #include <stdlib.h> > #include <stdint.h> > #include <stdio.h> > +#include <unistd.h> > > #define TRY(func) \ > do { \ > diff --git a/test/mpi/rma/Makefile.am b/test/mpi/rma/Makefile.am > index 579721a..76155c5 100644 > --- a/test/mpi/rma/Makefile.am > +++ b/test/mpi/rma/Makefile.am > @@ -155,5 +155,5 @@ get_accumulate_double_derived_SOURCES = get_accumulate.c > get_accumulate_int_derived_SOURCES = get_accumulate.c > > mutex_bench_SOURCES = mutex_bench.c mcs-mutex.c mcs-mutex.h > -mutex_bench_shared_CPPFLAGS = -DUSE_WIN_SHARED > +mutex_bench_shared_CPPFLAGS = -DUSE_WIN_SHARED $(AM_CPPFLAGS) > mutex_bench_shared_SOURCES = mutex_bench.c mcs-mutex.c mcs-mutex.h > diff --git a/test/mpi/rma/lockcontention3.c b/test/mpi/rma/lockcontention3.c > index 6204e94..196dfb5 100644 > --- a/test/mpi/rma/lockcontention3.c > +++ b/test/mpi/rma/lockcontention3.c > @@ -8,6 +8,7 @@ > #include "stdlib.h" > #include "mpitest.h" > #include <assert.h> > +#include <string.h> > > #define LAST_TEST 14 > #define RMA_SIZE 2048 > diff --git a/test/mpi/rma/mutex_bench.c b/test/mpi/rma/mutex_bench.c > index 098382e..2db24e4 100644 > --- a/test/mpi/rma/mutex_bench.c > +++ b/test/mpi/rma/mutex_bench.c > @@ -12,6 +12,7 @@ > #include <assert.h> > > #include <mpi.h> > +#include "mpitest.h" > #include "mcs-mutex.h" > > #define NUM_ITER 1000 > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > <test.patch>

1 0