March 2016 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Error with MPI_Register_datarep
by Eric Chamberland 13 Mar '16

13 Mar '16

Le 2016-03-10 18:04, Thakur, Rajeev a écrit : > Support for user-defined data representations has not been added to MPICH yet. There has not been much demand for it. Thanks for your answer... Is it a "big" task? Should it be scheduled so it may appear in a forthcoming version? Thanks, Eric _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

about the makefile
by Hardwood Bao 12 Mar '16

12 Mar '16

hello,everyone. I want to invoke some third-party libraries to send and receive the message. But i don't know how to modify the makefile and how to include the this is a example when using the library. g++ -o normFileRecv normFileRecv.cpp -D_FILE_OFFSET_BITS=64 -I../common/ \ -I../protolib/include -I../include/ ../lib/libnorm.a ../protolib/lib/libprotokit.a \ -lpthread the third-party libraries include libnorm.a, libprotokit.a, and lpthread. and the include files are under the directory of common and protolib/include. If i want to involve the API in the file: src\mpid\ch3\channels\nemesis\src\Ch3-istartmsg.c how should i modify the makefile? I am really appreciate your help. Best Regards. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] NAG fortran and mpich
by Kevin Olson 11 Mar '16

11 Mar '16

Hi Ken, I just wanted to let you know that the problem has been fixed. There is an option to the nagfor compiler (-C=undefined) that we used and we should not have. Here’s what the nagfor man page says about this option: The -C=undefined option is subject to a number of limitations and is not binary-compatible with code compiled without that option; see the Undefined Variable Detection section for further details. Kevin Olson HPC Software Developer Numerical Algorithms Group, Inc 801 Warrenville Road, Suite 185 Lisle, Illinois 60532-4332 T 267-664-9913 F 630-971-2706 www.nag.com On 3/10/16, 2:10 PM, "Kenneth Raffenetti" <raffenet(a)mcs.anl.gov> wrote: >Do you have a small reproducer program we could try on our systems? Also >your config.log file would be useful. > >We do test this compiler combination on Ubuntu in our nightly builds >without issue. > >Ken > >On 03/10/2016 09:34 AM, Kevin Olson wrote: >> Hello, >> >> I am having trouble running a fortran code sung the NAG fortran compiler >> and various versions of mpich. All fail in the same way: >> First the error I’ getting: >> >> Fatal error in PMPI_Comm_rank: Invalid argument, error stack: >> PMPI_Comm_rank(110): MPI_Comm_rank(MPI_COMM_WORLD, rank=(nil)) failed >> PMPI_Comm_rank(82).: Null pointer in parameter rank >> Makefile.spicer:414: recipe for target 'initial0' failed >> make: *** [initial0] Error 1 >> >> This comes from the first call to an MPI function after MPI_INIT(). >> >> I am attempting to run the code on a Linux (Ubuntu) machine with AMD >> processor. >> The code does run successfully on other systems with the same >> combination of compilers (A Mac for instance). >> Here are the specs for the compilers used. >> >> NAG Fortran Compiler Release 6.0(Hibiya) Build 1064 >> Product NPL6A60NA for x86-64 Linux >> Copyright 1990-2015 The Numerical Algorithms Group Ltd., Oxford, U.K. >> >> >> >> >> Using built-in specs. >> COLLECT_GCC=gcc >> COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.9/lto-wrapper >> Target: x86_64-linux-gnu >> Configured with: ../src/configure -v --with-pkgversion='Ubuntu >> 4.9.2-10ubuntu13' >> --with-bugurl=file:///usr/share/doc/gcc-4.9/README.Bugs >> --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr >> --program-suffix=-4.9 --enable-shared --enable-linker-build-id >> --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix >> --with-gxx-include-dir=/usr/include/c++/4.9 --libdir=/usr/lib >> --enable-nls --with-sysroot=/ --enable-clocale=gnu >> --enable-libstdcxx-debug --enable-libstdcxx-time=yes >> --enable-gnu-unique-object --disable-vtable-verify --enable-plugin >> --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk >> --enable-gtk-cairo >> --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64/jre >> --enable-java-home >> --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64 >> --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.9-amd64 >> --with-arch-directory=amd64 >> --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc >> --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 >> --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic >> --enable-checking=release --build=x86_64-linux-gnu >> --host=x86_64-linux-gnu --target=x86_64-linux-gnu >> Thread model: posix >> gcc version 4.9.2 (Ubuntu 4.9.2-10ubuntu13) >> >> Is there an inconsistency with the compilers on such a system? Do you >> have any other suggestions? >> Thanks! >> >> Kevin Olson >> HPC Software Developer >> Numerical Algorithms Group, Inc >> 801 Warrenville Road, Suite 185 >> Lisle, Illinois 60532-4332 >> T 267-664-9913 >> F 630-971-2706 >> www.nag.com >> Kevin Olson >> HPC Software Developer >> Numerical Algorithms Group, Inc >> 801 Warrenville Road, Suite 185 >> Lisle, Illinois 60532-4332 >> T 267-664-9913 >> F 630-971-2706 >> www.nag.com >> >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >_______________________________________________ >discuss mailing list discuss(a)mpich.org >To manage subscription options or unsubscribe: >https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Error with MPI_Register_datarep
by Thakur, Rajeev 10 Mar '16

10 Mar '16

Support for user-defined data representations has not been added to MPICH yet. There has not been much demand for it. Rajeev > On Mar 10, 2016, at 1:52 PM, Éric Chamberland <Eric.Chamberland(a)giref.ulaval.ca> wrote: > > Hi, > > Here is my little test program. > > Almost the same as chapter 9.1 of the book " Using MPI-2: Advanced Features of the Message-Passing Interface" from William Gropp, Ewing Lusk, and Rajeev Thakur. > > Thanks, > > Eric > > > Le 16-03-09 22:39, Éric Chamberland a écrit : >> Hi, >> >> I am trying to use the MPI_Register_datarep functionnality with mpich-3.2, but while executing, I have this error: >> >> ERROR Returned by MPI: 67148823 >> ERROR_string Returned by MPI: An error occurred in a user-defined data conversion function, error stack: >> MPI_REGISTER_DATAREP(108): Read and Write datarep conversions are currently not supported by MPI-IO >> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 >> >> I have configured mpich-3.2 with: ./configure --prefix=/opt/mpich-3.2_ch3sock --with-device=ch3:sock >> >> Is there anything I have to do to be able to use this functionality? >> >> Thanks, >> >> Eric >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > <int64.cc>_______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Error with MPI_Register_datarep
by Éric Chamberland 10 Mar '16

10 Mar '16

Hi, Here is my little test program. Almost the same as chapter 9.1 of the book " Using MPI-2: Advanced Features of the Message-Passing Interface" from William Gropp, Ewing Lusk, and Rajeev Thakur. Thanks, Eric Le 16-03-09 22:39, ï¿½ric Chamberland a ï¿½crit : > Hi, > > I am trying to use the MPI_Register_datarep functionnality with > mpich-3.2, but while executing, I have this error: > > ERROR Returned by MPI: 67148823 > ERROR_string Returned by MPI: An error occurred in a user-defined data > conversion function, error stack: > MPI_REGISTER_DATAREP(108): Read and Write datarep conversions are > currently not supported by MPI-IO > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 > > I have configured mpich-3.2 with: ./configure > --prefix=/opt/mpich-3.2_ch3sock --with-device=ch3:sock > > Is there anything I have to do to be able to use this functionality? > > Thanks, > > Eric > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] NAG fortran and mpich
by Kenneth Raffenetti 10 Mar '16

10 Mar '16

Do you have a small reproducer program we could try on our systems? Also your config.log file would be useful. We do test this compiler combination on Ubuntu in our nightly builds without issue. Ken On 03/10/2016 09:34 AM, Kevin Olson wrote: > Hello, > > I am having trouble running a fortran code sung the NAG fortran compiler > and various versions of mpich. All fail in the same way: > First the error I’ getting: > > Fatal error in PMPI_Comm_rank: Invalid argument, error stack: > PMPI_Comm_rank(110): MPI_Comm_rank(MPI_COMM_WORLD, rank=(nil)) failed > PMPI_Comm_rank(82).: Null pointer in parameter rank > Makefile.spicer:414: recipe for target 'initial0' failed > make: *** [initial0] Error 1 > > This comes from the first call to an MPI function after MPI_INIT(). > > I am attempting to run the code on a Linux (Ubuntu) machine with AMD > processor. > The code does run successfully on other systems with the same > combination of compilers (A Mac for instance). > Here are the specs for the compilers used. > > NAG Fortran Compiler Release 6.0(Hibiya) Build 1064 > Product NPL6A60NA for x86-64 Linux > Copyright 1990-2015 The Numerical Algorithms Group Ltd., Oxford, U.K. > > > > > Using built-in specs. > COLLECT_GCC=gcc > COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.9/lto-wrapper > Target: x86_64-linux-gnu > Configured with: ../src/configure -v --with-pkgversion='Ubuntu > 4.9.2-10ubuntu13' > --with-bugurl=file:///usr/share/doc/gcc-4.9/README.Bugs > --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr > --program-suffix=-4.9 --enable-shared --enable-linker-build-id > --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix > --with-gxx-include-dir=/usr/include/c++/4.9 --libdir=/usr/lib > --enable-nls --with-sysroot=/ --enable-clocale=gnu > --enable-libstdcxx-debug --enable-libstdcxx-time=yes > --enable-gnu-unique-object --disable-vtable-verify --enable-plugin > --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk > --enable-gtk-cairo > --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64/jre > --enable-java-home > --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64 > --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.9-amd64 > --with-arch-directory=amd64 > --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc > --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 > --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic > --enable-checking=release --build=x86_64-linux-gnu > --host=x86_64-linux-gnu --target=x86_64-linux-gnu > Thread model: posix > gcc version 4.9.2 (Ubuntu 4.9.2-10ubuntu13) > > Is there an inconsistency with the compilers on such a system? Do you > have any other suggestions? > Thanks! > > Kevin Olson > HPC Software Developer > Numerical Algorithms Group, Inc > 801 Warrenville Road, Suite 185 > Lisle, Illinois 60532-4332 > T 267-664-9913 > F 630-971-2706 > www.nag.com > Kevin Olson > HPC Software Developer > Numerical Algorithms Group, Inc > 801 Warrenville Road, Suite 185 > Lisle, Illinois 60532-4332 > T 267-664-9913 > F 630-971-2706 > www.nag.com > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

NAG fortran and mpich
by Kevin Olson 10 Mar '16

10 Mar '16

Hello, I am having trouble running a fortran code sung the NAG fortran compiler and various versions of mpich. All fail in the same way: First the error I’ getting: Fatal error in PMPI_Comm_rank: Invalid argument, error stack: PMPI_Comm_rank(110): MPI_Comm_rank(MPI_COMM_WORLD, rank=(nil)) failed PMPI_Comm_rank(82).: Null pointer in parameter rank Makefile.spicer:414: recipe for target 'initial0' failed make: *** [initial0] Error 1 This comes from the first call to an MPI function after MPI_INIT(). I am attempting to run the code on a Linux (Ubuntu) machine with AMD processor. The code does run successfully on other systems with the same combination of compilers (A Mac for instance). Here are the specs for the compilers used. NAG Fortran Compiler Release 6.0(Hibiya) Build 1064 Product NPL6A60NA for x86-64 Linux Copyright 1990-2015 The Numerical Algorithms Group Ltd., Oxford, U.K. Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.9/lto-wrapper Target: x86_64-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.9.2-10ubuntu13' --with-bugurl=file:///usr/share/doc/gcc-4.9/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.9 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.9 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.9-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix gcc version 4.9.2 (Ubuntu 4.9.2-10ubuntu13) Is there an inconsistency with the compilers on such a system? Do you have any other suggestions? Thanks! Kevin Olson HPC Software Developer Numerical Algorithms Group, Inc 801 Warrenville Road, Suite 185 Lisle, Illinois 60532-4332 T 267-664-9913 F 630-971-2706 www.nag.com Kevin Olson HPC Software Developer Numerical Algorithms Group, Inc 801 Warrenville Road, Suite 185 Lisle, Illinois 60532-4332 T 267-664-9913 F 630-971-2706 www.nag.com _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Error with MPI_Register_datarep
by Éric Chamberland 09 Mar '16

09 Mar '16

Hi, I am trying to use the MPI_Register_datarep functionnality with mpich-3.2, but while executing, I have this error: ERROR Returned by MPI: 67148823 ERROR_string Returned by MPI: An error occurred in a user-defined data conversion function, error stack: MPI_REGISTER_DATAREP(108): Read and Write datarep conversions are currently not supported by MPI-IO application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 I have configured mpich-3.2 with: ./configure --prefix=/opt/mpich-3.2_ch3sock --with-device=ch3:sock Is there anything I have to do to be able to use this functionality? Thanks, Eric _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] One-sided semantics
by Palmer, Bruce J 08 Mar '16

08 Mar '16

I reran the tests using 2 processors on only 1 SMP node on our infinband cluster and everything works fine. I also built MPICH on my Dell quad-core workstation and ran the tests there as well. They also worked. The failures seem to only happen when you have to go off-node and actually use the infiniband network. Bruce -----Original Message----- From: Balaji, Pavan [mailto:[email protected]] Sent: Friday, March 04, 2016 8:02 PM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] One-sided semantics And, of course, I assume you'll eventually fix all the performance shortcomings with respect to doing extra flush/flush_locals or datatype commits inside the for loop. We'll be happy to review the Comex port for you, once it's ready, if you like. -- Pavan > On Mar 4, 2016, at 9:59 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > Bruce, > > You are missing an MPI_Barrier on line 81 (after the initialization). Without this, a remote process might update your buffer while you are still initializing. The program works with the barrier. > > -- Pavan > >> On Mar 4, 2016, at 6:22 PM, Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> wrote: >> >> Hi, >> >> I’ve been working on a thin implementation of the COMEX runtime over MPI-3. The COMEX interface has been used by most of the MPI-based runtimes in GA. One of the COMEX tests has processors writing to and then immediately reading from neighboring processes multiple times. The GA semantics are that for multiple consecutive operations between the same pair of processes, the operations are ordered on the remote process in the same order as on the originating process. The test for this frequently fails for the MPI-3 based implementation. I’ve tried testing this independently of GA but the results are confusing. >> >> The implementation I’ve been working on uses three different strategies to implement one-sided communication calls that follow, or are at least close to, the GA communication semantics. The first uses MPI_Put/MPI_Get/MPI_Accumulate and surrounds these calls by and MPI_Lock and MPI_Unlock pair immediately before and after the one-sided communication call. My understanding is that this forces completion both locally and remotely. The second approach calls MPI_Win_lock_all on the MPI window immediately after creation and MPI_Win_unlock_all when the window is destroy so that the window is always in a passive synchronization epoch. The put/get/accumulate calls are implemented with the request-based calls MPI_Rput/MPI_Rget/MPI_Raccumulate and followed immediately by a call to MPI_Wait on the request handle. Again, from my understanding, this should force local completion of the operation but not necessarily remote completion. Finally, the last implementation is to again use the MPI_Win_lock_all to guarantee that a window is in a permanent passive synchronization epoch, use MPI_Put/MPI_Get/MPI_Accumulate to implement put/get/accumulate and use MPI_Win_flush_local to force completion locally. The first implementation should require only a barrier to force synchronization between all processors, the second two include a call to MPI_Win_flush_all in conjunction with a barrier to synchronize the data on all processors. >> >> I’ve written a small test code that implements all three schemes and attached it to this email. It creates a 200x200 array of doubles, fills each array with unique numbers, writes a portion of the array to the next higher rank using put and then reads it back using get (cyclic boundary conditions are used for the first and last ranks). This is repeated 2000 times, with each test using a slightly different set of numbers from the previous test. I’ve done this for all three implementations using both a synchronization between the put and the get and without synchronization. The code has been run on an Infinband cluster using 2 processors on 2 separate SMP nodes. The results I get are that the request-based implementation and the flush_local_all implementation without synchronization work pretty consistently while the tests with synchronization all fail. The lock/unlock implementation also fails both with and without synchronization. Most tests that fail get through at least a few put/get cycles before failing but they don’t do all 2000 iterations. >> >> I’ve also tried this using OpenMPI. In the OpenMPI case, there doesn’t appear to be much of an effect from using synchronization. In addition, the lock/unlock algorithm does not consistently fail, although it fails more frequently than the other two. >> >> Does anyone have a suggestion as to what I’m doing wrong here? From my understanding of the MPI-3 standard, all three implementations should work with synchronization. I’m not completely sure if they should work without synchronization. >> >> Bruce Palmer >> >> <testmpi.c>_______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] One-sided semantics
by Palmer, Bruce J 07 Mar '16

07 Mar '16

I put the barrier in at line 81, but it doesn't seem to be having much of an effect. I still seem to get the same results. I'm using MPICH 3.2 built with gcc 5.2.0 and am running on an infiniband cluster. What are you using? As far as the flush_locals/waits etc., I've put those in so that we can reproduce GA/ARMCI semantics. I've tried to get things to work without them, but I don't seem to be able to get communication progress without them. The GA/ARMCI semantics for blocking calls are that the local buffer is safe for use when the put/get/accumulate returns and the only way to achieve that in MPI, as far as I know, is to use the combinations that are in the test program. However, if you have any suggestions on how to get rid of some of them, I'm all ears. I didn't mention it in the previous email, but you can get the different options by commenting/uncommenting out the preprocessor symbols MPI_USE_REQUESTS, MPI_USE_FLUSH_LOCAL, and USE_SYNC. If you don't define either of the first two, the program uses the lock/unlock approach, if you define MPI_USE_REQUESTS it uses the request-based protocol with MPI_Wait and if you define MPI_USE_FLUSH_LOCAL it uses MPI_Win_flush_local on the remote processor. Defining or undefining USE_SYNC puts a synchronization operation between put and get. The COMEX port is in the GA developer build under comex/src-mpi3. I think that is available via anonymous SVN, but I'll check. Bruce -----Original Message----- From: Balaji, Pavan [mailto:[email protected]] Sent: Friday, March 04, 2016 8:02 PM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] One-sided semantics And, of course, I assume you'll eventually fix all the performance shortcomings with respect to doing extra flush/flush_locals or datatype commits inside the for loop. We'll be happy to review the Comex port for you, once it's ready, if you like. -- Pavan > On Mar 4, 2016, at 9:59 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > Bruce, > > You are missing an MPI_Barrier on line 81 (after the initialization). Without this, a remote process might update your buffer while you are still initializing. The program works with the barrier. > > -- Pavan > >> On Mar 4, 2016, at 6:22 PM, Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> wrote: >> >> Hi, >> >> I’ve been working on a thin implementation of the COMEX runtime over MPI-3. The COMEX interface has been used by most of the MPI-based runtimes in GA. One of the COMEX tests has processors writing to and then immediately reading from neighboring processes multiple times. The GA semantics are that for multiple consecutive operations between the same pair of processes, the operations are ordered on the remote process in the same order as on the originating process. The test for this frequently fails for the MPI-3 based implementation. I’ve tried testing this independently of GA but the results are confusing. >> >> The implementation I’ve been working on uses three different strategies to implement one-sided communication calls that follow, or are at least close to, the GA communication semantics. The first uses MPI_Put/MPI_Get/MPI_Accumulate and surrounds these calls by and MPI_Lock and MPI_Unlock pair immediately before and after the one-sided communication call. My understanding is that this forces completion both locally and remotely. The second approach calls MPI_Win_lock_all on the MPI window immediately after creation and MPI_Win_unlock_all when the window is destroy so that the window is always in a passive synchronization epoch. The put/get/accumulate calls are implemented with the request-based calls MPI_Rput/MPI_Rget/MPI_Raccumulate and followed immediately by a call to MPI_Wait on the request handle. Again, from my understanding, this should force local completion of the operation but not necessarily remote completion. Finally, the last implementation is to again use the MPI_Win_lock_all to guarantee that a window is in a permanent passive synchronization epoch, use MPI_Put/MPI_Get/MPI_Accumulate to implement put/get/accumulate and use MPI_Win_flush_local to force completion locally. The first implementation should require only a barrier to force synchronization between all processors, the second two include a call to MPI_Win_flush_all in conjunction with a barrier to synchronize the data on all processors. >> >> I’ve written a small test code that implements all three schemes and attached it to this email. It creates a 200x200 array of doubles, fills each array with unique numbers, writes a portion of the array to the next higher rank using put and then reads it back using get (cyclic boundary conditions are used for the first and last ranks). This is repeated 2000 times, with each test using a slightly different set of numbers from the previous test. I’ve done this for all three implementations using both a synchronization between the put and the get and without synchronization. The code has been run on an Infinband cluster using 2 processors on 2 separate SMP nodes. The results I get are that the request-based implementation and the flush_local_all implementation without synchronization work pretty consistently while the tests with synchronization all fail. The lock/unlock implementation also fails both with and without synchronization. Most tests that fail get through at least a few put/get cycles before failing but they don’t do all 2000 iterations. >> >> I’ve also tried this using OpenMPI. In the OpenMPI case, there doesn’t appear to be much of an effect from using synchronization. In addition, the lock/unlock algorithm does not consistently fail, although it fails more frequently than the other two. >> >> Does anyone have a suggestion as to what I’m doing wrong here? From my understanding of the MPI-3 standard, all three implementations should work with synchronization. I’m not completely sure if they should work without synchronization. >> >> Bruce Palmer >> >> <testmpi.c>_______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0