November 2014 - discuss - mailman3.cels.anl.gov

Isend and Recv
by Zhen Wang 05 Nov '14

05 Nov '14

Hi MPIers, I have some questions regarding MPI_Isend() and MPI_Recv(). MPICH-3.1.3 is used to compile and run the attached code on Red Hat Enterprise Linux 6.3 (a shared memory machine). While n = 9999, the MPI_Recv() finishes immediately after MPI_Isend(): (This is what I understand and expect) MPI 1: Recv started at 09:53:53. MPI 0: Isend started at 09:53:53. MPI 1: Recv finished at 09:53:53. MPI 0: Isend finished at 09:53:58. When n = 99999, I get the following. The MPI_Recv() finishes after MPI_Wait(): MPI 1: Recv started at 09:47:56. MPI 0: Isend started at 09:47:56. MPI 0: Isend finished at 09:48:01. MPI 1: Recv finished at 09:48:01. But with OpenMPI 1.8 and n = 99999, MPI_Recv() finishes immediately after MPI_Isend(): MPI 0: Isend started at 09:55:28. MPI 1: Recv started at 09:55:28. MPI 1: Recv finished at 09:55:28. MPI 0: Isend finished at 09:55:33. Am I misunderstanding something here? In case the attached code is dropped, the code is included. Thanks in advance. #include "mpi.h" #include <unistd.h> #include <stdio.h> #include "vector" #include <time.h> int main(int argc, char* argv[]) { MPI_Init(&argc, &argv); int rank; MPI_Comm_rank(MPI_COMM_WORLD, &rank); int n = 9999; std::vector<int> vec(n); MPI_Request mpiRequest; MPI_Status mpiStatus; char tt[9] = {0}; MPI_Barrier(MPI_COMM_WORLD); if (rank == 0) { MPI_Isend(&vec[0], n, MPI_INT, 1, 0, MPI_COMM_WORLD, &mpiRequest); time_t t = time(0); strftime(tt, 9, "%H:%M:%S", localtime(&t)); printf("MPI %d: Isend started at %s.\n", rank, tt); //int done = 0; //while (done == 0) //{ // MPI_Test(&mpiRequest, &done, &mpiStatus); //} sleep(5); MPI_Wait(&mpiRequest, &mpiStatus); t = time(0); strftime(tt, 9, "%H:%M:%S", localtime(&t)); printf("MPI %d: Isend finished at %s.\n", rank, tt); } else { time_t t = time(0); strftime(tt, 9, "%H:%M:%S", localtime(&t)); printf("MPI %d: Recv started at %s.\n", rank, tt); MPI_Recv(&vec[0], n, MPI_INT, 0, 0, MPI_COMM_WORLD, &mpiStatus); t = time(0); strftime(tt, 9, "%H:%M:%S", localtime(&t)); printf("MPI %d: Recv finished at %s.\n", rank, tt); } MPI_Finalize(); return 0; } Best regards, Zhen _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPICH examples
by Michael L. Stokes 05 Nov '14

05 Nov '14

Under examples/rma there are several .c files that are not listed in testlist (for example, nb_test.c). When I look at the source for nb_test.c it appears it should require at least 2 processes to run correctly, but when I run it with n>=2 it hangs. Do 2 questions. 1) What what is going on with applications not listed in testlist, and 2) for is going on with nb_test.c? Thanks Mike _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Question regarding MPICH Examples
by William Gropp 04 Nov '14

04 Nov '14

Note that some of these are not environment variables. The mpiversion number is compared to the test level, and controls whether that test is run. The timelimit option is really meaningful only for a script - its intended to terminate hangs; people are pretty good at that when running these manually. Bill On Nov 4, 2014, at 2:05 PM, Michael L. Stokes <Michael.Stokes(a)uah.edu> wrote: > Kenneth, > > If I were to run these examples outside of the provided scripts, how would I used > these environment variables? There seems to be a multitude of ways environment > variables can be specified. > > Thanks > Mike > > On 11/04/2014 12:12 PM, Kenneth Raffenetti wrote: >> Hi Mike, >> >> Those testlist files are interpreted by the mpich/test/mpi/runtests script to set actual MPICH/Hydra envvars. >> >> Ken >> >> On 11/04/2014 12:06 PM, Michael L. Stokes wrote: >>> In the file testlist (i.e rma directory) you have what looks like >>> environment variables like timeLimit=600, mpiversion=3.0 >>> listed by the names of the tests (as well as the number of processors to >>> pass to mpirun). >>> How are these values used? I looked in the mtest.c routines and there >>> are environment variables (i.e. MPITEST_DEBUG, MPITEST_VERBOSE) that you >>> read >>> to control execution of the tests, but I don't see the ones listed here. >>> >>> Thanks >>> Mike >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Question regarding MPICH Examples
by Kenneth Raffenetti 04 Nov '14

04 Nov '14

This wiki page has info for Hydra's (mpiexec) envvars. https://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager#Envi… For all others, see README.envvar in the base MPICH source directory. Ken On 11/04/2014 02:05 PM, Michael L. Stokes wrote: > Kenneth, > > If I were to run these examples outside of the provided scripts, how > would I used > these environment variables? There seems to be a multitude of ways > environment > variables can be specified. > > Thanks > Mike > > On 11/04/2014 12:12 PM, Kenneth Raffenetti wrote: >> Hi Mike, >> >> Those testlist files are interpreted by the mpich/test/mpi/runtests >> script to set actual MPICH/Hydra envvars. >> >> Ken >> >> On 11/04/2014 12:06 PM, Michael L. Stokes wrote: >>> In the file testlist (i.e rma directory) you have what looks like >>> environment variables like timeLimit=600, mpiversion=3.0 >>> listed by the names of the tests (as well as the number of processors to >>> pass to mpirun). >>> How are these values used? I looked in the mtest.c routines and there >>> are environment variables (i.e. MPITEST_DEBUG, MPITEST_VERBOSE) that you >>> read >>> to control execution of the tests, but I don't see the ones listed here. >>> >>> Thanks >>> Mike >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Question regarding MPICH Examples
by Michael L. Stokes 04 Nov '14

04 Nov '14

Kenneth, If I were to run these examples outside of the provided scripts, how would I used these environment variables? There seems to be a multitude of ways environment variables can be specified. Thanks Mike On 11/04/2014 12:12 PM, Kenneth Raffenetti wrote: > Hi Mike, > > Those testlist files are interpreted by the mpich/test/mpi/runtests > script to set actual MPICH/Hydra envvars. > > Ken > > On 11/04/2014 12:06 PM, Michael L. Stokes wrote: >> In the file testlist (i.e rma directory) you have what looks like >> environment variables like timeLimit=600, mpiversion=3.0 >> listed by the names of the tests (as well as the number of processors to >> pass to mpirun). >> How are these values used? I looked in the mtest.c routines and there >> are environment variables (i.e. MPITEST_DEBUG, MPITEST_VERBOSE) that you >> read >> to control execution of the tests, but I don't see the ones listed here. >> >> Thanks >> Mike >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Question regarding MPICH Examples
by Kenneth Raffenetti 04 Nov '14

04 Nov '14

Hi Mike, Those testlist files are interpreted by the mpich/test/mpi/runtests script to set actual MPICH/Hydra envvars. Ken On 11/04/2014 12:06 PM, Michael L. Stokes wrote: > In the file testlist (i.e rma directory) you have what looks like > environment variables like timeLimit=600, mpiversion=3.0 > listed by the names of the tests (as well as the number of processors to > pass to mpirun). > How are these values used? I looked in the mtest.c routines and there > are environment variables (i.e. MPITEST_DEBUG, MPITEST_VERBOSE) that you > read > to control execution of the tests, but I don't see the ones listed here. > > Thanks > Mike > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Question regarding MPICH Examples
by Michael L. Stokes 04 Nov '14

04 Nov '14

In the file testlist (i.e rma directory) you have what looks like environment variables like timeLimit=600, mpiversion=3.0 listed by the names of the tests (as well as the number of processors to pass to mpirun). How are these values used? I looked in the mtest.c routines and there are environment variables (i.e. MPITEST_DEBUG, MPITEST_VERBOSE) that you read to control execution of the tests, but I don't see the ones listed here. Thanks Mike _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Assertion Failure using MPICH3 RMA
by Wesley Bland 03 Nov '14

03 Nov '14

Have you tried updating to use the latest version of MPICH? You said you haven’t seen the issue when you run with 3.1. We’re now on 3.1.3 and I believe a number of RMA bug fixes have gone in recently. Thanks, Wesley > On Nov 3, 2014, at 3:14 PM, Corey A. Henderson <cahenderson(a)wisc.edu> wrote: > > About 1 in 100 runs on my local desktop, while developing an MPI code that uses MPI 3-0 RMA shared-lock features, I see the following assertion failure. The assertion does not fail at the same point in a run, or following any pattern that I can see. It has not happened on a cluster I use that is running MPICH3.1, but I haven't run my code there very often yet. > > Failure text: > > Assertion failed in file <snip>/mpich-3.0.4/src/mpid/ch3/src/ch3u_rma_sync.c at line 2803: win_ptr->targets[target_rank].remote_lock_state == MPIDI_CH3_WIN_LOCK_REQUESTED || win_ptr->targets[target_rank].remote_lock_state == MPIDI_CH3_WIN_LOCK_GRANTED > internal ABORT - process 0 > > I have not attempted to recreate the issue with a smaller code snippet because I am not sure where to even begin to do so. Can anyone suggest to me where I might start to look for the cause of this? > > Some notes on what the code does: > > - One window per node of fixed size opened (MPI_Win_allocate) at program start. > - All windows locked (shared) after creation with MPI_Win_lock_all > - Code uses MPI_Fetch_and_op and MPI_Compare_and_swap on a few concurrently-accessed MPI_INT locations in a node's window > - Code uses GET/PUT for data access to the rest of the window on any node (those portions that are not accessed concurrently) > > MPICH is v3.0.4 on 64-bit Ubuntu 12.04 LTS in a single-machine configuration. > > The error occurs regardless of how many MPI processes I may be testing with at any given time. I have not nailed down where in the code to trace to see why the error occurs because I don't know what could cause this (which is why I'm asking). The MPI messaging portion of the code hasn't changed in a couple of months, but I'm starting to run my code more often and for longer periods now as it nears completion. > > Any help on where to begin tracing to fix this would be great. > > > -- > Corey A. Henderson > PhD Candidate and NSF Graduate Fellow > Dept. of Engineering Physics > Univ. of Wisconsin - Madison > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Assertion Failure using MPICH3 RMA
by Corey A. Henderson 03 Nov '14

03 Nov '14

About 1 in 100 runs on my local desktop, while developing an MPI code that uses MPI 3-0 RMA shared-lock features, I see the following assertion failure. The assertion does not fail at the same point in a run, or following any pattern that I can see. It has not happened on a cluster I use that is running MPICH3.1, but I haven't run my code there very often yet. Failure text: Assertion failed in file <snip>/mpich-3.0.4/src/mpid/ch3/src/ch3u_rma_sync.c at line 2803: win_ptr->targets[target_rank].remote_lock_state == MPIDI_CH3_WIN_LOCK_REQUESTED || win_ptr->targets[target_rank].remote_lock_state == MPIDI_CH3_WIN_LOCK_GRANTED internal ABORT - process 0 I have not attempted to recreate the issue with a smaller code snippet because I am not sure where to even begin to do so. Can anyone suggest to me where I might start to look for the cause of this? Some notes on what the code does: - One window per node of fixed size opened (MPI_Win_allocate) at program start. - All windows locked (shared) after creation with MPI_Win_lock_all - Code uses MPI_Fetch_and_op and MPI_Compare_and_swap on a few concurrently-accessed MPI_INT locations in a node's window - Code uses GET/PUT for data access to the rest of the window on any node (those portions that are not accessed concurrently) MPICH is v3.0.4 on 64-bit Ubuntu 12.04 LTS in a single-machine configuration. The error occurs regardless of how many MPI processes I may be testing with at any given time. I have not nailed down where in the code to trace to see why the error occurs because I don't know what could cause this (which is why I'm asking). The MPI messaging portion of the code hasn't changed in a couple of months, but I'm starting to run my code more often and for longer periods now as it nears completion. Any help on where to begin tracing to fix this would be great. -- Corey A. Henderson PhD Candidate and NSF Graduate Fellow Dept. of Engineering Physics Univ. of Wisconsin - Madison _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Build Problem mpich-3.1.3
by Seo, Sangmin 02 Nov '14

02 Nov '14

Hi Hirak, I’d like to say again this is a bug of configure because it does nothing to correct the situation when the valgrind headers are broken, even though it causes make to fail. I have created a ticket for this problem, https://trac.mpich.org/projects/mpich/ticket/2195, and cc’ed you to the ticket. Thanks, Sangmin On Nov 2, 2014, at 9:53 AM, Seo, Sangmin <sseo(a)anl.gov<mailto:[email protected]>> wrote: Hi Hirak, It may not be a bug. It happens when the valgrind headers are broken or too old. Can you send us c.txt (configure logs) that you got on the compilation error? Regards, Sangmin On Nov 2, 2014, at 8:52 AM, Roy, Hirak <Hirak_Roy(a)mentor.com<mailto:[email protected]>> wrote: Thanks Sangmin. ‘—without-valgrind’ worked for me. However, is this a bug ? Thanks, Hirak Hi Hirak, Can you send us c.txt as described in http://www.mpich.org/static/downloads/3.1.3/mpich-3.1.3-installguide.pdf? In the meantime, you can also try configure with '--without-valgrind’. I think it can be a problem of valgrind version. Regards, Sangmin On Oct 31, 2014, at 5:44 AM, Hirak Roy <hirak_roy at mentor.com<https://lists.mpich.org/mailman/listinfo/discuss><mailto:hirak_roy at mentor.com<https://lists.mpich.org/mailman/listinfo/discuss>>> wrote: Hi Team, I could not complete the build. Could you please let me know if it is config issue or anything else. Thanks, Hirak make all-recursive make[1]: Entering directory `/net/zinzeng17/export/hroy/software/MPI/mpich-3.1.3' Making all in /net/zinzeng17/export/hroy/software/MPI/mpich-3.1.3/src/mpl make[2]: Entering directory `/net/zinzeng17/export/hroy/software/MPI/mpich-3.1.3/src/mpl' CC src/mpltrmem.lo src/mpltrmem.c: In function 'MPL_trdump': src/mpltrmem.c:657:9: error: 'old_head' undeclared (first use in this function) src/mpltrmem.c:657:9: note: each undeclared identifier is reported only once for each function it appears in make[2]: *** [src/mpltrmem.lo] Error 1 make[2]: Leaving directory `/net/zinzeng17/export/hroy/software/MPI/mpich-3.1.3/src/mpl' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/net/zinzeng17/export/hroy/software/MPI/mpich-3.1.3' make: *** [all] Error 2 My linux distribution : Linux zinblade62 2.6.9-67.ELsmp #1 SMP Wed Nov 7 13:56:44 EST 2007 x86_64 x86_64 x86_64 GNU/Linux Config settings: Configuring with : mpich2 version = 3.1.3 platform = linux_x86_64 Source directory = /net/zinzeng17/export/hroy/software/MPI/mpich-3.1.3 Install directory = /home/hroy/local//mpich-3.1.3/linux_x86_64 CC = /u/prod/gnu/gcc/20121129/gcc-4.5.0-linux_x86_64/bin/gcc CFLAGS = -O3 -fPIC CXXFLAGS = -O3 -fPIC LDFLAGS = CONFIG_OPTIONS = --disable-f77 --disable-fc --disable-f90modules --disable-cxx --enable-fast=nochkmsg --enable-fast=notiming --enable-fast=ndebug --enable-fast=O3 --with-device=ch3:sock --enable-g=dbg --disable-fortran _______________________________________________ discuss mailing list discuss at mpich.org<https://lists.mpich.org/mailman/listinfo/discuss><mailto:discuss at mpich.org<https://lists.mpich.org/mailman/listinfo/discuss>> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0