May 2015 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] libmpich.so.10: cannot open shared object file
by Jorge Pachon 04 May '15

04 May '15

Hi Ken, I did some test and MPICH seems to be working OK, at least I ran the CPI example and worked fine. However, I have problems running an 'n' process CPI job on multiple nodes. It seems I have not created a machinefile yet to succeed. Is this step absolutely necessary? Moreover, my problem running WRF still exists, because the libmpich.so.10 file is not successfully created when installing MPICH 3.0.4. I'm attaching all the configure.log files from my installation as well as the output from mpiexce -info Your help is greatly appreciated. Jorge On Tue, Apr 28, 2015 at 9:30 PM, Kenneth Raffenetti <raffenet(a)mcs.anl.gov> wrote: > Can you confirm your MPICH installation working correctly? Have you tried > compiling and running the cpi.c program from the examples directory from > the tarball? > > The MPICH 3.0.4 installation should contain a libmpich.so.10 file, so it > will be helpful to know things are installed and working well. > > Ken > > > On 04/28/2015 04:58 PM, Jorge Pachon wrote: > >> Hi all, >> >> I´m trying to run WRF for which I´m using mpich libraries. I recently >> installed mpich-3.0.4 and mpich-3.1.4 (with ifort) hoping to install the >> libmpich.so.10 file, but neither of them have it ,and I always got this >> error >> >> ./real.exe: error while loading shared libraries: libmpich.so.10: cannot >> open shared object file: No such file or directory >> >> I searched in the mpich folders: /usr/local/lib and >> /usr/local/bin/mpich, but cannot find the file >> >> any help will be appreciated >> >> Jorge >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_Allreduce is slow for 7 processes or more
by Junchao Zhang 04 May '15

04 May '15

Could you try mpich3.2b2? I tested your code with it on my laptop. My timing is np time(s) 2 102.57 4 51.285 8 25.6425 16 12.8212 --Junchao Zhang On Mon, May 4, 2015 at 10:14 AM, David Froger <david.froger.ml(a)mailoo.org> wrote: > Thanks Junchao. > > > I don't see you measure MPI_Allreduce. > > You're right, let's call my code "a simple example to reproduce a bug" > rather > than a benchmark. > > > Basically you only measured some random > numbers across processes. > > The usleep simulate the time to perform computation in my real code > (Computational Fluid Dynamic software). bench_mpi.cxx only do a > usleep(microseconds) then call MPI_Allreduce. microseconds is a constant > base > time, divised by mpi_size (+ a random overhead between 0% and 5%, so that > MPI_Allreduce is not called at the same wall clock on all proceses, but I > thing a should have use a different seed on each proc). > > So because what the code do is only usleep(base_time / mpi_size), I expect > the > wall clock time to be half with twice processor. > > With MPiCH 3.1.4, the wall clock time increase with 7 or more processes. > MPI_Allreduce become very slow without a reason. I'm triying to understand > why. > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Shared Memory Segfaults
by Brian Cornille 04 May '15

04 May '15

Jeff, I agree that the code I supplied has no interesting parallelism. That code was simply an attempt to diagnose or replicated an error I was getting with a more complicated code which also had shared memory structures pointing to other shared memory structures. The linked list was just a quick example of a similar structure to test what the issue could be. It ended up having very similar error behavior as the original program so I thought I would share it instead of a more complicated project. For the project I was working on I wanted to create large shared memory structures that would be read-only once the initial set-up was done. Some of these shared memory structures were to have shared memory structures allocated within them, which appeared to generate the errors I was encountering. If you are interested in the original code that produced these errors it is posted at: https://github.com/bcornille/mpi-shared-ne506/tree/develop/src (mostly in geoms.c) There are likely ways that I could change the structure of the code to work around this problem, but I was hoping to try to understand what was causing the very strange errors that I produced. I encountered errors when I tried to make assignments to shared memory regions that were pointed to from other shared memory regions. I am hoping that the error in the code I provided is the mirror image of the error in my original code. Thanks and best, Brian Cornille On Sat, May 2 at 5:46 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > >I do not understand what this program is supposed to do. >MPI_Win_allocate_shared is collective, as is MPI_Win_fence, so I don't >see how any interesting parallelism emerges from this type of >implementation of a linked-list. > >Are you just using the linked-list to manage a collection of shared >memory windows (GMR does this in >http://git.mpich.org/armci-mpi.git/blob/HEAD:/src/gmr.c, for example)? > >If you want to do a distributed linked-list using RMA, I recall there >is a linked-list example in the MPI-3 spec or perhaps somewhere else >(somebody on this list will remember, or I can look it up). And this >example probably uses MPI_Win_create_dynamic and MPI_Win_attach, which >means no interprocess load-store. > >Jeff > >On Sat, May 2, 2015 at 4:06 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: >> I can reproduce the segfault with the latest mpich. In Clean_List(), I think >> there is a data race, since all shmem members update the head node >> head->next_win = temp_win; >> head->next = temp_next; >> I tried to simplify Clean_List() further as follows, >> >> void Clean_List() >> { >> Node *cur_node = head; >> MPI_Win cur_win = head_win; >> Node *next_node; >> MPI_Win next_win; >> >> while (cur_node) { >> next_node = cur_node->next; >> next_win = cur_node->next_win; >> MPI_Win_free(&cur_win); >> cur_node = next_node; >> cur_win = next_win; >> } >> >> head = tail = NULL; >> } >> >> But I still met segfault. With gdb, the segfault disappears. If I comment >> out the call to Clean_List in main(), the error also disappear. >> I Cc'ed our local RMA expert Xin to see if she has new findings. >> >> --Junchao Zhang >> >> On Sat, May 2, 2015 at 10:26 AM, Brian Cornille <bcornille(a)wisc.edu> wrote: >>> >>> Hello, >>> >>> >>> In working on a project that is attempting to use MPI shared memory (from >>> MPI_Win_allocate_shared) I began getting inconsistent segfaults in portions >>> of the code that appeared to have no memory errors when investigated with >>> gdb. I believe I have somewhat reproduced this error in a small code >>> (attached) that creates a linked list of MPI shared memory allocated >>> elements. >>> >>> >>> The attached program segfaults for me when run with more than one process. >>> However, will not segfault if run in gdb (e.g. mpirun -n 2 xterm -e gdb >>> ./mpi_shmem_ll). I have done what I can to eliminate any apparent race >>> conditions. Any help in this matter would be much appreciated. >>> >>> >>> Thanks and best, >>> >>> Brian Cornille >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > >-- >Jeff Hammond >jeff.science(a)gmail.com >http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_Allreduce is slow for 7 processes or more
by David Froger 04 May '15

04 May '15

Thanks Junchao. > I don't see you measure MPI_Allreduce. You're right, let's call my code "a simple example to reproduce a bug" rather than a benchmark. > Basically you only measured some random > numbers across processes. The usleep simulate the time to perform computation in my real code (Computational Fluid Dynamic software). bench_mpi.cxx only do a usleep(microseconds) then call MPI_Allreduce. microseconds is a constant base time, divised by mpi_size (+ a random overhead between 0% and 5%, so that MPI_Allreduce is not called at the same wall clock on all proceses, but I thing a should have use a different seed on each proc). So because what the code do is only usleep(base_time / mpi_size), I expect the wall clock time to be half with twice processor. With MPiCH 3.1.4, the wall clock time increase with 7 or more processes. MPI_Allreduce become very slow without a reason. I'm triying to understand why. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_Allreduce is slow for 7 processes or more
by Junchao Zhang 04 May '15

04 May '15

I don't see you measure MPI_Allreduce. Basically you only measured some random numbers across processes. See osu_allreduce in OSU microbenchmark ( http://mvapich.cse.ohio-state.edu/benchmarks/) for a MPI_Allreduce benchmark. --Junchao Zhang On Mon, May 4, 2015 at 2:35 AM, David Froger <david.froger.ml(a)mailoo.org> wrote: > Hello, > > I've written a little MPI_Allreduce benchmarks [0]. > > The results with MPICH 1.4.1 [1] and Open MPI 1.8.4 [2] (1 column for the > number of processors, and 1 columns for the corresponding wall clock time) > show > that clock time is half with twice processors. > > But with `MPICH 3.1.4`, the wall clock time increase for 7, 8 or more > processes [3]. > > In my real code and for all of the 3 above MPI implementation, I observe > the > same problem for 7, 8 or more processes, while I expect my code to be > scallable > to at least 8 or 16 processes. > > So I'm trying to understand what could happen with the little benchmark and > MPICH 3.1.4? > > Thanks for reading. > > Best regards, > David > > [0] > https://github.com/dfroger/issue/blob/8b8bdd8e4b2b5e265c25fc2ba7077f6a108bb… > [1] > https://github.com/dfroger/issue/blob/8b8bdd8e4b2b5e265c25fc2ba7077f6a108bb… > [2] > https://github.com/dfroger/issue/blob/8b8bdd8e4b2b5e265c25fc2ba7077f6a108bb… > [3] > https://github.com/dfroger/issue/blob/8b8bdd8e4b2b5e265c25fc2ba7077f6a108bb… > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPI_Allreduce is slow for 7 processes or more
by David Froger 04 May '15

04 May '15

Hello, I've written a little MPI_Allreduce benchmarks [0]. The results with MPICH 1.4.1 [1] and Open MPI 1.8.4 [2] (1 column for the number of processors, and 1 columns for the corresponding wall clock time) show that clock time is half with twice processors. But with `MPICH 3.1.4`, the wall clock time increase for 7, 8 or more processes [3]. In my real code and for all of the 3 above MPI implementation, I observe the same problem for 7, 8 or more processes, while I expect my code to be scallable to at least 8 or 16 processes. So I'm trying to understand what could happen with the little benchmark and MPICH 3.1.4? Thanks for reading. Best regards, David [0] https://github.com/dfroger/issue/blob/8b8bdd8e4b2b5e265c25fc2ba7077f6a108bb… [1] https://github.com/dfroger/issue/blob/8b8bdd8e4b2b5e265c25fc2ba7077f6a108bb… [2] https://github.com/dfroger/issue/blob/8b8bdd8e4b2b5e265c25fc2ba7077f6a108bb… [3] https://github.com/dfroger/issue/blob/8b8bdd8e4b2b5e265c25fc2ba7077f6a108bb… _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Shared Memory Segfaults
by Jeff Hammond 02 May '15

02 May '15

I do not understand what this program is supposed to do. MPI_Win_allocate_shared is collective, as is MPI_Win_fence, so I don't see how any interesting parallelism emerges from this type of implementation of a linked-list. Are you just using the linked-list to manage a collection of shared memory windows (GMR does this in http://git.mpich.org/armci-mpi.git/blob/HEAD:/src/gmr.c, for example)? If you want to do a distributed linked-list using RMA, I recall there is a linked-list example in the MPI-3 spec or perhaps somewhere else (somebody on this list will remember, or I can look it up). And this example probably uses MPI_Win_create_dynamic and MPI_Win_attach, which means no interprocess load-store. Jeff On Sat, May 2, 2015 at 4:06 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: > I can reproduce the segfault with the latest mpich. In Clean_List(), I think > there is a data race, since all shmem members update the head node > head->next_win = temp_win; > head->next = temp_next; > I tried to simplify Clean_List() further as follows, > > void Clean_List() > { > Node *cur_node = head; > MPI_Win cur_win = head_win; > Node *next_node; > MPI_Win next_win; > > while (cur_node) { > next_node = cur_node->next; > next_win = cur_node->next_win; > MPI_Win_free(&cur_win); > cur_node = next_node; > cur_win = next_win; > } > > head = tail = NULL; > } > > But I still met segfault. With gdb, the segfault disappears. If I comment > out the call to Clean_List in main(), the error also disappear. > I Cc'ed our local RMA expert Xin to see if she has new findings. > > --Junchao Zhang > > On Sat, May 2, 2015 at 10:26 AM, Brian Cornille <bcornille(a)wisc.edu> wrote: >> >> Hello, >> >> >> In working on a project that is attempting to use MPI shared memory (from >> MPI_Win_allocate_shared) I began getting inconsistent segfaults in portions >> of the code that appeared to have no memory errors when investigated with >> gdb. I believe I have somewhat reproduced this error in a small code >> (attached) that creates a linked list of MPI shared memory allocated >> elements. >> >> >> The attached program segfaults for me when run with more than one process. >> However, will not segfault if run in gdb (e.g. mpirun -n 2 xterm -e gdb >> ./mpi_shmem_ll). I have done what I can to eliminate any apparent race >> conditions. Any help in this matter would be much appreciated. >> >> >> Thanks and best, >> >> Brian Cornille >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Shared Memory Segfaults
by Junchao Zhang 02 May '15

02 May '15

I can reproduce the segfault with the latest mpich. In Clean_List(), I think there is a data race, since all shmem members update the head node head->next_win = temp_win; head->next = temp_next; I tried to simplify Clean_List() further as follows, void Clean_List() { Node *cur_node = head; MPI_Win cur_win = head_win; Node *next_node; MPI_Win next_win; while (cur_node) { next_node = cur_node->next; next_win = cur_node->next_win; MPI_Win_free(&cur_win); cur_node = next_node; cur_win = next_win; } head = tail = NULL; } But I still met segfault. With gdb, the segfault disappears. If I comment out the call to Clean_List in main(), the error also disappear. I Cc'ed our local RMA expert Xin to see if she has new findings. --Junchao Zhang On Sat, May 2, 2015 at 10:26 AM, Brian Cornille <bcornille(a)wisc.edu> wrote: > Hello, > > > In working on a project that is attempting to use MPI shared memory > (from MPI_Win_allocate_shared) I began getting inconsistent segfaults in > portions of the code that appeared to have no memory errors when > investigated with gdb. I believe I have somewhat reproduced this error in > a small code (attached) that creates a linked list of MPI shared memory > allocated elements. > > > The attached program segfaults for me when run with more than one > process. However, will not segfault if run in gdb (e.g. mpirun -n 2 xterm > -e gdb ./mpi_shmem_ll). I have done what I can to eliminate any apparent > race conditions. Any help in this matter would be much appreciated. > > > Thanks and best, > > Brian Cornille > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Shared Memory Segfaults
by Brian Cornille 02 May '15

02 May '15

Hello, In working on a project that is attempting to use MPI shared memory (from MPI_Win_allocate_shared) I began getting inconsistent segfaults in portions of the code that appeared to have no memory errors when investigated with gdb. I believe I have somewhat reproduced this error in a small code (attached) that creates a linked list of MPI shared memory allocated elements. The attached program segfaults for me when run with more than one process. However, will not segfault if run in gdb (e.g. mpirun -n 2 xterm -e gdb ./mpi_shmem_ll). I have done what I can to eliminate any apparent race conditions. Any help in this matter would be much appreciated. Thanks and best, Brian Cornille _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0