January 2013 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] can't MPI_Abort a program due to deadlock at pthread_mutex_lock
by Jeff Hammond 02 Jan '13

02 Jan '13

I'm using Mac OS but I imagine that the BSD implementation is similar. I'll accept that this is an incorrect program on the basis of http://www.mcs.anl.gov/research/projects/mpi/mpi-standard/mpi-report-2.0/no… (this text persists in more recent versions of the standard) and find a better way to implement it. Jeff On Wed, Jan 2, 2013 at 10:01 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Wed, Jan 2, 2013 at 9:49 AM, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: >> >> I wrote a program that calls MPI from a pthread. MPI_THREAD_MULTIPLE >> is used. I cancel and then join the pthread back into the main >> program. After I do this, any calls to MPI, including MPI_Abort (see >> first gdb+source below) and MPI_Barrier (see second gdb+source below) >> deadlock on pthread_mutex_lock. >> >> I can understand that canceling the thread while it is holding the MPI >> pthread mutex is a bad idea, but I'm not sure this is an incorrect >> program. Is pthread_cancel using signals? I recall that MPICH >> functions are not guaranteed to be signal-safe. > > > On Linux, pthread_cancel is implemented using tgkill ("man tgkill"). > >> >> Is it possible to implement MPI_Abort such that acquisition of the MPI >> mutex is not required in order to allow deadlocked programs to >> terminate? It seems rather unfortunate that MPI_Abort does not >> perform the desired action here. Obviously, I can use abort(), but on >> some implementations, this will leave the system resources in a dirty >> state. >> >> I know a bunch of ways to work around this, but it would be helpful to >> understand exactly what the problem is. I wonder if there is a way to >> modify MPICH such that the mutex is released when a thread holding it >> is cancelled. >> >> Is it possible for MPICH to provide an MPIX call or otherwise document >> an internal, MPICH-specific function to clear the MPI mutex? I could >> resolve this problem if I could merely call MPIX_Mutex_force_unlock() >> and let the program terminate as desired. >> >> Thanks, >> >> Jeff >> >> $ make >> mpicc -g -O0 -Wall -std=gnu99 -c bug.c -o bug.o >> mpicc -g -O0 -Wall -std=gnu99 safemalloc.o bug.o -lm -o bug.x >> rm bug.o >> >> ======== stuck in abort/finalize ======== >> >> $ gdb ./bug.x >> GNU gdb 6.3.50-20050815 (Apple version gdb-1752) (Sat Jan 28 03:02:46 UTC >> 2012) >> Copyright 2004 Free Software Foundation, Inc. >> GDB is free software, covered by the GNU General Public License, and you >> are >> welcome to change it and/or distribute copies of it under certain >> conditions. >> Type "show copying" to see the conditions. >> There is absolutely no warranty for GDB. Type "show warranty" for >> details. >> This GDB was configured as "x86_64-apple-darwin"...Reading symbols for >> shared libraries ...... done >> >> warning: Could not find object file >> "/Users/jhammond/eclipse/OSPRI/trunk/tests/devices/mpi-pt/bug.o" - no >> debug information available for "bug.c". >> >> >> (gdb) run >> Starting program: >> /Users/jhammond/eclipse/OSPRI/trunk/tests/devices/mpi-pt/bug.x >> Reading symbols for shared libraries +++++........................ done >> test done except for pthread shutdown >> 0: Progress >> all done >> >> ^C >> Program received signal SIGINT, Interrupt. >> 0x00007fff912fdbf2 in __psynch_mutexwait () >> (gdb) bt >> #0 0x00007fff912fdbf2 in __psynch_mutexwait () >> #1 0x00007fff8c80a1a1 in pthread_mutex_lock () >> #2 0x000000010004912b in MPIU_Thread_CS_enter_lockname_recursive_impl_ () >> #3 0x0000000100048d64 in MPI_Abort () >> #4 0x0000000100000cc0 in main () >> (gdb) quit >> The program is running. Exit anyway? (y or n) y >> >> $ cat bug.c >> #include <stdio.h> >> #include <stdlib.h> >> #include <unistd.h> >> #include <string.h> >> #include <assert.h> >> #include <limits.h> >> #include <pthread.h> >> >> #include <mpi.h> >> >> #define DEBUG >> >> pthread_t Progress_thread; >> >> static void * Progress_function(void * dummy) >> { >> int rank; >> MPI_Comm_rank(MPI_COMM_WORLD, &rank); >> >> while (1) >> { >> #ifdef DEBUG >> printf("%d: Progress \n", rank); >> #endif >> MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, >> MPI_STATUS_IGNORE); >> //usleep(500); >> } >> >> return NULL; >> } >> >> int main(int argc, char * argv[]) >> { >> int rc; >> int rank, size; >> int provided; >> >> MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided); >> if (provided!=MPI_THREAD_MULTIPLE) MPI_Abort(MPI_COMM_WORLD, 1); >> >> MPI_Comm_rank(MPI_COMM_WORLD, &rank); >> MPI_Comm_size(MPI_COMM_WORLD, &size); >> >> rc = pthread_create(&Progress_thread, NULL, &Progress_function, NULL); >> if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); >> >> MPI_Barrier(MPI_COMM_WORLD); >> >> printf("test done except for pthread shutdown \n"); >> fflush(stdout); >> >> MPI_Barrier(MPI_COMM_WORLD); >> >> rc = pthread_cancel(Progress_thread); >> if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); >> >> void * rv; >> rc = pthread_join(Progress_thread, &rv); >> if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); >> >> printf("all done \n"); >> fflush(stdout); >> >> MPI_Abort(MPI_COMM_SELF, 2); >> MPI_Finalize(); >> >> return 0; >> } >> >> ======== stuck in barrier ======== >> >> $ gdb ./bug.x >> GNU gdb 6.3.50-20050815 (Apple version gdb-1752) (Sat Jan 28 03:02:46 UTC >> 2012) >> Copyright 2004 Free Software Foundation, Inc. >> GDB is free software, covered by the GNU General Public License, and you >> are >> welcome to change it and/or distribute copies of it under certain >> conditions. >> Type "show copying" to see the conditions. >> There is absolutely no warranty for GDB. Type "show warranty" for >> details. >> This GDB was configured as "x86_64-apple-darwin"...Reading symbols for >> shared libraries ...... done >> >> warning: Could not find object file >> "/Users/jhammond/eclipse/OSPRI/trunk/tests/devices/mpi-pt/bug.o" - no >> debug information available for "bug.c". >> >> >> (gdb) run >> Starting program: >> /Users/jhammond/eclipse/OSPRI/trunk/tests/devices/mpi-pt/bug.x >> Reading symbols for shared libraries +++++........................ done >> before pthread shutdown >> 0: Progress >> after pthread shutdown >> ^C >> Program received signal SIGINT, Interrupt. >> 0x00007fff912fdbf2 in __psynch_mutexwait () >> (gdb) bt >> #0 0x00007fff912fdbf2 in __psynch_mutexwait () >> #1 0x00007fff8c80a1a1 in pthread_mutex_lock () >> #2 0x000000010000f7eb in MPIU_Thread_CS_enter_lockname_recursive_impl_ () >> #3 0x000000010000f4f7 in MPI_Barrier () >> #4 0x0000000100000c4d in main () >> (gdb) quit >> The program is running. Exit anyway? (y or n) y >> Jeffs-MacBook-Pro:mpi-pt jhammond$ cat bug.c >> #include <stdio.h> >> #include <stdlib.h> >> #include <unistd.h> >> #include <string.h> >> #include <assert.h> >> #include <limits.h> >> #include <pthread.h> >> >> #include <mpi.h> >> >> #define DEBUG >> >> pthread_t Progress_thread; >> >> static void * Progress_function(void * dummy) >> { >> int rank; >> MPI_Comm_rank(MPI_COMM_WORLD, &rank); >> >> while (1) >> { >> #ifdef DEBUG >> printf("%d: Progress \n", rank); >> #endif >> MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, >> MPI_STATUS_IGNORE); >> //usleep(500); >> } >> >> return NULL; >> } >> >> int main(int argc, char * argv[]) >> { >> int rc; >> int rank, size; >> int provided; >> >> MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided); >> if (provided!=MPI_THREAD_MULTIPLE) MPI_Abort(MPI_COMM_WORLD, 1); >> >> MPI_Comm_rank(MPI_COMM_WORLD, &rank); >> MPI_Comm_size(MPI_COMM_WORLD, &size); >> >> rc = pthread_create(&Progress_thread, NULL, &Progress_function, NULL); >> if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); >> >> MPI_Barrier(MPI_COMM_WORLD); >> >> printf("before pthread shutdown \n"); >> fflush(stdout); >> >> MPI_Barrier(MPI_COMM_WORLD); >> >> rc = pthread_cancel(Progress_thread); >> if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); >> >> void * rv; >> rc = pthread_join(Progress_thread, &rv); >> if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); >> >> printf("after pthread shutdown \n"); >> fflush(stdout); >> >> MPI_Barrier(MPI_COMM_WORLD); >> >> printf("before MPI Abort/Finalize \n"); >> fflush(stdout); >> >> MPI_Abort(MPI_COMM_SELF, 2); >> MPI_Finalize(); >> >> printf("after MPI Abort/Finalize \n"); >> fflush(stdout); >> >> return 0; >> } >> >> -- >> Jeff Hammond >> Argonne Leadership Computing Facility >> University of Chicago Computation Institute >> jhammond(a)alcf.anl.gov / (630) 252-5381 >> http://www.linkedin.com/in/jeffhammond >> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-discuss] can't MPI_Abort a program due to deadlock at pthread_mutex_lock
by Jed Brown 02 Jan '13

02 Jan '13

On Wed, Jan 2, 2013 at 9:49 AM, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: > I wrote a program that calls MPI from a pthread. MPI_THREAD_MULTIPLE > is used. I cancel and then join the pthread back into the main > program. After I do this, any calls to MPI, including MPI_Abort (see > first gdb+source below) and MPI_Barrier (see second gdb+source below) > deadlock on pthread_mutex_lock. > > I can understand that canceling the thread while it is holding the MPI > pthread mutex is a bad idea, but I'm not sure this is an incorrect > program. Is pthread_cancel using signals? I recall that MPICH > functions are not guaranteed to be signal-safe. > On Linux, pthread_cancel is implemented using tgkill ("man tgkill"). > Is it possible to implement MPI_Abort such that acquisition of the MPI > mutex is not required in order to allow deadlocked programs to > terminate? It seems rather unfortunate that MPI_Abort does not > perform the desired action here. Obviously, I can use abort(), but on > some implementations, this will leave the system resources in a dirty > state. > > I know a bunch of ways to work around this, but it would be helpful to > understand exactly what the problem is. I wonder if there is a way to > modify MPICH such that the mutex is released when a thread holding it > is cancelled. > > Is it possible for MPICH to provide an MPIX call or otherwise document > an internal, MPICH-specific function to clear the MPI mutex? I could > resolve this problem if I could merely call MPIX_Mutex_force_unlock() > and let the program terminate as desired. > > Thanks, > > Jeff > > $ make > mpicc -g -O0 -Wall -std=gnu99 -c bug.c -o bug.o > mpicc -g -O0 -Wall -std=gnu99 safemalloc.o bug.o -lm -o bug.x > rm bug.o > > ======== stuck in abort/finalize ======== > > $ gdb ./bug.x > GNU gdb 6.3.50-20050815 (Apple version gdb-1752) (Sat Jan 28 03:02:46 UTC > 2012) > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you > are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "x86_64-apple-darwin"...Reading symbols for > shared libraries ...... done > > warning: Could not find object file > "/Users/jhammond/eclipse/OSPRI/trunk/tests/devices/mpi-pt/bug.o" - no > debug information available for "bug.c". > > > (gdb) run > Starting program: > /Users/jhammond/eclipse/OSPRI/trunk/tests/devices/mpi-pt/bug.x > Reading symbols for shared libraries +++++........................ done > test done except for pthread shutdown > 0: Progress > all done > > ^C > Program received signal SIGINT, Interrupt. > 0x00007fff912fdbf2 in __psynch_mutexwait () > (gdb) bt > #0 0x00007fff912fdbf2 in __psynch_mutexwait () > #1 0x00007fff8c80a1a1 in pthread_mutex_lock () > #2 0x000000010004912b in MPIU_Thread_CS_enter_lockname_recursive_impl_ () > #3 0x0000000100048d64 in MPI_Abort () > #4 0x0000000100000cc0 in main () > (gdb) quit > The program is running. Exit anyway? (y or n) y > > $ cat bug.c > #include <stdio.h> > #include <stdlib.h> > #include <unistd.h> > #include <string.h> > #include <assert.h> > #include <limits.h> > #include <pthread.h> > > #include <mpi.h> > > #define DEBUG > > pthread_t Progress_thread; > > static void * Progress_function(void * dummy) > { > int rank; > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > > while (1) > { > #ifdef DEBUG > printf("%d: Progress \n", rank); > #endif > MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, > MPI_STATUS_IGNORE); > //usleep(500); > } > > return NULL; > } > > int main(int argc, char * argv[]) > { > int rc; > int rank, size; > int provided; > > MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided); > if (provided!=MPI_THREAD_MULTIPLE) MPI_Abort(MPI_COMM_WORLD, 1); > > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > MPI_Comm_size(MPI_COMM_WORLD, &size); > > rc = pthread_create(&Progress_thread, NULL, &Progress_function, NULL); > if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); > > MPI_Barrier(MPI_COMM_WORLD); > > printf("test done except for pthread shutdown \n"); > fflush(stdout); > > MPI_Barrier(MPI_COMM_WORLD); > > rc = pthread_cancel(Progress_thread); > if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); > > void * rv; > rc = pthread_join(Progress_thread, &rv); > if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); > > printf("all done \n"); > fflush(stdout); > > MPI_Abort(MPI_COMM_SELF, 2); > MPI_Finalize(); > > return 0; > } > > ======== stuck in barrier ======== > > $ gdb ./bug.x > GNU gdb 6.3.50-20050815 (Apple version gdb-1752) (Sat Jan 28 03:02:46 UTC > 2012) > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you > are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "x86_64-apple-darwin"...Reading symbols for > shared libraries ...... done > > warning: Could not find object file > "/Users/jhammond/eclipse/OSPRI/trunk/tests/devices/mpi-pt/bug.o" - no > debug information available for "bug.c". > > > (gdb) run > Starting program: > /Users/jhammond/eclipse/OSPRI/trunk/tests/devices/mpi-pt/bug.x > Reading symbols for shared libraries +++++........................ done > before pthread shutdown > 0: Progress > after pthread shutdown > ^C > Program received signal SIGINT, Interrupt. > 0x00007fff912fdbf2 in __psynch_mutexwait () > (gdb) bt > #0 0x00007fff912fdbf2 in __psynch_mutexwait () > #1 0x00007fff8c80a1a1 in pthread_mutex_lock () > #2 0x000000010000f7eb in MPIU_Thread_CS_enter_lockname_recursive_impl_ () > #3 0x000000010000f4f7 in MPI_Barrier () > #4 0x0000000100000c4d in main () > (gdb) quit > The program is running. Exit anyway? (y or n) y > Jeffs-MacBook-Pro:mpi-pt jhammond$ cat bug.c > #include <stdio.h> > #include <stdlib.h> > #include <unistd.h> > #include <string.h> > #include <assert.h> > #include <limits.h> > #include <pthread.h> > > #include <mpi.h> > > #define DEBUG > > pthread_t Progress_thread; > > static void * Progress_function(void * dummy) > { > int rank; > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > > while (1) > { > #ifdef DEBUG > printf("%d: Progress \n", rank); > #endif > MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, > MPI_STATUS_IGNORE); > //usleep(500); > } > > return NULL; > } > > int main(int argc, char * argv[]) > { > int rc; > int rank, size; > int provided; > > MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided); > if (provided!=MPI_THREAD_MULTIPLE) MPI_Abort(MPI_COMM_WORLD, 1); > > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > MPI_Comm_size(MPI_COMM_WORLD, &size); > > rc = pthread_create(&Progress_thread, NULL, &Progress_function, NULL); > if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); > > MPI_Barrier(MPI_COMM_WORLD); > > printf("before pthread shutdown \n"); > fflush(stdout); > > MPI_Barrier(MPI_COMM_WORLD); > > rc = pthread_cancel(Progress_thread); > if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); > > void * rv; > rc = pthread_join(Progress_thread, &rv); > if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); > > printf("after pthread shutdown \n"); > fflush(stdout); > > MPI_Barrier(MPI_COMM_WORLD); > > printf("before MPI Abort/Finalize \n"); > fflush(stdout); > > MPI_Abort(MPI_COMM_SELF, 2); > MPI_Finalize(); > > printf("after MPI Abort/Finalize \n"); > fflush(stdout); > > return 0; > } > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

can't MPI_Abort a program due to deadlock at pthread_mutex_lock
by Jeff Hammond 02 Jan '13

02 Jan '13

I wrote a program that calls MPI from a pthread. MPI_THREAD_MULTIPLE is used. I cancel and then join the pthread back into the main program. After I do this, any calls to MPI, including MPI_Abort (see first gdb+source below) and MPI_Barrier (see second gdb+source below) deadlock on pthread_mutex_lock. I can understand that canceling the thread while it is holding the MPI pthread mutex is a bad idea, but I'm not sure this is an incorrect program. Is pthread_cancel using signals? I recall that MPICH functions are not guaranteed to be signal-safe. Is it possible to implement MPI_Abort such that acquisition of the MPI mutex is not required in order to allow deadlocked programs to terminate? It seems rather unfortunate that MPI_Abort does not perform the desired action here. Obviously, I can use abort(), but on some implementations, this will leave the system resources in a dirty state. I know a bunch of ways to work around this, but it would be helpful to understand exactly what the problem is. I wonder if there is a way to modify MPICH such that the mutex is released when a thread holding it is cancelled. Is it possible for MPICH to provide an MPIX call or otherwise document an internal, MPICH-specific function to clear the MPI mutex? I could resolve this problem if I could merely call MPIX_Mutex_force_unlock() and let the program terminate as desired. Thanks, Jeff $ make mpicc -g -O0 -Wall -std=gnu99 -c bug.c -o bug.o mpicc -g -O0 -Wall -std=gnu99 safemalloc.o bug.o -lm -o bug.x rm bug.o ======== stuck in abort/finalize ======== $ gdb ./bug.x GNU gdb 6.3.50-20050815 (Apple version gdb-1752) (Sat Jan 28 03:02:46 UTC 2012) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-apple-darwin"...Reading symbols for shared libraries ...... done warning: Could not find object file "/Users/jhammond/eclipse/OSPRI/trunk/tests/devices/mpi-pt/bug.o" - no debug information available for "bug.c". (gdb) run Starting program: /Users/jhammond/eclipse/OSPRI/trunk/tests/devices/mpi-pt/bug.x Reading symbols for shared libraries +++++........................ done test done except for pthread shutdown 0: Progress all done ^C Program received signal SIGINT, Interrupt. 0x00007fff912fdbf2 in __psynch_mutexwait () (gdb) bt #0 0x00007fff912fdbf2 in __psynch_mutexwait () #1 0x00007fff8c80a1a1 in pthread_mutex_lock () #2 0x000000010004912b in MPIU_Thread_CS_enter_lockname_recursive_impl_ () #3 0x0000000100048d64 in MPI_Abort () #4 0x0000000100000cc0 in main () (gdb) quit The program is running. Exit anyway? (y or n) y $ cat bug.c #include <stdio.h> #include <stdlib.h> #include <unistd.h> #include <string.h> #include <assert.h> #include <limits.h> #include <pthread.h> #include <mpi.h> #define DEBUG pthread_t Progress_thread; static void * Progress_function(void * dummy) { int rank; MPI_Comm_rank(MPI_COMM_WORLD, &rank); while (1) { #ifdef DEBUG printf("%d: Progress \n", rank); #endif MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE); //usleep(500); } return NULL; } int main(int argc, char * argv[]) { int rc; int rank, size; int provided; MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided); if (provided!=MPI_THREAD_MULTIPLE) MPI_Abort(MPI_COMM_WORLD, 1); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); rc = pthread_create(&Progress_thread, NULL, &Progress_function, NULL); if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); MPI_Barrier(MPI_COMM_WORLD); printf("test done except for pthread shutdown \n"); fflush(stdout); MPI_Barrier(MPI_COMM_WORLD); rc = pthread_cancel(Progress_thread); if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); void * rv; rc = pthread_join(Progress_thread, &rv); if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); printf("all done \n"); fflush(stdout); MPI_Abort(MPI_COMM_SELF, 2); MPI_Finalize(); return 0; } ======== stuck in barrier ======== $ gdb ./bug.x GNU gdb 6.3.50-20050815 (Apple version gdb-1752) (Sat Jan 28 03:02:46 UTC 2012) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-apple-darwin"...Reading symbols for shared libraries ...... done warning: Could not find object file "/Users/jhammond/eclipse/OSPRI/trunk/tests/devices/mpi-pt/bug.o" - no debug information available for "bug.c". (gdb) run Starting program: /Users/jhammond/eclipse/OSPRI/trunk/tests/devices/mpi-pt/bug.x Reading symbols for shared libraries +++++........................ done before pthread shutdown 0: Progress after pthread shutdown ^C Program received signal SIGINT, Interrupt. 0x00007fff912fdbf2 in __psynch_mutexwait () (gdb) bt #0 0x00007fff912fdbf2 in __psynch_mutexwait () #1 0x00007fff8c80a1a1 in pthread_mutex_lock () #2 0x000000010000f7eb in MPIU_Thread_CS_enter_lockname_recursive_impl_ () #3 0x000000010000f4f7 in MPI_Barrier () #4 0x0000000100000c4d in main () (gdb) quit The program is running. Exit anyway? (y or n) y Jeffs-MacBook-Pro:mpi-pt jhammond$ cat bug.c #include <stdio.h> #include <stdlib.h> #include <unistd.h> #include <string.h> #include <assert.h> #include <limits.h> #include <pthread.h> #include <mpi.h> #define DEBUG pthread_t Progress_thread; static void * Progress_function(void * dummy) { int rank; MPI_Comm_rank(MPI_COMM_WORLD, &rank); while (1) { #ifdef DEBUG printf("%d: Progress \n", rank); #endif MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE); //usleep(500); } return NULL; } int main(int argc, char * argv[]) { int rc; int rank, size; int provided; MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided); if (provided!=MPI_THREAD_MULTIPLE) MPI_Abort(MPI_COMM_WORLD, 1); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); rc = pthread_create(&Progress_thread, NULL, &Progress_function, NULL); if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); MPI_Barrier(MPI_COMM_WORLD); printf("before pthread shutdown \n"); fflush(stdout); MPI_Barrier(MPI_COMM_WORLD); rc = pthread_cancel(Progress_thread); if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); void * rv; rc = pthread_join(Progress_thread, &rv); if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); printf("after pthread shutdown \n"); fflush(stdout); MPI_Barrier(MPI_COMM_WORLD); printf("before MPI Abort/Finalize \n"); fflush(stdout); MPI_Abort(MPI_COMM_SELF, 2); MPI_Finalize(); printf("after MPI Abort/Finalize \n"); fflush(stdout); return 0; } -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-discuss] MPI_Accumulate with MAXLOC does not error for incompatible types
by Jeff Hammond 02 Jan '13

02 Jan '13

> Perhaps, but my reading is that the predefined Op is applied to the basic > type from which I constructed MY_2INT (i.e., MPI_INT). Since MPI_MAXLOC does > not apply to MPI_INT, this would be an error. I wouldn't know how to write > precise language specifying the behavior that would lead to MPI_MAXLOC > working with MY_2INT. You're right. My comment was erroneous. Jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-discuss] [PATCH] Use attribute layout_compatible for pair types
by Jed Brown 01 Jan '13

01 Jan '13

In addition, I suggest guarding these definitions. Leaving these in increases the total number of symbols in an example executable linking PETSc by a factor of 2. (They're all read-only, but they're still there.) Clang is smart enough to remove these, presumably because it understands the special attributes. static const MPI_Datatype mpich_mpi_float_int MPICH_ATTR_TYPE_TAG_LAYOUT_COMPATIBLE(struct mpich_struct_mpi_float_int) = MPI_FLOAT_INT; static const MPI_Datatype mpich_mpi_double_int MPICH_ATTR_TYPE_TAG_LAYOUT_COMPATIBLE(struct mpich_struct_mpi_double_int) = MPI_DOUBLE_INT; static const MPI_Datatype mpich_mpi_long_int MPICH_ATTR_TYPE_TAG_LAYOUT_COMPATIBLE(struct mpich_struct_mpi_long_int) = MPI_LONG_INT; static const MPI_Datatype mpich_mpi_short_int MPICH_ATTR_TYPE_TAG_LAYOUT_COMPATIBLE(struct mpich_struct_mpi_short_int) = MPI_SHORT_INT; static const MPI_Datatype mpich_mpi_2int MPICH_ATTR_TYPE_TAG_LAYOUT_COMPATIBLE(struct mpich_struct_mpi_2int) = MPI_2INT; static const MPI_Datatype mpich_mpi_long_double_int MPICH_ATTR_TYPE_TAG_LAYOUT_COMPATIBLE(struct mpich_struct_mpi_long_double_int) = MPI_LONG_DOUBLE_INT; > > > On Tue, Jan 1, 2013 at 11:48 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > >> --- >> src/include/mpi.h.in | 12 ++++++------ >> 1 file changed, 6 insertions(+), 6 deletions(-) >> >> >

1 0

Re: [mpich-discuss] MPI_Accumulate with MAXLOC does not error for incompatible types
by Jed Brown 01 Jan '13

01 Jan '13

On Tue, Jan 1, 2013 at 7:49 PM, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: > MPI_Accumulate permits the use of pre-defined reduction operations > with user-defined datatypes, provided the user-defined datatypes are > homogeneous. Your use of MPI_MAXLOC with MY_2INT for MPI_Accumulate > is legal in my reading of the standard. It may not be legal for > MPI_Reduce because of the stupid asymmetry (that I believe you have > pointed out in the past) wherein MPI_Reduce (and MPI_Allreduce) cannot > do things that MPI_Accumulate can do w.r.t. built-in MPI_Op with > user-defined MPI_Datatype. > > Perhaps MPI_Accumulate is correct in not generating but providing the > wrong answer for MPI_MAXLOC, which would be a bug of a different > nature. > Perhaps, but my reading is that the predefined Op is applied to the basic type from which I constructed MY_2INT (i.e., MPI_INT). Since MPI_MAXLOC does not apply to MPI_INT, this would be an error. I wouldn't know how to write precise language specifying the behavior that would lead to MPI_MAXLOC working with MY_2INT.

1 0

Re: [mpich-discuss] MPI_Accumulate with MAXLOC does not error for incompatible types
by Jeff Hammond 01 Jan '13

01 Jan '13

MPI_Accumulate permits the use of pre-defined reduction operations with user-defined datatypes, provided the user-defined datatypes are homogeneous. Your use of MPI_MAXLOC with MY_2INT for MPI_Accumulate is legal in my reading of the standard. It may not be legal for MPI_Reduce because of the stupid asymmetry (that I believe you have pointed out in the past) wherein MPI_Reduce (and MPI_Allreduce) cannot do things that MPI_Accumulate can do w.r.t. built-in MPI_Op with user-defined MPI_Datatype. Perhaps MPI_Accumulate is correct in not generating but providing the wrong answer for MPI_MAXLOC, which would be a bug of a different nature. Jeff On Tue, Jan 1, 2013 at 1:09 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > With MPI_Reduce, we get a nice helpful error message: > > Fatal error in PMPI_Reduce: Invalid MPI_Op, error stack: > PMPI_Reduce(1217)...........: MPI_Reduce(sbuf=0x7fff3e39fa50, > rbuf=0x7fff3e39fa40, count=1, dtype=USER<contig>, MPI_MAXLOC, root=0, > MPI_COMM_WORLD) failed > MPIR_MAXLOC_check_dtype(151): MPI_Op MPI_MAXLOC operation not defined for > this datatype > > > With MPI_Accumulate, we just get wrong results. See attached code using a > user-defined contiguous results. > > I care about this because I'm stuck [1] dispatching into alternative > mechanisms for type combinations that are not directly supported by MPI > (e.g., struct {int u; long long k;} and __float128). I missed a spot and had > a confusing bug that I think MPICH should be catching. > > > [1] As usual, the crud would evaporate if only MPI_Accumulate were > extensible over datatypes. > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

MPI_Accumulate with MAXLOC does not error for incompatible types
by Jed Brown 01 Jan '13

01 Jan '13

With MPI_Reduce, we get a nice helpful error message: Fatal error in PMPI_Reduce: Invalid MPI_Op, error stack: PMPI_Reduce(1217)...........: MPI_Reduce(sbuf=0x7fff3e39fa50, rbuf=0x7fff3e39fa40, count=1, dtype=USER<contig>, MPI_MAXLOC, root=0, MPI_COMM_WORLD) failed MPIR_MAXLOC_check_dtype(151): MPI_Op MPI_MAXLOC operation not defined for this datatype With MPI_Accumulate, we just get wrong results. See attached code using a user-defined contiguous results. I care about this because I'm stuck [1] dispatching into alternative mechanisms for type combinations that are not directly supported by MPI (e.g., struct {int u; long long k;} and __float128). I missed a spot and had a confusing bug that I think MPICH should be catching. [1] As usual, the crud would evaporate if only MPI_Accumulate were extensible over datatypes.

1 0

[PATCH] Use attribute layout_compatible for pair types
by Jed Brown 01 Jan '13

01 Jan '13

--- src/include/mpi.h.in | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)

1 0

Re: [mpich-discuss] Error in c.txt nemesis does not exist
by Pavan Balaji 01 Jan '13

01 Jan '13

On 01/01/2013 07:58 AM US Central Time, behzad moradi wrote: > can't exec > '/Applications/Xcode.app/Contents/Developer/usr/bin/llvm-gcc-4.2' (No > such file or directory) > What is this error? Please don't use this mailing list unless it's an MPICH related error. FWIW, you just told us what the error is. You are not able to execute llvm-gcc-4.2. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0