January 2013 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Indexed MPI_Reduce_local
by Jed Brown 02 Jan '13

02 Jan '13

On Wed, Jan 2, 2013 at 11:35 AM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: > > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/318 > > I actually did a bunch of work in a private branch to support these types, > but stopped short of finishing it when the overall Forum opinion on them > turned rather sour. Perhaps we should restart the effort on this (both > standardization and implementation) now that MPI-3.0 is out. > I actually think they shouldn't be standardized, but instead, user-defined types should be restored to first-class status like they were in MPI-1. ;-) But as long as the Forum remains heavily committed to discriminating against user-defined types, I'd like to find a way to support them. > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/319 > > I don't have any objection to these types, they just need to be > implemented. We can put them in as MPIX_ types for now. > > > Any other suggestions? > > > > In lieu of a more elegant solution, I'm going to match on the types and > ops that I need to support and write the sparse reduction by hand. > > > > In the longer term, I'm curious whether the MPICH developers, and > eventually the Forum, may be interested in providing better support for > higher level comm libraries like this. > > Like motherhood and apple pie, of course the answer is yes :) The devil > is always in "to what extent". > I finished re-implementing my "SF" comm component over plain Isend/Irecv so that all the types I need work without any extras from MPI. My API still accepts MPI_Datatype and MPI_Op, and I'd eventually like it to support all combinations (including user-defined types and ops defined by the caller, but that I didn't have prior knowledge of) over all backends (one-sided, basic two-sided, and (soon) neighborhood collectives). In principle, there would be good reasons to use different backends in different contexts because they have different setup vs. execution costs, and different tradeoffs associated with packing.

1 0

Re: [mpich-discuss] Extensible Clang type annotation
by Dmitri Gribenko 02 Jan '13

02 Jan '13

[resending to the list, sorry for duplicates] On Wed, Jan 2, 2013 at 7:38 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > Also, could there be a way (maybe in a future clang release) for me to get > static type checking for my own types, such as __float128 or pair types like > {long,long} (which are not in the MPI standard)? It can't truly be a static > value because I have to create the type at run-time, but perhaps there could > be a way to tell clang "I promise to set this type up like so." Sorry, I did not understand your requirements completely, but you can annotate your own 'MPI_Datatype's (as I described in the previous email). Is that sufficient for your task? Dmitri -- main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if (j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr(a)gmail.com>*/

1 0

Re: [mpich-discuss] Extensible Clang type annotation
by Dmitri Gribenko 02 Jan '13

02 Jan '13

[resending to the list, sorry for duplicates] On Wed, Jan 2, 2013 at 7:31 PM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: > [adding Hal Finkel and Dmitri Gribenko, who did most of the annotations work] > > Hal/Dmitri, > > Do you have any ideas about how to handle Jed's case? Will the MPICH_ATTR_TYPE_TAG_LAYOUT_COMPATIBLE macro (reproduced below) help in this case? > > ----8<---- > # define MPICH_ATTR_TYPE_TAG_LAYOUT_COMPATIBLE(type) __attribute__((type_tag_for_datatype(MPI,type,layout_compatible))) > ----8<---- > > -Dave > > On Dec 30, 2012, at 11:13 PM CST, Jed Brown wrote: > >> I upgraded to clang-3.2 and the new type annotations are great. I'd like to have a identified for enum types, e.g. >> >> typedef enum {SOME, VALUES} MyEnum; >> #define MY_MPI_ENUM MPI_INT // or define some other way >> static const MPI_Datatype my_mpi_enum MPICH_ATTR_TYPE_TAG(MyEnum) = MY_MPI_ENUM; >> >> >> Of course the above does not work because the new definition masks the old one, so all the normal 'int' uses are hidden. Unfortunately, the annotation syntax requires compile-time constants, which are not accessible to me. >> >> Is there any hope for something like this, or do I have to choose between (a) ugly cast at all MPI call points and (b) make MY_MPI_ENUM a variable so that the compiler can't check it statically? Hi everyone, I don't think we can make it work with the way MPICH currently defines type tags (MPI_INT etc). The issue is that these are preprocessor macros, so in the AST we can only see the values they expand to. So MPI_INT and MY_MPI_ENUM become indistinguishable. But, if you create a different datatype (with MPI_Type_contigous, probably), it should be possible. enum MyEnum { Foo }; MPI_Datatype MyEnumDatatype MPICH_ATTR_TYPE_TAG(MyEnum); int main() { ... MPI_Type_contiguous(1, MPI_INT, MyEnumDatatype); } I haven't tried it. Please tell me if it does not work. Dmitri -- main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if (j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr(a)gmail.com>*/

1 0

Re: [mpich-discuss] Extensible Clang type annotation
by Jed Brown 02 Jan '13

02 Jan '13

Also, could there be a way (maybe in a future clang release) for me to get static type checking for my own types, such as __float128 or pair types like {long,long} (which are not in the MPI standard)? It can't truly be a static value because I have to create the type at run-time, but perhaps there could be a way to tell clang "I promise to set this type up like so." On Wed, Jan 2, 2013 at 11:31 AM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: > [adding Hal Finkel and Dmitri Gribenko, who did most of the annotations > work] > > Hal/Dmitri, > > Do you have any ideas about how to handle Jed's case? Will the > MPICH_ATTR_TYPE_TAG_LAYOUT_COMPATIBLE macro (reproduced below) help in this > case? > > ----8<---- > # define MPICH_ATTR_TYPE_TAG_LAYOUT_COMPATIBLE(type) > __attribute__((type_tag_for_datatype(MPI,type,layout_compatible))) > ----8<---- > > -Dave > > On Dec 30, 2012, at 11:13 PM CST, Jed Brown wrote: > > > I upgraded to clang-3.2 and the new type annotations are great. I'd like > to have a identified for enum types, e.g. > > > > typedef enum {SOME, VALUES} MyEnum; > > #define MY_MPI_ENUM MPI_INT // or define some other way > > static const MPI_Datatype my_mpi_enum MPICH_ATTR_TYPE_TAG(MyEnum) = > MY_MPI_ENUM; > > > > > > Of course the above does not work because the new definition masks the > old one, so all the normal 'int' uses are hidden. Unfortunately, the > annotation syntax requires compile-time constants, which are not accessible > to me. > > > > Is there any hope for something like this, or do I have to choose > between (a) ugly cast at all MPI call points and (b) make MY_MPI_ENUM a > variable so that the compiler can't check it statically? > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Indexed MPI_Reduce_local
by Dave Goodell 02 Jan '13

02 Jan '13

On Dec 28, 2012, at 7:33 PM CST, Jed Brown wrote: > The context is a comm library on top of MPI. I provide sparse communication with respect to a user-provided "unit" datatype (usually basic types or small contiguous structs). […] > I could implement this operation in-place using MPI_Accumulate on COMM_SELF, but I'd rather avoid MPI_Accumulate because it doesn't work with quad-precision or with MINLOC on {long; long}. Related tickets include: > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/338 We definitely want to support this case (#338), but for a number of reasons haven't gotten around to implementing it yet. > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/318 I actually did a bunch of work in a private branch to support these types, but stopped short of finishing it when the overall Forum opinion on them turned rather sour. Perhaps we should restart the effort on this (both standardization and implementation) now that MPI-3.0 is out. > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/319 I don't have any objection to these types, they just need to be implemented. We can put them in as MPIX_ types for now. > Any other suggestions? > > In lieu of a more elegant solution, I'm going to match on the types and ops that I need to support and write the sparse reduction by hand. > > In the longer term, I'm curious whether the MPICH developers, and eventually the Forum, may be interested in providing better support for higher level comm libraries like this. Like motherhood and apple pie, of course the answer is yes :) The devil is always in "to what extent". -Dave

1 0

Re: [mpich-discuss] Extensible Clang type annotation
by Dave Goodell 02 Jan '13

02 Jan '13

[adding Hal Finkel and Dmitri Gribenko, who did most of the annotations work] Hal/Dmitri, Do you have any ideas about how to handle Jed's case? Will the MPICH_ATTR_TYPE_TAG_LAYOUT_COMPATIBLE macro (reproduced below) help in this case? ----8<---- # define MPICH_ATTR_TYPE_TAG_LAYOUT_COMPATIBLE(type) __attribute__((type_tag_for_datatype(MPI,type,layout_compatible))) ----8<---- -Dave On Dec 30, 2012, at 11:13 PM CST, Jed Brown wrote: > I upgraded to clang-3.2 and the new type annotations are great. I'd like to have a identified for enum types, e.g. > > typedef enum {SOME, VALUES} MyEnum; > #define MY_MPI_ENUM MPI_INT // or define some other way > static const MPI_Datatype my_mpi_enum MPICH_ATTR_TYPE_TAG(MyEnum) = MY_MPI_ENUM; > > > Of course the above does not work because the new definition masks the old one, so all the normal 'int' uses are hidden. Unfortunately, the annotation syntax requires compile-time constants, which are not accessible to me. > > Is there any hope for something like this, or do I have to choose between (a) ugly cast at all MPI call points and (b) make MY_MPI_ENUM a variable so that the compiler can't check it statically? > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] can't MPI_Abort a program due to deadlock at pthread_mutex_lock
by Jeff Hammond 02 Jan '13

02 Jan '13

> If you really need this to work, you can also disable pthread cancellation > when the thread is in MPI calls: > > pthread_setcanceltype(PTHREAD_CANCEL_DISABLE, &cancel_val); > MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, > MPI_STATUS_IGNORE); > pthread_setcanceltype(cancel_val, &junk); > > With that change, the program you sent works correctly. Thanks. This is useful info but I have a better way to work around this issue in the code that I actually care about. I'm writing active messages over p2p and it's trivial to tell the polling thread to pthread_exit via an active-message. > It think that adding setcanceltype() to the MPICH thread safety macros might > be all that we have to do to make the library cancel-safe, but I'm not a > pthreads expert. It could have a nonzero performance cost, so it would > likely need to be a compile-time option. This might be useful as a debugging option but I wouldn't prioritize it. >> Pthread_cancel is unsafe in any context where the cancelled thread can >> hold a mutex needed by other threads. This looks to me like a bad >> programming practice (e.g. you will also likely leak memory, since you >> can't free MPI objects allocated by the thread from inside of a >> cancellation handler), not an MPICH-specific problem; there are safer >> ways to cancel/cleanup threads. I don't care about memory leaks since the use case was merely to exit the program. >> Do you have a good case for why such functionality is needed? The use of pthread_cancel is entirely optional. I knew there were many ways to work around this and I've already implemented the one that makes the most sense to me. Thanks for all your comments. Best, Jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-discuss] unsubscribe
by Dave Goodell 02 Jan '13

02 Jan '13

On Dec 24, 2012, at 6:28 AM CST, Nachiket Gokhale wrote: > Unsubscribe Please use the link at the bottom of every mail posted to this list in order to unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss -Dave

1 0

Re: [mpich-discuss] can't MPI_Abort a program due to deadlock at pthread_mutex_lock
by Jim Dinan 02 Jan '13

02 Jan '13

Jeff, If you really need this to work, you can also disable pthread cancellation when the thread is in MPI calls: pthread_setcanceltype(PTHREAD_CANCEL_DISABLE, &cancel_val); MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE); pthread_setcanceltype(cancel_val, &junk); With that change, the program you sent works correctly. It think that adding setcanceltype() to the MPICH thread safety macros might be all that we have to do to make the library cancel-safe, but I'm not a pthreads expert. It could have a nonzero performance cost, so it would likely need to be a compile-time option. ~Jim. On 1/2/13 10:25 AM, Jim Dinan wrote: > Jeff, > > Pthread_cancel is unsafe in any context where the cancelled thread can > hold a mutex needed by other threads. This looks to me like a bad > programming practice (e.g. you will also likely leak memory, since you > can't free MPI objects allocated by the thread from inside of a > cancellation handler), not an MPICH-specific problem; there are safer > ways to cancel/cleanup threads. > > Do you have a good case for why such functionality is needed? > > ~Jim. > > On 1/2/13 9:49 AM, Jeff Hammond wrote: >> I wrote a program that calls MPI from a pthread. MPI_THREAD_MULTIPLE >> is used. I cancel and then join the pthread back into the main >> program. After I do this, any calls to MPI, including MPI_Abort (see >> first gdb+source below) and MPI_Barrier (see second gdb+source below) >> deadlock on pthread_mutex_lock. >> >> I can understand that canceling the thread while it is holding the MPI >> pthread mutex is a bad idea, but I'm not sure this is an incorrect >> program. Is pthread_cancel using signals? I recall that MPICH >> functions are not guaranteed to be signal-safe. >> >> Is it possible to implement MPI_Abort such that acquisition of the MPI >> mutex is not required in order to allow deadlocked programs to >> terminate? It seems rather unfortunate that MPI_Abort does not >> perform the desired action here. Obviously, I can use abort(), but on >> some implementations, this will leave the system resources in a dirty >> state. >> >> I know a bunch of ways to work around this, but it would be helpful to >> understand exactly what the problem is. I wonder if there is a way to >> modify MPICH such that the mutex is released when a thread holding it >> is cancelled. >> >> Is it possible for MPICH to provide an MPIX call or otherwise document >> an internal, MPICH-specific function to clear the MPI mutex? I could >> resolve this problem if I could merely call MPIX_Mutex_force_unlock() >> and let the program terminate as desired. >> >> Thanks, >> >> Jeff >> >> $ make >> mpicc -g -O0 -Wall -std=gnu99 -c bug.c -o bug.o >> mpicc -g -O0 -Wall -std=gnu99 safemalloc.o bug.o -lm -o bug.x >> rm bug.o >> >> ======== stuck in abort/finalize ======== >> >> $ gdb ./bug.x >> GNU gdb 6.3.50-20050815 (Apple version gdb-1752) (Sat Jan 28 03:02:46 >> UTC 2012) >> Copyright 2004 Free Software Foundation, Inc. >> GDB is free software, covered by the GNU General Public License, and >> you are >> welcome to change it and/or distribute copies of it under certain >> conditions. >> Type "show copying" to see the conditions. >> There is absolutely no warranty for GDB. Type "show warranty" for >> details. >> This GDB was configured as "x86_64-apple-darwin"...Reading symbols for >> shared libraries ...... done >> >> warning: Could not find object file >> "/Users/jhammond/eclipse/OSPRI/trunk/tests/devices/mpi-pt/bug.o" - no >> debug information available for "bug.c". >> >> >> (gdb) run >> Starting program: >> /Users/jhammond/eclipse/OSPRI/trunk/tests/devices/mpi-pt/bug.x >> Reading symbols for shared libraries +++++........................ done >> test done except for pthread shutdown >> 0: Progress >> all done >> >> ^C >> Program received signal SIGINT, Interrupt. >> 0x00007fff912fdbf2 in __psynch_mutexwait () >> (gdb) bt >> #0 0x00007fff912fdbf2 in __psynch_mutexwait () >> #1 0x00007fff8c80a1a1 in pthread_mutex_lock () >> #2 0x000000010004912b in >> MPIU_Thread_CS_enter_lockname_recursive_impl_ () >> #3 0x0000000100048d64 in MPI_Abort () >> #4 0x0000000100000cc0 in main () >> (gdb) quit >> The program is running. Exit anyway? (y or n) y >> >> $ cat bug.c >> #include <stdio.h> >> #include <stdlib.h> >> #include <unistd.h> >> #include <string.h> >> #include <assert.h> >> #include <limits.h> >> #include <pthread.h> >> >> #include <mpi.h> >> >> #define DEBUG >> >> pthread_t Progress_thread; >> >> static void * Progress_function(void * dummy) >> { >> int rank; >> MPI_Comm_rank(MPI_COMM_WORLD, &rank); >> >> while (1) >> { >> #ifdef DEBUG >> printf("%d: Progress \n", rank); >> #endif >> MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, >> MPI_STATUS_IGNORE); >> //usleep(500); >> } >> >> return NULL; >> } >> >> int main(int argc, char * argv[]) >> { >> int rc; >> int rank, size; >> int provided; >> >> MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided); >> if (provided!=MPI_THREAD_MULTIPLE) MPI_Abort(MPI_COMM_WORLD, 1); >> >> MPI_Comm_rank(MPI_COMM_WORLD, &rank); >> MPI_Comm_size(MPI_COMM_WORLD, &size); >> >> rc = pthread_create(&Progress_thread, NULL, &Progress_function, >> NULL); >> if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); >> >> MPI_Barrier(MPI_COMM_WORLD); >> >> printf("test done except for pthread shutdown \n"); >> fflush(stdout); >> >> MPI_Barrier(MPI_COMM_WORLD); >> >> rc = pthread_cancel(Progress_thread); >> if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); >> >> void * rv; >> rc = pthread_join(Progress_thread, &rv); >> if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); >> >> printf("all done \n"); >> fflush(stdout); >> >> MPI_Abort(MPI_COMM_SELF, 2); >> MPI_Finalize(); >> >> return 0; >> } >> >> ======== stuck in barrier ======== >> >> $ gdb ./bug.x >> GNU gdb 6.3.50-20050815 (Apple version gdb-1752) (Sat Jan 28 03:02:46 >> UTC 2012) >> Copyright 2004 Free Software Foundation, Inc. >> GDB is free software, covered by the GNU General Public License, and >> you are >> welcome to change it and/or distribute copies of it under certain >> conditions. >> Type "show copying" to see the conditions. >> There is absolutely no warranty for GDB. Type "show warranty" for >> details. >> This GDB was configured as "x86_64-apple-darwin"...Reading symbols for >> shared libraries ...... done >> >> warning: Could not find object file >> "/Users/jhammond/eclipse/OSPRI/trunk/tests/devices/mpi-pt/bug.o" - no >> debug information available for "bug.c". >> >> >> (gdb) run >> Starting program: >> /Users/jhammond/eclipse/OSPRI/trunk/tests/devices/mpi-pt/bug.x >> Reading symbols for shared libraries +++++........................ done >> before pthread shutdown >> 0: Progress >> after pthread shutdown >> ^C >> Program received signal SIGINT, Interrupt. >> 0x00007fff912fdbf2 in __psynch_mutexwait () >> (gdb) bt >> #0 0x00007fff912fdbf2 in __psynch_mutexwait () >> #1 0x00007fff8c80a1a1 in pthread_mutex_lock () >> #2 0x000000010000f7eb in >> MPIU_Thread_CS_enter_lockname_recursive_impl_ () >> #3 0x000000010000f4f7 in MPI_Barrier () >> #4 0x0000000100000c4d in main () >> (gdb) quit >> The program is running. Exit anyway? (y or n) y >> Jeffs-MacBook-Pro:mpi-pt jhammond$ cat bug.c >> #include <stdio.h> >> #include <stdlib.h> >> #include <unistd.h> >> #include <string.h> >> #include <assert.h> >> #include <limits.h> >> #include <pthread.h> >> >> #include <mpi.h> >> >> #define DEBUG >> >> pthread_t Progress_thread; >> >> static void * Progress_function(void * dummy) >> { >> int rank; >> MPI_Comm_rank(MPI_COMM_WORLD, &rank); >> >> while (1) >> { >> #ifdef DEBUG >> printf("%d: Progress \n", rank); >> #endif >> MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, >> MPI_STATUS_IGNORE); >> //usleep(500); >> } >> >> return NULL; >> } >> >> int main(int argc, char * argv[]) >> { >> int rc; >> int rank, size; >> int provided; >> >> MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided); >> if (provided!=MPI_THREAD_MULTIPLE) MPI_Abort(MPI_COMM_WORLD, 1); >> >> MPI_Comm_rank(MPI_COMM_WORLD, &rank); >> MPI_Comm_size(MPI_COMM_WORLD, &size); >> >> rc = pthread_create(&Progress_thread, NULL, &Progress_function, >> NULL); >> if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); >> >> MPI_Barrier(MPI_COMM_WORLD); >> >> printf("before pthread shutdown \n"); >> fflush(stdout); >> >> MPI_Barrier(MPI_COMM_WORLD); >> >> rc = pthread_cancel(Progress_thread); >> if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); >> >> void * rv; >> rc = pthread_join(Progress_thread, &rv); >> if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); >> >> printf("after pthread shutdown \n"); >> fflush(stdout); >> >> MPI_Barrier(MPI_COMM_WORLD); >> >> printf("before MPI Abort/Finalize \n"); >> fflush(stdout); >> >> MPI_Abort(MPI_COMM_SELF, 2); >> MPI_Finalize(); >> >> printf("after MPI Abort/Finalize \n"); >> fflush(stdout); >> >> return 0; >> } >> > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] can't MPI_Abort a program due to deadlock at pthread_mutex_lock
by Jim Dinan 02 Jan '13

02 Jan '13

Jeff, Pthread_cancel is unsafe in any context where the cancelled thread can hold a mutex needed by other threads. This looks to me like a bad programming practice (e.g. you will also likely leak memory, since you can't free MPI objects allocated by the thread from inside of a cancellation handler), not an MPICH-specific problem; there are safer ways to cancel/cleanup threads. Do you have a good case for why such functionality is needed? ~Jim. On 1/2/13 9:49 AM, Jeff Hammond wrote: > I wrote a program that calls MPI from a pthread. MPI_THREAD_MULTIPLE > is used. I cancel and then join the pthread back into the main > program. After I do this, any calls to MPI, including MPI_Abort (see > first gdb+source below) and MPI_Barrier (see second gdb+source below) > deadlock on pthread_mutex_lock. > > I can understand that canceling the thread while it is holding the MPI > pthread mutex is a bad idea, but I'm not sure this is an incorrect > program. Is pthread_cancel using signals? I recall that MPICH > functions are not guaranteed to be signal-safe. > > Is it possible to implement MPI_Abort such that acquisition of the MPI > mutex is not required in order to allow deadlocked programs to > terminate? It seems rather unfortunate that MPI_Abort does not > perform the desired action here. Obviously, I can use abort(), but on > some implementations, this will leave the system resources in a dirty > state. > > I know a bunch of ways to work around this, but it would be helpful to > understand exactly what the problem is. I wonder if there is a way to > modify MPICH such that the mutex is released when a thread holding it > is cancelled. > > Is it possible for MPICH to provide an MPIX call or otherwise document > an internal, MPICH-specific function to clear the MPI mutex? I could > resolve this problem if I could merely call MPIX_Mutex_force_unlock() > and let the program terminate as desired. > > Thanks, > > Jeff > > $ make > mpicc -g -O0 -Wall -std=gnu99 -c bug.c -o bug.o > mpicc -g -O0 -Wall -std=gnu99 safemalloc.o bug.o -lm -o bug.x > rm bug.o > > ======== stuck in abort/finalize ======== > > $ gdb ./bug.x > GNU gdb 6.3.50-20050815 (Apple version gdb-1752) (Sat Jan 28 03:02:46 UTC 2012) > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "x86_64-apple-darwin"...Reading symbols for > shared libraries ...... done > > warning: Could not find object file > "/Users/jhammond/eclipse/OSPRI/trunk/tests/devices/mpi-pt/bug.o" - no > debug information available for "bug.c". > > > (gdb) run > Starting program: > /Users/jhammond/eclipse/OSPRI/trunk/tests/devices/mpi-pt/bug.x > Reading symbols for shared libraries +++++........................ done > test done except for pthread shutdown > 0: Progress > all done > > ^C > Program received signal SIGINT, Interrupt. > 0x00007fff912fdbf2 in __psynch_mutexwait () > (gdb) bt > #0 0x00007fff912fdbf2 in __psynch_mutexwait () > #1 0x00007fff8c80a1a1 in pthread_mutex_lock () > #2 0x000000010004912b in MPIU_Thread_CS_enter_lockname_recursive_impl_ () > #3 0x0000000100048d64 in MPI_Abort () > #4 0x0000000100000cc0 in main () > (gdb) quit > The program is running. Exit anyway? (y or n) y > > $ cat bug.c > #include <stdio.h> > #include <stdlib.h> > #include <unistd.h> > #include <string.h> > #include <assert.h> > #include <limits.h> > #include <pthread.h> > > #include <mpi.h> > > #define DEBUG > > pthread_t Progress_thread; > > static void * Progress_function(void * dummy) > { > int rank; > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > > while (1) > { > #ifdef DEBUG > printf("%d: Progress \n", rank); > #endif > MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, > MPI_STATUS_IGNORE); > //usleep(500); > } > > return NULL; > } > > int main(int argc, char * argv[]) > { > int rc; > int rank, size; > int provided; > > MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided); > if (provided!=MPI_THREAD_MULTIPLE) MPI_Abort(MPI_COMM_WORLD, 1); > > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > MPI_Comm_size(MPI_COMM_WORLD, &size); > > rc = pthread_create(&Progress_thread, NULL, &Progress_function, NULL); > if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); > > MPI_Barrier(MPI_COMM_WORLD); > > printf("test done except for pthread shutdown \n"); > fflush(stdout); > > MPI_Barrier(MPI_COMM_WORLD); > > rc = pthread_cancel(Progress_thread); > if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); > > void * rv; > rc = pthread_join(Progress_thread, &rv); > if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); > > printf("all done \n"); > fflush(stdout); > > MPI_Abort(MPI_COMM_SELF, 2); > MPI_Finalize(); > > return 0; > } > > ======== stuck in barrier ======== > > $ gdb ./bug.x > GNU gdb 6.3.50-20050815 (Apple version gdb-1752) (Sat Jan 28 03:02:46 UTC 2012) > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "x86_64-apple-darwin"...Reading symbols for > shared libraries ...... done > > warning: Could not find object file > "/Users/jhammond/eclipse/OSPRI/trunk/tests/devices/mpi-pt/bug.o" - no > debug information available for "bug.c". > > > (gdb) run > Starting program: > /Users/jhammond/eclipse/OSPRI/trunk/tests/devices/mpi-pt/bug.x > Reading symbols for shared libraries +++++........................ done > before pthread shutdown > 0: Progress > after pthread shutdown > ^C > Program received signal SIGINT, Interrupt. > 0x00007fff912fdbf2 in __psynch_mutexwait () > (gdb) bt > #0 0x00007fff912fdbf2 in __psynch_mutexwait () > #1 0x00007fff8c80a1a1 in pthread_mutex_lock () > #2 0x000000010000f7eb in MPIU_Thread_CS_enter_lockname_recursive_impl_ () > #3 0x000000010000f4f7 in MPI_Barrier () > #4 0x0000000100000c4d in main () > (gdb) quit > The program is running. Exit anyway? (y or n) y > Jeffs-MacBook-Pro:mpi-pt jhammond$ cat bug.c > #include <stdio.h> > #include <stdlib.h> > #include <unistd.h> > #include <string.h> > #include <assert.h> > #include <limits.h> > #include <pthread.h> > > #include <mpi.h> > > #define DEBUG > > pthread_t Progress_thread; > > static void * Progress_function(void * dummy) > { > int rank; > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > > while (1) > { > #ifdef DEBUG > printf("%d: Progress \n", rank); > #endif > MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, > MPI_STATUS_IGNORE); > //usleep(500); > } > > return NULL; > } > > int main(int argc, char * argv[]) > { > int rc; > int rank, size; > int provided; > > MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided); > if (provided!=MPI_THREAD_MULTIPLE) MPI_Abort(MPI_COMM_WORLD, 1); > > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > MPI_Comm_size(MPI_COMM_WORLD, &size); > > rc = pthread_create(&Progress_thread, NULL, &Progress_function, NULL); > if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); > > MPI_Barrier(MPI_COMM_WORLD); > > printf("before pthread shutdown \n"); > fflush(stdout); > > MPI_Barrier(MPI_COMM_WORLD); > > rc = pthread_cancel(Progress_thread); > if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); > > void * rv; > rc = pthread_join(Progress_thread, &rv); > if (rc!=0) MPI_Abort(MPI_COMM_WORLD, rc); > > printf("after pthread shutdown \n"); > fflush(stdout); > > MPI_Barrier(MPI_COMM_WORLD); > > printf("before MPI Abort/Finalize \n"); > fflush(stdout); > > MPI_Abort(MPI_COMM_SELF, 2); > MPI_Finalize(); > > printf("after MPI Abort/Finalize \n"); > fflush(stdout); > > return 0; > } >

1 0