- devel - mailman3.cels.anl.gov

Re: [mpich-devel] Two issues with MPI_File error handling
by Lisandro Dalcin 08 May '15

08 May '15

On 7 May 2015 at 22:00, Rob Latham <robl(a)mcs.anl.gov> wrote: > > > Thanks for reporting these two issues. These two patches (attached) address > your test case, but it's a part of the code I don't play in often, so I'm > going to need a review on them before committing: > Rob, I think you patch 2 is not right. It will endup calling the error handler set in COMM_WORLD. So if you set the FATAL errhandler in a file, but the RETURNS in comm world, then things will not work as expected. I think a better approach would be to somehow call handleFatalError(NULL, "MPI_File_call_errhandler", errorcode); in errutil.c, but that would require some code refactoring to access that function. -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459

1 0

Re: [mpich-devel] Two issues with MPI_File error handling
by Rob Latham 07 May '15

07 May '15

On 05/07/2015 06:58 AM, Lisandro Dalcin wrote: > This is using the preview release 3.2b2, I discovered this issue > months ago and it seems I forgot to report it. Thanks for reporting these two issues. These two patches (attached) address your test case, but it's a part of the code I don't play in often, so I'm going to need a review on them before committing: ==rob > > 1) MPI_File_call_errhandler(file, value) should return MPI_SUCCESS > instead of value under normal operation (as the MPI spec says), and > particularly if the error handler is set to MPI_ERRORS_RETURN. This > bug is trivial to fix. > > 2) Setting the predefined error handler MPI_ERRORS_ARE_FATAL and > calling MPI_File_call_errhandler(file, errcode) should "gracefully" > abort the processes the MPI way, instead of segfaulting. The problem > here is that ROMIO has to somehow handle MPI_ERRORS_ARE_FATAL in some > special way I'm not sure how to implement, right now it is trying to > invoke a callback that is set to NULL. > > I'm attaching a test case showcasing the two issues, and pasting output below: > > $ cat file_errhdl.c > #include <mpi.h> > #include <stdio.h> > > int main(int argc, char *argv[]) > { > int ierr; > MPI_File fh; > MPI_Init(&argc, &argv); > > MPI_File_open(MPI_COMM_WORLD,"/tmp/datafile", > MPI_MODE_CREATE|MPI_MODE_RDWR|MPI_MODE_DELETE_ON_CLOSE, > MPI_INFO_NULL,&fh); > > MPI_File_set_errhandler(fh,MPI_ERRORS_RETURN); > ierr = MPI_File_call_errhandler(fh,MPI_ERR_FILE); > printf("ierr: %d, expected: %d\n", ierr, (int)MPI_SUCCESS); > > MPI_File_set_errhandler(fh,MPI_ERRORS_ARE_FATAL); > MPI_File_call_errhandler(fh,MPI_ERR_FILE); /* should abort */ > MPI_File_close(&fh); > > MPI_Finalize(); > return 0; > } > > $ mpicc file_errhdl.c > > $ ./a.out > ierr: 27, expected: 0 > Segmentation fault (core dumped) > > $ valgrind -q ./a.out > ierr: 27, expected: 0 > ==30036== Jump to the invalid address stated on the next line > ==30036== at 0x0: ??? > ==30036== by 0x4CE8EDC: PMPI_File_call_errhandler > (file_call_errhandler.c:105) > ==30036== by 0x400951: main (in /home/dalcinl/Devel/BUGS-MPI/mpich/a.out) > ==30036== Address 0x0 is not stack'd, malloc'd or (recently) free'd > ==30036== > ==30036== > ==30036== Process terminating with default action of signal 11 (SIGSEGV) > ==30036== Bad permissions for mapped region at address 0x0 > ==30036== at 0x0: ??? > ==30036== by 0x4CE8EDC: PMPI_File_call_errhandler > (file_call_errhandler.c:105) > ==30036== by 0x400951: main (in /home/dalcinl/Devel/BUGS-MPI/mpich/a.out) > Segmentation fault (core dumped) > > > > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-devel] Two issues with MPI_File error handling
by Rob Latham 07 May '15

07 May '15

On 05/07/2015 11:34 AM, Lisandro Dalcin wrote: > On 7 May 2015 at 18:44, Rob Latham <robl(a)mcs.anl.gov> wrote: >> >>> 1) MPI_File_call_errhandler(file, value) should return MPI_SUCCESS >>> instead of value under normal operation (as the MPI spec says), and >>> particularly if the error handler is set to MPI_ERRORS_RETURN. This >>> bug is trivial to fix. >> >> >> Trivial to fix...if we only had to worry about C and Fortran! > > I'm probably missing something here, so I have to ask. > > I said this was "trivial to fix" for the particular case of > MPI_ERRORS_RETURN. How should C++ bindings behave in case > MPI_ERRORS_RETURN is set? In other cases, I agree with you, fixing > this beast is not easy. However, you have a point about C++ bindings > being deprecated. It is I who was missing something. I read the code too fast, saw the special C++ case, and thought it mattered for MPI_ERRORS_RETURN. This one is easy. The core dumping one is a little more work but I see an angle of attack now. ==rob > > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-devel] Two issues with MPI_File error handling
by Lisandro Dalcin 07 May '15

07 May '15

On 7 May 2015 at 18:44, Rob Latham <robl(a)mcs.anl.gov> wrote: > >> 1) MPI_File_call_errhandler(file, value) should return MPI_SUCCESS >> instead of value under normal operation (as the MPI spec says), and >> particularly if the error handler is set to MPI_ERRORS_RETURN. This >> bug is trivial to fix. > > > Trivial to fix...if we only had to worry about C and Fortran! I'm probably missing something here, so I have to ask. I said this was "trivial to fix" for the particular case of MPI_ERRORS_RETURN. How should C++ bindings behave in case MPI_ERRORS_RETURN is set? In other cases, I agree with you, fixing this beast is not easy. However, you have a point about C++ bindings being deprecated. -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459

1 0

Re: [mpich-devel] Two issues with MPI_File error handling
by Rob Latham 07 May '15

07 May '15

On 05/07/2015 06:58 AM, Lisandro Dalcin wrote: > This is using the preview release 3.2b2, I discovered this issue > months ago and it seems I forgot to report it. We really should get mpi4py into the nightly tests. You seem to find a lot of interesting cases. > 1) MPI_File_call_errhandler(file, value) should return MPI_SUCCESS > instead of value under normal operation (as the MPI spec says), and > particularly if the error handler is set to MPI_ERRORS_RETURN. This > bug is trivial to fix. Trivial to fix...if we only had to worry about C and Fortran! (and maybe we can ignore C++ regressions now that the bindings are deprecated? ) It looks like we have hijacked the error code (years and years ago, in to send information to the C++ binidngs: http://git.mpich.org/mpich.git/blob/HEAD:/src/mpi/errhan/file_call_errhandl… The change dates back to 2008: http://git.mpich.org/mpich.git/commit/abbe596a and it looks like the Comm and Window error handlers stomp on the error code, too. This is ticket #2259 if you want to follow along https://trac.mpich.org/projects/mpich/ticket/2259 I'll respond to #2 in another email ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Issue with too many calls to MPIR_Add_finalize
by Lisandro Dalcin 07 May '15

07 May '15

I'm running the testsuite of my Python wrappers with MPICH 3.2b2. The runs are aborting this way: $ python test/runtests.py -q overflow in finalize stack! internal ABORT - process 0 [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=13 : system msg for write_line failure : Bad file descriptor It seems that the routine MPIR_Add_finalize() is being called more than 32 times, exhausting the static allocation space of 32 entries. I'm attaching a text file with a gdb sessions showing all the calls to this routine. -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459

1 0

Two issues with MPI_File error handling
by Lisandro Dalcin 07 May '15

07 May '15

This is using the preview release 3.2b2, I discovered this issue months ago and it seems I forgot to report it. 1) MPI_File_call_errhandler(file, value) should return MPI_SUCCESS instead of value under normal operation (as the MPI spec says), and particularly if the error handler is set to MPI_ERRORS_RETURN. This bug is trivial to fix. 2) Setting the predefined error handler MPI_ERRORS_ARE_FATAL and calling MPI_File_call_errhandler(file, errcode) should "gracefully" abort the processes the MPI way, instead of segfaulting. The problem here is that ROMIO has to somehow handle MPI_ERRORS_ARE_FATAL in some special way I'm not sure how to implement, right now it is trying to invoke a callback that is set to NULL. I'm attaching a test case showcasing the two issues, and pasting output below: $ cat file_errhdl.c #include <mpi.h> #include <stdio.h> int main(int argc, char *argv[]) { int ierr; MPI_File fh; MPI_Init(&argc, &argv); MPI_File_open(MPI_COMM_WORLD,"/tmp/datafile", MPI_MODE_CREATE|MPI_MODE_RDWR|MPI_MODE_DELETE_ON_CLOSE, MPI_INFO_NULL,&fh); MPI_File_set_errhandler(fh,MPI_ERRORS_RETURN); ierr = MPI_File_call_errhandler(fh,MPI_ERR_FILE); printf("ierr: %d, expected: %d\n", ierr, (int)MPI_SUCCESS); MPI_File_set_errhandler(fh,MPI_ERRORS_ARE_FATAL); MPI_File_call_errhandler(fh,MPI_ERR_FILE); /* should abort */ MPI_File_close(&fh); MPI_Finalize(); return 0; } $ mpicc file_errhdl.c $ ./a.out ierr: 27, expected: 0 Segmentation fault (core dumped) $ valgrind -q ./a.out ierr: 27, expected: 0 ==30036== Jump to the invalid address stated on the next line ==30036== at 0x0: ??? ==30036== by 0x4CE8EDC: PMPI_File_call_errhandler (file_call_errhandler.c:105) ==30036== by 0x400951: main (in /home/dalcinl/Devel/BUGS-MPI/mpich/a.out) ==30036== Address 0x0 is not stack'd, malloc'd or (recently) free'd ==30036== ==30036== ==30036== Process terminating with default action of signal 11 (SIGSEGV) ==30036== Bad permissions for mapped region at address 0x0 ==30036== at 0x0: ??? ==30036== by 0x4CE8EDC: PMPI_File_call_errhandler (file_call_errhandler.c:105) ==30036== by 0x400951: main (in /home/dalcinl/Devel/BUGS-MPI/mpich/a.out) Segmentation fault (core dumped) -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459

1 0

Re: [mpich-devel] patches for createf90types.c to test DP
by Junchao Zhang 30 Apr '15

30 Apr '15

On Thu, Apr 23, 2015 at 2:39 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > Thanks. Please also apply the attached patch (0004) to fix a trivial bug. > > The other patch (0003) will make this test XFAIL. You should not > apply it yet, but that's where I'm going... > Jeff, I applied your 0001, 0002, 0004 patches and pushed them to mpich/master. > > Jeff > > > On Thu, Apr 23, 2015 at 10:44 AM, Junchao Zhang <jczhang(a)mcs.anl.gov> > wrote: > > I will add your patch. Thanks. > > > > --Junchao Zhang > > > > On Thu, Apr 23, 2015 at 12:31 PM, Jeff Hammond <jeff.science(a)gmail.com> > > wrote: > >> > >> Patches for test/mpi/f90/f90types/createf90types.c that test > >> MPI_Type_create_f90_complex for 64b floats are attached. > >> > >> If these patches are accepted, I want to extend them to support the > >> case of REAL128 (http://fortranwiki.org/fortran/show/Real+precision) > >> which is part of Fortran 2008 and therefore should be supported by > >> MPICH. Tests for REAL128 will fail since MPICH does not support it > >> yet, but I'm hoping to inspire this to change. And I might end up > >> doing it myself if I have to. > >> > >> Thanks, > >> > >> Jeff > >> > >> -- > >> Jeff Hammond > >> jeff.science(a)gmail.com > >> http://jeffhammond.github.io/ > >> > >> _______________________________________________ > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/devel > > > > > > > > _______________________________________________ > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/devel > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel >

1 0

Re: [mpich-devel] call PMI before MPI_Init_thread?
by Jeff Hammond 25 Apr '15

25 Apr '15

Sorry, I meant "change the PMI spec" instead of "implement", but I guess it doesn't matter if it works everywhere. Jeff On Sat, Apr 25, 2015 at 4:23 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > It works in the implementation, so I’m not sure what you mean. I just mean it’s not allowed per the “spec” (or whatever sparse documentation we have that others use for implementing PMI). > > — Pavan > >> On Apr 24, 2015, at 7:22 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> >> Why not implement the equivalent of the ref-counting init proposal or >> at least the simple things that allows multiple inits but only one >> finalize? Is it that you don't see the effort being worthwhile or >> because there is a principled reason (PMI semantics should match MPI)? >> >> Given that e.g. PGAS runtimes may want to use PMI for bootstrapping (I >> do this with DMAPP code on Cray, for example), it seems to benefit >> user experience to support this behavior. >> >> Thanks, >> >> Jeff >> >> On Fri, Apr 24, 2015 at 1:12 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: >>> >>> Multiple PMI Inits are not allowed. It might work in the current >>> implementation within MPICH, but is not safe. >>> >>> If you are looking for MPICH specific behavior, you can look for the PMI_ID >>> and PMI_RANK environment variables instead to get your rank. >>> >>> — Pavan >>> >>> On Apr 21, 2015, at 11:17 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>> >>> So MPICH doesn't fail when I try this on my Mac, but it would be good >>> to know if I have a reasonable expectation of doing this in general. >>> >>> Thanks, >>> >>> Jeff >>> >>> jrhammon-mac01:ENDPOINTS jrhammon$ mpiexec -n 4 ./a.out >>> PMI says I am 2 of 4 >>> PMI says I am 0 of 4 >>> PMI says I am 1 of 4 >>> PMI says I am 3 of 4 >>> MPI says I am 0 of 4 (provided=1) >>> MPI says I am 2 of 4 (provided=1) >>> MPI says I am 3 of 4 (provided=3) >>> MPI says I am 1 of 4 (provided=3) >>> >>> jrhammon-mac01:ENDPOINTS jrhammon$ cat pmi.c >>> #include <stdio.h> >>> #include <stdlib.h> >>> >>> /*#include <pmi.h>*/ >>> int PMI_Init( int *spawned ); >>> int PMI_Finalize( void ); >>> int PMI_Get_size( int *size ); >>> int PMI_Get_rank( int *rank ); >>> >>> #include <mpi.h> >>> >>> int main(int argc, char * argv[]) >>> { >>> int has_parent; >>> PMI_Init(&has_parent); >>> >>> int rank, size; >>> PMI_Get_size(&size); >>> PMI_Get_rank(&rank); >>> >>> printf("PMI says I am %d of %d \n", rank, size); >>> fflush(stdout); >>> >>> int requested = (rank%2==0) ? MPI_THREAD_FUNNELED : MPI_THREAD_MULTIPLE; >>> int provided; >>> MPI_Init_thread(&argc, &argv, requested, &provided); >>> >>> int wsize, wrank; >>> MPI_Comm_size(MPI_COMM_WORLD, &wsize); >>> MPI_Comm_rank(MPI_COMM_WORLD, &wrank); >>> >>> printf("MPI says I am %d of %d (provided=%d)\n", wrank, wsize, provided); >>> fflush(stdout); >>> >>> MPI_Finalize(); >>> //PMI_Finalize(); /* required? */ >>> >>> return 0; >>> } >>> >>> On Tue, Apr 21, 2015 at 9:09 PM, Jeff Hammond <jeff.science(a)gmail.com> >>> wrote: >>> >>> Is it valid in _MPICH_ to make calls to PMI before MPI is initialized? >>> I am interested in PMI_Init, PMI_Get_size and PMI_Get_rank. >>> >>> The motivating use case is to initialize e.g. half my ranks with >>> MPI_THREAD_FUNNELED and the other half with MPI_THREAD_MULTIPLE but >>> not use MPMD launching. >>> >>> I have two reasons for wanting to do it this way. First, it is >>> otherwise unnecessary to use MPMD since I can branch at the top of my >>> code and meet the requirements of FUNNELED and MULTIPLE. Second, I >>> might be interested in running my code on a supercomputer with an >>> MPICH-based MPI implementation that does not support MPMD launching in >>> an unrestricted manner. >>> >>> If MPICH can handle this, then I will worry about whether or not the >>> aforementioned supercomputer can. It is my assumption that, if MPICH >>> cannot handle this, then the MPICH derivative in question cannot >>> either. >>> >>> Thanks, >>> >>> Jeff >>> >>> -- >>> Jeff Hammond >>> jeff.science(a)gmail.com >>> http://jeffhammond.github.io/ >>> >>> >>> >>> >>> -- >>> Jeff Hammond >>> jeff.science(a)gmail.com >>> http://jeffhammond.github.io/ >>> _______________________________________________ >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/devel >>> >>> >>> >>> _______________________________________________ >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/devel >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> http://jeffhammond.github.io/ >> _______________________________________________ >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/devel > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-devel] call PMI before MPI_Init_thread?
by Balaji, Pavan 25 Apr '15

25 Apr '15

It works in the implementation, so I’m not sure what you mean. I just mean it’s not allowed per the “spec” (or whatever sparse documentation we have that others use for implementing PMI). — Pavan > On Apr 24, 2015, at 7:22 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > Why not implement the equivalent of the ref-counting init proposal or > at least the simple things that allows multiple inits but only one > finalize? Is it that you don't see the effort being worthwhile or > because there is a principled reason (PMI semantics should match MPI)? > > Given that e.g. PGAS runtimes may want to use PMI for bootstrapping (I > do this with DMAPP code on Cray, for example), it seems to benefit > user experience to support this behavior. > > Thanks, > > Jeff > > On Fri, Apr 24, 2015 at 1:12 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: >> >> Multiple PMI Inits are not allowed. It might work in the current >> implementation within MPICH, but is not safe. >> >> If you are looking for MPICH specific behavior, you can look for the PMI_ID >> and PMI_RANK environment variables instead to get your rank. >> >> — Pavan >> >> On Apr 21, 2015, at 11:17 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> >> So MPICH doesn't fail when I try this on my Mac, but it would be good >> to know if I have a reasonable expectation of doing this in general. >> >> Thanks, >> >> Jeff >> >> jrhammon-mac01:ENDPOINTS jrhammon$ mpiexec -n 4 ./a.out >> PMI says I am 2 of 4 >> PMI says I am 0 of 4 >> PMI says I am 1 of 4 >> PMI says I am 3 of 4 >> MPI says I am 0 of 4 (provided=1) >> MPI says I am 2 of 4 (provided=1) >> MPI says I am 3 of 4 (provided=3) >> MPI says I am 1 of 4 (provided=3) >> >> jrhammon-mac01:ENDPOINTS jrhammon$ cat pmi.c >> #include <stdio.h> >> #include <stdlib.h> >> >> /*#include <pmi.h>*/ >> int PMI_Init( int *spawned ); >> int PMI_Finalize( void ); >> int PMI_Get_size( int *size ); >> int PMI_Get_rank( int *rank ); >> >> #include <mpi.h> >> >> int main(int argc, char * argv[]) >> { >> int has_parent; >> PMI_Init(&has_parent); >> >> int rank, size; >> PMI_Get_size(&size); >> PMI_Get_rank(&rank); >> >> printf("PMI says I am %d of %d \n", rank, size); >> fflush(stdout); >> >> int requested = (rank%2==0) ? MPI_THREAD_FUNNELED : MPI_THREAD_MULTIPLE; >> int provided; >> MPI_Init_thread(&argc, &argv, requested, &provided); >> >> int wsize, wrank; >> MPI_Comm_size(MPI_COMM_WORLD, &wsize); >> MPI_Comm_rank(MPI_COMM_WORLD, &wrank); >> >> printf("MPI says I am %d of %d (provided=%d)\n", wrank, wsize, provided); >> fflush(stdout); >> >> MPI_Finalize(); >> //PMI_Finalize(); /* required? */ >> >> return 0; >> } >> >> On Tue, Apr 21, 2015 at 9:09 PM, Jeff Hammond <jeff.science(a)gmail.com> >> wrote: >> >> Is it valid in _MPICH_ to make calls to PMI before MPI is initialized? >> I am interested in PMI_Init, PMI_Get_size and PMI_Get_rank. >> >> The motivating use case is to initialize e.g. half my ranks with >> MPI_THREAD_FUNNELED and the other half with MPI_THREAD_MULTIPLE but >> not use MPMD launching. >> >> I have two reasons for wanting to do it this way. First, it is >> otherwise unnecessary to use MPMD since I can branch at the top of my >> code and meet the requirements of FUNNELED and MULTIPLE. Second, I >> might be interested in running my code on a supercomputer with an >> MPICH-based MPI implementation that does not support MPMD launching in >> an unrestricted manner. >> >> If MPICH can handle this, then I will worry about whether or not the >> aforementioned supercomputer can. It is my assumption that, if MPICH >> cannot handle this, then the MPICH derivative in question cannot >> either. >> >> Thanks, >> >> Jeff >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> http://jeffhammond.github.io/ >> >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> http://jeffhammond.github.io/ >> _______________________________________________ >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/devel >> >> >> >> _______________________________________________ >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/devel > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel

1 0