January 2014 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] broken stuff in next please fix
by Satish Balay 04 Jan '14

04 Jan '14

On Sat, 4 Jan 2014, Satish Balay wrote: > On Sat, 4 Jan 2014, Jed Brown wrote: > > > Barry Smith <bsmith(a)mcs.anl.gov> writes: > > > Ahh, what should we be using for nightly tests? We can use a release or any development version. Currently we don’t seem to have a policy of what to track,. > > > At the moment I have it set to http://www.mpich.org/static/downloads/3.0/mpich-3.0.tar.gz > > > > Use their latest (3.0.4). I fixed this a year ago and it is in > > mpich-3.0.2 and later. > > > > http://git.mpich.org/mpich.git/commit/d440abbb025d3fb4e468bc43714346bd2c113… > > Barry, > > We were previously using v3.0.4 [with some bug fixes for some issues that > came up in petsc testing]. > > http://ftp.mcs.anl.gov/pub/petsc/tmp/mpich-master-v3.0.4-106-g3adb59c.tar.gz BTW: This is the version used in 'maint'/petsc-3.4 aswell. Satish

1 0

Re: [petsc-dev] broken stuff in next please fix
by Satish Balay 04 Jan '14

04 Jan '14

On Sat, 4 Jan 2014, Jed Brown wrote: > Barry Smith <bsmith(a)mcs.anl.gov> writes: > > Ahh, what should we be using for nightly tests? We can use a release or any development version. Currently we don’t seem to have a policy of what to track,. > > At the moment I have it set to http://www.mpich.org/static/downloads/3.0/mpich-3.0.tar.gz > > Use their latest (3.0.4). I fixed this a year ago and it is in > mpich-3.0.2 and later. > > http://git.mpich.org/mpich.git/commit/d440abbb025d3fb4e468bc43714346bd2c113… Barry, We were previously using v3.0.4 [with some bug fixes for some issues that came up in petsc testing]. http://ftp.mcs.anl.gov/pub/petsc/tmp/mpich-master-v3.0.4-106-g3adb59c.tar.gz Satish

1 0

Re: [petsc-dev] broken stuff in next please fix
by Jed Brown 04 Jan '14

04 Jan '14

Barry Smith <bsmith(a)mcs.anl.gov> writes: > Ahh, what should we be using for nightly tests? We can use a release or any development version. Currently we don’t seem to have a policy of what to track,. > At the moment I have it set to http://www.mpich.org/static/downloads/3.0/mpich-3.0.tar.gz Use their latest (3.0.4). I fixed this a year ago and it is in mpich-3.0.2 and later. http://git.mpich.org/mpich.git/commit/d440abbb025d3fb4e468bc43714346bd2c113…

1 0

Re: [petsc-dev] broken stuff in next please fix
by Barry Smith 04 Jan '14

04 Jan '14

On Jan 4, 2014, at 8:41 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > Matthew Knepley <knepley(a)gmail.com> writes: > >> On Sat, Jan 4, 2014 at 6:01 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >> >>> http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2014/01/04/filtered-ma… >>> >>> /sandbox/petsc/petsc.clone/src/sys/objects/pinit.c:224:37: warning: >>> argument type 'struct <anonymous struct at >>> /sandbox/petsc/petsc.clone/src/sys/objects/pinit.c:217:3> *' doesn't match >>> specified 'MPI' type tag that requires 'struct mpich_struct_mpi_2int *' >>> [-Wtype-safety] >>> 1 warning generated. >>> /sandbox/petsc/petsc.clone/src/vec/is/sf/interface/sf.c:459:49: warning: >>> argument type 'PetscSFNode *' doesn't match specified 'MPI' type tag that >>> requires 'struct mpich_struct_mpi_2int *' [-Wtype-safety] >>> >> >> Can we shut off this warning? > > What version of MPICH and Clang are you using? Current MPICH 'master' > is broken, but earlier versions with clang-3.3 are not producing this > warning. Ahh, what should we be using for nightly tests? We can use a release or any development version. Currently we don’t seem to have a policy of what to track,. At the moment I have it set to http://www.mpich.org/static/downloads/3.0/mpich-3.0.tar.gz Barry

1 0

Re: [petsc-dev] broken stuff in next please fix
by Jed Brown 04 Jan '14

04 Jan '14

Matthew Knepley <knepley(a)gmail.com> writes: > On Sat, Jan 4, 2014 at 6:01 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > >> http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2014/01/04/filtered-ma… >> >> /sandbox/petsc/petsc.clone/src/sys/objects/pinit.c:224:37: warning: >> argument type 'struct <anonymous struct at >> /sandbox/petsc/petsc.clone/src/sys/objects/pinit.c:217:3> *' doesn't match >> specified 'MPI' type tag that requires 'struct mpich_struct_mpi_2int *' >> [-Wtype-safety] >> 1 warning generated. >> /sandbox/petsc/petsc.clone/src/vec/is/sf/interface/sf.c:459:49: warning: >> argument type 'PetscSFNode *' doesn't match specified 'MPI' type tag that >> requires 'struct mpich_struct_mpi_2int *' [-Wtype-safety] >> > > Can we shut off this warning? What version of MPICH and Clang are you using? Current MPICH 'master' is broken, but earlier versions with clang-3.3 are not producing this warning.

1 0

Re: [petsc-dev] broken stuff in next please fix
by Matthew Knepley 04 Jan '14

04 Jan '14

On Sat, Jan 4, 2014 at 6:01 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2014/01/04/filtered-ma… > > /usr/home/balay/petsc.clone-3/arch-freebsd-cxx-cmplx-pkgs-dbg/bin/mpicxx > -Dpetsc_EXPORTS -Wall -Wwrite-strings -Wno-strict-aliasing > -Wno-unknown-pragmas -g -O0 -fPIC -fPIC > -I/usr/home/balay/petsc.clone-3/include > -I/usr/home/balay/petsc.clone-3/arch-freebsd-cxx-cmplx-pkgs-dbg/include > -I/usr/local/include -o CMakeFiles/petsc.dir/src/sys/error/errtrace.c.o -c > /usr/home/balay/petsc.clone-3/src/sys/error/errtrace.c > /usr/home/balay/petsc.clone-3/src/sys/error/err.c: In function 'void > PetscCxxErrorThrow()': > /usr/home/balay/petsc.clone-3/src/sys/error/err.c:310: error: no matching > function for call to 'std::exception::exception(const char*&)' > /usr/include/c++/4.2/exception:59: note: candidates are: > std::exception::exception() > /usr/include/c++/4.2/exception:57: note: > std::exception::exception(const std::exception&) > *** [CMakeFiles/petsc.dir/src/sys/error/err.c.o] Error code 1 > > > > http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2014/01/04/filtered-ma… > > /sandbox/petsc/petsc.clone/src/sys/objects/pinit.c:224:37: warning: > argument type 'struct <anonymous struct at > /sandbox/petsc/petsc.clone/src/sys/objects/pinit.c:217:3> *' doesn't match > specified 'MPI' type tag that requires 'struct mpich_struct_mpi_2int *' > [-Wtype-safety] > 1 warning generated. > /sandbox/petsc/petsc.clone/src/vec/is/sf/interface/sf.c:459:49: warning: > argument type 'PetscSFNode *' doesn't match specified 'MPI' type tag that > requires 'struct mpich_struct_mpi_2int *' [-Wtype-safety] > Can we shut off this warning? Matt > /sandbox/petsc/petsc.clone/src/dm/impls/plex/plex.c:3334:59: warning: > argument type 'PetscSFNode *' doesn't match specified 'MPI' type tag that > requires 'struct mpich_struct_mpi_2int *' [-Wtype-safety] > /sandbox/petsc/petsc.clone/src/dm/impls/plex/plex.c:3334:50: warning: > argument type 'PetscSFNode *' doesn't match specified 'MPI' type tag that > requires 'struct mpich_struct_mpi_2int *' [-Wtype-safety] > /sandbox/petsc/petsc.clone > > /sandbox/petsc/petsc.clone/src/dm/impls/plex/plexsubmesh.c:2753:69: > warning: argument type 'PetscSFNode *' doesn't match specified 'MPI' type > tag that requires 'struct mpich_struct_mpi_2int *' [-Wtype-safety] > > /sandbox/petsc/petsc.clone/src/ksp/pc/impls/gasm/gasm.c:328:42: warning: > argument type 'struct <anonymous struct at > /sandbox/petsc/petsc.clone/src/ksp/pc/impls/gasm/gasm.c:324:7> *' doesn't > match specified 'MPI' type tag that requires 'struct mpich_struct_mpi_2int > *' [-Wtype-safety] > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

broken stuff in next please fix
by Barry Smith 04 Jan '14

04 Jan '14

http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2014/01/04/filtered-ma… /usr/home/balay/petsc.clone-3/arch-freebsd-cxx-cmplx-pkgs-dbg/bin/mpicxx -Dpetsc_EXPORTS -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g -O0 -fPIC -fPIC -I/usr/home/balay/petsc.clone-3/include -I/usr/home/balay/petsc.clone-3/arch-freebsd-cxx-cmplx-pkgs-dbg/include -I/usr/local/include -o CMakeFiles/petsc.dir/src/sys/error/errtrace.c.o -c /usr/home/balay/petsc.clone-3/src/sys/error/errtrace.c /usr/home/balay/petsc.clone-3/src/sys/error/err.c: In function 'void PetscCxxErrorThrow()': /usr/home/balay/petsc.clone-3/src/sys/error/err.c:310: error: no matching function for call to 'std::exception::exception(const char*&)' /usr/include/c++/4.2/exception:59: note: candidates are: std::exception::exception() /usr/include/c++/4.2/exception:57: note: std::exception::exception(const std::exception&) *** [CMakeFiles/petsc.dir/src/sys/error/err.c.o] Error code 1 http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2014/01/04/filtered-ma… /sandbox/petsc/petsc.clone/src/sys/objects/pinit.c:224:37: warning: argument type 'struct <anonymous struct at /sandbox/petsc/petsc.clone/src/sys/objects/pinit.c:217:3> *' doesn't match specified 'MPI' type tag that requires 'struct mpich_struct_mpi_2int *' [-Wtype-safety] 1 warning generated. /sandbox/petsc/petsc.clone/src/vec/is/sf/interface/sf.c:459:49: warning: argument type 'PetscSFNode *' doesn't match specified 'MPI' type tag that requires 'struct mpich_struct_mpi_2int *' [-Wtype-safety] /sandbox/petsc/petsc.clone/src/dm/impls/plex/plex.c:3334:59: warning: argument type 'PetscSFNode *' doesn't match specified 'MPI' type tag that requires 'struct mpich_struct_mpi_2int *' [-Wtype-safety] /sandbox/petsc/petsc.clone/src/dm/impls/plex/plex.c:3334:50: warning: argument type 'PetscSFNode *' doesn't match specified 'MPI' type tag that requires 'struct mpich_struct_mpi_2int *' [-Wtype-safety] /sandbox/petsc/petsc.clone /sandbox/petsc/petsc.clone/src/dm/impls/plex/plexsubmesh.c:2753:69: warning: argument type 'PetscSFNode *' doesn't match specified 'MPI' type tag that requires 'struct mpich_struct_mpi_2int *' [-Wtype-safety] /sandbox/petsc/petsc.clone/src/ksp/pc/impls/gasm/gasm.c:328:42: warning: argument type 'struct <anonymous struct at /sandbox/petsc/petsc.clone/src/ksp/pc/impls/gasm/gasm.c:324:7> *' doesn't match specified 'MPI' type tag that requires 'struct mpich_struct_mpi_2int *' [-Wtype-safety]

1 0

[Jeff Hammond] Re: VPATH builds broken
by Jed Brown 03 Jan '14

03 Jan '14

This reminds me that PETSc's build system is tied to PETSC_ARCH, thus does not support a read-only source tree. To do this, we would have to make tests and Fortran stubs go into the build directory (we want that already) and to lessen our dependence on PETSC_ARCH. I think we should be able to make it possible to run /path/to/petsc/configure with no special arguments and have it set up a normal out-of-source build that would be independent of PETSC_DIR and PETSC_ARCH. If instead, you run configure from PETSC_DIR, it would continue to use PETSC_ARCH as the build directory. (To make these otherwise uniform, PETSC_ARCH would probably become just a hint to run make from the appropriate directory.) Any reason we should not do this once the testing and Fortran stuff works that way?

1 0

Re: [petsc-dev] IMP prototype
by Jed Brown 03 Jan '14

03 Jan '14

Victor Eijkhout <eijkhout(a)tacc.utexas.edu> writes: > On Jan 2, 2014, at 10:12 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > >> the execution model is BSP > > No it's not. There are no barriers or syncs. You have a communication phase followed by computation, followed by more communication (in general). Looks like BSP without explicit barriers (which would be semantically meaningless if you added them). An example of a less structured communication pattern is Mark's asynchronous Gauss-Seidel. http://crd.lbl.gov/assets/Uploads/ANAG/gs.pdf > The callbacks are there just to make the code uniform. I've edited the > document to reflect that in the MPI case you can dispense with them. I thought you were targeting hybrid MPI/threads? >> your transformation is recognizing a common >> pattern of communication into temporary buffers, followed by >> computation, followed by post-communication and putting a declarative >> syntax on it > > Somewhat simplified, but not wrong. I'm kind of interested in the > question what practically relevant algorithms do not conform to that > model. The GS above is one example. More simply, how does MatMult_MPI look in your model (note overlapped communication and computation)? Also, you can't implement an efficient setup for your communication pattern without richer semantics, see PetscCommBuildTwoSided_Ibarrier and the paper http://unixer.de/publications/img/hoefler-dsde-protocols.pdf >> Your abstraction is not uniform >> if you need to index into owned parts of shared data structures or >> perform optimizations like cooperative prefetch. > > Not sure what you're saying here. In case you dug into my code deeply, > at the moment gathering the halo region involves one send-to-self that > should be optimized away in the future. Look, it's a prototype, all > right? I don't care about that. What does the hybrid case look like? Do you prepare separate work buffers for each thread or do the threads work on parts of a shared data structure, perhaps cooperating at higher frequency than the MPI? I thought you were creating strict independence and private buffers, which would be MPI-like semantics using threads (totally possible, and I think usually a good thing, but most threaded programming models are explicitly trying to avoid it). >> If you're going to use >> separate buffers, why is a system like MPI not sufficient? > > 1. No idea why you're so fixated on buffers. I've just been going for > the simplest implementation that makes it work on this one example. It > can all be optimized. What would the optimized version look like? Someone has to decide how to index into the data structures. You're not providing much uniformity relative to MPI+X (e.g., X=OpenMP) if the threaded part is always sorted out by the user in an application-dependent way. > 2. Why is MPI not sufficient: because you have to spell out too > much. Calculating a halo in the case of a block-cyclically distributed > vector is way too much work. An extension of IMP would make this very > easy to specify, and all the hard work is done under the covers. VecScatter and PetscSF are less intrusive interfaces on top of MPI. I agree that MPI_Isend is pretty low level, but what are you providing relative to these less intrusive abstractions? You said that you were NOT looking for 'a better notation for VecScatters", so let's assume that such interfaces are available to the MPI programmer. >> What semantic does your abstraction provide for hybrid >> distributed/shared memory that imperative communication systems cannot? > > How about having the exact same code that I've shown you, except that > you specify that the processes are organized as N nodes times C cores? > Right now I've not implemented hybrid programming, but it shouldn't be > hard. How does the code the user writes in the callback remain the same for MPI and threads without the copies that AMPI or shared-memory MPI would have done and without requiring the user to explicitly deal with the threads (working on shared data structures as with OpenMP/TBB/pthreads)? I'm looking for a precise statement of something user code can be ignorant of with your model, yet reap the benefits of a model where they explicitly used that information (e.g., a semantic not possible with shared-memory MPI/AMPI, and possible using MPI+X only with some onerous complexity that cannot be easily tucked into a generic library function).

1 0

Re: [petsc-dev] IMP prototype
by Victor Eijkhout 02 Jan '14

02 Jan '14

On Jan 2, 2014, at 10:12 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > the execution model is BSP No it's not. There are no barriers or syncs. > it's not clear what is gained by switching > to callbacks. The callbacks are there just to make the code uniform. I've edited the document to reflect that in the MPI case you can dispense with them. > your transformation is recognizing a common > pattern of communication into temporary buffers, followed by > computation, followed by post-communication and putting a declarative > syntax on it Somewhat simplified, but not wrong. I'm kind of interested in the question what practically relevant algorithms do not conform to that model. > (with callbacks for the computation) They are incidental, at least in the MPI case. Ignore them. > Your abstraction is not uniform > if you need to index into owned parts of shared data structures or > perform optimizations like cooperative prefetch. Not sure what you're saying here. In case you dug into my code deeply, at the moment gathering the halo region involves one send-to-self that should be optimized away in the future. Look, it's a prototype, all right? > If you're going to use > separate buffers, why is a system like MPI not sufficient? 1. No idea why you're so fixated on buffers. I've just been going for the simplest implementation that makes it work on this one example. It can all be optimized. 2. Why is MPI not sufficient: because you have to spell out too much. Calculating a halo in the case of a block-cyclically distributed vector is way too much work. An extension of IMP would make this very easy to specify, and all the hard work is done under the covers. > What semantic does your abstraction provide for hybrid > distributed/shared memory that imperative communication systems cannot? How about having the exact same code that I've shown you, except that you specify that the processes are organized as N nodes times C cores? Right now I've not implemented hybrid programming, but it shouldn't be hard. Victor. -- Victor Eijkhout, 512 471 5809 (w) Texas Advanced Computing Center, The University of Texas at Austin

1 0