April 2011 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] [petsc-maint #69143] petsc-dev performance issues
by Barry Smith 05 Apr '11

05 Apr '11

On Apr 5, 2011, at 8:09 AM, Dmitry Karpeev wrote: > > > On Tue, Apr 5, 2011 at 7:31 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > On Apr 5, 2011, at 1:31 AM, Satish Balay wrote: > > > On Mon, 4 Apr 2011, Dmitry Karpeev wrote: > > > >> What is the patch level petsc-3.1 being used? The dlopen() code that > >> generates the > >> error was added to petsc-dev after the initial release of 3.1. > >> > >> Typically (on many systems) dlopen(NULL,dlflags) returns the handle of the > >> main executable, as if it were > >> a dynamically-loaded library, and dlsym() is then run on that handle. This > >> doesn't work on all systems, > >> though (e.g., OS X defines a specific handle for the main executable; > >> enabling that means augmenting > >> configure). In particular, on this system, dlopen() with a NULL first > >> argument appears to throw a confusing error. > >> > >> The original release of petsc-3.1 would use dlsym(0) in this case, skipping > >> dlopen(NULL,dlflags), > >> so that is, perhaps, what works on XT5. To revert to that behavior it > >> should be sufficient > >> to apply the following patch to petsc-dev/src/sys/dll/dlimpl.c (or simply to > >> remove lines 269 through 308 inclusive). > > > > For One - I don't think any of the default PETSc code should be doing > > dlopen(executalbe) - if --with-dynamic-loading=0 is set [Otherwise why > > have this option? suppress dlopen(libpetsc.so) - but enable > > dlopen(executable)?] > > > > So I've changed the fix below and disable it with: > > > > #if defined(PETSC_HAVE_DLOPEN) && defined(PETSC_USE_DYNAMIC_LIBRARIES) > > > > wrt dlsym(0) - we ignore the dlerror() from it - so I guess its ok.. > > > > This works fine on the cray - so pusing this change. > > > > [I guess we still need to fix this for --with-dynamic-loading=1] > > I thought about this also, but it is the wrong fix. HAVE_DYNAMIC means that dynamic libraries work and the user can pass strings as functions; USE_DYNAMIC means make PETSc libraries dynamic and always use strings for functions. The problem is only that we do not have the configure tests in places for various versions of dlopen() on the executable. The correct fix is to make configure tests for dlopen() and change the PETSc code to do the various dlopen(0,...) variants based on the configure test. Now we can punt on doing it correctly, but we'd be punting only out of lazyness, not because it is the right model. > > BTW: someone should submit a bug report to Cray (and IBM if it doesn't work there also). > > Okay, but what's going on with petsc-3.1 on that machine? It doesn't fail. Is that because dlsym(0,string) works? > Or was dynamic loading disabled in it? As I explained in my previous mail, before petsc-dev there was no PETSC_HAVE_DYNAMIC_LIBRARIES only PETSC_USE_DYNAMIC_LIBRARIES so PETSc did not try to dlopen(0) unless --with-dynamic was passed in and hence the error in the Cray went undetected. Barry > > Dmitry. > > > Barry > > > > > > Satish > > > >> > >> Dmitry. > >> > >> @@ -266,46 +266,7 @@ > >> } > >> else { > >> dlhandle = (dlhandle_t) 0; > >> - > >> -#if defined(PETSC_HAVE_DLOPEN) > >> - /* Attempt to retrieve the main executable's dlhandle. */ > >> - { int dlflags1 = 0, dlflags2 = 0; > >> -#if defined(PETSC_HAVE_RTLD_LAZY) > >> - dlflags1 = RTLD_LAZY; > >> -#endif > >> - if(!dlflags1) { > >> -#if defined(PETSC_HAVE_RTLD_NOW) > >> - dlflags1 = RTLD_NOW; > >> -#endif > >> - } > >> -#if defined(PETSC_HAVE_RTLD_LOCAL) > >> - dlflags2 = RTLD_LOCAL; > >> -#endif > >> - if(!dlflags2) { > >> -#if defined(PETSC_HAVE_RTLD_GLOBAL) > >> - dlflags2 = RTLD_GLOBAL; > >> -#endif > >> - } > >> -#if defined(PETSC_HAVE_DLERROR) > >> -#if defined(PETSC_HAVE_VALGRIND) > >> - if (!(RUNNING_ON_VALGRIND)) { > >> -#endif > >> - dlerror(); /* clear any previous error; valgrind does not like this > >> */ > >> -#if defined(PETSC_HAVE_VALGRIND) > >> - } > >> -#endif > >> -#endif > >> - /* Attempt to open the main executable as a dynamic library. */ > >> - dlhandle = dlopen(0, dlflags1|dlflags2); > >> - } > >> -#if defined(PETSC_HAVE_DLERROR) > >> - { const char *e = (const char*) dlerror(); > >> - if(e){ > >> - SETERRQ1(PETSC_COMM_SELF, PETSC_ERR_ARG_WRONG, "Error opening main > >> executable as a dynamic library:\n Error message from dlopen(): '%s'\n", > >> e); > >> - } > >> - } > >> -#endif > >> -#endif /* PETSC_HAVE_DLOPEN */ > >> + > >> } > >> #if defined(PETSC_HAVE_DLERROR) > >> dlerror(); /* clear any previous error */ > >> > >> > >> > >> On Mon, Apr 4, 2011 at 7:26 PM, Matthew Knepley <petsc-maint(a)mcs.anl.gov>wrote: > >> > >>> Now it is really hard for me to understand what the problem is since both > >>> 3.1 and dev check for this > >>> function in the same way. Moreover, it does not depend on shared libraries. > >>> > >>> Satish, have you seen this error before on the XT5? > >>> > >>> Matt > >>> > >>> On Mon, Apr 4, 2011 at 6:45 PM, Satish Balay <petsc-maint(a)mcs.anl.gov > >>>> wrote: > >>> > >>>> On Mon, 4 Apr 2011, Sebastian Steiger wrote: > >>>> > >>>>> On 04/04/2011 05:42 PM, Satish Balay wrote: > >>>>>> Could you run both the binaries in the same node-allocation - with a > >>>>>> single batch file and send the '-log_summary' for them? > >>>>>> run petsc-dev > >>>>>> run petsc-31 > >>>>>> run petsc-dev > >>>>>> run petsc-31 > >>>>> Do you mean exactly the same physical nodes? I don't know how to do > >>>>> that. My batch files for petsc-dev and petsc-3.1-p4 are identical > >>> except > >>>>> for the static executable. > >>>> > >>>> > >>>> Haven't used batch stuff on ornl machine - but if the usual batch > >>>> script file is something like: [for eg: pbs] > >>>> > >>>>>>> > >>>> #!/bin/sh > >>>> #PBS -N hello > >>>> #PBS -l nodes=1:ppn=8 > >>>> #PBS -l walltime=0:00:15 > >>>> #PBS -j oe > >>>> > >>>> cd $PBS_O_WORKDIR > >>>> mpiexec ./ex1 > >>>> <<<<< > >>>> > >>>> you could change it to: > >>>> > >>>>>>>>> > >>>> #!/bin/sh > >>>> #PBS -N hello > >>>> #PBS -l nodes=1:ppn=8 > >>>> #PBS -l walltime=0:00:30 > >>>> #PBS -j oe > >>>> > >>>> cd $PBS_O_WORKDIR > >>>> mpiexec ./ex1 > >>>> mpiexec ./ex2 > >>>> mpiexec ./ex1 > >>>> mpiexec ./ex2 > >>>> <<<<<<< > >>>> > >>>> and run it with a single allocation.. > >>>> > >>>> Satish > >>>> > >>>> > >>> > >>> > >>> -- > >>> What most experimenters take for granted before they begin their > >>> experiments > >>> is infinitely more interesting than any results to which their experiments > >>> lead. > >>> -- Norbert Wiener > >>> > >>> > >>> Now it is really hard for me to understand what the problem is since both > >>> 3.1 and dev check for this > >>> function in the same way. Moreover, it does not depend on shared libraries. > >>> > >>> Satish, have you seen this error before on the XT5? > >>> > >>> Matt > >>> > >>> On Mon, Apr 4, 2011 at 6:45 PM, Satish Balay <petsc-maint(a)mcs.anl.gov>wrote: > >>> > >>>> On Mon, 4 Apr 2011, Sebastian Steiger wrote: > >>>> > >>>>> On 04/04/2011 05:42 PM, Satish Balay wrote: > >>>>>> Could you run both the binaries in the same node-allocation - with a > >>>>>> single batch file and send the '-log_summary' for them? > >>>>>> run petsc-dev > >>>>>> run petsc-31 > >>>>>> run petsc-dev > >>>>>> run petsc-31 > >>>>> Do you mean exactly the same physical nodes? I don't know how to do > >>>>> that. My batch files for petsc-dev and petsc-3.1-p4 are identical except > >>>>> for the static executable. > >>>> > >>>> > >>>> Haven't used batch stuff on ornl machine - but if the usual batch > >>>> script file is something like: [for eg: pbs] > >>>> > >>>>>>> > >>>> #!/bin/sh > >>>> #PBS -N hello > >>>> #PBS -l nodes=1:ppn=8 > >>>> #PBS -l walltime=0:00:15 > >>>> #PBS -j oe > >>>> > >>>> cd $PBS_O_WORKDIR > >>>> mpiexec ./ex1 > >>>> <<<<< > >>>> > >>>> you could change it to: > >>>> > >>>>>>>>> > >>>> #!/bin/sh > >>>> #PBS -N hello > >>>> #PBS -l nodes=1:ppn=8 > >>>> #PBS -l walltime=0:00:30 > >>>> #PBS -j oe > >>>> > >>>> cd $PBS_O_WORKDIR > >>>> mpiexec ./ex1 > >>>> mpiexec ./ex2 > >>>> mpiexec ./ex1 > >>>> mpiexec ./ex2 > >>>> <<<<<<< > >>>> > >>>> and run it with a single allocation.. > >>>> > >>>> Satish > >>>> > >>>> > >>> > >>> > >>> -- > >>> What most experimenters take for granted before they begin their > >>> experiments is infinitely more interesting than any results to which their > >>> experiments lead. > >>> -- Norbert Wiener > >>> > >>> > >> > >> > > > > > >

1 0

Re: [petsc-dev] [petsc-maint #69143] petsc-dev performance issues
by Dmitry Karpeev 05 Apr '11

05 Apr '11

On Tue, Apr 5, 2011 at 7:31 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > On Apr 5, 2011, at 1:31 AM, Satish Balay wrote: > > > On Mon, 4 Apr 2011, Dmitry Karpeev wrote: > > > >> What is the patch level petsc-3.1 being used? The dlopen() code that > >> generates the > >> error was added to petsc-dev after the initial release of 3.1. > >> > >> Typically (on many systems) dlopen(NULL,dlflags) returns the handle of > the > >> main executable, as if it were > >> a dynamically-loaded library, and dlsym() is then run on that handle. > This > >> doesn't work on all systems, > >> though (e.g., OS X defines a specific handle for the main executable; > >> enabling that means augmenting > >> configure). In particular, on this system, dlopen() with a NULL first > >> argument appears to throw a confusing error. > >> > >> The original release of petsc-3.1 would use dlsym(0) in this case, > skipping > >> dlopen(NULL,dlflags), > >> so that is, perhaps, what works on XT5. To revert to that behavior it > >> should be sufficient > >> to apply the following patch to petsc-dev/src/sys/dll/dlimpl.c (or > simply to > >> remove lines 269 through 308 inclusive). > > > > For One - I don't think any of the default PETSc code should be doing > > dlopen(executalbe) - if --with-dynamic-loading=0 is set [Otherwise why > > have this option? suppress dlopen(libpetsc.so) - but enable > > dlopen(executable)?] > > > > So I've changed the fix below and disable it with: > > > > #if defined(PETSC_HAVE_DLOPEN) && defined(PETSC_USE_DYNAMIC_LIBRARIES) > > > > wrt dlsym(0) - we ignore the dlerror() from it - so I guess its ok.. > > > > This works fine on the cray - so pusing this change. > > > > [I guess we still need to fix this for --with-dynamic-loading=1] > > I thought about this also, but it is the wrong fix. HAVE_DYNAMIC means > that dynamic libraries work and the user can pass strings as functions; > USE_DYNAMIC means make PETSc libraries dynamic and always use strings for > functions. The problem is only that we do not have the configure tests in > places for various versions of dlopen() on the executable. The correct fix > is to make configure tests for dlopen() and change the PETSc code to do the > various dlopen(0,...) variants based on the configure test. Now we can punt > on doing it correctly, but we'd be punting only out of lazyness, not because > it is the right model. > > BTW: someone should submit a bug report to Cray (and IBM if it doesn't > work there also). > Okay, but what's going on with petsc-3.1 on that machine? It doesn't fail. Is that because dlsym(0,string) works? Or was dynamic loading disabled in it? Dmitry. > > Barry > > > > > > Satish > > > >> > >> Dmitry. > >> > >> @@ -266,46 +266,7 @@ > >> } > >> else { > >> dlhandle = (dlhandle_t) 0; > >> - > >> -#if defined(PETSC_HAVE_DLOPEN) > >> - /* Attempt to retrieve the main executable's dlhandle. */ > >> - { int dlflags1 = 0, dlflags2 = 0; > >> -#if defined(PETSC_HAVE_RTLD_LAZY) > >> - dlflags1 = RTLD_LAZY; > >> -#endif > >> - if(!dlflags1) { > >> -#if defined(PETSC_HAVE_RTLD_NOW) > >> - dlflags1 = RTLD_NOW; > >> -#endif > >> - } > >> -#if defined(PETSC_HAVE_RTLD_LOCAL) > >> - dlflags2 = RTLD_LOCAL; > >> -#endif > >> - if(!dlflags2) { > >> -#if defined(PETSC_HAVE_RTLD_GLOBAL) > >> - dlflags2 = RTLD_GLOBAL; > >> -#endif > >> - } > >> -#if defined(PETSC_HAVE_DLERROR) > >> -#if defined(PETSC_HAVE_VALGRIND) > >> - if (!(RUNNING_ON_VALGRIND)) { > >> -#endif > >> - dlerror(); /* clear any previous error; valgrind does not like > this > >> */ > >> -#if defined(PETSC_HAVE_VALGRIND) > >> - } > >> -#endif > >> -#endif > >> - /* Attempt to open the main executable as a dynamic library. */ > >> - dlhandle = dlopen(0, dlflags1|dlflags2); > >> - } > >> -#if defined(PETSC_HAVE_DLERROR) > >> - { const char *e = (const char*) dlerror(); > >> - if(e){ > >> - SETERRQ1(PETSC_COMM_SELF, PETSC_ERR_ARG_WRONG, "Error opening > main > >> executable as a dynamic library:\n Error message from dlopen(): > '%s'\n", > >> e); > >> - } > >> - } > >> -#endif > >> -#endif /* PETSC_HAVE_DLOPEN */ > >> + > >> } > >> #if defined(PETSC_HAVE_DLERROR) > >> dlerror(); /* clear any previous error */ > >> > >> > >> > >> On Mon, Apr 4, 2011 at 7:26 PM, Matthew Knepley < > petsc-maint(a)mcs.anl.gov>wrote: > >> > >>> Now it is really hard for me to understand what the problem is since > both > >>> 3.1 and dev check for this > >>> function in the same way. Moreover, it does not depend on shared > libraries. > >>> > >>> Satish, have you seen this error before on the XT5? > >>> > >>> Matt > >>> > >>> On Mon, Apr 4, 2011 at 6:45 PM, Satish Balay <petsc-maint(a)mcs.anl.gov > >>>> wrote: > >>> > >>>> On Mon, 4 Apr 2011, Sebastian Steiger wrote: > >>>> > >>>>> On 04/04/2011 05:42 PM, Satish Balay wrote: > >>>>>> Could you run both the binaries in the same node-allocation - with a > >>>>>> single batch file and send the '-log_summary' for them? > >>>>>> run petsc-dev > >>>>>> run petsc-31 > >>>>>> run petsc-dev > >>>>>> run petsc-31 > >>>>> Do you mean exactly the same physical nodes? I don't know how to do > >>>>> that. My batch files for petsc-dev and petsc-3.1-p4 are identical > >>> except > >>>>> for the static executable. > >>>> > >>>> > >>>> Haven't used batch stuff on ornl machine - but if the usual batch > >>>> script file is something like: [for eg: pbs] > >>>> > >>>>>>> > >>>> #!/bin/sh > >>>> #PBS -N hello > >>>> #PBS -l nodes=1:ppn=8 > >>>> #PBS -l walltime=0:00:15 > >>>> #PBS -j oe > >>>> > >>>> cd $PBS_O_WORKDIR > >>>> mpiexec ./ex1 > >>>> <<<<< > >>>> > >>>> you could change it to: > >>>> > >>>>>>>>> > >>>> #!/bin/sh > >>>> #PBS -N hello > >>>> #PBS -l nodes=1:ppn=8 > >>>> #PBS -l walltime=0:00:30 > >>>> #PBS -j oe > >>>> > >>>> cd $PBS_O_WORKDIR > >>>> mpiexec ./ex1 > >>>> mpiexec ./ex2 > >>>> mpiexec ./ex1 > >>>> mpiexec ./ex2 > >>>> <<<<<<< > >>>> > >>>> and run it with a single allocation.. > >>>> > >>>> Satish > >>>> > >>>> > >>> > >>> > >>> -- > >>> What most experimenters take for granted before they begin their > >>> experiments > >>> is infinitely more interesting than any results to which their > experiments > >>> lead. > >>> -- Norbert Wiener > >>> > >>> > >>> Now it is really hard for me to understand what the problem is since > both > >>> 3.1 and dev check for this > >>> function in the same way. Moreover, it does not depend on shared > libraries. > >>> > >>> Satish, have you seen this error before on the XT5? > >>> > >>> Matt > >>> > >>> On Mon, Apr 4, 2011 at 6:45 PM, Satish Balay <petsc-maint(a)mcs.anl.gov > >wrote: > >>> > >>>> On Mon, 4 Apr 2011, Sebastian Steiger wrote: > >>>> > >>>>> On 04/04/2011 05:42 PM, Satish Balay wrote: > >>>>>> Could you run both the binaries in the same node-allocation - with a > >>>>>> single batch file and send the '-log_summary' for them? > >>>>>> run petsc-dev > >>>>>> run petsc-31 > >>>>>> run petsc-dev > >>>>>> run petsc-31 > >>>>> Do you mean exactly the same physical nodes? I don't know how to do > >>>>> that. My batch files for petsc-dev and petsc-3.1-p4 are identical > except > >>>>> for the static executable. > >>>> > >>>> > >>>> Haven't used batch stuff on ornl machine - but if the usual batch > >>>> script file is something like: [for eg: pbs] > >>>> > >>>>>>> > >>>> #!/bin/sh > >>>> #PBS -N hello > >>>> #PBS -l nodes=1:ppn=8 > >>>> #PBS -l walltime=0:00:15 > >>>> #PBS -j oe > >>>> > >>>> cd $PBS_O_WORKDIR > >>>> mpiexec ./ex1 > >>>> <<<<< > >>>> > >>>> you could change it to: > >>>> > >>>>>>>>> > >>>> #!/bin/sh > >>>> #PBS -N hello > >>>> #PBS -l nodes=1:ppn=8 > >>>> #PBS -l walltime=0:00:30 > >>>> #PBS -j oe > >>>> > >>>> cd $PBS_O_WORKDIR > >>>> mpiexec ./ex1 > >>>> mpiexec ./ex2 > >>>> mpiexec ./ex1 > >>>> mpiexec ./ex2 > >>>> <<<<<<< > >>>> > >>>> and run it with a single allocation.. > >>>> > >>>> Satish > >>>> > >>>> > >>> > >>> > >>> -- > >>> What most experimenters take for granted before they begin their > >>> experiments is infinitely more interesting than any results to which > their > >>> experiments lead. > >>> -- Norbert Wiener > >>> > >>> > >> > >> > > > > > >

1 0

Re: [petsc-dev] Compiling C code with C compiler
by Barry Smith 05 Apr '11

05 Apr '11

Jed, If it is as simple as you say then I am fine with adding support BUT it will require "yet another flag" and hence increase complexity. --with-c-for-c-code=1 or something and either require a compiler that is smart enough to switch based on extension or two lists of files to compile with the two compilers. These choices are already complicated for some users and the logic in PETSc will get messy. Barry On Apr 5, 2011, at 2:03 AM, Jed Brown wrote: > On Tue, Apr 5, 2011 at 04:33, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > Since the sieve c++ compiled code uses the C code (like VecXXX) the C++ compiler would have to know that the other parts of C are not compiled with C++. At a minimum you would have to extern c it in the header files other wise the C++ compiled code will call the C routines with name-mangling turned on while the C compiler doesn't know about name mangling and hence would generate routines without name mangling and the c++ code wouldn't find the c routines. > > But this is already what happens --with-c-support. Why is it ever necessary to compile the C source with a C++ compiler (that is, why support with-c-support=0)? The C++ convenience functions (PetscPolymorphicFunction) can easily be defined whenever a C++ compiler is used, they will be name mangled (and usually inlined). I think all you would need to do is remove the PETSC_USE_EXTERN_CXX clause from > > #if !defined(PETSC_USE_EXTERN_CXX) && defined(__cplusplus) > > In other words, even without issues of struct layouts currently everything in PETSc needs to be built with the same compiler. If sieve was made a completely separate package form PETSc then one could easily arrange things so that sieve compiled with c++ and PETSc with c. > > I don't see any complexity in always building C source with a C compiler and C++ source with a C++ compiler. It seems simpler to me. And then people could build a PETSc that includes Sieve and still call it from C (e.g. when they happen to not be using Sieve). > > > The only problem I can think of, and which might be justification for building C code with a C++ compiler, is that C++ compilers seem to be stricter about complex. C99 complex is binary compatible with C++ complex, but type checking is weaker. I don't know if this is implicit in the standard or if the GNU compilers are unnecessarily lenient about C99 complex. This is purely a matter of quality error/warning messages, but I'm loathe to give up any static type checking.

1 0

Re: [petsc-dev] [petsc-maint #69143] petsc-dev performance issues
by Barry Smith 05 Apr '11

05 Apr '11

On Apr 5, 2011, at 1:31 AM, Satish Balay wrote: > On Mon, 4 Apr 2011, Dmitry Karpeev wrote: > >> What is the patch level petsc-3.1 being used? The dlopen() code that >> generates the >> error was added to petsc-dev after the initial release of 3.1. >> >> Typically (on many systems) dlopen(NULL,dlflags) returns the handle of the >> main executable, as if it were >> a dynamically-loaded library, and dlsym() is then run on that handle. This >> doesn't work on all systems, >> though (e.g., OS X defines a specific handle for the main executable; >> enabling that means augmenting >> configure). In particular, on this system, dlopen() with a NULL first >> argument appears to throw a confusing error. >> >> The original release of petsc-3.1 would use dlsym(0) in this case, skipping >> dlopen(NULL,dlflags), >> so that is, perhaps, what works on XT5. To revert to that behavior it >> should be sufficient >> to apply the following patch to petsc-dev/src/sys/dll/dlimpl.c (or simply to >> remove lines 269 through 308 inclusive). > > For One - I don't think any of the default PETSc code should be doing > dlopen(executalbe) - if --with-dynamic-loading=0 is set [Otherwise why > have this option? suppress dlopen(libpetsc.so) - but enable > dlopen(executable)?] > > So I've changed the fix below and disable it with: > > #if defined(PETSC_HAVE_DLOPEN) && defined(PETSC_USE_DYNAMIC_LIBRARIES) > > wrt dlsym(0) - we ignore the dlerror() from it - so I guess its ok.. > > This works fine on the cray - so pusing this change. > > [I guess we still need to fix this for --with-dynamic-loading=1] I thought about this also, but it is the wrong fix. HAVE_DYNAMIC means that dynamic libraries work and the user can pass strings as functions; USE_DYNAMIC means make PETSc libraries dynamic and always use strings for functions. The problem is only that we do not have the configure tests in places for various versions of dlopen() on the executable. The correct fix is to make configure tests for dlopen() and change the PETSc code to do the various dlopen(0,...) variants based on the configure test. Now we can punt on doing it correctly, but we'd be punting only out of lazyness, not because it is the right model. BTW: someone should submit a bug report to Cray (and IBM if it doesn't work there also). Barry > > Satish > >> >> Dmitry. >> >> @@ -266,46 +266,7 @@ >> } >> else { >> dlhandle = (dlhandle_t) 0; >> - >> -#if defined(PETSC_HAVE_DLOPEN) >> - /* Attempt to retrieve the main executable's dlhandle. */ >> - { int dlflags1 = 0, dlflags2 = 0; >> -#if defined(PETSC_HAVE_RTLD_LAZY) >> - dlflags1 = RTLD_LAZY; >> -#endif >> - if(!dlflags1) { >> -#if defined(PETSC_HAVE_RTLD_NOW) >> - dlflags1 = RTLD_NOW; >> -#endif >> - } >> -#if defined(PETSC_HAVE_RTLD_LOCAL) >> - dlflags2 = RTLD_LOCAL; >> -#endif >> - if(!dlflags2) { >> -#if defined(PETSC_HAVE_RTLD_GLOBAL) >> - dlflags2 = RTLD_GLOBAL; >> -#endif >> - } >> -#if defined(PETSC_HAVE_DLERROR) >> -#if defined(PETSC_HAVE_VALGRIND) >> - if (!(RUNNING_ON_VALGRIND)) { >> -#endif >> - dlerror(); /* clear any previous error; valgrind does not like this >> */ >> -#if defined(PETSC_HAVE_VALGRIND) >> - } >> -#endif >> -#endif >> - /* Attempt to open the main executable as a dynamic library. */ >> - dlhandle = dlopen(0, dlflags1|dlflags2); >> - } >> -#if defined(PETSC_HAVE_DLERROR) >> - { const char *e = (const char*) dlerror(); >> - if(e){ >> - SETERRQ1(PETSC_COMM_SELF, PETSC_ERR_ARG_WRONG, "Error opening main >> executable as a dynamic library:\n Error message from dlopen(): '%s'\n", >> e); >> - } >> - } >> -#endif >> -#endif /* PETSC_HAVE_DLOPEN */ >> + >> } >> #if defined(PETSC_HAVE_DLERROR) >> dlerror(); /* clear any previous error */ >> >> >> >> On Mon, Apr 4, 2011 at 7:26 PM, Matthew Knepley <petsc-maint(a)mcs.anl.gov>wrote: >> >>> Now it is really hard for me to understand what the problem is since both >>> 3.1 and dev check for this >>> function in the same way. Moreover, it does not depend on shared libraries. >>> >>> Satish, have you seen this error before on the XT5? >>> >>> Matt >>> >>> On Mon, Apr 4, 2011 at 6:45 PM, Satish Balay <petsc-maint(a)mcs.anl.gov >>>> wrote: >>> >>>> On Mon, 4 Apr 2011, Sebastian Steiger wrote: >>>> >>>>> On 04/04/2011 05:42 PM, Satish Balay wrote: >>>>>> Could you run both the binaries in the same node-allocation - with a >>>>>> single batch file and send the '-log_summary' for them? >>>>>> run petsc-dev >>>>>> run petsc-31 >>>>>> run petsc-dev >>>>>> run petsc-31 >>>>> Do you mean exactly the same physical nodes? I don't know how to do >>>>> that. My batch files for petsc-dev and petsc-3.1-p4 are identical >>> except >>>>> for the static executable. >>>> >>>> >>>> Haven't used batch stuff on ornl machine - but if the usual batch >>>> script file is something like: [for eg: pbs] >>>> >>>>>>> >>>> #!/bin/sh >>>> #PBS -N hello >>>> #PBS -l nodes=1:ppn=8 >>>> #PBS -l walltime=0:00:15 >>>> #PBS -j oe >>>> >>>> cd $PBS_O_WORKDIR >>>> mpiexec ./ex1 >>>> <<<<< >>>> >>>> you could change it to: >>>> >>>>>>>>> >>>> #!/bin/sh >>>> #PBS -N hello >>>> #PBS -l nodes=1:ppn=8 >>>> #PBS -l walltime=0:00:30 >>>> #PBS -j oe >>>> >>>> cd $PBS_O_WORKDIR >>>> mpiexec ./ex1 >>>> mpiexec ./ex2 >>>> mpiexec ./ex1 >>>> mpiexec ./ex2 >>>> <<<<<<< >>>> >>>> and run it with a single allocation.. >>>> >>>> Satish >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments >>> is infinitely more interesting than any results to which their experiments >>> lead. >>> -- Norbert Wiener >>> >>> >>> Now it is really hard for me to understand what the problem is since both >>> 3.1 and dev check for this >>> function in the same way. Moreover, it does not depend on shared libraries. >>> >>> Satish, have you seen this error before on the XT5? >>> >>> Matt >>> >>> On Mon, Apr 4, 2011 at 6:45 PM, Satish Balay <petsc-maint(a)mcs.anl.gov>wrote: >>> >>>> On Mon, 4 Apr 2011, Sebastian Steiger wrote: >>>> >>>>> On 04/04/2011 05:42 PM, Satish Balay wrote: >>>>>> Could you run both the binaries in the same node-allocation - with a >>>>>> single batch file and send the '-log_summary' for them? >>>>>> run petsc-dev >>>>>> run petsc-31 >>>>>> run petsc-dev >>>>>> run petsc-31 >>>>> Do you mean exactly the same physical nodes? I don't know how to do >>>>> that. My batch files for petsc-dev and petsc-3.1-p4 are identical except >>>>> for the static executable. >>>> >>>> >>>> Haven't used batch stuff on ornl machine - but if the usual batch >>>> script file is something like: [for eg: pbs] >>>> >>>>>>> >>>> #!/bin/sh >>>> #PBS -N hello >>>> #PBS -l nodes=1:ppn=8 >>>> #PBS -l walltime=0:00:15 >>>> #PBS -j oe >>>> >>>> cd $PBS_O_WORKDIR >>>> mpiexec ./ex1 >>>> <<<<< >>>> >>>> you could change it to: >>>> >>>>>>>>> >>>> #!/bin/sh >>>> #PBS -N hello >>>> #PBS -l nodes=1:ppn=8 >>>> #PBS -l walltime=0:00:30 >>>> #PBS -j oe >>>> >>>> cd $PBS_O_WORKDIR >>>> mpiexec ./ex1 >>>> mpiexec ./ex2 >>>> mpiexec ./ex1 >>>> mpiexec ./ex2 >>>> <<<<<<< >>>> >>>> and run it with a single allocation.. >>>> >>>> Satish >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> >> >> > >

1 0

Re: [petsc-dev] [petsc-users] Is there a way to do row/column scaling of jacobian matrix
by Barry Smith 05 Apr '11

05 Apr '11

We should write a FAQ question that addresses all theses issues. I admit I don't really want to add these scalings to PETSc it makes the code more complicated, unless we really need it. Barry On Apr 5, 2011, at 1:24 AM, Jed Brown wrote: > On Tue, Apr 5, 2011 at 06:10, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > They are done to change the relative importance of different entries in the vector to determine stopping conditions and search directions in Newton's method. > > Note that weighting fields differently is equivalent to choosing units for the different fields. I think it is generally a good idea to make the units a runtime option anyway since it lets you check that the code is dimensionally correct.

1 0

Re: [petsc-dev] Preconditioner for Schur complement (pcfieldsplit)
by Thomas Witkowski 05 Apr '11

05 Apr '11

Jed Brown wrote: > On Tue, Mar 29, 2011 at 15:29, Thomas Witkowski > <thomas.witkowski(a)tu-dresden.de > <mailto:[email protected]>> wrote: > > Is there any example which makes use of it? > > > src/snes/examples/tutorials/ex28.c is a (contrived) multi-physics > prototype that uses MatNest tranparently if you run with > -pack_dm_mat_type nest (currently, you should add -pc_type fieldsplit > -pc_fieldsplit_type additive because off-diagonal blocks are not > automatically preallocated). > > There are also two tests that use the low-level API: > > $ grep -l MatNest src/**/tests/*.c > src/ksp/ksp/examples/tests/ex22.c > src/snes/examples/tests/ex17.c So, I reimplemented my code to assemble the matrices to a MatNest matrix. This seems to work fine. But how to define the FieldSplits? I did it in the following way: KSP solver; PC pc; KSPCreate(PETSC_COMM_WORLD, &solver); KSPGetPC(solver, &pc); const PetscInt interiorField[1] = {0}; const PetscInt boundaryField[1] = {1}; PCFieldSplitSetFields(pc, "interior", 1, interiorField); PCFieldSplitSetFields(pc, "boundary", 1, boundaryField); When I run my code with the options "-ksp_type preonly -pc_type fieldsplit -pc_fieldsplit_type schur", I get the following errors in KSPSolve: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Petsc has generated inconsistent data! [0]PETSC ERROR: Unhandled case, must have at least two fields! Is the creation of the split wrong? If not, what is a good way to figure out the problem? Thomas

1 0

Re: [petsc-dev] Compiling C code with C compiler
by Jed Brown 05 Apr '11

05 Apr '11

On Tue, Apr 5, 2011 at 04:33, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > Since the sieve c++ compiled code uses the C code (like VecXXX) the C++ > compiler would have to know that the other parts of C are not compiled with > C++. At a minimum you would have to extern c it in the header files other > wise the C++ compiled code will call the C routines with name-mangling > turned on while the C compiler doesn't know about name mangling and hence > would generate routines without name mangling and the c++ code wouldn't find > the c routines. But this is already what happens --with-c-support. Why is it ever necessary to compile the C source with a C++ compiler (that is, why support with-c-support=0)? The C++ convenience functions (PetscPolymorphicFunction) can easily be defined whenever a C++ compiler is used, they will be name mangled (and usually inlined). I think all you would need to do is remove the PETSC_USE_EXTERN_CXX clause from #if !defined(PETSC_USE_EXTERN_CXX) && defined(__cplusplus) > In other words, even without issues of struct layouts currently everything > in PETSc needs to be built with the same compiler. If sieve was made a > completely separate package form PETSc then one could easily arrange things > so that sieve compiled with c++ and PETSc with c. I don't see any complexity in always building C source with a C compiler and C++ source with a C++ compiler. It seems simpler to me. And then people could build a PETSc that includes Sieve and still call it from C (e.g. when they happen to not be using Sieve). The only problem I can think of, and which might be justification for building C code with a C++ compiler, is that C++ compilers seem to be stricter about complex. C99 complex is binary compatible with C++ complex, but type checking is weaker. I don't know if this is implicit in the standard or if the GNU compilers are unnecessarily lenient about C99 complex. This is purely a matter of quality error/warning messages, but I'm loathe to give up any static type checking.

1 0

Re: [petsc-dev] Compiling C code with C compiler
by Jed Brown 05 Apr '11

05 Apr '11

On Tue, Apr 5, 2011 at 05:22, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > But Matt aparently forgot his laptop when he went out to the bar tonight so > hasn't seen the discussion yet. You must not be aware that Matt does not go to the bar anymore. He must have forgotten to bring his laptop to "date night".

1 0

Re: [petsc-dev] Compiling C code with C compiler
by Jed Brown 05 Apr '11

05 Apr '11

On Tue, Apr 5, 2011 at 04:24, Dmitry Karpeev <karpeev(a)mcs.anl.gov> wrote: > I don't know whether C++ does the same (there is the issue of where it > sticks the vtable, and I'm not sure what else). > I imagine there will be no problem, since we've been compiling the PETSc C > code with C++ compilers for a decade now, > but I don't know whether the C++ standard guarantees that. Jed is the only > person I know that actually reads the standard > (or maybe it's part of the ABI). > As far as the implementation, there is no vtable because we don't put C++ virtual methods in the struct. The standard does not say anything about the existence of a vtable, but it does make the same sort of guarantees about struct layout. I do not know if you can guarantee that a struct within an extern "C" block is binary compatible with a struct outside an extern "C" block that does not happen to contain a vtable.

1 0

Re: [petsc-dev] Compiling C code with C compiler
by Barry Smith 04 Apr '11

04 Apr '11

On Apr 4, 2011, at 9:50 PM, Lisandro Dalcin wrote: > On 4 April 2011 23:46:38 UTC-3, Dmitry Karpeev <karpeev(a)mcs.anl.gov> wrote: >> >> >> On Mon, Apr 4, 2011 at 9:33 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >>> >>> On Apr 4, 2011, at 9:25 PM, Sean Farley wrote: >>> >>>> What did you expect when you used --with-clanguage=C++ for your buid? Or >>>> were you misunderstanding what this options means wrt PETSc build? >>>> >>>> I understand what it was trying to do but it is necessary for sieve >>>> (especially with certain mesh generators). Why does all of PETSc need to be >>>> compiled with C++ in order to use sieve is the real question? >>> >>> Since the sieve c++ compiled code uses the C code (like VecXXX) the C++ >>> compiler would have to know that the other parts of C are not compiled with >>> C++. At a minimum you would have to extern c it in the header files other >>> wise the C++ compiled code will call the C routines with name-mangling >>> turned on while the C compiler doesn't know about name mangling and hence >>> would generate routines without name mangling and the c++ code wouldn't find >>> the c routines. In other words, even without issues of struct layouts >>> currently everything in PETSc needs to be built with the same compiler. If >>> sieve was made a completely separate package form PETSc then one could >>> easily arrange things so that sieve compiled with c++ and PETSc with c. >>> >>> Why is everyone talking about this unimportant issue when we got serious >>> dlopen() issues to fix? >> >> It seems to me that on Jaguar passing in a 0 handle should work, since >> that's what 3.1 does (I'm not sure about 3.1-p4, though). No, in 3.1 it only ever does the dlopen() if --with-dynamic flag is set. In petsc-dev if BuildSystem "thinks" that dlopen() works it will do dlopen(). >> Also, if we are interested in comparing performance, why not just build 3.1 >> without shared? >> Dmitry. >> > > Sorry, what dlopen() issues are you talking about? The culprit himself :-) Apparently dlopen(0,flags) doesn't work on some systems but BuildSystem has no check for this (it only checks that dlopen(somelibrary,flags) works). Your code that always turns on PETSC_HAVE_DYNAMIC_LIBRARIES causes the dlopen(0,flags) to be used and hence cause grief. The fix is, of course, to add the additional check to BuildSystem. But Matt aparently forgot his laptop when he went out to the bar tonight so hasn't seen the discussion yet. Barry > > > -- > Lisandro Dalcin > --------------- > CIMEC (INTEC/CONICET-UNL) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > 3000 Santa Fe, Argentina > Tel: +54-342-4511594 (ext 1011) > Tel/Fax: +54-342-4511169

1 0