On Apr 5, 2011, at 8:09 AM, Dmitry Karpeev wrote:
On Tue, Apr 5, 2011 at 7:31 AM, Barry Smith <[email protected]> wrote:
On Apr 5, 2011, at 1:31 AM, Satish Balay wrote:
On Mon, 4 Apr 2011, Dmitry Karpeev wrote:
What is the patch level petsc-3.1 being used? The dlopen() code that generates the error was added to petsc-dev after the initial release of 3.1.
Typically (on many systems) dlopen(NULL,dlflags) returns the handle of the main executable, as if it were a dynamically-loaded library, and dlsym() is then run on that handle. This doesn't work on all systems, though (e.g., OS X defines a specific handle for the main executable; enabling that means augmenting configure). In particular, on this system, dlopen() with a NULL first argument appears to throw a confusing error.
The original release of petsc-3.1 would use dlsym(0) in this case, skipping dlopen(NULL,dlflags), so that is, perhaps, what works on XT5. To revert to that behavior it should be sufficient to apply the following patch to petsc-dev/src/sys/dll/dlimpl.c (or simply to remove lines 269 through 308 inclusive).
For One - I don't think any of the default PETSc code should be doing dlopen(executalbe) - if --with-dynamic-loading=0 is set [Otherwise why have this option? suppress dlopen(libpetsc.so) - but enable dlopen(executable)?]
So I've changed the fix below and disable it with:
#if defined(PETSC_HAVE_DLOPEN) && defined(PETSC_USE_DYNAMIC_LIBRARIES)
wrt dlsym(0) - we ignore the dlerror() from it - so I guess its ok..
This works fine on the cray - so pusing this change.
[I guess we still need to fix this for --with-dynamic-loading=1]
I thought about this also, but it is the wrong fix. HAVE_DYNAMIC means that dynamic libraries work and the user can pass strings as functions; USE_DYNAMIC means make PETSc libraries dynamic and always use strings for functions. The problem is only that we do not have the configure tests in places for various versions of dlopen() on the executable. The correct fix is to make configure tests for dlopen() and change the PETSc code to do the various dlopen(0,...) variants based on the configure test. Now we can punt on doing it correctly, but we'd be punting only out of lazyness, not because it is the right model.
BTW: someone should submit a bug report to Cray (and IBM if it doesn't work there also).
Okay, but what's going on with petsc-3.1 on that machine? It doesn't fail. Is that because dlsym(0,string) works? Or was dynamic loading disabled in it?
As I explained in my previous mail, before petsc-dev there was no PETSC_HAVE_DYNAMIC_LIBRARIES only PETSC_USE_DYNAMIC_LIBRARIES so PETSc did not try to dlopen(0) unless --with-dynamic was passed in and hence the error in the Cray went undetected. Barry
Dmitry.
Barry
Satish
Dmitry.
@@ -266,46 +266,7 @@ } else { dlhandle = (dlhandle_t) 0; - -#if defined(PETSC_HAVE_DLOPEN) - /* Attempt to retrieve the main executable's dlhandle. */ - { int dlflags1 = 0, dlflags2 = 0; -#if defined(PETSC_HAVE_RTLD_LAZY) - dlflags1 = RTLD_LAZY; -#endif - if(!dlflags1) { -#if defined(PETSC_HAVE_RTLD_NOW) - dlflags1 = RTLD_NOW; -#endif - } -#if defined(PETSC_HAVE_RTLD_LOCAL) - dlflags2 = RTLD_LOCAL; -#endif - if(!dlflags2) { -#if defined(PETSC_HAVE_RTLD_GLOBAL) - dlflags2 = RTLD_GLOBAL; -#endif - } -#if defined(PETSC_HAVE_DLERROR) -#if defined(PETSC_HAVE_VALGRIND) - if (!(RUNNING_ON_VALGRIND)) { -#endif - dlerror(); /* clear any previous error; valgrind does not like this */ -#if defined(PETSC_HAVE_VALGRIND) - } -#endif -#endif - /* Attempt to open the main executable as a dynamic library. */ - dlhandle = dlopen(0, dlflags1|dlflags2); - } -#if defined(PETSC_HAVE_DLERROR) - { const char *e = (const char*) dlerror(); - if(e){ - SETERRQ1(PETSC_COMM_SELF, PETSC_ERR_ARG_WRONG, "Error opening main executable as a dynamic library:\n Error message from dlopen(): '%s'\n", e); - } - } -#endif -#endif /* PETSC_HAVE_DLOPEN */ + } #if defined(PETSC_HAVE_DLERROR) dlerror(); /* clear any previous error */
On Mon, Apr 4, 2011 at 7:26 PM, Matthew Knepley <[email protected]>wrote:
Now it is really hard for me to understand what the problem is since both 3.1 and dev check for this function in the same way. Moreover, it does not depend on shared libraries.
Satish, have you seen this error before on the XT5?
Matt
On Mon, Apr 4, 2011 at 6:45 PM, Satish Balay <[email protected]
wrote:
On Mon, 4 Apr 2011, Sebastian Steiger wrote:
On 04/04/2011 05:42 PM, Satish Balay wrote:
Could you run both the binaries in the same node-allocation - with a single batch file and send the '-log_summary' for them? run petsc-dev run petsc-31 run petsc-dev run petsc-31 Do you mean exactly the same physical nodes? I don't know how to do that. My batch files for petsc-dev and petsc-3.1-p4 are identical except for the static executable.
Haven't used batch stuff on ornl machine - but if the usual batch script file is something like: [for eg: pbs]
#!/bin/sh #PBS -N hello #PBS -l nodes=1:ppn=8 #PBS -l walltime=0:00:15 #PBS -j oe
cd $PBS_O_WORKDIR mpiexec ./ex1 <<<<<
you could change it to:
#!/bin/sh #PBS -N hello #PBS -l nodes=1:ppn=8 #PBS -l walltime=0:00:30 #PBS -j oe
cd $PBS_O_WORKDIR mpiexec ./ex1 mpiexec ./ex2 mpiexec ./ex1 mpiexec ./ex2 <<<<<<<
and run it with a single allocation..
Satish
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
Now it is really hard for me to understand what the problem is since both 3.1 and dev check for this function in the same way. Moreover, it does not depend on shared libraries.
Satish, have you seen this error before on the XT5?
Matt
On Mon, Apr 4, 2011 at 6:45 PM, Satish Balay <[email protected]>wrote:
On Mon, 4 Apr 2011, Sebastian Steiger wrote:
On 04/04/2011 05:42 PM, Satish Balay wrote:
Could you run both the binaries in the same node-allocation - with a single batch file and send the '-log_summary' for them? run petsc-dev run petsc-31 run petsc-dev run petsc-31 Do you mean exactly the same physical nodes? I don't know how to do that. My batch files for petsc-dev and petsc-3.1-p4 are identical except for the static executable.
Haven't used batch stuff on ornl machine - but if the usual batch script file is something like: [for eg: pbs]
#!/bin/sh #PBS -N hello #PBS -l nodes=1:ppn=8 #PBS -l walltime=0:00:15 #PBS -j oe
cd $PBS_O_WORKDIR mpiexec ./ex1 <<<<<
you could change it to:
#!/bin/sh #PBS -N hello #PBS -l nodes=1:ppn=8 #PBS -l walltime=0:00:30 #PBS -j oe
cd $PBS_O_WORKDIR mpiexec ./ex1 mpiexec ./ex2 mpiexec ./ex1 mpiexec ./ex2 <<<<<<<
and run it with a single allocation..
Satish
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener