April 2011 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] PetscOptionsGetString
by Satish Balay 15 Apr '11

15 Apr '11

Ah - missed the 'one char' part.. I guess you'll have to do: char Mode [2]; PetscOptionsGetString(PETSC_NULL,"-Mode",Mode,2,&flag); (and use Mode[0]) Satish On Fri, 15 Apr 2011, Satish Balay wrote: > Hm - if you have: > > char Mode [64]; > > you should be using: > > PetscOptionsGetString(PETSC_NULL,"-Mode",&Mode,64,&flag); > > You can check an example - say src/ksp/ksp/examples/tutorials/ex10.c > > Satish > > > On Fri, 15 Apr 2011, Shiyuan wrote: > > > Hi, > > In the version of 3.1-p5, PetscOptionsGetString() can be used to get a char > > input. It seems that it doesn't work any more in development version. > > PetscOptionsGetString(PETSC_NULL,"-Mode",&Mode,1,&flag); > > This gives me a \0. How can I get the input of only one character in dev? > > Thanks. > > > >

1 0

Re: [petsc-dev] PetscOptionsGetString
by Satish Balay 15 Apr '11

15 Apr '11

Hm - if you have: char Mode [64]; you should be using: PetscOptionsGetString(PETSC_NULL,"-Mode",&Mode,64,&flag); You can check an example - say src/ksp/ksp/examples/tutorials/ex10.c Satish On Fri, 15 Apr 2011, Shiyuan wrote: > Hi, > In the version of 3.1-p5, PetscOptionsGetString() can be used to get a char > input. It seems that it doesn't work any more in development version. > PetscOptionsGetString(PETSC_NULL,"-Mode",&Mode,1,&flag); > This gives me a \0. How can I get the input of only one character in dev? > Thanks. >

1 0

PetscOptionsGetString
by Shiyuan 15 Apr '11

15 Apr '11

Hi, In the version of 3.1-p5, PetscOptionsGetString() can be used to get a char input. It seems that it doesn't work any more in development version. PetscOptionsGetString(PETSC_NULL,"-Mode",&Mode,1,&flag); This gives me a \0. How can I get the input of only one character in dev? Thanks.

1 0

Isogeometric elements
by Matthew Knepley 15 Apr '11

15 Apr '11

Check out the very cool isogeometric simulation of Navier-Stokes-Korteweg from Nathan Collier at KAUST. cd src/ts/examples/tutorials make ex18 ./ex18 -snes_mf -ts_monitor_solution -ts_monitor -snes_monitor He has a 2D and 3D Cahn-Hilliard which I will get converted soon. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] mvapich and petsc-dev
by Ethan Coon 14 Apr '11

14 Apr '11

On Thu, 2011-04-14 at 11:11 -0500, Satish Balay wrote: > Can you run the mvapich codde with -vecscatter_alltoall and see if > it goes through? > > satish Hmm, this seems to be breaking in a different way -- it gets to what I suspect is the same spot and then just hangs, waiting for a communication I suspect. No seg fault, but something is still breaking. On Thu, 2011-04-14 at 11:11 -0500, Barry Smith wrote: Ethan, > > First valgrind the heck out of the code on a system where you can and make sure it is completely clean. > It's clean except for a few string problems -- like setting names of things with Fortran using TRIM() > On the bad system see if the code crashes with no optimization turned on. Yep. > > Does it always crash at the same place? Or seemingly at some random place. If the same place can you do some kind of restart file so that it crashes soon after you start instead of after many time-steps. > Yes it does, I'll dump out a restart file which helps a bit. We have some debugging support on this machine, but I'm having issues getting totalview to play nicely with mvapich. Will bug people here. On another note, I found a valgrind hidden in a generic tools module, and am getting some new memcheck errors from mvapich. First, in the initialization stage of the program: ==18207== Conditional jump or move depends on uninitialised value(s) ==18207== at 0xC3D7AD: intra_shmem_Allreduce (in /usr/projects/porescale/research-lbm/hybrid-lbm-pflotran-lobo/tests/fracture-micromodel2/runLBMSimulation) ==18207== by 0xC32827: PMPI_Allreduce (in /usr/projects/porescale/research-lbm/hybrid-lbm-pflotran-lobo/tests/fracture-micromodel2/runLBMSimulation) ==18207== by 0x6F503A: PetscSplitOwnership (in /usr/projects/porescale/research-lbm/hybrid-lbm-pflotran-lobo/tests/fracture-micromodel2/runLBMSimulation) ==18207== by 0x51B7B5: PetscLayoutSetUp (in /usr/projects/porescale/research-lbm/hybrid-lbm-pflotran-lobo/tests/fracture-micromodel2/runLBMSimulation) ==18207== by 0x505026: VecCreate_Seq_Private (in /usr/projects/porescale/research-lbm/hybrid-lbm-pflotran-lobo/tests/fracture-micromodel2/runLBMSimulation) ==18207== by 0x50D87D: VecCreate_Seq (in /usr/projects/porescale/research-lbm/hybrid-lbm-pflotran-lobo/tests/fracture-micromodel2/runLBMSimulation) ==18207== by 0x4F8885: VecSetType (in /usr/projects/porescale/research-lbm/hybrid-lbm-pflotran-lobo/tests/fracture-micromodel2/runLBMSimulation) ==18207== by 0x510163: VecCreate_Standard (in /usr/projects/porescale/research-lbm/hybrid-lbm-pflotran-lobo/tests/fracture-micromodel2/runLBMSimulation) ==18207== by 0x4F8885: VecSetType (in /usr/projects/porescale/research-lbm/hybrid-lbm-pflotran-lobo/tests/fracture-micromodel2/runLBMSimulation) ==18207== by 0x594941: DMCreateLocalVector_DA (in /usr/projects/porescale/research-lbm/hybrid-lbm-pflotran-lobo/tests/fracture-micromodel2/runLBMSimulation) ==18207== by 0x4A0F65: DMCreateLocalVector (in /usr/projects/porescale/research-lbm/hybrid-lbm-pflotran-lobo/tests/fracture-micromodel2/runLBMSimulation) ==18207== by 0x4A3E56: dmcreatelocalvector_ (in /usr/projects/porescale/research-lbm/hybrid-lbm-pflotran-lobo/tests/fracture-micromodel2/runLBMSimulation) ==18207== And then, when the error actually occurs: [7]PETSC ERROR: ------------------------------------------------------------------------ [7]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[7]PET… ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [7]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [7]PETSC ERROR: to get more information on the crash. ==18207== Invalid read of size 4 ==18207== at 0xC552B5: MPID_IsendContig (in /usr/projects/porescale/research-lbm/hybrid-lbm-pflotran-lobo/tests/fracture-micromodel2/runLBMSimulation) ==18207== by 0xC59770: MPID_IsendDatatype (in /usr/projects/porescale/research-lbm/hybrid-lbm-pflotran-lobo/tests/fracture-micromodel2/runLBMSimulation) ==18207== by 0xC2209F: PMPI_Start (in /usr/projects/porescale/research-lbm/hybrid-lbm-pflotran-lobo/tests/fracture-micromodel2/runLBMSimulation) ==18207== by 0x4E1FD0: VecScatterBegin_1 (in /usr/projects/porescale/research-lbm/hybrid-lbm-pflotran-lobo/tests/fracture-micromodel2/runLBMSimulation) ==18207== by 0x4DBB1F: VecScatterBegin (in /usr/projects/porescale/research-lbm/hybrid-lbm-pflotran-lobo/tests/fracture-micromodel2/runLBMSimulation) ==18207== by 0x592350: DMDALocalToLocalBegin (in /usr/projects/porescale/research-lbm/hybrid-lbm-pflotran-lobo/tests/fracture-micromodel2/runLBMSimulation) ==18207== by 0x49D94E: dmdalocaltolocalbegin_ (in /usr/projects/porescale/research-lbm/hybrid-lbm-pflotran-lobo/tests/fracture-micromodel2/runLBMSimulation) ==18207== by 0x415B7C: lbm_distribution_function_module_distributioncommunicatedensity_ (in /usr/projects/porescale/research-lbm/hybrid-lbm-pflotran-lobo/tests/fracture-micromodel2/runLBMSimulation) ==18207== by 0x42B27A: lbm_module_lbmrun_ (in /usr/projects/porescale/research-lbm/hybrid-lbm-pflotran-lobo/tests/fracture-micromodel2/runLBMSimulation) ==18207== by 0x42FB06: MAIN_ (in /usr/projects/porescale/research-lbm/hybrid-lbm-pflotran-lobo/tests/fracture-micromodel2/runLBMSimulation) ==18207== by 0x40D6DF: main (in /usr/projects/porescale/research-lbm/hybrid-lbm-pflotran-lobo/tests/fracture-micromodel2/runLBMSimulation) ==18207== Address 0x38cd9980 is 3390064 bytes inside data symbol "temporary" ==18207== Surely this is an mvapich bug? Thanks, Ethan > My guess is the problem is a combination of the mvapich and possibly the hardware. Maybe bug the systems people about upgrades on the system? > > Barry > > > On Thu, 14 Apr 2011, Ethan Coon wrote: > > > I'm a bit grasping at straws here, because I'm completely stymied, so > > please bear with me. > > > > > > I'm running a program in two locations -- on local workstations with > > mpich2 and on a supercomputer with mvapich. > > > > On the workstation, the program runs, in all cases I've tested, > > including 8 processes (the number of cores), and up to 64 processes > > (multiple procs per core). > > > > On the supercomputer, it runs on 16 cores (one full node). With 64 > > cores, it seg-faults and core dumps many timesteps in to the > > simulation. > > > > Using a debugger, a debug-enabled petsc-dev, but with no access to > > debugging symbols in the mvapich installation, I've looked at the core. > > It appears to dump during VecScatterBegin_1 (within a DMDALocalToLocal() > > with xin = xout). The Vec I pass in as both input and output appears > > normal. > > > > The stack looks something like: > > > > MPIR_HBT_lookup, FP=7fff1010f740 > > PMPI_Attr_get, FP=7fff1010f780 > > PetscCommDuplicate, FP=7fff1010f7d0 > > PetscViewerASCIIGetStdout, FP=7fff1010f800 > > PETSC_VIEWER_STDOUT_, FP=7fff1010f820 > > PetscDefaultSignalHandler, FP=7fff1010fa70 > > PetscSignalHandler_Private, FP=7fff1010fa90 > > **** Signal Stack Frame ****************** > > MPID_IsendContig, FP=7fff1010ff20 > > MPID_IsendDatatype, FP=7fff1010ffa0 > > PMPI_Start, FP=7fff1010fff0 > > VecScatterBegin_1, FP=7fff10110080 > > VecScatterBegin, FP=7fff101100e0 > > DMDALocalToLocalBegin, FP=7fff10110120 > > dmdalocaltolocalbegin_, FP=7fff10110160 > > > > > > Has anyone run into anything like this before? I have no clue even how > > to proceed, and I doubt this is a PETSc problem, but I figured you guys > > might have enough experience in these types of issues to know where to > > look from here... > > > > Thanks, > > > > Ethan > > > > > > > -- ------------------------------------ Ethan Coon Post-Doctoral Researcher Applied Mathematics - T-5 Los Alamos National Laboratory 505-665-8289 http://www.ldeo.columbia.edu/~ecoon/ ------------------------------------

1 0

Re: [petsc-dev] mvapich and petsc-dev
by Satish Balay 14 Apr '11

14 Apr '11

Can you run the mvapich codde with -vecscatter_alltoall and see if it goes through? satish On Thu, 14 Apr 2011, Ethan Coon wrote: > I'm a bit grasping at straws here, because I'm completely stymied, so > please bear with me. > > > I'm running a program in two locations -- on local workstations with > mpich2 and on a supercomputer with mvapich. > > On the workstation, the program runs, in all cases I've tested, > including 8 processes (the number of cores), and up to 64 processes > (multiple procs per core). > > On the supercomputer, it runs on 16 cores (one full node). With 64 > cores, it seg-faults and core dumps many timesteps in to the > simulation. > > Using a debugger, a debug-enabled petsc-dev, but with no access to > debugging symbols in the mvapich installation, I've looked at the core. > It appears to dump during VecScatterBegin_1 (within a DMDALocalToLocal() > with xin = xout). The Vec I pass in as both input and output appears > normal. > > The stack looks something like: > > MPIR_HBT_lookup, FP=7fff1010f740 > PMPI_Attr_get, FP=7fff1010f780 > PetscCommDuplicate, FP=7fff1010f7d0 > PetscViewerASCIIGetStdout, FP=7fff1010f800 > PETSC_VIEWER_STDOUT_, FP=7fff1010f820 > PetscDefaultSignalHandler, FP=7fff1010fa70 > PetscSignalHandler_Private, FP=7fff1010fa90 > **** Signal Stack Frame ****************** > MPID_IsendContig, FP=7fff1010ff20 > MPID_IsendDatatype, FP=7fff1010ffa0 > PMPI_Start, FP=7fff1010fff0 > VecScatterBegin_1, FP=7fff10110080 > VecScatterBegin, FP=7fff101100e0 > DMDALocalToLocalBegin, FP=7fff10110120 > dmdalocaltolocalbegin_, FP=7fff10110160 > > > Has anyone run into anything like this before? I have no clue even how > to proceed, and I doubt this is a PETSc problem, but I figured you guys > might have enough experience in these types of issues to know where to > look from here... > > Thanks, > > Ethan > > >

1 0

Re: [petsc-dev] mvapich and petsc-dev
by Barry Smith 14 Apr '11

14 Apr '11

Ethan, First valgrind the heck out of the code on a system where you can and make sure it is completely clean. On the bad system see if the code crashes with no optimization turned on. Does it always crash at the same place? Or seemingly at some random place. If the same place can you do some kind of restart file so that it crashes soon after you start instead of after many time-steps. My guess is the problem is a combination of the mvapich and possibly the hardware. Maybe bug the systems people about upgrades on the system? Barry On Apr 14, 2011, at 11:04 AM, Ethan Coon wrote: > I'm a bit grasping at straws here, because I'm completely stymied, so > please bear with me. > > > I'm running a program in two locations -- on local workstations with > mpich2 and on a supercomputer with mvapich. > > On the workstation, the program runs, in all cases I've tested, > including 8 processes (the number of cores), and up to 64 processes > (multiple procs per core). > > On the supercomputer, it runs on 16 cores (one full node). With 64 > cores, it seg-faults and core dumps many timesteps in to the > simulation. > > Using a debugger, a debug-enabled petsc-dev, but with no access to > debugging symbols in the mvapich installation, I've looked at the core. > It appears to dump during VecScatterBegin_1 (within a DMDALocalToLocal() > with xin = xout). The Vec I pass in as both input and output appears > normal. > > The stack looks something like: > > MPIR_HBT_lookup, FP=7fff1010f740 > PMPI_Attr_get, FP=7fff1010f780 > PetscCommDuplicate, FP=7fff1010f7d0 > PetscViewerASCIIGetStdout, FP=7fff1010f800 > PETSC_VIEWER_STDOUT_, FP=7fff1010f820 > PetscDefaultSignalHandler, FP=7fff1010fa70 > PetscSignalHandler_Private, FP=7fff1010fa90 > **** Signal Stack Frame ****************** > MPID_IsendContig, FP=7fff1010ff20 > MPID_IsendDatatype, FP=7fff1010ffa0 > PMPI_Start, FP=7fff1010fff0 > VecScatterBegin_1, FP=7fff10110080 > VecScatterBegin, FP=7fff101100e0 > DMDALocalToLocalBegin, FP=7fff10110120 > dmdalocaltolocalbegin_, FP=7fff10110160 > > > Has anyone run into anything like this before? I have no clue even how > to proceed, and I doubt this is a PETSc problem, but I figured you guys > might have enough experience in these types of issues to know where to > look from here... > > Thanks, > > Ethan > > > -- > ------------------------------------ > Ethan Coon > Post-Doctoral Researcher > Applied Mathematics - T-5 > Los Alamos National Laboratory > 505-665-8289 > > http://www.ldeo.columbia.edu/~ecoon/ > ------------------------------------ >

1 0

mvapich and petsc-dev
by Ethan Coon 14 Apr '11

14 Apr '11

I'm a bit grasping at straws here, because I'm completely stymied, so please bear with me. I'm running a program in two locations -- on local workstations with mpich2 and on a supercomputer with mvapich. On the workstation, the program runs, in all cases I've tested, including 8 processes (the number of cores), and up to 64 processes (multiple procs per core). On the supercomputer, it runs on 16 cores (one full node). With 64 cores, it seg-faults and core dumps many timesteps in to the simulation. Using a debugger, a debug-enabled petsc-dev, but with no access to debugging symbols in the mvapich installation, I've looked at the core. It appears to dump during VecScatterBegin_1 (within a DMDALocalToLocal() with xin = xout). The Vec I pass in as both input and output appears normal. The stack looks something like: MPIR_HBT_lookup, FP=7fff1010f740 PMPI_Attr_get, FP=7fff1010f780 PetscCommDuplicate, FP=7fff1010f7d0 PetscViewerASCIIGetStdout, FP=7fff1010f800 PETSC_VIEWER_STDOUT_, FP=7fff1010f820 PetscDefaultSignalHandler, FP=7fff1010fa70 PetscSignalHandler_Private, FP=7fff1010fa90 **** Signal Stack Frame ****************** MPID_IsendContig, FP=7fff1010ff20 MPID_IsendDatatype, FP=7fff1010ffa0 PMPI_Start, FP=7fff1010fff0 VecScatterBegin_1, FP=7fff10110080 VecScatterBegin, FP=7fff101100e0 DMDALocalToLocalBegin, FP=7fff10110120 dmdalocaltolocalbegin_, FP=7fff10110160 Has anyone run into anything like this before? I have no clue even how to proceed, and I doubt this is a PETSc problem, but I figured you guys might have enough experience in these types of issues to know where to look from here... Thanks, Ethan -- ------------------------------------ Ethan Coon Post-Doctoral Researcher Applied Mathematics - T-5 Los Alamos National Laboratory 505-665-8289 http://www.ldeo.columbia.edu/~ecoon/ ------------------------------------

1 0

dlopen(0,) business
by Barry Smith 13 Apr '11

13 Apr '11

Since no one else showed any interest I modified dlimpl.c to use RTLD_DEFAULT if available and only use dlopen(0,) if RTLD_DEFAULT is not available. Not well tested, in particular I do not know what will happen on the Cray, but at least the new code is better than the old and it no longer has Satish's bad hack of requiring USE_DYNAMIC_LIBRARIES Barry

1 0

Re: [petsc-dev] memory leaks in BCGSL when using KSPReset()
by Lisandro Dalcin 12 Apr '11

12 Apr '11

On 12 April 2011 10:26, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > On Apr 11, 2011, at 10:02 PM, Lisandro Dalcin wrote: > >> After adding this test in petsc4py-dev/test/test_ksp.py : >> >> def testResetAndSolve(self): >> self.ksp.reset() >> self.testSolve() >> self.ksp.reset() >> self.testSolve() >> self.ksp.reset() >> >> it seems there is some memory bleeding: >> >> [0]Total space allocated 256 bytes >> [ 0]80 bytes KSPSetUp_BCGSL() line 476 in >> /home/devel/petsc/dev/src/ksp/ksp/impls/bcgsl/bcgsl.c >> [0] KSPSetUp() line 184 in >> /home/devel/petsc/dev/src/ksp/ksp/interface/itfunc.c >> [0] KSPSolve() line 332 in >> /home/devel/petsc/dev/src/ksp/ksp/interface/itfunc.c >> >> These are workspace arrays that are re-allocated at every KSPSetup_BCGSL call. >> >> Not sure about the proper way to fix this, but I would deallocate the >> arrays at KSPReset_BCGSL. Can anyone help me with this? >> > This could be a problem in many places: for example if eigenvalue computing is done with CG then I think those arrays will be reallocated. > Indeed. I'm not testing eigenvalue computing. I could try to add the options to my testsuite and see what happens. > Previously I guess KSPSetUp_XXX() never had a reason to be called more than once, with reset now it is. So that means that all arrays allocated in KSPSetUp_XXX() should be freed in KSPReset_XXX()? If this is correct then it is easily fixed by moving them from KSPDestroy_XXX(). > Well, In my latest push I was lazy enough as to not remove the PetscFree() call to Reset_XXX. After all, PetscFree() nullifies the pointer, so we are safe. If you want me to fix this, let me know. PS: Could any of you check BCGLS for a few "ksp->setupstage = KSP_SETUP_NEW;" lines? That really smells, such hack forces a full (not just in the subype) KSPSetup() in the next KSPSolve(). -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169

1 0