HMPI usage in PETSc relies on sending function pointer addresses between MPI processes; this use to be fine since you start up several copies of the same executable on a machine each has the same virtual addresses for instructions. Yes, this a bit hacky but it worked great. With ASLR this model breaks. On Linux I found a way supposedly to turn off ASLR using sysctl -w kernel.randomize_va_space=0 but on Apple there does not seem to be a system wide way to turn it off easily. What to do? It looks like I have to switch to requiring shared libraries and passing string function names and dynamically access them. Any thoughts on alternatives. Barry Begin forwarded message:
From: Barry Smith <[email protected]> Subject: Re: [petsc-users] pctype hmpi Date: August 31, 2012 1:17:07 PM CDT To: PETSc users list <[email protected]> Reply-To: PETSc users list <[email protected]>
On Aug 30, 2012, at 10:02 PM, George Pau <[email protected]> wrote:
Hi Barry,
I tried with the addition of
-hmpi_spawn_size 3
but I am still getting the same error though.
The EXACT same error? Or some other error?
What happens if you run with the -hmpi_merge_size <size> option instead?
Barry
1) I am getting a crash with the spawn version I suspect is due to bugs in the MPICH version I am using related to spawn.
2) I am getting errors with the merge version due to Apple's ASLR which they make hard to turn off.
I am using mpich2. Any other options to try?
George
On Aug 30, 2012, at 7:28 PM, Barry Smith wrote:
On Aug 30, 2012, at 7:24 PM, George Pau <[email protected]> wrote:
Hi,
I have some issues using the -pctype hmpi. I used the same setting found at
http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHMPI.html
i.e. -pc_type hmpi -ksp_type preonly -hmpi_ksp_type cg -hmpi_pc_type hypre -hmpi_pc_hypre_type boomeramg
My command is
mpiexec -n 1 myprogram
Sorry the documentation doesn't make this clearer. You need to start PETSc with special options to get the "worker" processes initialized. From the manual page for PCHMPI it has
See PetscHMPIMerge() and PetscHMPISpawn() for two ways to start up MPI for use with this preconditioner
This will tell you want option to start PETSc up with.
I will fix the PC so that it prints a far more useful error message.
Barry
But, I get
[gilbert:4041] *** An error occurred in MPI_Bcast [gilbert:4041] *** on communicator MPI_COMM_WORLD [gilbert:4041] *** MPI_ERR_COMM: invalid communicator [gilbert:4041] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
with openmpi. I get similar error with mpich2
Fatal error in PMPI_Bcast: Invalid communicator, error stack: PMPI_Bcast(1478): MPI_Bcast(buf=0x7fffb683479c, count=1, MPI_INT, root=0, comm=0x0) failed PMPI_Bcast(1418): Invalid communicator
I couldn't figure out what is wrong. My petsc is version 3.3.3 and the configuration is -with-debugging=0 --with-mpi-dir=/usr/lib/openmpi --download-hypre=1 and I am on a Ubuntu machine.
Note that with the default pc_type and ksp_type, everything is fine. It was also tested with multiple processors. I wondering whether there are some options that I am not specifying correctly?
-- George Pau Earth Sciences Division Lawrence Berkeley National Laboratory One Cyclotron, MS 74-120 Berkeley, CA 94720
(510) 486-7196 [email protected] http://esd.lbl.gov/about/staff/georgepau/