On Thu, Aug 12, 2021 at 11:22 AM Barry Smith <[email protected]> wrote:
User visible communicators generally do not have a keyval attached. Rather the keyval is attached to the inner communicator; because we don't want both PETSc and the user doing MPI operations on the same communicator (to prevent mixing up tags).
I think PetscShmCommGet() is wrong. I think it should not call MPI_Comm_get_attr(globcomm,Petsc_Counter_keyval,&counter,&flg); but should call PetscCommDuplicate() and use that communicator to stash the pshmcomm; then you would not have the problem you are having.
Barry, I think it over and find the problem is: if I PetscCommDuplicate() the outer comm in PetscShmCommGet(), then I can not find a place to destroy the inner communicator (note usually petsc inner communciators are destroyed with petsc objects. Doing what you said breaks the rule) PetscShmCommGet() was designed to help doing OpenMP multithreading on a communicator that some petsc objects live in. So requiring the input communicator to be petsc comm is not totally nonsense. I tried another approach: Don't check whether the input comm in PetscShmCommGet(globcomm,&pshmcomm) is an outer comm or an inner comm. We instead check its Petsc_ShmComm_keyval. If it does not have one, we just create one for it (along with a new shared memory communicator) With that, one is able to call PetscShmCommGet(PETSC_COMM_WORLD, ...). The problem is we attached an attribute to PETSC_COMM_WORLD. It is deleted inside MPI_Finalize(). PETSc -malloc_dump complains of unfreed memory (since I used PetscMalloc inside PetscShmCommGet). I could bypass PetscMalloc and directly use malloc() to avoid this situation. Is it worthy?
Barry
On Aug 12, 2021, at 11:05 AM, Pierre Jolivet <[email protected]> wrote:
Hello, Is there a specific reason why PETSC_COMM_WORLD is not a PETSc communicator, i.e., has no Petsc_Counter_keyval attached? ierr = PetscOmpCtrlCreate(PETSC_COMM_WORLD,nthreads,&ctrl);CHKERRQ(ierr); yields [0]PETSC ERROR: Bad MPI communicator supplied must be a PETSc communicator [0]PETSC ERROR: #1 PetscShmCommGet() at src/sys/utils/mpishm.c:60 [0]PETSC ERROR: #2 PetscOmpCtrlCreate() at src/sys/utils/mpishm.c:340
Thanks, Pierre