On Fri, 16 Apr 2021, Matthew Knepley wrote:

Hmm, this looks like an MPIUNI memory leak with split collectives. I will have to spin up that configuration and track it down.

with mpich: Breakpoint 2, PetscSplitReductionDestroy (sr=sr@entry=0xd457d0) at /home/balay/petsc/src/vec/vec/utils/comb.c:267 267 { (gdb) where #0 PetscSplitReductionDestroy (sr=sr@entry=0xd457d0) at /home/balay/petsc/src/vec/vec/utils/comb.c:267 #1 0x00007ffff66f0a3a in Petsc_DelReduction (comm=<optimized out>, keyval=<optimized out>, attr_val=0xd457d0, extra_state=<optimized out>) at /home/balay/petsc/src/vec/vec/utils/comb.c:291 #2 0x00007ffff53dec55 in MPII_Attr_delete_c_proxy (user_function=0x7ffff66f095e <Petsc_DelReduction>, handle=-2080374784, keyval=-1539309563, attrib_type=MPIR_ATTR_PTR, attrib=0xd457d0, extra_state=0x0) at src/mpi/attr/attrutil.c:347 #3 0x00007ffff53de385 in MPIR_Call_attr_delete (handle=-2080374784, attr_p=0x7ffff56dd5e8 <MPID_Attr_direct+392>) at src/mpi/attr/attrutil.c:95 #4 0x00007ffff53de6ea in MPIR_Attr_delete_list (handle=-2080374784, attr=0x7ffff56de208 <MPIR_Comm_direct+72>) at src/mpi/attr/attrutil.c:266 #5 0x00007ffff545208b in MPIR_Comm_delete_internal (comm_ptr=0x7ffff56de1c0 <MPIR_Comm_direct>) at src/mpi/comm/commutil.c:756 #6 0x00007ffff5331000 in MPIR_Comm_release (comm_ptr=0x7ffff56de1c0 <MPIR_Comm_direct>) at ./src/include/mpir_comm.h:233 #7 0x00007ffff53310a1 in MPIR_Comm_free_impl (comm_ptr=0x7ffff56de1c0 <MPIR_Comm_direct>) at src/mpi/comm/comm_free.c:35 #8 0x00007ffff53314a0 in PMPI_Comm_free (comm=0x7fffffffa178) at src/mpi/comm/comm_free.c:127 #9 0x00007ffff65401fa in PetscCommDestroy (comm=comm@entry=0x625920) at /home/balay/petsc/src/sys/objects/tagm.c:217 #10 0x00007ffff6547c48 in PetscHeaderDestroy_Private (h=0x6258e0) at /home/balay/petsc/src/sys/objects/inherit.c:121 #11 0x00007ffff7017f3e in PetscP4estFinalize () at /home/balay/petsc/src/dm/impls/forest/p4est/petsc_p4est_package.c:47 #12 0x00007ffff6543270 in PetscRegisterFinalizeAll () at /home/balay/petsc/src/sys/objects/destroy.c:389 #13 0x00007ffff657b7b1 in PetscFinalize () at /home/balay/petsc/src/sys/objects/pinit.c:1476 #14 0x000000000040b51a in main (argc=<optimized out>, argv=<optimized out>) at ex12.c:1249 (gdb) with mpiuni - there is no call to PetscSplitReductionDestroy() I guess this mechanism doesn't work with mpiuni Satish ------ PETSC_EXTERN void petsc_mpi_comm_free_(int *comm,int *ierr) { *ierr = MPI_SUCCESS; }