I see this error on BGQ at scale (nproc=524288). Abort(1) on node 492082 (rank 492082 in comm 1140850688): Fatal error in PMPI_Comm_group: Other MPI error, error stack: PMPI_Comm_group(174)....: MPI_Comm_group(MPI_COMM_WORLD, group=0x1fffffb9b4) failed MPIR_Comm_group_impl(45): MPIR_Group_create(83)...: Unable to allocate 8388608 bytes of memory for newgroup->lrank_to_lpid (probably out of memory) The culprit appears to be (*new_group_ptr)->lrank_to_lpid = (MPID_Group_pmap_t *)MPIU_Malloc( nproc * sizeof(MPID_Group_pmap_t) ); and typedef struct MPID_Group_pmap_t { int lrank; /* Local rank in group (between 0 and size-1) */ int lpid; /* local process id, from VCONN */ int next_lpid; /* Index of next lpid (in lpid order) */ int flag; /* marker, used to implement group operations */ } MPID_Group_pmap_t; I haven't thought about it very deeply yet, but is there a way to implement group operations without using O(p) memory? Thanks, Jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute [email protected] / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond