MPICH uses PMI 1 by default. How is your MPICH configured? And how do you run your program, is it via srun? -- Hui Zhou From: Mccall, Kurt E. (MSFC-EV41) via discuss <[email protected]> Date: Friday, January 7, 2022 at 2:21 PM To: [email protected] <[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <[email protected]> Subject: [mpich-discuss] Slurm and MPI_Comm_spawn My MPICH/Slurm job is failing when the call to MPI_Comm_spawn is made. The Slurm MPI guide https://slurm.schedmd.com/mpi_guide.html#mpich2<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Fmpi_guide.html%23mpich2&data=04%7C01%7Ckurt.e.mccall%40nasa.gov%7Ce8eee3b5265a434ed5e908d9d219ec6f%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637771831057086290%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=J027PCERirTUMWvkhgqD8LJ%2FjHqGtQvDdpvApqmrlPg%3D&reserved=0> specifically states that MPI_Comm_spawn will work going through Hydra’s PMI 1.1 interface. How do I ensure that it goes through that interface? Maybe we’ll have to rebuild Slurm to support PMI 1.1. This Slurm command yields the following and PMI 1.1 is not mentioned, although PMI 2 is. $ srun –mpi=list srun: MPI types are... srun: cray_shasta srun: pmi2 srun: none