Hi Shuwei, See my comments/questions between the quotes below: On Nov 13, 2018, at 12:33 AM, Shuwei Zhao <[email protected]<mailto:[email protected]>> wrote: Hi, Hui, Thanks for your response, I have already read through the docs that you said. It's very useful. although I did some research still have some questions: IMHO, there are 2 possible ways to do dynamic scheduling of cpu cores: Could you clarify on what do you mean by “dynamic scheduling of cpu cores”? 1. using mpi_comm_spawn to spawn a new hydra_pmi_proxy on a different port or using mpi_comm_spawn_multiple to spawn new processes on the same port The port — the one shown on the hydra_pmi_proxy command line — is implementation details for the communication between proxy and control; it has nothing to do with how the process is spawned and on which host. For MPICH, each host (node) will have one hydra_pmi_proxy running (spawning one or more executables). If you only have one single program (executable) to be run on spawned nodes, then you only need use `mpi_comm_spawn`. `mpi_comm_spawn_multiple` is used when you need run multiple different executables. To control which host to spawn the process, you pass that information through the MPI_Info object. but my question is that how to start the new processes on a remote machine, using farm resource manager(qsub,bsub,etc)? Is there a reserved keyword in the MPI_info that can qualify for this function? Use MPI_Info argument, which is a set of key/value hints. In particular, you may set `host` for where to launch the new process, or `hosfile` for a file which provide a list of hosts. You may also set `weir` to specify working directory for the new process. Without the hints, the default is the information you passed on the command line or set by resource manager. 2. using mpi_comm_connect to connect an already started mpi application to connect to another mpi application, but I didn't find some useful testcases yet. Do you know if this way is possible? Those are a set of routines to establish new communicators. You only need use them if the existing communicators (MPI_COMM_WORLD and the comms that returned from mpi_comm_spawn and mpi_comm_parent) are insufficient. They are independent of spawning processes. Thanks, Shuwei