Hi, Zhou, 

Sorry for the late reply, I was busy with some customer requests. Yes my system is UGE similar to SGE. 
Your explanation makes a lot senses. I haven't had a chance to verify it. I'll try it later. 

Thank you a lot for your explanations. I have great learning experiences in the discussion forum. 

Wish you and your team member have a nice holiday!

Thanks,
Shuwei


On Thu, Nov 29, 2018 at 5:17 PM Zhou, Hui <zhouh@anl.gov> wrote:
What is your system? PBS? You have to prescribe a pool of nodes with qsub and run mpiexec -n 1 (for example). Then when you launch additional mpi process dynamically, hydra will spawn new processes within the prescribed pool. If your original pool is a single host, then all new dynamic process will be on that single host. If you describe multiple hosts, I believe hydra will do round-robin style allocation, i.e. the next processes will go to host 2, host 3, …., host N, host 1 (circle back).

This is my current understanding. Let me know whether you can confirm.
 
— 
Hui Zhou



On Nov 29, 2018, at 4:41 PM, Shuwei Zhao <shuweizhao1991@gmail.com> wrote:

Hi Hui 

I cannot run mpiexec directly on my hosts because both rsh and ssh connections are disabled. I have to use qsub -P bnormal -pe dp 4 mpiexec ./my_exe to run my binary on remote hosts. 

Thanks 
Shuwei