Corrected error message as below: mt Dp Mpich 3.2.1 pass Pass Mpich 3.3.2 Fail error: executing task of job 490150 failed: execution daemon on host "host1" didn't accept task Fail [proxy:0:0@host1] HYDU_sock_write (utils/sock/sock.c:289): write error ( Bad file descriptor) 2 [proxy:0:0@host1] main (pm/pmiserv/pmip.c:189): unable to send the proxyID to the server On Thu, May 28, 2020 at 9:40 PM Shuwei Zhao <[email protected]> wrote:

Hi

I was trying to upgrade our mpich version from 3.2.1 to 3.3.2 to consume the latest stable version of mpich, however the new mpich version run get failed.

We have 2 parallel environment configuration as below:

mt

dp

pe_name mt

slots 2000000

used_slots 1181

bound_slots 0

user_lists NONE

xuser_lists NONE

start_proc_args /bin/true

stop_proc_args /bin/true

per_pe_task_prolog NONE

per_pe_task_epilog NONE

allocation_rule $pe_slots

control_slaves FALSE

job_is_first_task FALSE

urgency_slots min

accounting_summary FALSE

daemon_forks_slaves FALSE

master_forks_slaves TRUE

pe_name dp

slots 10000

used_slots 0

bound_slots 0

user_lists NONE

xuser_lists NONE

start_proc_args /bin/true

stop_proc_args /bin/true

per_pe_task_prolog NONE

per_pe_task_epilog NONE

allocation_rule $round_robin

control_slaves TRUE

job_is_first_task FALSE

urgency_slots min

accounting_summary FALSE

daemon_forks_slaves FALSE

master_forks_slaves FALSE

Running command:

We are using qsub to submit workers and master-worker connection will be established using MPI_COMM_ACCEPT and MPI_COMM_CONNECT

qsub -P bnormal -pe mt 1 -e sge_err -o sge_out mpiexec -n 1 /path/to/my/binary binary_arguments

qsub -P bnormal -pe dp 1 -e sge_err -o sge_out mpiexec -n 1 /path/to/my/binary binary_arguments

Running result:

mt

Dp

Mpich 3.2.1

pass

Pass

Mpich 3.3.2

Fail

[proxy:0:0@host1] HYDU_sock_write (utils/sock/sock.c:289): write error ( Bad file descriptor)

2 [proxy:0:0@host1] main (pm/pmiserv/pmip.c:189): unable to send the proxyID to the server

Fail

error: executing task of job 490150 failed: execution daemon on host "host1" didn't accept task

Could you please explain why the run get failed in mpich-3.3 and any solution we can use to get the run pass with mpich-3.3.2?

Appreciate any help.

Thanks