- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Hydra WARNING: too many ssh connections
by Zhou, Hui 01 Apr '22

01 Apr '22

Every time you call MPI_Comm_spawn, hydra will launch a ssh (for each host) to create a proxy. It is certainly not ideal for applications relying on spawning many processes. ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Sent: Friday, April 1, 2022 4:08 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] Hydra WARNING: too many ssh connections Hi, you provided the following information about the warning “too many ssh connections”: The particular warning is issued by hydra, MPICH’s process manager. Following excerpt is the comment in that source code: /* ssh has many types of security controls that do not allow a * user to ssh to the same node multiple times very * quickly. If this happens, the ssh daemons disables ssh * connections causing the job to fail. This is basically a * hack to slow down ssh connections to the same node. We * check for offset == 0 before applying this hack, so we only * slow down the cases where ssh is being used, and not the * cases where we fall back to fork. */ Is this just during an initial ssh connection attempt? I’m trying to figure out where my code is triggering this warning. Could it be from 1. MPI_Intercomm_create 2. MPI_Comm_spawn 3. others? I’m calling mpiexec with the “—launcher ssh” option, MPICH 4.0.1. Thanks, Kurt

1 0

Hydra WARNING: too many ssh connections
by Mccall, Kurt E. (MSFC-EV41) 01 Apr '22

01 Apr '22

Hi, you provided the following information about the warning "too many ssh connections": The particular warning is issued by hydra, MPICH's process manager. Following excerpt is the comment in that source code: /* ssh has many types of security controls that do not allow a * user to ssh to the same node multiple times very * quickly. If this happens, the ssh daemons disables ssh * connections causing the job to fail. This is basically a * hack to slow down ssh connections to the same node. We * check for offset == 0 before applying this hack, so we only * slow down the cases where ssh is being used, and not the * cases where we fall back to fork. */ Is this just during an initial ssh connection attempt? I'm trying to figure out where my code is triggering this warning. Could it be from 1. MPI_Intercomm_create 2. MPI_Comm_spawn 3. others? I'm calling mpiexec with the "-launcher ssh" option, MPICH 4.0.1. Thanks, Kurt

1 0

Re: [mpich-discuss] Error message
by Zhou, Hui 30 Mar '22

30 Mar '22

Hi Bruce, The call stack says you called MPI_Intercomm_create. Try get a full stack trace using debugger. -- Hui ________________________________ From: Palmer, Bruce J via discuss <discuss(a)mpich.org> Sent: Wednesday, March 30, 2022 6:28 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: [mpich-discuss] Error message Does anyone have any idea what the following error message is supposed to be telling me? It appears to be coming from an attempt to make a copy of MPI_COMM_SELF using MPI_Comm_dup. Abort(66703622) on node 4 (rank 4 in comm 0): Fatal error in internal_Intercomm_create: Invalid rank, error stack: internal_Intercomm_create(134): MPI_Intercomm_create(comm=0x84000005, 0, comm=0x84000003, 3, 0, newintercomm=0x7fffffffcd68) failed internal_Intercomm_create(102): Local and remote leaders must be different processes Bruce Palmer

1 0

Error message
by Palmer, Bruce J 30 Mar '22

30 Mar '22

Does anyone have any idea what the following error message is supposed to be telling me? It appears to be coming from an attempt to make a copy of MPI_COMM_SELF using MPI_Comm_dup. Abort(66703622) on node 4 (rank 4 in comm 0): Fatal error in internal_Intercomm_create: Invalid rank, error stack: internal_Intercomm_create(134): MPI_Intercomm_create(comm=0x84000005, 0, comm=0x84000003, 3, 0, newintercomm=0x7fffffffcd68) failed internal_Intercomm_create(102): Local and remote leaders must be different processes Bruce Palmer

1 0

Re: [mpich-discuss] error in installing MPICH-4.0.1: libhwloc.so.15: cannot open shared object file: No such file or directory
by Zhou, Hui 26 Mar '22

26 Mar '22

Hi Bruce, libhwloc must be installed on the host where you build MPICH but is missing on one of your nodes. By default when you build MPICH, it checks for system hwloc and will use the one it found rather than use the embedded one. To force use embedded -- I recommend it in your case -- add --with-hwloc=embedded to your configure option. Cheers, -- Hui ________________________________ From: Bruce Rout via discuss <discuss(a)mpich.org> Sent: Saturday, March 26, 2022 9:59 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Bruce Rout <bbrout(a)rasl.ca> Subject: [mpich-discuss] error in installing MPICH-4.0.1: libhwloc.so.15: cannot open shared object file: No such file or directory Greetings, I have been trying for about a week to get MPICH to work. I am running Ubuntu 20.04.4 on an HP Z-600 as a master with three nodes all under NFS. NFS is working with ssh passwordless access and common hard drive directory between all four computers, (master plus three nodes). MPICH had some problems with HYDRA Proxies however a re-install with apt-get install mpich solved the problem and it worked with processors accessed on the entire array. I installed a hard drive for a different user on the master node, leaving the master user mpiuser unchanged. There was some difficulty in getting the drive to mount on the different user, but now mpich ohny works on the master node. I have reinstalled a number of times and am installing by downloading mpich-4.0.1 from http://www.mpich.org. The running of a single node works: mpiuser@BD-Main:~$ mpiexec -n 3 ./examples/cpi Invalid MIT-MAGIC-COOKIE-1 keyInvalid MIT-MAGIC-COOKIE-1 keyInvalid MIT-MAGIC-COOKIE-1 keyProcess 0 of 3 is on BD-Main Process 1 of 3 is on BD-Main Process 2 of 3 is on BD-Main pi is approximately 3.1415926544231318, Error is 0.0000000008333387 wall clock time = 0.002317 mpiuser@BD-Main:~$ and there is an invalid key somewhere in that output. Machinefile contains the following: (This is the host file???) mpiuser@BD-Main:~$ more machinefile node1:4 node2:2 node3:4 mpiuser@BD-Main:~$ And there is the following error with multiple nodes: mpiuser@BD-Main:~$ mpiexec -f machinefile -n 3 ./examples/cpi /home/mpiuser/mpich-install/bin/hydra_pmi_proxy: error while loading shared libraries: libhwloc.so.15: cannot open shared object file: No such file or directory ^C[mpiexec@BD-Main] Sending Ctrl-C to processes as requested [mpiexec@BD-Main] Press Ctrl-C again to force abort [mpiexec@BD-Main] HYDU_sock_write (utils/sock/sock.c:254): write error (Bad file descriptor) [mpiexec@BD-Main] HYD_pmcd_pmiserv_send_signal (pm/pmiserv/pmiserv_cb.c:176): unable to write data to proxy [mpiexec@BD-Main] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:42): unable to send signal downstream [mpiexec@BD-Main] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec@BD-Main] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:160): error waiting for event [mpiexec@BD-Main] main (ui/mpich/mpiexec.c:325): process manager error waiting for completion mpiuser@BD-Main:~$ The program hangs after the first error message /home/mpiuser/mpich-install/bin/hydra_pmi_proxy: error while loading shared libraries: libhwloc.so.15: cannot open shared object file: No such file or directory and had to exit with ctl-c. I have hunted everywhere for libhwloc.so.15 and found two rpm files hwloc-libs-1.11.8-4.el7.x86_64.rpm and hwloc-2.7.1.tar.bz2 which I managed to download but cannot install the librarires. Synaptic is not help since the repository for hwloc is not installed. Another indication of a problem is that the file cp mpich4.0.1/src/pm/hydra/tools/topo/hwloc/hwloc/config.log is missing from the ./configure step. Obviously it is not included in the zipped files attached here. The configure, make and make install steps ran without a hitch and without error. This is the output of mpich -info mpiuser@BD-Main:~$ mpiexec -info HYDRA build details: Version: 4.0.1 Release Date: Tue Feb 22 16:37:51 CST 2022 CC: gcc Configure options: '--disable-option-checking' '--prefix=/home/mpiuser/mpich-install' '--cache-file=/dev/null' '--srcdir=.' 'CC=gcc' 'CFLAGS= ' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS= ' Process Manager: pmi Launchers available: ssh rsh fork slurm ll lsf sge manual persist Topology libraries available: hwloc Resource management kernels available: user slurm ll lsf sge pbs cobalt Demux engines available: poll select mpiuser@BD-Main:~$ The master node is BD-Main and this is the output of /etc/hosts which is similar on all nodes: mpiuser@BD-Main:~$ more /etc/hosts 127.0.0.1 localhost 127.0.1.1 BD-Main # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters # The following sets up the local network for cluster 10.0.0.1 master 10.0.0.2 node1 10.0.0.3 node2 10.0.0.4 node3 As I have mentioned before, this set-up seems to have worked and NFS is working well. I have purged both hydra and mpich and then reinstalled mpich-4.0.1 according to the instructions in the README fiel. Any help would be greatly appreciated. Thank you for your time. Yours, Bruce

1 0

error in installing MPICH-4.0.1: libhwloc.so.15: cannot open shared object file: No such file or directory
by Bruce Rout 26 Mar '22

26 Mar '22

Greetings, I have been trying for about a week to get MPICH to work. I am running Ubuntu 20.04.4 on an HP Z-600 as a master with three nodes all under NFS. NFS is working with ssh passwordless access and common hard drive directory between all four computers, (master plus three nodes). MPICH had some problems with HYDRA Proxies however a re-install with apt-get install mpich solved the problem and it worked with processors accessed on the entire array. I installed a hard drive for a different user on the master node, leaving the master user mpiuser unchanged. There was some difficulty in getting the drive to mount on the different user, but now mpich ohny works on the master node. I have reinstalled a number of times and am installing by downloading mpich-4.0.1 from http://www.mpich.org. The running of a single node works: mpiuser@BD-Main:~$ mpiexec -n 3 ./examples/cpi Invalid MIT-MAGIC-COOKIE-1 keyInvalid MIT-MAGIC-COOKIE-1 keyInvalid MIT-MAGIC-COOKIE-1 keyProcess 0 of 3 is on BD-Main Process 1 of 3 is on BD-Main Process 2 of 3 is on BD-Main pi is approximately 3.1415926544231318, Error is 0.0000000008333387 wall clock time = 0.002317 mpiuser@BD-Main:~$ and there is an invalid key somewhere in that output. Machinefile contains the following: (This is the host file???) mpiuser@BD-Main:~$ more machinefile node1:4 node2:2 node3:4 mpiuser@BD-Main:~$ And there is the following error with multiple nodes: mpiuser@BD-Main:~$ mpiexec -f machinefile -n 3 ./examples/cpi /home/mpiuser/mpich-install/bin/hydra_pmi_proxy: error while loading shared libraries: libhwloc.so.15: cannot open shared object file: No such file or directory ^C[mpiexec@BD-Main] Sending Ctrl-C to processes as requested [mpiexec@BD-Main] Press Ctrl-C again to force abort [mpiexec@BD-Main] HYDU_sock_write (utils/sock/sock.c:254): write error (Bad file descriptor) [mpiexec@BD-Main] HYD_pmcd_pmiserv_send_signal (pm/pmiserv/pmiserv_cb.c:176): unable to write data to proxy [mpiexec@BD-Main] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:42): unable to send signal downstream [mpiexec@BD-Main] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec@BD-Main] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:160): error waiting for event [mpiexec@BD-Main] main (ui/mpich/mpiexec.c:325): process manager error waiting for completion mpiuser@BD-Main:~$ The program hangs after the first error message /home/mpiuser/mpich-install/bin/hydra_pmi_proxy: error while loading shared libraries: libhwloc.so.15: cannot open shared object file: No such file or directory and had to exit with ctl-c. I have hunted everywhere for libhwloc.so.15 and found two rpm files hwloc-libs-1.11.8-4.el7.x86_64.rpm and hwloc-2.7.1.tar.bz2 which I managed to download but cannot install the librarires. Synaptic is not help since the repository for hwloc is not installed. Another indication of a problem is that the file cp mpich4.0.1/src/pm/hydra/tools/topo/hwloc/hwloc/config.log is missing from the ./configure step. Obviously it is not included in the zipped files attached here. The configure, make and make install steps ran without a hitch and without error. This is the output of mpich -info mpiuser@BD-Main:~$ mpiexec -info HYDRA build details: Version: 4.0.1 Release Date: Tue Feb 22 16:37:51 CST 2022 CC: gcc Configure options: '--disable-option-checking' '--prefix=/home/mpiuser/mpich-install' '--cache-file=/dev/null' '--srcdir=.' 'CC=gcc' 'CFLAGS= ' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS= ' Process Manager: pmi Launchers available: ssh rsh fork slurm ll lsf sge manual persist Topology libraries available: hwloc Resource management kernels available: user slurm ll lsf sge pbs cobalt Demux engines available: poll select mpiuser@BD-Main:~$ The master node is BD-Main and this is the output of /etc/hosts which is similar on all nodes: mpiuser@BD-Main:~$ more /etc/hosts 127.0.0.1 localhost 127.0.1.1 BD-Main # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters # The following sets up the local network for cluster 10.0.0.1 master 10.0.0.2 node1 10.0.0.3 node2 10.0.0.4 node3 As I have mentioned before, this set-up seems to have worked and NFS is working well. I have purged both hydra and mpich and then reinstalled mpich-4.0.1 according to the instructions in the README fiel. Any help would be greatly appreciated. Thank you for your time. Yours, Bruce

1 0

Re: [mpich-discuss] not able to run mpi jobs
by Zhou, Hui 22 Mar '22

22 Mar '22

To use mpich with srun, please follow: ``` ./configure --with-pmi=slurm --with-pm=no --with-slurm=[path_to_slurm_install] ``` ref: https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_How_do… -- Hui ________________________________ From: masber masber via discuss <discuss(a)mpich.org> Sent: Tuesday, March 22, 2022 12:20 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: masber masber <masber(a)hotmail.com> Subject: [mpich-discuss] Fw: not able to run mpi jobs Dear mpich community, I am quite new to mpi but I got a small slurm cluster with 3 compute nodes running. I can run simple jobs like `srun -N3 hostname` and I am trying now to run an mpi helloworld app. My issue is that the job hangs and fails after a few seconds. # srun -N2 -n4 /scratch/helloworld-mpi srun: error: mpi/pmi2: failed to send temp kvs to compute nodes srun: Job step aborted: Waiting up to 32 seconds for job step to finish. slurmstepd: error: *** STEP 2.0 ON nid001001-cluster-1 CANCELLED AT 2022-03-22T16:43:07 *** srun: error: nid001002-cluster-1: task 3: Killed srun: launch/slurm: _step_signal: Terminating StepId=2.0 srun: error: nid001001-cluster-1: tasks 0-2: Killed I can see this in the slurmd logs: slurmd: debug3: CPUs=40 Boards=1 Sockets=40 Cores=1 Threads=1 Memory=515174 TmpDisk=211436 Uptime=1920011 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null) slurmd: debug: _handle_node_reg_resp: slurmctld sent back 8 TRES. slurmd: debug3: in the service_connection slurmd: debug2: Start processing RPC: REQUEST_LAUNCH_TASKS slurmd: debug2: Processing RPC: REQUEST_LAUNCH_TASKS slurmd: launch task StepId=2.0 request from UID:0 GID:0 HOST:172.29.113.47 PORT:40062 slurmd: debug: Checking credential with 468 bytes of sig data slurmd: debug2: _group_cache_lookup_internal: no entry found for root slurmd: debug: task/affinity: task_p_slurmd_launch_request: task affinity : before lllp distribution cpu bind method is '(null type)' ((null)) slurmd: debug3: task/affinity: _get_avail_map: slurmctld s 40 c 1; hw s 40 c 1 t 1 slurmd: debug3: task/affinity: _get_avail_map: StepId=2.0 core mask from slurmctld: 0x0000000007 slurmd: debug3: task/affinity: _get_avail_map: StepId=2.0 CPU final mask for local node: 0x0000000000000007 slurmd: debug3: task/affinity: _lllp_map_abstract_masks: _lllp_map_abstract_masks slurmd: debug: task/affinity: lllp_distribution: binding tasks:3 to nodes:0 sockets:3:0 cores:3:0 threads:3 slurmd: task/affinity: lllp_distribution: JobId=2 implicit auto binding: sockets,one_thread, dist 8192 slurmd: debug2: task/affinity: lllp_distribution: JobId=2 will use lllp_cyclic because of SelectTypeParameters slurmd: task/affinity: _task_layout_lllp_cyclic: _task_layout_lllp_cyclic slurmd: debug3: task/affinity: _get_avail_map: slurmctld s 40 c 1; hw s 40 c 1 t 1 slurmd: debug3: task/affinity: _get_avail_map: StepId=2.0 core mask from slurmctld: 0x0000000007 slurmd: debug3: task/affinity: _get_avail_map: StepId=2.0 CPU final mask for local node: 0x0000000000000007 slurmd: debug3: task/affinity: _task_layout_display_masks: _task_layout_display_masks jobid [2:0] 0x0000000000000001 slurmd: debug3: task/affinity: _task_layout_display_masks: _task_layout_display_masks jobid [2:1] 0x0000000000000002 slurmd: debug3: task/affinity: _task_layout_display_masks: _task_layout_display_masks jobid [2:2] 0x0000000000000004 slurmd: debug3: task/affinity: _lllp_map_abstract_masks: _lllp_map_abstract_masks slurmd: debug3: task/affinity: _task_layout_display_masks: _task_layout_display_masks jobid [2:0] 0x0000000000000001 slurmd: debug3: task/affinity: _task_layout_display_masks: _task_layout_display_masks jobid [2:1] 0x0000000100000000 slurmd: debug3: task/affinity: _task_layout_display_masks: _task_layout_display_masks jobid [2:2] 0x0000000000000002 slurmd: debug3: task/affinity: _lllp_generate_cpu_bind: 3 19 58 slurmd: task/affinity: _lllp_generate_cpu_bind: _lllp_generate_cpu_bind jobid [2]: mask_cpu,one_thread, 0x0000000000000001,0x0000000100000000,0x0000000000000002 slurmd: debug: task/affinity: task_p_slurmd_launch_request: task affinity : after lllp distribution cpu bind method is 'mask_cpu,one_thread' (0x0000000000000001,0x0000000100000000,0x0000000000000002) slurmd: debug2: _insert_job_state: we already have a job state for job 2. No big deal, just an FYI. slurmd: debug3: _rpc_launch_tasks: call to _forkexec_slurmstepd slurmd: debug2: _read_slurm_cgroup_conf_int: No cgroup.conf file (/etc/slurm/cgroup.conf) slurmd: debug3: slurmstepd rank 0 (nid001001-cluster-1), parent rank -1 (NONE), children 0, depth 0, max_depth 0 slurmd: debug3: _rpc_launch_tasks: return from _forkexec_slurmstepd slurmd: debug2: Finish processing RPC: REQUEST_LAUNCH_TASKS slurmd: debug3: in the service_connection slurmd: debug2: Start processing RPC: REQUEST_FORWARD_DATA slurmd: debug2: Processing RPC: REQUEST_FORWARD_DATA slurmd: debug3: Entering _rpc_forward_data, address: /var/spool/slurmd/sock.pmi2.2.0, len: 84 slurmd: debug2: slurmd: debug3: CPUs=40 Boards=1 Sockets=40 Cores=1 Threads=1 Memory=515174 TmpDisk=211436 Uptime=1920011 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null) slurmd: debug: _handle_node_reg_resp: slurmctld sent back 8 TRES. slurmd: debug3: in the service_connection slurmd: debug2: Start processing RPC: REQUEST_LAUNCH_TASKS slurmd: debug2: Processing RPC: REQUEST_LAUNCH_TASKS slurmd: launch task StepId=2.0 request from UID:0 GID:0 HOST:172.29.113.47 PORT:40062 slurmd: debug: Checking credential with 468 bytes of sig data slurmd: debug2: _group_cache_lookup_internal: no entry found for root slurmd: debug: task/affinity: task_p_slurmd_launch_request: task affinity : before lllp distribution cpu bind method is '(null type)' ((null)) slurmd: debug3: task/affinity: _get_avail_map: slurmctld s 40 c 1; hw s 40 c 1 t 1 slurmd: debug3: task/affinity: _get_avail_map: StepId=2.0 core mask from slurmctld: 0x0000000007 slurmd: debug3: task/affinity: _get_avail_map: StepId=2.0 CPU final mask for local node: 0x0000000000000007 slurmd: debug3: task/affinity: _lllp_map_abstract_masks: _lllp_map_abstract_masks slurmd: debug: task/affinity: lllp_distribution: binding tasks:3 to nodes:0 sockets:3:0 cores:3:0 threads:3 slurmd: task/affinity: lllp_distribution: JobId=2 implicit auto binding: sockets,one_thread, dist 8192 slurmd: debug2: task/affinity: lllp_distribution: JobId=2 will use lllp_cyclic because of SelectTypeParameters slurmd: task/affinity: _task_layout_lllp_cyclic: _task_layout_lllp_cyclic slurmd: debug3: task/affinity: _get_avail_map: slurmctld s 40 c 1; hw s 40 c 1 t 1 slurmd: debug3: task/affinity: _get_avail_map: StepId=2.0 core mask from slurmctld: 0x0000000007 slurmd: debug3: task/affinity: _get_avail_map: StepId=2.0 CPU final mask for local node: 0x0000000000000007 slurmd: debug3: task/affinity: _task_layout_display_masks: _task_layout_display_masks jobid [2:0] 0x0000000000000001 slurmd: debug3: task/affinity: _task_layout_display_masks: _task_layout_display_masks jobid [2:1] 0x0000000000000002 slurmd: debug3: task/affinity: _task_layout_display_masks: _task_layout_display_masks jobid [2:2] 0x0000000000000004 slurmd: debug3: task/affinity: _lllp_map_abstract_masks: _lllp_map_abstract_masks slurmd: debug3: task/affinity: _task_layout_display_masks: _task_layout_display_masks jobid [2:0] 0x0000000000000001 slurmd: debug3: task/affinity: _task_layout_display_masks: _task_layout_display_masks jobid [2:1] 0x0000000100000000 slurmd: debug3: task/affinity: _task_layout_display_masks: _task_layout_display_masks jobid [2:2] 0x0000000000000002 slurmd: debug3: task/affinity: _lllp_generate_cpu_bind: 3 19 58 slurmd: task/affinity: _lllp_generate_cpu_bind: _lllp_generate_cpu_bind jobid [2]: mask_cpu,one_thread, 0x0000000000000001,0x0000000100000000,0x0000000000000002 slurmd: debug: task/affinity: task_p_slurmd_launch_request: task affinity : after lllp distribution cpu bind method is 'mask_cpu,one_thread' (0x0000000000000001,0x0000000100000000,0x0000000000000002) slurmd: debug2: _insert_job_state: we already have a job state for job 2. No big deal, just an FYI. slurmd: debug3: _rpc_launch_tasks: call to _forkexec_slurmstepd slurmd: debug2: _read_slurm_cgroup_conf_int: No cgroup.conf file (/etc/slurm/cgroup.conf) slurmd: debug3: slurmstepd rank 0 (nid001001-cluster-1), parent rank -1 (NONE), children 0, depth 0, max_depth 0 slurmd: debug3: _rpc_launch_tasks: return from _forkexec_slurmstepd slurmd: debug2: Finish processing RPC: REQUEST_LAUNCH_TASKS slurmd: debug3: in the service_connection slurmd: debug2: Start processing RPC: REQUEST_FORWARD_DATA slurmd: debug2: Processing RPC: REQUEST_FORWARD_DATA slurmd: debug3: Entering _rpc_forward_data, address: /var/spool/slurmd/sock.pmi2.2.0, len: 84 slurmd: debug2: failed connecting to specified socket '/var/spool/slurmd/sock.pmi2.2.0': Connection refused ... I compiled mpich-4.0.0 and I can run mpi jobs outside slurm # mpirun -ppn 2 --hosts nid001001-cluster-1,nid001003-cluster-1,nid001003-cluster-1 /scratch/helloworld-mpi Warning: Permanently added 'nid001003-cluster-1,172.29.9.83' (ECDSA) to the list of known hosts. Hello world from processor nid001003-cluster-1, rank 2 out of 4 processors Hello world from processor nid001003-cluster-1, rank 3 out of 4 processors Hello world from processor nid001001-cluster-1, rank 0 out of 4 processors Hello world from processor nid001001-cluster-1, rank 1 out of 4 processors Could someone please give me a hint of what to look in regards running mpi jobs in slurm? thank you very much

1 0

Re: [mpich-discuss] MPICH -- too many open files
by Zhou, Hui 22 Mar '22

22 Mar '22

MPICH currently launches individual proxy for each spawn. I think that probably attributes to the flood of fds on the server. For now, I guess the solution is to ask system admin to increase the fd limit. Feel free to open a issue at https://github.com/pmodels/mpich/issues. We'll prioritize to get it enhanced. -- Hui [https://opengraph.githubassets.com/7cbf378518c55cfe4549cfe44dfb9b74ceb606f9…]<https://github.com/pmodels/mpich/issues> Issues · pmodels/mpich - GitHub<https://github.com/pmodels/mpich/issues> Official MPICH Repository. Contribute to pmodels/mpich development by creating an account on GitHub. github.com ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Sent: Tuesday, March 22, 2022 12:55 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] MPICH -- too many open files My application, which spawns multiple subprocesses via MPI_Comm_spawn, eventually fails on one Slurm cluster as I scale up the number of processes, with the error: [mpiexec(a)n002.cluster.pssclabs.com] HYDU_create_process (../../../../mpich-4.0.1/src/pm/hydra/utils/launch/launch.c:21): pipe error (Too many open files) [mpiexec(a)n002.cluster.pssclabs.com] HYDT_bscd_common_launch_procs (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/external/external_common_launch.c:296): create process returned error free(): invalid pointer /var/spool/slurm/job235999/slurm_script: line 296: 3778907 Aborted (core dumped) It works fine on a different (Torque) cluster for very large job sizes. “ulimit -n” (number of open files) on both machines returns 1024. I’m hoping that there is some other system setting on the Slurm cluster that would allow larger jobs. I can provide the “-verbose” output file if that would help. Thanks, Kurt

1 0

MPICH -- too many open files
by Mccall, Kurt E. (MSFC-EV41) 22 Mar '22

22 Mar '22

My application, which spawns multiple subprocesses via MPI_Comm_spawn, eventually fails on one Slurm cluster as I scale up the number of processes, with the error: [mpiexec(a)n002.cluster.pssclabs.com] HYDU_create_process (../../../../mpich-4.0.1/src/pm/hydra/utils/launch/launch.c:21): pipe error (Too many open files) [mpiexec(a)n002.cluster.pssclabs.com] HYDT_bscd_common_launch_procs (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/external/external_common_launch.c:296): create process returned error free(): invalid pointer /var/spool/slurm/job235999/slurm_script: line 296: 3778907 Aborted (core dumped) It works fine on a different (Torque) cluster for very large job sizes. "ulimit -n" (number of open files) on both machines returns 1024. I'm hoping that there is some other system setting on the Slurm cluster that would allow larger jobs. I can provide the "-verbose" output file if that would help. Thanks, Kurt

1 0

Fw: not able to run mpi jobs
by masber masber 22 Mar '22

22 Mar '22

Dear mpich community, I am quite new to mpi but I got a small slurm cluster with 3 compute nodes running. I can run simple jobs like `srun -N3 hostname` and I am trying now to run an mpi helloworld app. My issue is that the job hangs and fails after a few seconds. # srun -N2 -n4 /scratch/helloworld-mpi srun: error: mpi/pmi2: failed to send temp kvs to compute nodes srun: Job step aborted: Waiting up to 32 seconds for job step to finish. slurmstepd: error: *** STEP 2.0 ON nid001001-cluster-1 CANCELLED AT 2022-03-22T16:43:07 *** srun: error: nid001002-cluster-1: task 3: Killed srun: launch/slurm: _step_signal: Terminating StepId=2.0 srun: error: nid001001-cluster-1: tasks 0-2: Killed I can see this in the slurmd logs: slurmd: debug3: CPUs=40 Boards=1 Sockets=40 Cores=1 Threads=1 Memory=515174 TmpDisk=211436 Uptime=1920011 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null) slurmd: debug: _handle_node_reg_resp: slurmctld sent back 8 TRES. slurmd: debug3: in the service_connection slurmd: debug2: Start processing RPC: REQUEST_LAUNCH_TASKS slurmd: debug2: Processing RPC: REQUEST_LAUNCH_TASKS slurmd: launch task StepId=2.0 request from UID:0 GID:0 HOST:172.29.113.47 PORT:40062 slurmd: debug: Checking credential with 468 bytes of sig data slurmd: debug2: _group_cache_lookup_internal: no entry found for root slurmd: debug: task/affinity: task_p_slurmd_launch_request: task affinity : before lllp distribution cpu bind method is '(null type)' ((null)) slurmd: debug3: task/affinity: _get_avail_map: slurmctld s 40 c 1; hw s 40 c 1 t 1 slurmd: debug3: task/affinity: _get_avail_map: StepId=2.0 core mask from slurmctld: 0x0000000007 slurmd: debug3: task/affinity: _get_avail_map: StepId=2.0 CPU final mask for local node: 0x0000000000000007 slurmd: debug3: task/affinity: _lllp_map_abstract_masks: _lllp_map_abstract_masks slurmd: debug: task/affinity: lllp_distribution: binding tasks:3 to nodes:0 sockets:3:0 cores:3:0 threads:3 slurmd: task/affinity: lllp_distribution: JobId=2 implicit auto binding: sockets,one_thread, dist 8192 slurmd: debug2: task/affinity: lllp_distribution: JobId=2 will use lllp_cyclic because of SelectTypeParameters slurmd: task/affinity: _task_layout_lllp_cyclic: _task_layout_lllp_cyclic slurmd: debug3: task/affinity: _get_avail_map: slurmctld s 40 c 1; hw s 40 c 1 t 1 slurmd: debug3: task/affinity: _get_avail_map: StepId=2.0 core mask from slurmctld: 0x0000000007 slurmd: debug3: task/affinity: _get_avail_map: StepId=2.0 CPU final mask for local node: 0x0000000000000007 slurmd: debug3: task/affinity: _task_layout_display_masks: _task_layout_display_masks jobid [2:0] 0x0000000000000001 slurmd: debug3: task/affinity: _task_layout_display_masks: _task_layout_display_masks jobid [2:1] 0x0000000000000002 slurmd: debug3: task/affinity: _task_layout_display_masks: _task_layout_display_masks jobid [2:2] 0x0000000000000004 slurmd: debug3: task/affinity: _lllp_map_abstract_masks: _lllp_map_abstract_masks slurmd: debug3: task/affinity: _task_layout_display_masks: _task_layout_display_masks jobid [2:0] 0x0000000000000001 slurmd: debug3: task/affinity: _task_layout_display_masks: _task_layout_display_masks jobid [2:1] 0x0000000100000000 slurmd: debug3: task/affinity: _task_layout_display_masks: _task_layout_display_masks jobid [2:2] 0x0000000000000002 slurmd: debug3: task/affinity: _lllp_generate_cpu_bind: 3 19 58 slurmd: task/affinity: _lllp_generate_cpu_bind: _lllp_generate_cpu_bind jobid [2]: mask_cpu,one_thread, 0x0000000000000001,0x0000000100000000,0x0000000000000002 slurmd: debug: task/affinity: task_p_slurmd_launch_request: task affinity : after lllp distribution cpu bind method is 'mask_cpu,one_thread' (0x0000000000000001,0x0000000100000000,0x0000000000000002) slurmd: debug2: _insert_job_state: we already have a job state for job 2. No big deal, just an FYI. slurmd: debug3: _rpc_launch_tasks: call to _forkexec_slurmstepd slurmd: debug2: _read_slurm_cgroup_conf_int: No cgroup.conf file (/etc/slurm/cgroup.conf) slurmd: debug3: slurmstepd rank 0 (nid001001-cluster-1), parent rank -1 (NONE), children 0, depth 0, max_depth 0 slurmd: debug3: _rpc_launch_tasks: return from _forkexec_slurmstepd slurmd: debug2: Finish processing RPC: REQUEST_LAUNCH_TASKS slurmd: debug3: in the service_connection slurmd: debug2: Start processing RPC: REQUEST_FORWARD_DATA slurmd: debug2: Processing RPC: REQUEST_FORWARD_DATA slurmd: debug3: Entering _rpc_forward_data, address: /var/spool/slurmd/sock.pmi2.2.0, len: 84 slurmd: debug2: slurmd: debug3: CPUs=40 Boards=1 Sockets=40 Cores=1 Threads=1 Memory=515174 TmpDisk=211436 Uptime=1920011 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null) slurmd: debug: _handle_node_reg_resp: slurmctld sent back 8 TRES. slurmd: debug3: in the service_connection slurmd: debug2: Start processing RPC: REQUEST_LAUNCH_TASKS slurmd: debug2: Processing RPC: REQUEST_LAUNCH_TASKS slurmd: launch task StepId=2.0 request from UID:0 GID:0 HOST:172.29.113.47 PORT:40062 slurmd: debug: Checking credential with 468 bytes of sig data slurmd: debug2: _group_cache_lookup_internal: no entry found for root slurmd: debug: task/affinity: task_p_slurmd_launch_request: task affinity : before lllp distribution cpu bind method is '(null type)' ((null)) slurmd: debug3: task/affinity: _get_avail_map: slurmctld s 40 c 1; hw s 40 c 1 t 1 slurmd: debug3: task/affinity: _get_avail_map: StepId=2.0 core mask from slurmctld: 0x0000000007 slurmd: debug3: task/affinity: _get_avail_map: StepId=2.0 CPU final mask for local node: 0x0000000000000007 slurmd: debug3: task/affinity: _lllp_map_abstract_masks: _lllp_map_abstract_masks slurmd: debug: task/affinity: lllp_distribution: binding tasks:3 to nodes:0 sockets:3:0 cores:3:0 threads:3 slurmd: task/affinity: lllp_distribution: JobId=2 implicit auto binding: sockets,one_thread, dist 8192 slurmd: debug2: task/affinity: lllp_distribution: JobId=2 will use lllp_cyclic because of SelectTypeParameters slurmd: task/affinity: _task_layout_lllp_cyclic: _task_layout_lllp_cyclic slurmd: debug3: task/affinity: _get_avail_map: slurmctld s 40 c 1; hw s 40 c 1 t 1 slurmd: debug3: task/affinity: _get_avail_map: StepId=2.0 core mask from slurmctld: 0x0000000007 slurmd: debug3: task/affinity: _get_avail_map: StepId=2.0 CPU final mask for local node: 0x0000000000000007 slurmd: debug3: task/affinity: _task_layout_display_masks: _task_layout_display_masks jobid [2:0] 0x0000000000000001 slurmd: debug3: task/affinity: _task_layout_display_masks: _task_layout_display_masks jobid [2:1] 0x0000000000000002 slurmd: debug3: task/affinity: _task_layout_display_masks: _task_layout_display_masks jobid [2:2] 0x0000000000000004 slurmd: debug3: task/affinity: _lllp_map_abstract_masks: _lllp_map_abstract_masks slurmd: debug3: task/affinity: _task_layout_display_masks: _task_layout_display_masks jobid [2:0] 0x0000000000000001 slurmd: debug3: task/affinity: _task_layout_display_masks: _task_layout_display_masks jobid [2:1] 0x0000000100000000 slurmd: debug3: task/affinity: _task_layout_display_masks: _task_layout_display_masks jobid [2:2] 0x0000000000000002 slurmd: debug3: task/affinity: _lllp_generate_cpu_bind: 3 19 58 slurmd: task/affinity: _lllp_generate_cpu_bind: _lllp_generate_cpu_bind jobid [2]: mask_cpu,one_thread, 0x0000000000000001,0x0000000100000000,0x0000000000000002 slurmd: debug: task/affinity: task_p_slurmd_launch_request: task affinity : after lllp distribution cpu bind method is 'mask_cpu,one_thread' (0x0000000000000001,0x0000000100000000,0x0000000000000002) slurmd: debug2: _insert_job_state: we already have a job state for job 2. No big deal, just an FYI. slurmd: debug3: _rpc_launch_tasks: call to _forkexec_slurmstepd slurmd: debug2: _read_slurm_cgroup_conf_int: No cgroup.conf file (/etc/slurm/cgroup.conf) slurmd: debug3: slurmstepd rank 0 (nid001001-cluster-1), parent rank -1 (NONE), children 0, depth 0, max_depth 0 slurmd: debug3: _rpc_launch_tasks: return from _forkexec_slurmstepd slurmd: debug2: Finish processing RPC: REQUEST_LAUNCH_TASKS slurmd: debug3: in the service_connection slurmd: debug2: Start processing RPC: REQUEST_FORWARD_DATA slurmd: debug2: Processing RPC: REQUEST_FORWARD_DATA slurmd: debug3: Entering _rpc_forward_data, address: /var/spool/slurmd/sock.pmi2.2.0, len: 84 slurmd: debug2: failed connecting to specified socket '/var/spool/slurmd/sock.pmi2.2.0': Connection refused ... I compiled mpich-4.0.0 and I can run mpi jobs outside slurm # mpirun -ppn 2 --hosts nid001001-cluster-1,nid001003-cluster-1,nid001003-cluster-1 /scratch/helloworld-mpi Warning: Permanently added 'nid001003-cluster-1,172.29.9.83' (ECDSA) to the list of known hosts. Hello world from processor nid001003-cluster-1, rank 2 out of 4 processors Hello world from processor nid001003-cluster-1, rank 3 out of 4 processors Hello world from processor nid001001-cluster-1, rank 0 out of 4 processors Hello world from processor nid001001-cluster-1, rank 1 out of 4 processors Could someone please give me a hint of what to look in regards running mpi jobs in slurm? thank you very much

1 0