Hi Hui, Any suggestion on this ? Regards, Niyaz From: Niyaz Murshed via discuss <[email protected]> Date: Tuesday, July 2, 2024 at 10:48 AM To: Zhou, Hui <[email protected]>, [email protected] <[email protected]>, Jenke, Joachim <[email protected]> Cc: Niyaz Murshed <[email protected]>, nd <[email protected]> Subject: Re: [mpich-discuss] Custom rank for processes I deleted and tried again. root@ ampere-altra-2-1: /# cd /dev/shm/ root@ ampere-altra-2-1: /dev/shm# ls -l total 24 -rw------- 1 root root 256 Jul 2 15: 07 mpich_shar_tmp7LFvuU -rw------- 1 root root 256 Jul 2 02: 01 mpich_shar_tmpDXxqHv -rw------- ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I deleted and tried again. root@ampere-altra-2-1:/# cd /dev/shm/ root@ampere-altra-2-1:/dev/shm# ls -l total 24 -rw------- 1 root root 256 Jul 2 15:07 mpich_shar_tmp7LFvuU -rw------- 1 root root 256 Jul 2 02:01 mpich_shar_tmpDXxqHv -rw------- 1 root root 256 Jul 2 15:07 mpich_shar_tmpEivSSF -rw------- 1 root root 192 Jul 2 15:07 mpich_shar_tmpXaqJui -rw------- 1 root root 192 Jul 2 02:01 mpich_shar_tmpg2Rank -rw------- 1 root root 192 Jul 2 15:07 mpich_shar_tmpxwlNrm root@ampere-altra-2-1:/dev/shm# rm -rf * root@ampere-altra-2-1:/dev/shm# mpirun -np 5 -rankmap 1,1,0,0,0 -hosts 192.168.2.100,192.168.2.200 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Abort(678028815) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory Abort(678028815) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory ^C[mpiexec@ampere-altra-2-1] Sending Ctrl-C to processes as requested [mpiexec@ampere-altra-2-1] Press Ctrl-C again to force abort [mpiexec@ampere-altra-2-1] HYDU_sock_write (lib/utils/sock.c:250): write error (Bad file descriptor) [mpiexec@ampere-altra-2-1] send_hdr_downstream (mpiexec/pmiserv_cb.c:28): sock write error [mpiexec@ampere-altra-2-1] HYD_pmcd_pmiserv_send_signal (mpiexec/pmiserv_cb.c:218): unable to write data to proxy [mpiexec@ampere-altra-2-1] ui_cmd_cb (mpiexec/pmiserv_pmci.c:61): unable to send signal downstream [mpiexec@ampere-altra-2-1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [mpiexec@ampere-altra-2-1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:173): error waiting for event [mpiexec@ampere-altra-2-1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion root@ampere-altra-2-1:/dev/shm# ls -l total 8 -rw------- 1 root root 256 Jul 2 15:46 mpich_shar_tmp0eFIGw -rw------- 1 root root 192 Jul 2 15:46 mpich_shar_tmpd4W7pC From: Zhou, Hui <[email protected]> Date: Tuesday, July 2, 2024 at 10:20 AM To: Niyaz Murshed <[email protected]>, [email protected] <[email protected]>, Jenke, Joachim <[email protected]> Cc: nd <[email protected]> Subject: Re: Custom rank for processes Not sure why it can't create shared memory. Do you have /dev/shm? Is it full? -- Hui ________________________________ From: Niyaz Murshed <[email protected]> Sent: Tuesday, July 2, 2024 10:09 AM To: Zhou, Hui <[email protected]>; [email protected] <[email protected]>; Jenke, Joachim <[email protected]> Cc: nd <[email protected]> Subject: Re: Custom rank for processes Tried both ways but gives error. root@ ampere-altra-2-1: /# mpirun -np 5 -rankmap '(vector,1,1,0,0,0)' -hosts 192. 168. 2. 100,192. 168. 2. 200 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Abort(678028815) on node 1: Fatal error in internal_Init: ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Tried both ways but gives error. root@ampere-altra-2-1:/# mpirun -np 5 -rankmap '(vector,1,1,0,0,0)' -hosts 192.168.2.100,192.168.2.200 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Abort(678028815) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory Abort(678028815) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory ^C[mpiexec@ampere-altra-2-1] Sending Ctrl-C to processes as requested [mpiexec@ampere-altra-2-1] Press Ctrl-C again to force abort [mpiexec@ampere-altra-2-1] HYDU_sock_write (lib/utils/sock.c:250): write error (Bad file descriptor) [mpiexec@ampere-altra-2-1] send_hdr_downstream (mpiexec/pmiserv_cb.c:28): sock write error [mpiexec@ampere-altra-2-1] HYD_pmcd_pmiserv_send_signal (mpiexec/pmiserv_cb.c:218): unable to write data to proxy [mpiexec@ampere-altra-2-1] ui_cmd_cb (mpiexec/pmiserv_pmci.c:61): unable to send signal downstream [mpiexec@ampere-altra-2-1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [mpiexec@ampere-altra-2-1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:173): error waiting for event [mpiexec@ampere-altra-2-1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion root@ampere-altra-2-1:/# mpirun -np 5 -rankmap 1,1,0,0,0 -hosts 192.168.2.100,192.168.2.200 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Abort(678028815) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory Abort(678028815) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory ^C[mpiexec@ampere-altra-2-1] Sending Ctrl-C to processes as requested [mpiexec@ampere-altra-2-1] Press Ctrl-C again to force abort [mpiexec@ampere-altra-2-1] HYDU_sock_write (lib/utils/sock.c:250): write error (Bad file descriptor) [mpiexec@ampere-altra-2-1] send_hdr_downstream (mpiexec/pmiserv_cb.c:28): sock write error [mpiexec@ampere-altra-2-1] HYD_pmcd_pmiserv_send_signal (mpiexec/pmiserv_cb.c:218): unable to write data to proxy [mpiexec@ampere-altra-2-1] ui_cmd_cb (mpiexec/pmiserv_pmci.c:61): unable to send signal downstream [mpiexec@ampere-altra-2-1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [mpiexec@ampere-altra-2-1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:173): error waiting for event [mpiexec@ampere-altra-2-1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion From: Zhou, Hui <[email protected]> Date: Monday, July 1, 2024 at 9:39 PM To: Niyaz Murshed <[email protected]>, [email protected] <[email protected]>, Jenke, Joachim <[email protected]> Cc: nd <[email protected]> Subject: Re: Custom rank for processes Use quote. Try -rankmap '(vector,1,1,0,0,0)'. Actually, the vector part is optional, so this should also work: -rankmap 1,1,0,0,0. -- Hui ________________________________ From: Niyaz Murshed <[email protected]> Sent: Monday, July 1, 2024 9:11 PM To: [email protected] <[email protected]>; Zhou, Hui <[email protected]>; Jenke, Joachim <[email protected]> Cc: nd <[email protected]> Subject: Re: Custom rank for processes I get the below error when trying. . am I trying it worng? root@ ampere-altra-2-1: /# mpirun -np 5 -hosts 192. 168. 2. 100,192. 168. 2. 200 -rankmap (vector,1,1,0,0,0) /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world bash: syntax error near ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I get the below error when trying.. am I trying it worng? root@ampere-altra-2-1:/# mpirun -np 5 -hosts 192.168.2.100,192.168.2.200 -rankmap (vector,1,1,0,0,0) /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world bash: syntax error near unexpected token `(' From: Niyaz Murshed via discuss <[email protected]> Date: Monday, July 1, 2024 at 4:42 PM To: Zhou, Hui <[email protected]>, [email protected] <[email protected]>, Jenke, Joachim <[email protected]> Cc: Niyaz Murshed <[email protected]>, nd <[email protected]> Subject: Re: [mpich-discuss] Custom rank for processes Thank you so much Hui. Really appreciate it. I now understand it. From: Zhou, Hui <zhouh@ anl. gov> Date: Monday, July 1, 2024 at 4: 37 PM To: Niyaz Murshed <Niyaz. Murshed@ arm. com>, discuss@ mpich. org <discuss@ mpich. org>, Jenke, ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you so much Hui. Really appreciate it. I now understand it. From: Zhou, Hui <[email protected]> Date: Monday, July 1, 2024 at 4:37 PM To: Niyaz Murshed <[email protected]>, [email protected] <[email protected]>, Jenke, Joachim <[email protected]> Cc: nd <[email protected]> Subject: Re: Custom rank for processes Yes, you can use rankmap to simply list out the node assignment for each rank. "-rankmap (vector,1,1,0,0,0)" is a list of 5 node ids, one for each rank. So rank 0 gets node 1, rank 1 gets node 1, rank 2 gets node 0, and so on. The -hosts options is convenient if you have somewhat uniform assignment. If you want arbitrary assignment, just use -rankmap option. -- Hui ________________________________ From: Niyaz Murshed <[email protected]> Sent: Monday, July 1, 2024 4:30 PM To: Zhou, Hui <[email protected]>; [email protected] <[email protected]>; Jenke, Joachim <[email protected]> Cc: nd <[email protected]> Subject: Re: Custom rank for processes Sorry to bother you again with silly questions. How do I read the below : -rankmap (vector,1,1,0,0,0) I have 5 processes. 1 1 => means node1 will get first 2 processes ? -hosts 192. 168. 2. 100: 2,192. 168. 2. 200: 3 : this will give 2 process in ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Sorry to bother you again with silly questions. How do I read the below : -rankmap (vector,1,1,0,0,0) I have 5 processes. 1 1 => means node1 will get first 2 processes ? -hosts 192.168.2.100:2,192.168.2.200:3 : this will give 2 process in 100 and 3 in 200 .. Still won’t be able to specify which ranks goes to which node. Rank0 will be on node 100 .. what If I want Rank0 on node 200? From: Zhou, Hui <[email protected]> Date: Monday, July 1, 2024 at 4:09 PM To: Niyaz Murshed <[email protected]>, [email protected] <[email protected]>, Jenke, Joachim <[email protected]> Cc: nd <[email protected]> Subject: Re: Custom rank for processes
root@ampere-altra-2-1:/# mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world
Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors Hello world from processor dpr740, rank 0 out of 5 processors Hello world from processor dpr740, rank 4 out of 5 processors Hello world from processor dpr740, rank 2 out of 5 processors
-bind-to user:10,11,12,13
This would mean on host 192.168.2.100
P0=>10 , P2=>11
This would mean on host 192.168.2.200
P0=>10 , P2=>11, P3=12
Is this correct understanding ? Is it also possible to say which rank process will be pinned to which core ?
Yes, that is correct. The ranks are assigned to hosts as shown in the hello world output.
About the rankmap:, trying to understand if I can select where a particular rank would be from > list of hosts. Currently, the first host in the list always get rank0.
Can I specify the below ranks?
mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world
Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors => rank0
Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors. => rank1
Hello world from processor dpr740, rank 0 out of 5 processors =>rank2
Hello world from processor dpr740, rank 4 out of 5 processors =>rank3
Hello world from processor dpr740, rank 2 out of 5 processors =>rank4
Yes. You can use "-rankmap (vector,1,1,0,0,0)". Alternatively, you can use "-hosts 192.168.2.100:2,192.168.2.200:3", the colon syntax specifies how many processes you want to assign to each host. -- Hui