- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] yaksa: leaked handle pool objects
by Zhou, Hui 23 Jul '24

23 Jul '24

They are leak check warnings. They indicate not all datatype handles are freed (via MPI_Type_free). We understand not all users care about these handle leaks, thus this warning will be disabled by default in the next release. -- Hui Zhou From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Date: Tuesday, July 23, 2024 at 11:08 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] yaksa: leaked handle pool objects In my MPICH 4. 1. 2 application, I am getting a few of these warnings, when MPI_Finalize is called. [WARNING] yaksa: 2 leaked handle pool objects Is there any way to tell where/why they are occurring? Thanks, Kurt . ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd In my MPICH 4.1.2 application, I am getting a few of these warnings, when MPI_Finalize is called. [WARNING] yaksa: 2 leaked handle pool objects Is there any way to tell where/why they are occurring? Thanks, Kurt .

1 0

yaksa: leaked handle pool objects
by Mccall, Kurt E. (MSFC-EV41) 23 Jul '24

23 Jul '24

In my MPICH 4.1.2 application, I am getting a few of these warnings, when MPI_Finalize is called. [WARNING] yaksa: 2 leaked handle pool objects Is there any way to tell where/why they are occurring? Thanks, Kurt .

1 0

Re: [mpich-discuss] Seeking possible causes for an assertion error in socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || ...
by kumar.tarun＠siemens.com 17 Jul '24

17 Jul '24

Thanks. I had further discussion on this on github: https://urldefense.us/v3/__https://github.com/pmodels/mpich/discussions/705… I’m reposting my last response there as I need some answers. We got the confirmation from the customer that there was a security application which was somehow killing the application as the application(and it's executables) were not whitelisted. I have couple of questions: 1. Is the latest stable release of mpi4.2.2 stable against such instances? Is single machine run still require TCP ports or is it done using shared memory? 2. Is there anything that we can do while configuring mpich 3.2.1(the current version we are using) to make the application more robust against such instances? As I said it's a single machine run, so I'm still not sure why TCP ports are involved. Thanks Tarun From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Sunday, July 7, 2024 5:24 AM To: discuss(a)mpich.org Cc: Kumar, Tarun (DI SW ICS DVT RD QSCE) <kumar.tarun(a)siemens.com> Subject: Re: [mpich-discuss] Seeking possible causes for an assertion error in socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || ... Are you able to update the MPICH version? MPICH 3.2.1 was released in 2017 and is no longer actively supported/maintained. Ken From: "kumar.tarun--- via discuss" <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Saturday, July 6, 2024 at 3:49 PM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: "kumar.tarun(a)siemens.com<mailto:[email protected]>" <kumar.tarun(a)siemens.com<mailto:[email protected]>> Subject: [mpich-discuss] Seeking possible causes for an assertion error in socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || ... Hi, We are hitting following assertion: Assertion failed in file .. . nemesis/netmod/tcp/socksm. c at line 600: hdr. pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || .. . I looked at the assert and it looks like this in file …/nemesis/netmod/tcp/socksm. c ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, We are hitting following assertion: Assertion failed in file ...nemesis/netmod/tcp/socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || ... I looked at the assert and it looks like this in file …/nemesis/netmod/tcp/socksm.c MPIU_Assert(hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_TMPVC_INFO); We have tried multiple cores/partitions from 2 to 8 and the behaviour is same. Also a process is aborted and a message appears to suggest that. Mostly it's process 0 which is aborted but I have seen other processes as well reporting the crash. We are using mpich-3.2.1. I'm trying to understand possible causes for this error? I have explored the forum and no possible causes, like machine going out of memory etc are applicable here. Please suggest. Are there any debug/log/trace options I can use with mpiexec to further root cause? Regards Tarun

1 0

Re: [mpich-discuss] MPICH fails to allocate memory at beginning of job
by Zhou, Hui 08 Jul '24

08 Jul '24

The ucx netmod provides better support on infiniband. Could you try configure it with --with-device=ch4:ucx? -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Sent: Monday, July 8, 2024 3:00 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] MPICH fails to allocate memory at beginning of job I configured MPICH 4. 1. 2 as follows. Any clue as to what would fix the error below? $ cd mpich-4. 1. 2-build $ ./mpich-4. 1. 2/configure --prefix=/opt/mpich --with-device=ch4: ofi --with-libfabric-embedded --with-slurm -enable-debuginfo --enable-g=debug ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I configured MPICH 4.1.2 as follows. Any clue as to what would fix the error below? $ cd mpich-4.1.2-build $ ./mpich-4.1.2/configure --prefix=/opt/mpich --with-device=ch4:ofi --with-libfabric-embedded --with-slurm -enable-debuginfo --enable-g=debug 2>&1 | tee c.txt When I run a job, the error message on each node is: n007.cluster.pssclabs.com:rank6.HaystackMpiMM: Failed to modify UD QP to INIT on mlx5_0: Operation not permitted Abort(337761679): Fatal error in internal_Init: Other MPI error, error stack: internal_Init(66)........: MPI_Init(argc=0x7ffd93a663ec, argv=0x7ffd93a663e0) failed MPII_Init_thread(234)....: MPID_Init(513)...........: MPIDI_OFI_init_local(604): create_vni_context(982)..: OFI endpoint open failed (ofi_init.c:982:create_vni_context:Cannot allocate memory) The run command is: mpiexec -launcher ssh \ -print-all-exitcodes \ -wdir ${work_dir} \ -np ${num_proc} \ -ppn 1 \ my_program … Thanks, Kurt

1 0

MPICH fails to allocate memory at beginning of job
by Mccall, Kurt E. (MSFC-EV41) 08 Jul '24

08 Jul '24

I configured MPICH 4.1.2 as follows. Any clue as to what would fix the error below? $ cd mpich-4.1.2-build $ ./mpich-4.1.2/configure --prefix=/opt/mpich --with-device=ch4:ofi --with-libfabric-embedded --with-slurm -enable-debuginfo --enable-g=debug 2>&1 | tee c.txt When I run a job, the error message on each node is: n007.cluster.pssclabs.com:rank6.HaystackMpiMM: Failed to modify UD QP to INIT on mlx5_0: Operation not permitted Abort(337761679): Fatal error in internal_Init: Other MPI error, error stack: internal_Init(66)........: MPI_Init(argc=0x7ffd93a663ec, argv=0x7ffd93a663e0) failed MPII_Init_thread(234)....: MPID_Init(513)...........: MPIDI_OFI_init_local(604): create_vni_context(982)..: OFI endpoint open failed (ofi_init.c:982:create_vni_context:Cannot allocate memory) The run command is: mpiexec -launcher ssh \ -print-all-exitcodes \ -wdir ${work_dir} \ -np ${num_proc} \ -ppn 1 \ my_program ... Thanks, Kurt

1 0

Re: [mpich-discuss] Seeking possible causes for an assertion error in socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || ...
by Raffenetti, Ken 07 Jul '24

07 Jul '24

Are you able to update the MPICH version? MPICH 3.2.1 was released in 2017 and is no longer actively supported/maintained. Ken From: "kumar.tarun--- via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Saturday, July 6, 2024 at 3:49 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "kumar.tarun(a)siemens.com" <kumar.tarun(a)siemens.com> Subject: [mpich-discuss] Seeking possible causes for an assertion error in socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || ... Hi, We are hitting following assertion: Assertion failed in file .. . nemesis/netmod/tcp/socksm. c at line 600: hdr. pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || .. . I looked at the assert and it looks like this in file …/nemesis/netmod/tcp/socksm. c ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, We are hitting following assertion: Assertion failed in file ...nemesis/netmod/tcp/socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || ... I looked at the assert and it looks like this in file …/nemesis/netmod/tcp/socksm.c MPIU_Assert(hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_TMPVC_INFO); We have tried multiple cores/partitions from 2 to 8 and the behaviour is same. Also a process is aborted and a message appears to suggest that. Mostly it's process 0 which is aborted but I have seen other processes as well reporting the crash. We are using mpich-3.2.1. I'm trying to understand possible causes for this error? I have explored the forum and no possible causes, like machine going out of memory etc are applicable here. Please suggest. Are there any debug/log/trace options I can use with mpiexec to further root cause? Regards Tarun

1 0

Seeking possible causes for an assertion error in socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || ...
by kumar.tarun＠siemens.com 06 Jul '24

06 Jul '24

Hi, We are hitting following assertion: Assertion failed in file ...nemesis/netmod/tcp/socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || ... I looked at the assert and it looks like this in file .../nemesis/netmod/tcp/socksm.c MPIU_Assert(hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_TMPVC_INFO); We have tried multiple cores/partitions from 2 to 8 and the behaviour is same. Also a process is aborted and a message appears to suggest that. Mostly it's process 0 which is aborted but I have seen other processes as well reporting the crash. We are using mpich-3.2.1. I'm trying to understand possible causes for this error? I have explored the forum and no possible causes, like machine going out of memory etc are applicable here. Please suggest. Are there any debug/log/trace options I can use with mpiexec to further root cause? Regards Tarun

1 0

Re: [mpich-discuss] Custom rank for processes
by Zhou, Hui 05 Jul '24

05 Jul '24

Niyaz, I don't know what prevents your from using shared memory. But in the meantime if you want to turn off shared memory, you could try configure mpich with --without-ch4-shmmods. -- Hui ________________________________ From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Sent: Friday, July 5, 2024 9:14 AM To: discuss(a)mpich.org <discuss(a)mpich.org>; Zhou, Hui <zhouh(a)anl.gov>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Hi Hui, Any suggestion on this ? Regards, Niyaz From: Niyaz Murshed via discuss <discuss@ mpich. org> Date: Tuesday, July 2, 2024 at 10: 48 AM To: Zhou, Hui <zhouh@ anl. gov>, discuss@ mpich. org <discuss@ mpich. org>, Jenke, Joachim ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Hui, Any suggestion on this ? Regards, Niyaz From: Niyaz Murshed via discuss <discuss(a)mpich.org> Date: Tuesday, July 2, 2024 at 10:48 AM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: [mpich-discuss] Custom rank for processes I deleted and tried again. root@ ampere-altra-2-1: /# cd /dev/shm/ root@ ampere-altra-2-1: /dev/shm# ls -l total 24 -rw------- 1 root root 256 Jul 2 15: 07 mpich_shar_tmp7LFvuU -rw------- 1 root root 256 Jul 2 02: 01 mpich_shar_tmpDXxqHv -rw------- ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I deleted and tried again. root@ampere-altra-2-1:/# cd /dev/shm/ root@ampere-altra-2-1:/dev/shm# ls -l total 24 -rw------- 1 root root 256 Jul 2 15:07 mpich_shar_tmp7LFvuU -rw------- 1 root root 256 Jul 2 02:01 mpich_shar_tmpDXxqHv -rw------- 1 root root 256 Jul 2 15:07 mpich_shar_tmpEivSSF -rw------- 1 root root 192 Jul 2 15:07 mpich_shar_tmpXaqJui -rw------- 1 root root 192 Jul 2 02:01 mpich_shar_tmpg2Rank -rw------- 1 root root 192 Jul 2 15:07 mpich_shar_tmpxwlNrm root@ampere-altra-2-1:/dev/shm# rm -rf * root@ampere-altra-2-1:/dev/shm# mpirun -np 5 -rankmap 1,1,0,0,0 -hosts 192.168.2.100,192.168.2.200 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Abort(678028815) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory Abort(678028815) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory ^C[mpiexec@ampere-altra-2-1] Sending Ctrl-C to processes as requested [mpiexec@ampere-altra-2-1] Press Ctrl-C again to force abort [mpiexec@ampere-altra-2-1] HYDU_sock_write (lib/utils/sock.c:250): write error (Bad file descriptor) [mpiexec@ampere-altra-2-1] send_hdr_downstream (mpiexec/pmiserv_cb.c:28): sock write error [mpiexec@ampere-altra-2-1] HYD_pmcd_pmiserv_send_signal (mpiexec/pmiserv_cb.c:218): unable to write data to proxy [mpiexec@ampere-altra-2-1] ui_cmd_cb (mpiexec/pmiserv_pmci.c:61): unable to send signal downstream [mpiexec@ampere-altra-2-1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [mpiexec@ampere-altra-2-1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:173): error waiting for event [mpiexec@ampere-altra-2-1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion root@ampere-altra-2-1:/dev/shm# ls -l total 8 -rw------- 1 root root 256 Jul 2 15:46 mpich_shar_tmp0eFIGw -rw------- 1 root root 192 Jul 2 15:46 mpich_shar_tmpd4W7pC From: Zhou, Hui <zhouh(a)anl.gov> Date: Tuesday, July 2, 2024 at 10:20 AM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Not sure why it can't create shared memory. Do you have /dev/shm? Is it full? -- Hui ________________________________ From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Sent: Tuesday, July 2, 2024 10:09 AM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Tried both ways but gives error. root@ ampere-altra-2-1: /# mpirun -np 5 -rankmap '(vector,1,1,0,0,0)' -hosts 192. 168. 2. 100,192. 168. 2. 200 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Abort(678028815) on node 1: Fatal error in internal_Init: ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Tried both ways but gives error. root@ampere-altra-2-1:/# mpirun -np 5 -rankmap '(vector,1,1,0,0,0)' -hosts 192.168.2.100,192.168.2.200 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Abort(678028815) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory Abort(678028815) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory ^C[mpiexec@ampere-altra-2-1] Sending Ctrl-C to processes as requested [mpiexec@ampere-altra-2-1] Press Ctrl-C again to force abort [mpiexec@ampere-altra-2-1] HYDU_sock_write (lib/utils/sock.c:250): write error (Bad file descriptor) [mpiexec@ampere-altra-2-1] send_hdr_downstream (mpiexec/pmiserv_cb.c:28): sock write error [mpiexec@ampere-altra-2-1] HYD_pmcd_pmiserv_send_signal (mpiexec/pmiserv_cb.c:218): unable to write data to proxy [mpiexec@ampere-altra-2-1] ui_cmd_cb (mpiexec/pmiserv_pmci.c:61): unable to send signal downstream [mpiexec@ampere-altra-2-1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [mpiexec@ampere-altra-2-1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:173): error waiting for event [mpiexec@ampere-altra-2-1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion root@ampere-altra-2-1:/# mpirun -np 5 -rankmap 1,1,0,0,0 -hosts 192.168.2.100,192.168.2.200 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Abort(678028815) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory Abort(678028815) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory ^C[mpiexec@ampere-altra-2-1] Sending Ctrl-C to processes as requested [mpiexec@ampere-altra-2-1] Press Ctrl-C again to force abort [mpiexec@ampere-altra-2-1] HYDU_sock_write (lib/utils/sock.c:250): write error (Bad file descriptor) [mpiexec@ampere-altra-2-1] send_hdr_downstream (mpiexec/pmiserv_cb.c:28): sock write error [mpiexec@ampere-altra-2-1] HYD_pmcd_pmiserv_send_signal (mpiexec/pmiserv_cb.c:218): unable to write data to proxy [mpiexec@ampere-altra-2-1] ui_cmd_cb (mpiexec/pmiserv_pmci.c:61): unable to send signal downstream [mpiexec@ampere-altra-2-1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [mpiexec@ampere-altra-2-1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:173): error waiting for event [mpiexec@ampere-altra-2-1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 9:39 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Use quote. Try -rankmap '(vector,1,1,0,0,0)'. Actually, the vector part is optional, so this should also work: -rankmap 1,1,0,0,0. -- Hui ________________________________ From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Sent: Monday, July 1, 2024 9:11 PM To: discuss(a)mpich.org <discuss(a)mpich.org>; Zhou, Hui <zhouh(a)anl.gov>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes I get the below error when trying. . am I trying it worng? root@ ampere-altra-2-1: /# mpirun -np 5 -hosts 192. 168. 2. 100,192. 168. 2. 200 -rankmap (vector,1,1,0,0,0) /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world bash: syntax error near ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I get the below error when trying.. am I trying it worng? root@ampere-altra-2-1:/# mpirun -np 5 -hosts 192.168.2.100,192.168.2.200 -rankmap (vector,1,1,0,0,0) /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world bash: syntax error near unexpected token `(' From: Niyaz Murshed via discuss <discuss(a)mpich.org> Date: Monday, July 1, 2024 at 4:42 PM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: [mpich-discuss] Custom rank for processes Thank you so much Hui. Really appreciate it. I now understand it. From: Zhou, Hui <zhouh@ anl. gov> Date: Monday, July 1, 2024 at 4: 37 PM To: Niyaz Murshed <Niyaz. Murshed@ arm. com>, discuss@ mpich. org <discuss@ mpich. org>, Jenke, ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you so much Hui. Really appreciate it. I now understand it. From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 4:37 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Yes, you can use rankmap to simply list out the node assignment for each rank. "-rankmap (vector,1,1,0,0,0)" is a list of 5 node ids, one for each rank. So rank 0 gets node 1, rank 1 gets node 1, rank 2 gets node 0, and so on. The -hosts options is convenient if you have somewhat uniform assignment. If you want arbitrary assignment, just use -rankmap option. -- Hui ________________________________ From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Sent: Monday, July 1, 2024 4:30 PM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Sorry to bother you again with silly questions. How do I read the below : -rankmap (vector,1,1,0,0,0) I have 5 processes. 1 1 => means node1 will get first 2 processes ? -hosts 192. 168. 2. 100: 2,192. 168. 2. 200: 3 : this will give 2 process in ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Sorry to bother you again with silly questions. How do I read the below : -rankmap (vector,1,1,0,0,0) I have 5 processes. 1 1 => means node1 will get first 2 processes ? -hosts 192.168.2.100:2,192.168.2.200:3 : this will give 2 process in 100 and 3 in 200 .. Still won’t be able to specify which ranks goes to which node. Rank0 will be on node 100 .. what If I want Rank0 on node 200? From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 4:09 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes > root@ampere-altra-2-1:/# mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors Hello world from processor dpr740, rank 0 out of 5 processors Hello world from processor dpr740, rank 4 out of 5 processors Hello world from processor dpr740, rank 2 out of 5 processors > -bind-to user:10,11,12,13 > This would mean on host 192.168.2.100 > P0=>10 , P2=>11 > This would mean on host 192.168.2.200 > P0=>10 , P2=>11, P3=12 > Is this correct understanding ? Is it also possible to say which rank process will be pinned to which core ? Yes, that is correct. The ranks are assigned to hosts as shown in the hello world output. > About the rankmap:, trying to understand if I can select where a particular rank would be from > list of hosts. Currently, the first host in the list always get rank0. > Can I specify the below ranks? > mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world > > Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors => rank0 > Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors. => rank1 > Hello world from processor dpr740, rank 0 out of 5 processors =>rank2 > Hello world from processor dpr740, rank 4 out of 5 processors =>rank3 > Hello world from processor dpr740, rank 2 out of 5 processors =>rank4 Yes. You can use "-rankmap (vector,1,1,0,0,0)". Alternatively, you can use "-hosts 192.168.2.100:2,192.168.2.200:3", the colon syntax specifies how many processes you want to assign to each host. -- Hui

1 0

Re: [mpich-discuss] Custom rank for processes
by Niyaz Murshed 05 Jul '24

05 Jul '24

Hi Hui, Any suggestion on this ? Regards, Niyaz From: Niyaz Murshed via discuss <discuss(a)mpich.org> Date: Tuesday, July 2, 2024 at 10:48 AM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: [mpich-discuss] Custom rank for processes I deleted and tried again. root@ ampere-altra-2-1: /# cd /dev/shm/ root@ ampere-altra-2-1: /dev/shm# ls -l total 24 -rw------- 1 root root 256 Jul 2 15: 07 mpich_shar_tmp7LFvuU -rw------- 1 root root 256 Jul 2 02: 01 mpich_shar_tmpDXxqHv -rw------- ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I deleted and tried again. root@ampere-altra-2-1:/# cd /dev/shm/ root@ampere-altra-2-1:/dev/shm# ls -l total 24 -rw------- 1 root root 256 Jul 2 15:07 mpich_shar_tmp7LFvuU -rw------- 1 root root 256 Jul 2 02:01 mpich_shar_tmpDXxqHv -rw------- 1 root root 256 Jul 2 15:07 mpich_shar_tmpEivSSF -rw------- 1 root root 192 Jul 2 15:07 mpich_shar_tmpXaqJui -rw------- 1 root root 192 Jul 2 02:01 mpich_shar_tmpg2Rank -rw------- 1 root root 192 Jul 2 15:07 mpich_shar_tmpxwlNrm root@ampere-altra-2-1:/dev/shm# rm -rf * root@ampere-altra-2-1:/dev/shm# mpirun -np 5 -rankmap 1,1,0,0,0 -hosts 192.168.2.100,192.168.2.200 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Abort(678028815) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory Abort(678028815) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory ^C[mpiexec@ampere-altra-2-1] Sending Ctrl-C to processes as requested [mpiexec@ampere-altra-2-1] Press Ctrl-C again to force abort [mpiexec@ampere-altra-2-1] HYDU_sock_write (lib/utils/sock.c:250): write error (Bad file descriptor) [mpiexec@ampere-altra-2-1] send_hdr_downstream (mpiexec/pmiserv_cb.c:28): sock write error [mpiexec@ampere-altra-2-1] HYD_pmcd_pmiserv_send_signal (mpiexec/pmiserv_cb.c:218): unable to write data to proxy [mpiexec@ampere-altra-2-1] ui_cmd_cb (mpiexec/pmiserv_pmci.c:61): unable to send signal downstream [mpiexec@ampere-altra-2-1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [mpiexec@ampere-altra-2-1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:173): error waiting for event [mpiexec@ampere-altra-2-1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion root@ampere-altra-2-1:/dev/shm# ls -l total 8 -rw------- 1 root root 256 Jul 2 15:46 mpich_shar_tmp0eFIGw -rw------- 1 root root 192 Jul 2 15:46 mpich_shar_tmpd4W7pC From: Zhou, Hui <zhouh(a)anl.gov> Date: Tuesday, July 2, 2024 at 10:20 AM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Not sure why it can't create shared memory. Do you have /dev/shm? Is it full? -- Hui ________________________________ From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Sent: Tuesday, July 2, 2024 10:09 AM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Tried both ways but gives error. root@ ampere-altra-2-1: /# mpirun -np 5 -rankmap '(vector,1,1,0,0,0)' -hosts 192. 168. 2. 100,192. 168. 2. 200 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Abort(678028815) on node 1: Fatal error in internal_Init: ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Tried both ways but gives error. root@ampere-altra-2-1:/# mpirun -np 5 -rankmap '(vector,1,1,0,0,0)' -hosts 192.168.2.100,192.168.2.200 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Abort(678028815) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory Abort(678028815) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory ^C[mpiexec@ampere-altra-2-1] Sending Ctrl-C to processes as requested [mpiexec@ampere-altra-2-1] Press Ctrl-C again to force abort [mpiexec@ampere-altra-2-1] HYDU_sock_write (lib/utils/sock.c:250): write error (Bad file descriptor) [mpiexec@ampere-altra-2-1] send_hdr_downstream (mpiexec/pmiserv_cb.c:28): sock write error [mpiexec@ampere-altra-2-1] HYD_pmcd_pmiserv_send_signal (mpiexec/pmiserv_cb.c:218): unable to write data to proxy [mpiexec@ampere-altra-2-1] ui_cmd_cb (mpiexec/pmiserv_pmci.c:61): unable to send signal downstream [mpiexec@ampere-altra-2-1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [mpiexec@ampere-altra-2-1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:173): error waiting for event [mpiexec@ampere-altra-2-1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion root@ampere-altra-2-1:/# mpirun -np 5 -rankmap 1,1,0,0,0 -hosts 192.168.2.100,192.168.2.200 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Abort(678028815) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory Abort(678028815) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory ^C[mpiexec@ampere-altra-2-1] Sending Ctrl-C to processes as requested [mpiexec@ampere-altra-2-1] Press Ctrl-C again to force abort [mpiexec@ampere-altra-2-1] HYDU_sock_write (lib/utils/sock.c:250): write error (Bad file descriptor) [mpiexec@ampere-altra-2-1] send_hdr_downstream (mpiexec/pmiserv_cb.c:28): sock write error [mpiexec@ampere-altra-2-1] HYD_pmcd_pmiserv_send_signal (mpiexec/pmiserv_cb.c:218): unable to write data to proxy [mpiexec@ampere-altra-2-1] ui_cmd_cb (mpiexec/pmiserv_pmci.c:61): unable to send signal downstream [mpiexec@ampere-altra-2-1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [mpiexec@ampere-altra-2-1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:173): error waiting for event [mpiexec@ampere-altra-2-1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 9:39 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Use quote. Try -rankmap '(vector,1,1,0,0,0)'. Actually, the vector part is optional, so this should also work: -rankmap 1,1,0,0,0. -- Hui ________________________________ From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Sent: Monday, July 1, 2024 9:11 PM To: discuss(a)mpich.org <discuss(a)mpich.org>; Zhou, Hui <zhouh(a)anl.gov>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes I get the below error when trying. . am I trying it worng? root@ ampere-altra-2-1: /# mpirun -np 5 -hosts 192. 168. 2. 100,192. 168. 2. 200 -rankmap (vector,1,1,0,0,0) /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world bash: syntax error near ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I get the below error when trying.. am I trying it worng? root@ampere-altra-2-1:/# mpirun -np 5 -hosts 192.168.2.100,192.168.2.200 -rankmap (vector,1,1,0,0,0) /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world bash: syntax error near unexpected token `(' From: Niyaz Murshed via discuss <discuss(a)mpich.org> Date: Monday, July 1, 2024 at 4:42 PM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: [mpich-discuss] Custom rank for processes Thank you so much Hui. Really appreciate it. I now understand it. From: Zhou, Hui <zhouh@ anl. gov> Date: Monday, July 1, 2024 at 4: 37 PM To: Niyaz Murshed <Niyaz. Murshed@ arm. com>, discuss@ mpich. org <discuss@ mpich. org>, Jenke, ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you so much Hui. Really appreciate it. I now understand it. From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 4:37 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Yes, you can use rankmap to simply list out the node assignment for each rank. "-rankmap (vector,1,1,0,0,0)" is a list of 5 node ids, one for each rank. So rank 0 gets node 1, rank 1 gets node 1, rank 2 gets node 0, and so on. The -hosts options is convenient if you have somewhat uniform assignment. If you want arbitrary assignment, just use -rankmap option. -- Hui ________________________________ From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Sent: Monday, July 1, 2024 4:30 PM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Sorry to bother you again with silly questions. How do I read the below : -rankmap (vector,1,1,0,0,0) I have 5 processes. 1 1 => means node1 will get first 2 processes ? -hosts 192. 168. 2. 100: 2,192. 168. 2. 200: 3 : this will give 2 process in ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Sorry to bother you again with silly questions. How do I read the below : -rankmap (vector,1,1,0,0,0) I have 5 processes. 1 1 => means node1 will get first 2 processes ? -hosts 192.168.2.100:2,192.168.2.200:3 : this will give 2 process in 100 and 3 in 200 .. Still won’t be able to specify which ranks goes to which node. Rank0 will be on node 100 .. what If I want Rank0 on node 200? From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 4:09 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes > root@ampere-altra-2-1:/# mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors Hello world from processor dpr740, rank 0 out of 5 processors Hello world from processor dpr740, rank 4 out of 5 processors Hello world from processor dpr740, rank 2 out of 5 processors > -bind-to user:10,11,12,13 > This would mean on host 192.168.2.100 > P0=>10 , P2=>11 > This would mean on host 192.168.2.200 > P0=>10 , P2=>11, P3=12 > Is this correct understanding ? Is it also possible to say which rank process will be pinned to which core ? Yes, that is correct. The ranks are assigned to hosts as shown in the hello world output. > About the rankmap:, trying to understand if I can select where a particular rank would be from > list of hosts. Currently, the first host in the list always get rank0. > Can I specify the below ranks? > mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world > > Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors => rank0 > Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors. => rank1 > Hello world from processor dpr740, rank 0 out of 5 processors =>rank2 > Hello world from processor dpr740, rank 4 out of 5 processors =>rank3 > Hello world from processor dpr740, rank 2 out of 5 processors =>rank4 Yes. You can use "-rankmap (vector,1,1,0,0,0)". Alternatively, you can use "-hosts 192.168.2.100:2,192.168.2.200:3", the colon syntax specifies how many processes you want to assign to each host. -- Hui

1 0

Re: [mpich-discuss] All processes only find themselfs
by Raffenetti, Ken 03 Jul '24

03 Jul '24

I believe there has been an error in packaging, possibly resulting from a change in configure options, that has rendered the MPICH library incompatible with Hydra's mpiexec. ☹ I have no good solution besides building your own MPICH from source. I will reach out to the Ubuntu maintainer about the issue. Thanks for reporting. Ken -----Original Message----- From: Joachim Jenke <jenke(a)itc.rwth-aachen.de <mailto:[email protected]>> Date: Wednesday, July 3, 2024 at 12:20 PM To: "Raffenetti, Ken" <raffenet(a)anl.gov <mailto:[email protected]>>, "discuss(a)mpich.org <mailto:[email protected]>" <discuss(a)mpich.org <mailto:[email protected]>> Subject: Re: [mpich-discuss] All processes only find themselfs $ env MPIR_CVAR_PMI_VERSION=1 mpirun -np 2 -l env MPIR_CVAR_PMI_VERSION=1 ./a.out [0] Hello, I am 0 of 1 tasks. [1] Hello, I am 0 of 1 tasks. still the same. I was not sure whether that option is for mpiexec or the mpi library, so I set it twice just to be sure. Am 03.07.24 um 16:54 schrieb Raffenetti, Ken: > Hm, there is a PMIx dependency that is presumably coming from the MPICH library. MPICH's mpiexec is not compatible with an external PMIx, so I suspect that is the issue. Can you try setting MPIR_CVAR_PMI_VERSION=1 and see if that works? > > Ken > > -----Original Message----- > From: Joachim Jenke <jenke(a)itc.rwth-aachen.de <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> > Date: Wednesday, July 3, 2024 at 9:36 AM > To: "Raffenetti, Ken" <raffenet(a)anl.gov <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>>, "discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>" <discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> > Subject: Re: [mpich-discuss] All processes only find themselfs > > > $ ldd a.out > linux-vdso.so.1 (0x00007ffee8145000) > libmpich.so.12 => /lib/x86_64-linux-gnu/libmpich.so.12 (0x00007f15d3200000) > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f15d2e00000) > libhwloc.so.15 => /lib/x86_64-linux-gnu/libhwloc.so.15 (0x00007f15d5a13000) > libpmix.so.2 => /lib/x86_64-linux-gnu/libpmix.so.2 (0x00007f15d2a00000) > libucp.so.0 => /lib/x86_64-linux-gnu/libucp.so.0 (0x00007f15d309d000) > libucs.so.0 => /lib/x86_64-linux-gnu/libucs.so.0 (0x00007f15d59ab000) > /lib64/ld-linux-x86-64.so.2 (0x00007f15d5aa3000) > libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f15d2d17000) > libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f15d5978000) > libmunge.so.2 => /lib/x86_64-linux-gnu/libmunge.so.2 (0x00007f15d5970000) > libevent_core-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_core-2.1.so.7 > (0x00007f15d3068000) > libevent_pthreads-2.1.so.7 => > /lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x00007f15d5969000) > libuct.so.0 => /lib/x86_64-linux-gnu/libuct.so.0 (0x00007f15d3022000) > libucm.so.0 => /lib/x86_64-linux-gnu/libucm.so.0 (0x00007f15d594c000) > libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x00007f15d3015000) > > > $ apt-cache policy libmpich-dev > libmpich-dev: > Installed: 4.2.0-5build3 > Candidate: 4.2.0-5build3 > Version table: > *** 4.2.0-5build3 500 > 500 https://urldefense.us/v3/__http://de.archive.ubuntu.com/ubuntu__;!!G_uCfscf… <https://urldefense.us/v3/__http://de.archive.ubuntu.com/ubuntu__;!!G_uCfscf… > <https://urldefense.us/v3/__http://de.archive.ubuntu.com/ubuntu__;!!G_uCfscf… > <https://urldefense.us/v3/__http://de.archive.ubuntu.com/ubuntu>__;!!G_uC… > noble/universe amd64 > Packages > 100 /var/lib/dpkg/status > > > > > > > $ mpirun -np 2 --verbose ./a.out > host: ITC19206 > [mpiexec@ITC19206] Timeout set to -1 (-1 means infinite) > > > ================================================================================================== > mpiexec options: > ---------------- > Base path: /usr/bin/ > Launcher: (null) > Debug level: 1 > Enable X: -1 > > > Global environment: > ------------------- > SHELL=/bin/bash > > > SESSION_MANAGER=local/ITC19206:@/tmp/.ICE-unix/3236,unix/ITC19206:/tmp/.ICE-unix/3236 > QT_ACCESSIBILITY=1 > COLORTERM=truecolor > XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg > XDG_MENU_PREFIX=gnome- > GNOME_DESKTOP_SESSION_ID=this-is-deprecated > MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path > GNOME_SHELL_SESSION_MODE=ubuntu > SSH_AUTH_SOCK=/run/user/1000/keyring/ssh > MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA= > XMODIFIERS=@im=ibus > DESKTOP_SESSION=ubuntu > GTK_MODULES=gail:atk-bridge > XDG_SESSION_DESKTOP=ubuntu > XDG_SESSION_TYPE=x11 > MODULESHOME=/usr/share/modules > GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1 > SYSTEMD_EXEC_PID=3274 > XAUTHORITY=/run/user/1000/gdm/Xauthority > WINDOWPATH=2 > LANG=de_DE.UTF-8 > > > LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90: > XDG_CURRENT_DESKTOP=ubuntu:GNOME > > > MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/user.slice/user-1000.slice/user(a)1000.serv <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>ice/session.slice/org.gnome.Shell(a)x11.serv <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>ice/memory.pressure > VTE_VERSION=7600 > > > GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/63974876_8215_47b9_b16e_caeb54580fbc > QTWEBENGINE_DICTIONARIES_PATH=/usr/share/hunspell-bdic/ > LESSCLOSE=/usr/bin/lesspipe %s %s > XDG_SESSION_CLASS=user > TERM=xterm-256color > DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path > LESSOPEN=| /usr/bin/lesspipe %s > GNOME_TERMINAL_SERVICE=:1.95 > LOADEDMODULES= > SHLVL=1 > GSM_SKIP_SSH_AGENT_WORKAROUND=true > QT_IM_MODULE=ibus > XDG_RUNTIME_DIR=/run/user/1000 > __MODULES_LMINIT=module use --append > /etc/environment-modules/modules:module use --append > /usr/share/modules/versions:module use --append > /usr/share/modules/$MODULE_VERSION/modulefiles:module use --append > /usr/share/modules/modulefiles > DEBUGINFOD_URLS=https://urldefense.us/v3/__https://debuginfod.ubuntu.com__;!!G_uCfscf7eWS!euxkrFIlskGc4_et1CGV_3awQoswa8z0MgIGJoYCZiPO8hwHcXAmV-uUkpOo8Yl6IXlDO42OkpkcRunx$ <https://urldefense.us/v3/__https://debuginfod.ubuntu.com__;!!G_uCfscf7eWS!e… > <https://urldefense.us/v3/__https://debuginfod.ubuntu.com__;!!G_uCfscf7eWS!e… > <https://urldefense.us/v3/__https://debuginfod.ubuntu.com>__;!!G_uCfscf7e… > > > > XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop > > > PATH=/usr/lib/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin > > > MODULEPATH=/etc/environment-modules/modules:/usr/share/modules/versions:/usr/share/modules/$MODULE_VERSION/modulefiles:/usr/share/modules/modulefiles > GDMSESSION=ubuntu > DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus > MODULES_CMD=/usr/lib/x86_64-linux-gnu/modulecmd.tcl > BASH_FUNC_ml%%=() { module ml "$@" > } > BASH_FUNC_module%%=() { local _mlredir=1; > if [ -n "${MODULES_REDIRECT_OUTPUT+x}" ]; then > if [ "$MODULES_REDIRECT_OUTPUT" = '0' ]; then > _mlredir=0; > else > if [ "$MODULES_REDIRECT_OUTPUT" = '1' ]; then > _mlredir=1; > fi; > fi; > fi; > case " $@ " in > *' --no-redirect '*) > _mlredir=0 > ;; > *' --redirect '*) > _mlredir=1 > ;; > esac; > if [ $_mlredir -eq 0 ]; then > _module_raw "$@"; > else > _module_raw "$@" 2>&1; > fi > } > BASH_FUNC__module_raw%%=() { eval "$(/usr/bin/tclsh8.6 > '/usr/lib/x86_64-linux-gnu/modulecmd.tcl' bash "$@")"; > _mlstatus=$?; > return $_mlstatus > } > _=/usr/bin/mpirun > > > Hydra internal environment: > --------------------------- > GFORTRAN_UNBUFFERED_PRECONNECTED=y > > > > > Proxy information: > ********************* > [1] proxy: ITC19206 (1 cores) > Exec list: ./a.out (2 processes); > > > > > ================================================================================================== > > > > > Proxy launch args: /usr/bin/hydra_pmi_proxy --control-port > ITC19206:39745 --debug --rmk user --launcher ssh --demux poll --pgid 0 > --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 > --gpu-subdevs-per-proc -2 --proxy-id > > > Arguments being passed to proxy 0: > --version 4.2.0 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME > --hostname ITC19206 --global-core-map 0,1,1 --pmi-id-map 0,0 > --global-process-count 2 --auto-cleanup 1 --pmi-kvsname > kvs_718180_0_155951620_ITC19206 --pmi-process-mapping (vector,(0,1,2)) > --global-inherited-env 57 'SHELL=/bin/bash' > 'SESSION_MANAGER=local/ITC19206:@/tmp/.ICE-unix/3236,unix/ITC19206:/tmp/.ICE-unix/3236' > 'QT_ACCESSIBILITY=1' 'COLORTERM=truecolor' > 'XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg' 'XDG_MENU_PREFIX=gnome-' > 'GNOME_DESKTOP_SESSION_ID=this-is-deprecated' > 'MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path' > 'GNOME_SHELL_SESSION_MODE=ubuntu' > 'SSH_AUTH_SOCK=/run/user/1000/keyring/ssh' > 'MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA=' > 'XMODIFIERS=@im=ibus' 'DESKTOP_SESSION=ubuntu' > 'GTK_MODULES=gail:atk-bridge' 'XDG_SESSION_DESKTOP=ubuntu' > 'XDG_SESSION_TYPE=x11' 'MODULESHOME=/usr/share/modules' > 'GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1' > 'SYSTEMD_EXEC_PID=3274' 'XAUTHORITY=/run/user/1000/gdm/Xauthority' > 'WINDOWPATH=2' 'LANG=de_DE.UTF-8' > 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:' > 'XDG_CURRENT_DESKTOP=ubuntu:GNOME' > 'MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/user.slice/user-1000.slice/user(a)1000.serv <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>ice/session.slice/org.gnome.Shell(a)x11.serv <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>ice/memory.pressure' > 'VTE_VERSION=7600' > 'GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/63974876_8215_47b9_b16e_caeb54580fbc' > 'QTWEBENGINE_DICTIONARIES_PATH=/usr/share/hunspell-bdic/' > 'LESSCLOSE=/usr/bin/lesspipe %s %s' 'XDG_SESSION_CLASS=user' > 'TERM=xterm-256color' > 'DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path' 'LESSOPEN=| > /usr/bin/lesspipe %s' 'GNOME_TERMINAL_SERVICE=:1.95' 'LOADEDMODULES=' > 'SHLVL=1' 'GSM_SKIP_SSH_AGENT_WORKAROUND=true' 'QT_IM_MODULE=ibus' > 'XDG_RUNTIME_DIR=/run/user/1000' '__MODULES_LMINIT=module use --append > /etc/environment-modules/modules:module use --append > /usr/share/modules/versions:module use --append > /usr/share/modules/$MODULE_VERSION/modulefiles:module use --append > /usr/share/modules/modulefiles' > 'DEBUGINFOD_URLS=https://urldefense.us/v3/__https://debuginfod.ubuntu.com__;!!G_uCfscf7eWS!euxkrFIlskGc4_et1CGV_3awQoswa8z0MgIGJoYCZiPO8hwHcXAmV-uUkpOo8Yl6IXlDO42OkpkcRunx$ <https://urldefense.us/v3/__https://debuginfod.ubuntu.com__;!!G_uCfscf7eWS!e… > <https://urldefense.us/v3/__https://debuginfod.ubuntu.com__;!!G_uCfscf7eWS!e… > <https://urldefense.us/v3/__https://debuginfod.ubuntu.com>__;!!G_uCfscf7e… > ' > 'XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop' > 'PATH=/usr/lib/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin' > 'MODULEPATH=/etc/environment-modules/modules:/usr/share/modules/versions:/usr/share/modules/$MODULE_VERSION/modulefiles:/usr/share/modules/modulefiles' > 'GDMSESSION=ubuntu' > 'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus' > 'MODULES_CMD=/usr/lib/x86_64-linux-gnu/modulecmd.tcl' 'BASH_FUNC_ml%%=() > { module ml "$@" > }' 'BASH_FUNC_module%%=() { local _mlredir=1; > if [ -n "${MODULES_REDIRECT_OUTPUT+x}" ]; then > if [ "$MODULES_REDIRECT_OUTPUT" = '0' ]; then > _mlredir=0; > else > if [ "$MODULES_REDIRECT_OUTPUT" = '1' ]; then > _mlredir=1; > fi; > fi; > fi; > case " $@ " in > *' --no-redirect '*) > _mlredir=0 > ;; > *' --redirect '*) > _mlredir=1 > ;; > esac; > if [ $_mlredir -eq 0 ]; then > _module_raw "$@"; > else > _module_raw "$@" 2>&1; > fi > }' 'BASH_FUNC__module_raw%%=() { eval "$(/usr/bin/tclsh8.6 > '/usr/lib/x86_64-linux-gnu/modulecmd.tcl' bash "$@")"; > _mlstatus=$?; > return $_mlstatus > }' '_=/usr/bin/mpirun' --global-user-env 0 --global-system-env 1 > 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec > --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir [...] > --exec-args 1 ./a.out > > > [mpiexec@ITC19206] Launch arguments: /usr/bin/hydra_pmi_proxy > --control-port ITC19206:39745 --debug --rmk user --launcher ssh --demux > poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 > --gpu-subdevs-per-proc -2 --proxy-id 0 > [proxy:0@ITC19206] Sending upstream hdr.cmd = CMD_PID_LIST > [proxy:0@ITC19206] Sending upstream hdr.cmd = CMD_STDOUT > [proxy:0@ITC19206] Sending upstream hdr.cmd = CMD_STDOUT > Hello, I am 0 of 1 tasks. > Hello, I am 0 of 1 tasks. > [proxy:0@ITC19206] Sending upstream hdr.cmd = CMD_EXIT_STATUS > > > > > > > Am 03.07.24 um 16:25 schrieb Raffenetti, Ken: >> Can you confirm which MPI library you binary is linked with using ldd? Adding -v to the mpirun command should also provide some clues in the PMI client/server traffic. >> >> Ken >> >> -----Original Message----- >> From: Joachim Jenke via discuss <discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> <mailto:[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>>> >> Reply-To: "discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> <mailto:[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>>" <discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> <mailto:[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>>> >> Date: Wednesday, July 3, 2024 at 5:16 AM >> To: "discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> <mailto:[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>>" <discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> <mailto:[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>>> >> Cc: Joachim Jenke <jenke(a)itc.rwth-aachen.de <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> <mailto:[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>>> >> Subject: [mpich-discuss] All processes only find themselfs >> >> >> Hi all, >> >> >> after upgrading my notebook to Ubuntu 24.04, each processes thinks that >> MPI_COMM_WORLD only consists of its own process: >> >> >> $ mpirun -np 2 -l ./a.out >> [1] Hello, I am 0 of 1 tasks. >> [0] Hello, I am 0 of 1 tasks. >> >> >> Any suggestions what could go wrong? >> >> >> >> >> $ mpirun --version >> HYDRA build details: >> Version: 4.2.0 >> Release Date: Fri Feb 9 12:29:21 CST 2024 >> CC: gcc -Wdate-time >> -D_FORTIFY_SOURCE=3 -g -O2 -fno-omit-frame-pointer >> -mno-omit-leaf-frame-pointer >> -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto >> -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection >> -Wformat -Werror=format-security -fcf-protection >> -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3 >> -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro >> Configure options: '--with-hwloc-prefix=/usr' >> '--with-device=ch4:ucx' 'FFLAGS=-O2 -fno-omit-frame-pointer >> -mno-omit-leaf-frame-pointer >> -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto >> -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection >> -fcf-protection >> -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3 >> -fallow-invalid-boz -fallow-argument-mismatch' '--prefix=/usr' >> 'CFLAGS=-g -O2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer >> -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto >> -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection >> -Wformat -Werror=format-security -fcf-protection >> -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3' >> 'LDFLAGS=-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects >> -Wl,-z,relro' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=3' >> Process Manager: pmi >> Launchers available: ssh rsh fork slurm ll lsf >> sge manual persist >> Topology libraries available: hwloc >> Resource management kernels available: user slurm ll lsf sge pbs >> cobalt >> Demux engines available: poll select >> >> >> Best >> Joachim >> >> > > -- Dr. rer. nat. Joachim Jenke IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 jenke(a)itc.rwth-aachen.de <mailto:[email protected]> https://urldefense.us/v3/__http://www.itc.rwth-aachen.de__;!!G_uCfscf7eWS!e…

1 0