I believe there has been an error in packaging, possibly resulting from a change in configure options, that has rendered the MPICH library incompatible with Hydra's mpiexec. ☹ I have no good solution besides building your own MPICH from source. I will reach out to the Ubuntu maintainer about the issue. Thanks for reporting. Ken -----Original Message----- From: Joachim Jenke <[email protected] <mailto:[email protected]>> Date: Wednesday, July 3, 2024 at 12:20 PM To: "Raffenetti, Ken" <[email protected] <mailto:[email protected]>>, "[email protected] <mailto:[email protected]>" <[email protected] <mailto:[email protected]>> Subject: Re: [mpich-discuss] All processes only find themselfs $ env MPIR_CVAR_PMI_VERSION=1 mpirun -np 2 -l env MPIR_CVAR_PMI_VERSION=1 ./a.out [0] Hello, I am 0 of 1 tasks. [1] Hello, I am 0 of 1 tasks. still the same. I was not sure whether that option is for mpiexec or the mpi library, so I set it twice just to be sure. Am 03.07.24 um 16:54 schrieb Raffenetti, Ken:
Hm, there is a PMIx dependency that is presumably coming from the MPICH library. MPICH's mpiexec is not compatible with an external PMIx, so I suspect that is the issue. Can you try setting MPIR_CVAR_PMI_VERSION=1 and see if that works?
Ken
-----Original Message----- From: Joachim Jenke <[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> Date: Wednesday, July 3, 2024 at 9:36 AM To: "Raffenetti, Ken" <[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>>, "[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>" <[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> Subject: Re: [mpich-discuss] All processes only find themselfs
$ ldd a.out linux-vdso.so.1 (0x00007ffee8145000) libmpich.so.12 => /lib/x86_64-linux-gnu/libmpich.so.12 (0x00007f15d3200000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f15d2e00000) libhwloc.so.15 => /lib/x86_64-linux-gnu/libhwloc.so.15 (0x00007f15d5a13000) libpmix.so.2 => /lib/x86_64-linux-gnu/libpmix.so.2 (0x00007f15d2a00000) libucp.so.0 => /lib/x86_64-linux-gnu/libucp.so.0 (0x00007f15d309d000) libucs.so.0 => /lib/x86_64-linux-gnu/libucs.so.0 (0x00007f15d59ab000) /lib64/ld-linux-x86-64.so.2 (0x00007f15d5aa3000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f15d2d17000) libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f15d5978000) libmunge.so.2 => /lib/x86_64-linux-gnu/libmunge.so.2 (0x00007f15d5970000) libevent_core-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_core-2.1.so.7 (0x00007f15d3068000) libevent_pthreads-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x00007f15d5969000) libuct.so.0 => /lib/x86_64-linux-gnu/libuct.so.0 (0x00007f15d3022000) libucm.so.0 => /lib/x86_64-linux-gnu/libucm.so.0 (0x00007f15d594c000) libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x00007f15d3015000)
$ apt-cache policy libmpich-dev libmpich-dev: Installed: 4.2.0-5build3 Candidate: 4.2.0-5build3 Version table: *** 4.2.0-5build3 500 500 https://urldefense.us/v3/__http://de.archive.ubuntu.com/ubuntu__;!!G_uCfscf7... <https://urldefense.us/v3/__http://de.archive.ubuntu.com/ubuntu__;!!G_uCfscf7... > <https://urldefense.us/v3/__http://de.archive.ubuntu.com/ubuntu__;!!G_uCfscf7... > <https://urldefense.us/v3/__http://de.archive.ubuntu.com/ubuntu>__;!!G_uCfscf7eWS!euxkrFIlskGc4_et1CGV_3awQoswa8z0MgIGJoYCZiPO8hwHcXAmV-uUkpOo8Yl6IXlDO42OkjdqNFpx$ > noble/universe amd64 Packages 100 /var/lib/dpkg/status
$ mpirun -np 2 --verbose ./a.out host: ITC19206 [mpiexec@ITC19206] Timeout set to -1 (-1 means infinite)
================================================================================================== mpiexec options: ---------------- Base path: /usr/bin/ Launcher: (null) Debug level: 1 Enable X: -1
Global environment: ------------------- SHELL=/bin/bash
SESSION_MANAGER=local/ITC19206:@/tmp/.ICE-unix/3236,unix/ITC19206:/tmp/.ICE-unix/3236 QT_ACCESSIBILITY=1 COLORTERM=truecolor XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg XDG_MENU_PREFIX=gnome- GNOME_DESKTOP_SESSION_ID=this-is-deprecated MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path GNOME_SHELL_SESSION_MODE=ubuntu SSH_AUTH_SOCK=/run/user/1000/keyring/ssh MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA= XMODIFIERS=@im=ibus DESKTOP_SESSION=ubuntu GTK_MODULES=gail:atk-bridge XDG_SESSION_DESKTOP=ubuntu XDG_SESSION_TYPE=x11 MODULESHOME=/usr/share/modules GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1 SYSTEMD_EXEC_PID=3274 XAUTHORITY=/run/user/1000/gdm/Xauthority WINDOWPATH=2 LANG=de_DE.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90: XDG_CURRENT_DESKTOP=ubuntu:GNOME
MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/user.slice/user-1000.slice/[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>ice/session.slice/[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>ice/memory.pressure VTE_VERSION=7600
GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/63974876_8215_47b9_b16e_caeb54580fbc QTWEBENGINE_DICTIONARIES_PATH=/usr/share/hunspell-bdic/ LESSCLOSE=/usr/bin/lesspipe %s %s XDG_SESSION_CLASS=user TERM=xterm-256color DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path LESSOPEN=| /usr/bin/lesspipe %s GNOME_TERMINAL_SERVICE=:1.95 LOADEDMODULES= SHLVL=1 GSM_SKIP_SSH_AGENT_WORKAROUND=true QT_IM_MODULE=ibus XDG_RUNTIME_DIR=/run/user/1000 __MODULES_LMINIT=module use --append /etc/environment-modules/modules:module use --append /usr/share/modules/versions:module use --append /usr/share/modules/$MODULE_VERSION/modulefiles:module use --append /usr/share/modules/modulefiles DEBUGINFOD_URLS=https://urldefense.us/v3/__https://debuginfod.ubuntu.com__;!!G_uCfscf7eWS!eu... <https://urldefense.us/v3/__https://debuginfod.ubuntu.com__;!!G_uCfscf7eWS!eu... > <https://urldefense.us/v3/__https://debuginfod.ubuntu.com__;!!G_uCfscf7eWS!eu... > <https://urldefense.us/v3/__https://debuginfod.ubuntu.com>__;!!G_uCfscf7eWS!euxkrFIlskGc4_et1CGV_3awQoswa8z0MgIGJoYCZiPO8hwHcXAmV-uUkpOo8Yl6IXlDO42OkrjhIqm_$ >
XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop
PATH=/usr/lib/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin
MODULEPATH=/etc/environment-modules/modules:/usr/share/modules/versions:/usr/share/modules/$MODULE_VERSION/modulefiles:/usr/share/modules/modulefiles GDMSESSION=ubuntu DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus MODULES_CMD=/usr/lib/x86_64-linux-gnu/modulecmd.tcl BASH_FUNC_ml%%=() { module ml "$@" } BASH_FUNC_module%%=() { local _mlredir=1; if [ -n "${MODULES_REDIRECT_OUTPUT+x}" ]; then if [ "$MODULES_REDIRECT_OUTPUT" = '0' ]; then _mlredir=0; else if [ "$MODULES_REDIRECT_OUTPUT" = '1' ]; then _mlredir=1; fi; fi; fi; case " $@ " in *' --no-redirect '*) _mlredir=0 ;; *' --redirect '*) _mlredir=1 ;; esac; if [ $_mlredir -eq 0 ]; then _module_raw "$@"; else _module_raw "$@" 2>&1; fi } BASH_FUNC__module_raw%%=() { eval "$(/usr/bin/tclsh8.6 '/usr/lib/x86_64-linux-gnu/modulecmd.tcl' bash "$@")"; _mlstatus=$?; return $_mlstatus } _=/usr/bin/mpirun
Hydra internal environment: --------------------------- GFORTRAN_UNBUFFERED_PRECONNECTED=y
Proxy information: ********************* [1] proxy: ITC19206 (1 cores) Exec list: ./a.out (2 processes);
==================================================================================================
Proxy launch args: /usr/bin/hydra_pmi_proxy --control-port ITC19206:39745 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id
Arguments being passed to proxy 0: --version 4.2.0 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname ITC19206 --global-core-map 0,1,1 --pmi-id-map 0,0 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_718180_0_155951620_ITC19206 --pmi-process-mapping (vector,(0,1,2)) --global-inherited-env 57 'SHELL=/bin/bash' 'SESSION_MANAGER=local/ITC19206:@/tmp/.ICE-unix/3236,unix/ITC19206:/tmp/.ICE-unix/3236' 'QT_ACCESSIBILITY=1' 'COLORTERM=truecolor' 'XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg' 'XDG_MENU_PREFIX=gnome-' 'GNOME_DESKTOP_SESSION_ID=this-is-deprecated' 'MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path' 'GNOME_SHELL_SESSION_MODE=ubuntu' 'SSH_AUTH_SOCK=/run/user/1000/keyring/ssh' 'MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA=' 'XMODIFIERS=@im=ibus' 'DESKTOP_SESSION=ubuntu' 'GTK_MODULES=gail:atk-bridge' 'XDG_SESSION_DESKTOP=ubuntu' 'XDG_SESSION_TYPE=x11' 'MODULESHOME=/usr/share/modules' 'GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1' 'SYSTEMD_EXEC_PID=3274' 'XAUTHORITY=/run/user/1000/gdm/Xauthority' 'WINDOWPATH=2' 'LANG=de_DE.UTF-8' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:' 'XDG_CURRENT_DESKTOP=ubuntu:GNOME' 'MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/user.slice/user-1000.slice/[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>ice/session.slice/[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>ice/memory.pressure' 'VTE_VERSION=7600' 'GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/63974876_8215_47b9_b16e_caeb54580fbc' 'QTWEBENGINE_DICTIONARIES_PATH=/usr/share/hunspell-bdic/' 'LESSCLOSE=/usr/bin/lesspipe %s %s' 'XDG_SESSION_CLASS=user' 'TERM=xterm-256color' 'DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path' 'LESSOPEN=| /usr/bin/lesspipe %s' 'GNOME_TERMINAL_SERVICE=:1.95' 'LOADEDMODULES=' 'SHLVL=1' 'GSM_SKIP_SSH_AGENT_WORKAROUND=true' 'QT_IM_MODULE=ibus' 'XDG_RUNTIME_DIR=/run/user/1000' '__MODULES_LMINIT=module use --append /etc/environment-modules/modules:module use --append /usr/share/modules/versions:module use --append /usr/share/modules/$MODULE_VERSION/modulefiles:module use --append /usr/share/modules/modulefiles' 'DEBUGINFOD_URLS=https://urldefense.us/v3/__https://debuginfod.ubuntu.com__;!!G_uCfscf7eWS!eu... <https://urldefense.us/v3/__https://debuginfod.ubuntu.com__;!!G_uCfscf7eWS!eu... > <https://urldefense.us/v3/__https://debuginfod.ubuntu.com__;!!G_uCfscf7eWS!eu... > <https://urldefense.us/v3/__https://debuginfod.ubuntu.com>__;!!G_uCfscf7eWS!euxkrFIlskGc4_et1CGV_3awQoswa8z0MgIGJoYCZiPO8hwHcXAmV-uUkpOo8Yl6IXlDO42OkrjhIqm_$ > ' 'XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop' 'PATH=/usr/lib/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin' 'MODULEPATH=/etc/environment-modules/modules:/usr/share/modules/versions:/usr/share/modules/$MODULE_VERSION/modulefiles:/usr/share/modules/modulefiles' 'GDMSESSION=ubuntu' 'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus' 'MODULES_CMD=/usr/lib/x86_64-linux-gnu/modulecmd.tcl' 'BASH_FUNC_ml%%=() { module ml "$@" }' 'BASH_FUNC_module%%=() { local _mlredir=1; if [ -n "${MODULES_REDIRECT_OUTPUT+x}" ]; then if [ "$MODULES_REDIRECT_OUTPUT" = '0' ]; then _mlredir=0; else if [ "$MODULES_REDIRECT_OUTPUT" = '1' ]; then _mlredir=1; fi; fi; fi; case " $@ " in *' --no-redirect '*) _mlredir=0 ;; *' --redirect '*) _mlredir=1 ;; esac; if [ $_mlredir -eq 0 ]; then _module_raw "$@"; else _module_raw "$@" 2>&1; fi }' 'BASH_FUNC__module_raw%%=() { eval "$(/usr/bin/tclsh8.6 '/usr/lib/x86_64-linux-gnu/modulecmd.tcl' bash "$@")"; _mlstatus=$?; return $_mlstatus }' '_=/usr/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir [...] --exec-args 1 ./a.out
[mpiexec@ITC19206] Launch arguments: /usr/bin/hydra_pmi_proxy --control-port ITC19206:39745 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id 0 [proxy:0@ITC19206] Sending upstream hdr.cmd = CMD_PID_LIST [proxy:0@ITC19206] Sending upstream hdr.cmd = CMD_STDOUT [proxy:0@ITC19206] Sending upstream hdr.cmd = CMD_STDOUT Hello, I am 0 of 1 tasks. Hello, I am 0 of 1 tasks. [proxy:0@ITC19206] Sending upstream hdr.cmd = CMD_EXIT_STATUS
Am 03.07.24 um 16:25 schrieb Raffenetti, Ken:
Can you confirm which MPI library you binary is linked with using ldd? Adding -v to the mpirun command should also provide some clues in the PMI client/server traffic.
Ken
-----Original Message----- From: Joachim Jenke via discuss <[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> <mailto:[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>>> Reply-To: "[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> <mailto:[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>>" <[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> <mailto:[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>>> Date: Wednesday, July 3, 2024 at 5:16 AM To: "[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> <mailto:[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>>" <[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> <mailto:[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>>> Cc: Joachim Jenke <[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> <mailto:[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>>> Subject: [mpich-discuss] All processes only find themselfs
Hi all,
after upgrading my notebook to Ubuntu 24.04, each processes thinks that MPI_COMM_WORLD only consists of its own process:
$ mpirun -np 2 -l ./a.out [1] Hello, I am 0 of 1 tasks. [0] Hello, I am 0 of 1 tasks.
Any suggestions what could go wrong?
$ mpirun --version HYDRA build details: Version: 4.2.0 Release Date: Fri Feb 9 12:29:21 CST 2024 CC: gcc -Wdate-time -D_FORTIFY_SOURCE=3 -g -O2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3 -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro Configure options: '--with-hwloc-prefix=/usr' '--with-device=ch4:ucx' 'FFLAGS=-O2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -fcf-protection -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3 -fallow-invalid-boz -fallow-argument-mismatch' '--prefix=/usr' 'CFLAGS=-g -O2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3' 'LDFLAGS=-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=3' Process Manager: pmi Launchers available: ssh rsh fork slurm ll lsf sge manual persist Topology libraries available: hwloc Resource management kernels available: user slurm ll lsf sge pbs cobalt Demux engines available: poll select
Best Joachim
-- Dr. rer. nat. Joachim Jenke IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 [email protected] <mailto:[email protected]> https://urldefense.us/v3/__http://www.itc.rwth-aachen.de__;!!G_uCfscf7eWS!eu...