- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] All processes only find themselfs
by Joachim Jenke 03 Jul '24

03 Jul '24

$ env MPIR_CVAR_PMI_VERSION=1 mpirun -np 2 -l env MPIR_CVAR_PMI_VERSION=1 ./a.out [0] Hello, I am 0 of 1 tasks. [1] Hello, I am 0 of 1 tasks. still the same. I was not sure whether that option is for mpiexec or the mpi library, so I set it twice just to be sure. Am 03.07.24 um 16:54 schrieb Raffenetti, Ken: > Hm, there is a PMIx dependency that is presumably coming from the MPICH library. MPICH's mpiexec is not compatible with an external PMIx, so I suspect that is the issue. Can you try setting MPIR_CVAR_PMI_VERSION=1 and see if that works? > > Ken > > -----Original Message----- > From: Joachim Jenke <jenke(a)itc.rwth-aachen.de <mailto:[email protected]>> > Date: Wednesday, July 3, 2024 at 9:36 AM > To: "Raffenetti, Ken" <raffenet(a)anl.gov <mailto:[email protected]>>, "discuss(a)mpich.org <mailto:[email protected]>" <discuss(a)mpich.org <mailto:[email protected]>> > Subject: Re: [mpich-discuss] All processes only find themselfs > > > $ ldd a.out > linux-vdso.so.1 (0x00007ffee8145000) > libmpich.so.12 => /lib/x86_64-linux-gnu/libmpich.so.12 (0x00007f15d3200000) > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f15d2e00000) > libhwloc.so.15 => /lib/x86_64-linux-gnu/libhwloc.so.15 (0x00007f15d5a13000) > libpmix.so.2 => /lib/x86_64-linux-gnu/libpmix.so.2 (0x00007f15d2a00000) > libucp.so.0 => /lib/x86_64-linux-gnu/libucp.so.0 (0x00007f15d309d000) > libucs.so.0 => /lib/x86_64-linux-gnu/libucs.so.0 (0x00007f15d59ab000) > /lib64/ld-linux-x86-64.so.2 (0x00007f15d5aa3000) > libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f15d2d17000) > libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f15d5978000) > libmunge.so.2 => /lib/x86_64-linux-gnu/libmunge.so.2 (0x00007f15d5970000) > libevent_core-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_core-2.1.so.7 > (0x00007f15d3068000) > libevent_pthreads-2.1.so.7 => > /lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x00007f15d5969000) > libuct.so.0 => /lib/x86_64-linux-gnu/libuct.so.0 (0x00007f15d3022000) > libucm.so.0 => /lib/x86_64-linux-gnu/libucm.so.0 (0x00007f15d594c000) > libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x00007f15d3015000) > > > $ apt-cache policy libmpich-dev > libmpich-dev: > Installed: 4.2.0-5build3 > Candidate: 4.2.0-5build3 > Version table: > *** 4.2.0-5build3 500 > 500 http://de.archive.ubuntu.com/ubuntu <http://de.archive.ubuntu.com/ubuntu> noble/universe amd64 > Packages > 100 /var/lib/dpkg/status > > > > > > > $ mpirun -np 2 --verbose ./a.out > host: ITC19206 > [mpiexec@ITC19206] Timeout set to -1 (-1 means infinite) > > > ================================================================================================== > mpiexec options: > ---------------- > Base path: /usr/bin/ > Launcher: (null) > Debug level: 1 > Enable X: -1 > > > Global environment: > ------------------- > SHELL=/bin/bash > > > SESSION_MANAGER=local/ITC19206:@/tmp/.ICE-unix/3236,unix/ITC19206:/tmp/.ICE-unix/3236 > QT_ACCESSIBILITY=1 > COLORTERM=truecolor > XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg > XDG_MENU_PREFIX=gnome- > GNOME_DESKTOP_SESSION_ID=this-is-deprecated > MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path > GNOME_SHELL_SESSION_MODE=ubuntu > SSH_AUTH_SOCK=/run/user/1000/keyring/ssh > MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA= > XMODIFIERS=@im=ibus > DESKTOP_SESSION=ubuntu > GTK_MODULES=gail:atk-bridge > XDG_SESSION_DESKTOP=ubuntu > XDG_SESSION_TYPE=x11 > MODULESHOME=/usr/share/modules > GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1 > SYSTEMD_EXEC_PID=3274 > XAUTHORITY=/run/user/1000/gdm/Xauthority > WINDOWPATH=2 > LANG=de_DE.UTF-8 > > > LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90: > XDG_CURRENT_DESKTOP=ubuntu:GNOME > > > MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/user.slice/user-1000.slice/user(a)1000.serv <mailto:[email protected]>ice/session.slice/org.gnome.Shell(a)x11.serv <mailto:[email protected]>ice/memory.pressure > VTE_VERSION=7600 > > > GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/63974876_8215_47b9_b16e_caeb54580fbc > QTWEBENGINE_DICTIONARIES_PATH=/usr/share/hunspell-bdic/ > LESSCLOSE=/usr/bin/lesspipe %s %s > XDG_SESSION_CLASS=user > TERM=xterm-256color > DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path > LESSOPEN=| /usr/bin/lesspipe %s > GNOME_TERMINAL_SERVICE=:1.95 > LOADEDMODULES= > SHLVL=1 > GSM_SKIP_SSH_AGENT_WORKAROUND=true > QT_IM_MODULE=ibus > XDG_RUNTIME_DIR=/run/user/1000 > __MODULES_LMINIT=module use --append > /etc/environment-modules/modules:module use --append > /usr/share/modules/versions:module use --append > /usr/share/modules/$MODULE_VERSION/modulefiles:module use --append > /usr/share/modules/modulefiles > DEBUGINFOD_URLS=https://debuginfod.ubuntu.com <https://debuginfod.ubuntu.com> > > > XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop > > > PATH=/usr/lib/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin > > > MODULEPATH=/etc/environment-modules/modules:/usr/share/modules/versions:/usr/share/modules/$MODULE_VERSION/modulefiles:/usr/share/modules/modulefiles > GDMSESSION=ubuntu > DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus > MODULES_CMD=/usr/lib/x86_64-linux-gnu/modulecmd.tcl > BASH_FUNC_ml%%=() { module ml "$@" > } > BASH_FUNC_module%%=() { local _mlredir=1; > if [ -n "${MODULES_REDIRECT_OUTPUT+x}" ]; then > if [ "$MODULES_REDIRECT_OUTPUT" = '0' ]; then > _mlredir=0; > else > if [ "$MODULES_REDIRECT_OUTPUT" = '1' ]; then > _mlredir=1; > fi; > fi; > fi; > case " $@ " in > *' --no-redirect '*) > _mlredir=0 > ;; > *' --redirect '*) > _mlredir=1 > ;; > esac; > if [ $_mlredir -eq 0 ]; then > _module_raw "$@"; > else > _module_raw "$@" 2>&1; > fi > } > BASH_FUNC__module_raw%%=() { eval "$(/usr/bin/tclsh8.6 > '/usr/lib/x86_64-linux-gnu/modulecmd.tcl' bash "$@")"; > _mlstatus=$?; > return $_mlstatus > } > _=/usr/bin/mpirun > > > Hydra internal environment: > --------------------------- > GFORTRAN_UNBUFFERED_PRECONNECTED=y > > > > > Proxy information: > ********************* > [1] proxy: ITC19206 (1 cores) > Exec list: ./a.out (2 processes); > > > > > ================================================================================================== > > > > > Proxy launch args: /usr/bin/hydra_pmi_proxy --control-port > ITC19206:39745 --debug --rmk user --launcher ssh --demux poll --pgid 0 > --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 > --gpu-subdevs-per-proc -2 --proxy-id > > > Arguments being passed to proxy 0: > --version 4.2.0 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME > --hostname ITC19206 --global-core-map 0,1,1 --pmi-id-map 0,0 > --global-process-count 2 --auto-cleanup 1 --pmi-kvsname > kvs_718180_0_155951620_ITC19206 --pmi-process-mapping (vector,(0,1,2)) > --global-inherited-env 57 'SHELL=/bin/bash' > 'SESSION_MANAGER=local/ITC19206:@/tmp/.ICE-unix/3236,unix/ITC19206:/tmp/.ICE-unix/3236' > 'QT_ACCESSIBILITY=1' 'COLORTERM=truecolor' > 'XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg' 'XDG_MENU_PREFIX=gnome-' > 'GNOME_DESKTOP_SESSION_ID=this-is-deprecated' > 'MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path' > 'GNOME_SHELL_SESSION_MODE=ubuntu' > 'SSH_AUTH_SOCK=/run/user/1000/keyring/ssh' > 'MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA=' > 'XMODIFIERS=@im=ibus' 'DESKTOP_SESSION=ubuntu' > 'GTK_MODULES=gail:atk-bridge' 'XDG_SESSION_DESKTOP=ubuntu' > 'XDG_SESSION_TYPE=x11' 'MODULESHOME=/usr/share/modules' > 'GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1' > 'SYSTEMD_EXEC_PID=3274' 'XAUTHORITY=/run/user/1000/gdm/Xauthority' > 'WINDOWPATH=2' 'LANG=de_DE.UTF-8' > 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:' > 'XDG_CURRENT_DESKTOP=ubuntu:GNOME' > 'MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/user.slice/user-1000.slice/user(a)1000.serv <mailto:[email protected]>ice/session.slice/org.gnome.Shell(a)x11.serv <mailto:[email protected]>ice/memory.pressure' > 'VTE_VERSION=7600' > 'GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/63974876_8215_47b9_b16e_caeb54580fbc' > 'QTWEBENGINE_DICTIONARIES_PATH=/usr/share/hunspell-bdic/' > 'LESSCLOSE=/usr/bin/lesspipe %s %s' 'XDG_SESSION_CLASS=user' > 'TERM=xterm-256color' > 'DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path' 'LESSOPEN=| > /usr/bin/lesspipe %s' 'GNOME_TERMINAL_SERVICE=:1.95' 'LOADEDMODULES=' > 'SHLVL=1' 'GSM_SKIP_SSH_AGENT_WORKAROUND=true' 'QT_IM_MODULE=ibus' > 'XDG_RUNTIME_DIR=/run/user/1000' '__MODULES_LMINIT=module use --append > /etc/environment-modules/modules:module use --append > /usr/share/modules/versions:module use --append > /usr/share/modules/$MODULE_VERSION/modulefiles:module use --append > /usr/share/modules/modulefiles' > 'DEBUGINFOD_URLS=https://debuginfod.ubuntu.com <https://debuginfod.ubuntu.com> ' > 'XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop' > 'PATH=/usr/lib/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin' > 'MODULEPATH=/etc/environment-modules/modules:/usr/share/modules/versions:/usr/share/modules/$MODULE_VERSION/modulefiles:/usr/share/modules/modulefiles' > 'GDMSESSION=ubuntu' > 'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus' > 'MODULES_CMD=/usr/lib/x86_64-linux-gnu/modulecmd.tcl' 'BASH_FUNC_ml%%=() > { module ml "$@" > }' 'BASH_FUNC_module%%=() { local _mlredir=1; > if [ -n "${MODULES_REDIRECT_OUTPUT+x}" ]; then > if [ "$MODULES_REDIRECT_OUTPUT" = '0' ]; then > _mlredir=0; > else > if [ "$MODULES_REDIRECT_OUTPUT" = '1' ]; then > _mlredir=1; > fi; > fi; > fi; > case " $@ " in > *' --no-redirect '*) > _mlredir=0 > ;; > *' --redirect '*) > _mlredir=1 > ;; > esac; > if [ $_mlredir -eq 0 ]; then > _module_raw "$@"; > else > _module_raw "$@" 2>&1; > fi > }' 'BASH_FUNC__module_raw%%=() { eval "$(/usr/bin/tclsh8.6 > '/usr/lib/x86_64-linux-gnu/modulecmd.tcl' bash "$@")"; > _mlstatus=$?; > return $_mlstatus > }' '_=/usr/bin/mpirun' --global-user-env 0 --global-system-env 1 > 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec > --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir [...] > --exec-args 1 ./a.out > > > [mpiexec@ITC19206] Launch arguments: /usr/bin/hydra_pmi_proxy > --control-port ITC19206:39745 --debug --rmk user --launcher ssh --demux > poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 > --gpu-subdevs-per-proc -2 --proxy-id 0 > [proxy:0@ITC19206] Sending upstream hdr.cmd = CMD_PID_LIST > [proxy:0@ITC19206] Sending upstream hdr.cmd = CMD_STDOUT > [proxy:0@ITC19206] Sending upstream hdr.cmd = CMD_STDOUT > Hello, I am 0 of 1 tasks. > Hello, I am 0 of 1 tasks. > [proxy:0@ITC19206] Sending upstream hdr.cmd = CMD_EXIT_STATUS > > > > > > > Am 03.07.24 um 16:25 schrieb Raffenetti, Ken: >> Can you confirm which MPI library you binary is linked with using ldd? Adding -v to the mpirun command should also provide some clues in the PMI client/server traffic. >> >> Ken >> >> -----Original Message----- >> From: Joachim Jenke via discuss <discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> >> Reply-To: "discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>" <discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> >> Date: Wednesday, July 3, 2024 at 5:16 AM >> To: "discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>" <discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> >> Cc: Joachim Jenke <jenke(a)itc.rwth-aachen.de <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> >> Subject: [mpich-discuss] All processes only find themselfs >> >> >> Hi all, >> >> >> after upgrading my notebook to Ubuntu 24.04, each processes thinks that >> MPI_COMM_WORLD only consists of its own process: >> >> >> $ mpirun -np 2 -l ./a.out >> [1] Hello, I am 0 of 1 tasks. >> [0] Hello, I am 0 of 1 tasks. >> >> >> Any suggestions what could go wrong? >> >> >> >> >> $ mpirun --version >> HYDRA build details: >> Version: 4.2.0 >> Release Date: Fri Feb 9 12:29:21 CST 2024 >> CC: gcc -Wdate-time >> -D_FORTIFY_SOURCE=3 -g -O2 -fno-omit-frame-pointer >> -mno-omit-leaf-frame-pointer >> -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto >> -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection >> -Wformat -Werror=format-security -fcf-protection >> -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3 >> -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro >> Configure options: '--with-hwloc-prefix=/usr' >> '--with-device=ch4:ucx' 'FFLAGS=-O2 -fno-omit-frame-pointer >> -mno-omit-leaf-frame-pointer >> -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto >> -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection >> -fcf-protection >> -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3 >> -fallow-invalid-boz -fallow-argument-mismatch' '--prefix=/usr' >> 'CFLAGS=-g -O2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer >> -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto >> -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection >> -Wformat -Werror=format-security -fcf-protection >> -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3' >> 'LDFLAGS=-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects >> -Wl,-z,relro' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=3' >> Process Manager: pmi >> Launchers available: ssh rsh fork slurm ll lsf >> sge manual persist >> Topology libraries available: hwloc >> Resource management kernels available: user slurm ll lsf sge pbs >> cobalt >> Demux engines available: poll select >> >> >> Best >> Joachim >> >> > > -- Dr. rer. nat. Joachim Jenke IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 jenke(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpich-discuss] All processes only find themselfs
by Raffenetti, Ken 03 Jul '24

03 Jul '24

Hm, there is a PMIx dependency that is presumably coming from the MPICH library. MPICH's mpiexec is not compatible with an external PMIx, so I suspect that is the issue. Can you try setting MPIR_CVAR_PMI_VERSION=1 and see if that works? Ken -----Original Message----- From: Joachim Jenke <jenke(a)itc.rwth-aachen.de <mailto:[email protected]>> Date: Wednesday, July 3, 2024 at 9:36 AM To: "Raffenetti, Ken" <raffenet(a)anl.gov <mailto:[email protected]>>, "discuss(a)mpich.org <mailto:[email protected]>" <discuss(a)mpich.org <mailto:[email protected]>> Subject: Re: [mpich-discuss] All processes only find themselfs $ ldd a.out linux-vdso.so.1 (0x00007ffee8145000) libmpich.so.12 => /lib/x86_64-linux-gnu/libmpich.so.12 (0x00007f15d3200000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f15d2e00000) libhwloc.so.15 => /lib/x86_64-linux-gnu/libhwloc.so.15 (0x00007f15d5a13000) libpmix.so.2 => /lib/x86_64-linux-gnu/libpmix.so.2 (0x00007f15d2a00000) libucp.so.0 => /lib/x86_64-linux-gnu/libucp.so.0 (0x00007f15d309d000) libucs.so.0 => /lib/x86_64-linux-gnu/libucs.so.0 (0x00007f15d59ab000) /lib64/ld-linux-x86-64.so.2 (0x00007f15d5aa3000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f15d2d17000) libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f15d5978000) libmunge.so.2 => /lib/x86_64-linux-gnu/libmunge.so.2 (0x00007f15d5970000) libevent_core-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_core-2.1.so.7 (0x00007f15d3068000) libevent_pthreads-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x00007f15d5969000) libuct.so.0 => /lib/x86_64-linux-gnu/libuct.so.0 (0x00007f15d3022000) libucm.so.0 => /lib/x86_64-linux-gnu/libucm.so.0 (0x00007f15d594c000) libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x00007f15d3015000) $ apt-cache policy libmpich-dev libmpich-dev: Installed: 4.2.0-5build3 Candidate: 4.2.0-5build3 Version table: *** 4.2.0-5build3 500 500 https://urldefense.us/v3/__http://de.archive.ubuntu.com/ubuntu__;!!G_uCfscf… <https://urldefense.us/v3/__http://de.archive.ubuntu.com/ubuntu__;!!G_uCfscf… > noble/universe amd64 Packages 100 /var/lib/dpkg/status $ mpirun -np 2 --verbose ./a.out host: ITC19206 [mpiexec@ITC19206] Timeout set to -1 (-1 means infinite) ================================================================================================== mpiexec options: ---------------- Base path: /usr/bin/ Launcher: (null) Debug level: 1 Enable X: -1 Global environment: ------------------- SHELL=/bin/bash SESSION_MANAGER=local/ITC19206:@/tmp/.ICE-unix/3236,unix/ITC19206:/tmp/.ICE-unix/3236 QT_ACCESSIBILITY=1 COLORTERM=truecolor XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg XDG_MENU_PREFIX=gnome- GNOME_DESKTOP_SESSION_ID=this-is-deprecated MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path GNOME_SHELL_SESSION_MODE=ubuntu SSH_AUTH_SOCK=/run/user/1000/keyring/ssh MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA= XMODIFIERS=@im=ibus DESKTOP_SESSION=ubuntu GTK_MODULES=gail:atk-bridge XDG_SESSION_DESKTOP=ubuntu XDG_SESSION_TYPE=x11 MODULESHOME=/usr/share/modules GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1 SYSTEMD_EXEC_PID=3274 XAUTHORITY=/run/user/1000/gdm/Xauthority WINDOWPATH=2 LANG=de_DE.UTF-8 LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90: XDG_CURRENT_DESKTOP=ubuntu:GNOME MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/user.slice/user-1000.slice/user(a)1000.serv <mailto:[email protected]>ice/session.slice/org.gnome.Shell(a)x11.serv <mailto:[email protected]>ice/memory.pressure VTE_VERSION=7600 GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/63974876_8215_47b9_b16e_caeb54580fbc QTWEBENGINE_DICTIONARIES_PATH=/usr/share/hunspell-bdic/ LESSCLOSE=/usr/bin/lesspipe %s %s XDG_SESSION_CLASS=user TERM=xterm-256color DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path LESSOPEN=| /usr/bin/lesspipe %s GNOME_TERMINAL_SERVICE=:1.95 LOADEDMODULES= SHLVL=1 GSM_SKIP_SSH_AGENT_WORKAROUND=true QT_IM_MODULE=ibus XDG_RUNTIME_DIR=/run/user/1000 __MODULES_LMINIT=module use --append /etc/environment-modules/modules:module use --append /usr/share/modules/versions:module use --append /usr/share/modules/$MODULE_VERSION/modulefiles:module use --append /usr/share/modules/modulefiles DEBUGINFOD_URLS=https://urldefense.us/v3/__https://debuginfod.ubuntu.com__;!!G_uCfscf7eWS!dclXuitW1RCpXhgp-qZD83ijnwmbJiIbWEfRYNnsS-Cgb1iGZV0W3CPNYNLwn_snqDC0USZi-D6_fHTP$ <https://urldefense.us/v3/__https://debuginfod.ubuntu.com__;!!G_uCfscf7eWS!d… > XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop PATH=/usr/lib/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin MODULEPATH=/etc/environment-modules/modules:/usr/share/modules/versions:/usr/share/modules/$MODULE_VERSION/modulefiles:/usr/share/modules/modulefiles GDMSESSION=ubuntu DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus MODULES_CMD=/usr/lib/x86_64-linux-gnu/modulecmd.tcl BASH_FUNC_ml%%=() { module ml "$@" } BASH_FUNC_module%%=() { local _mlredir=1; if [ -n "${MODULES_REDIRECT_OUTPUT+x}" ]; then if [ "$MODULES_REDIRECT_OUTPUT" = '0' ]; then _mlredir=0; else if [ "$MODULES_REDIRECT_OUTPUT" = '1' ]; then _mlredir=1; fi; fi; fi; case " $@ " in *' --no-redirect '*) _mlredir=0 ;; *' --redirect '*) _mlredir=1 ;; esac; if [ $_mlredir -eq 0 ]; then _module_raw "$@"; else _module_raw "$@" 2>&1; fi } BASH_FUNC__module_raw%%=() { eval "$(/usr/bin/tclsh8.6 '/usr/lib/x86_64-linux-gnu/modulecmd.tcl' bash "$@")"; _mlstatus=$?; return $_mlstatus } _=/usr/bin/mpirun Hydra internal environment: --------------------------- GFORTRAN_UNBUFFERED_PRECONNECTED=y Proxy information: ********************* [1] proxy: ITC19206 (1 cores) Exec list: ./a.out (2 processes); ================================================================================================== Proxy launch args: /usr/bin/hydra_pmi_proxy --control-port ITC19206:39745 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id Arguments being passed to proxy 0: --version 4.2.0 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname ITC19206 --global-core-map 0,1,1 --pmi-id-map 0,0 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_718180_0_155951620_ITC19206 --pmi-process-mapping (vector,(0,1,2)) --global-inherited-env 57 'SHELL=/bin/bash' 'SESSION_MANAGER=local/ITC19206:@/tmp/.ICE-unix/3236,unix/ITC19206:/tmp/.ICE-unix/3236' 'QT_ACCESSIBILITY=1' 'COLORTERM=truecolor' 'XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg' 'XDG_MENU_PREFIX=gnome-' 'GNOME_DESKTOP_SESSION_ID=this-is-deprecated' 'MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path' 'GNOME_SHELL_SESSION_MODE=ubuntu' 'SSH_AUTH_SOCK=/run/user/1000/keyring/ssh' 'MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA=' 'XMODIFIERS=@im=ibus' 'DESKTOP_SESSION=ubuntu' 'GTK_MODULES=gail:atk-bridge' 'XDG_SESSION_DESKTOP=ubuntu' 'XDG_SESSION_TYPE=x11' 'MODULESHOME=/usr/share/modules' 'GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1' 'SYSTEMD_EXEC_PID=3274' 'XAUTHORITY=/run/user/1000/gdm/Xauthority' 'WINDOWPATH=2' 'LANG=de_DE.UTF-8' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:' 'XDG_CURRENT_DESKTOP=ubuntu:GNOME' 'MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/user.slice/user-1000.slice/user(a)1000.serv <mailto:[email protected]>ice/session.slice/org.gnome.Shell(a)x11.serv <mailto:[email protected]>ice/memory.pressure' 'VTE_VERSION=7600' 'GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/63974876_8215_47b9_b16e_caeb54580fbc' 'QTWEBENGINE_DICTIONARIES_PATH=/usr/share/hunspell-bdic/' 'LESSCLOSE=/usr/bin/lesspipe %s %s' 'XDG_SESSION_CLASS=user' 'TERM=xterm-256color' 'DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path' 'LESSOPEN=| /usr/bin/lesspipe %s' 'GNOME_TERMINAL_SERVICE=:1.95' 'LOADEDMODULES=' 'SHLVL=1' 'GSM_SKIP_SSH_AGENT_WORKAROUND=true' 'QT_IM_MODULE=ibus' 'XDG_RUNTIME_DIR=/run/user/1000' '__MODULES_LMINIT=module use --append /etc/environment-modules/modules:module use --append /usr/share/modules/versions:module use --append /usr/share/modules/$MODULE_VERSION/modulefiles:module use --append /usr/share/modules/modulefiles' 'DEBUGINFOD_URLS=https://urldefense.us/v3/__https://debuginfod.ubuntu.com__;!!G_uCfscf7eWS!dclXuitW1RCpXhgp-qZD83ijnwmbJiIbWEfRYNnsS-Cgb1iGZV0W3CPNYNLwn_snqDC0USZi-D6_fHTP$ <https://urldefense.us/v3/__https://debuginfod.ubuntu.com__;!!G_uCfscf7eWS!d… > ' 'XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop' 'PATH=/usr/lib/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin' 'MODULEPATH=/etc/environment-modules/modules:/usr/share/modules/versions:/usr/share/modules/$MODULE_VERSION/modulefiles:/usr/share/modules/modulefiles' 'GDMSESSION=ubuntu' 'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus' 'MODULES_CMD=/usr/lib/x86_64-linux-gnu/modulecmd.tcl' 'BASH_FUNC_ml%%=() { module ml "$@" }' 'BASH_FUNC_module%%=() { local _mlredir=1; if [ -n "${MODULES_REDIRECT_OUTPUT+x}" ]; then if [ "$MODULES_REDIRECT_OUTPUT" = '0' ]; then _mlredir=0; else if [ "$MODULES_REDIRECT_OUTPUT" = '1' ]; then _mlredir=1; fi; fi; fi; case " $@ " in *' --no-redirect '*) _mlredir=0 ;; *' --redirect '*) _mlredir=1 ;; esac; if [ $_mlredir -eq 0 ]; then _module_raw "$@"; else _module_raw "$@" 2>&1; fi }' 'BASH_FUNC__module_raw%%=() { eval "$(/usr/bin/tclsh8.6 '/usr/lib/x86_64-linux-gnu/modulecmd.tcl' bash "$@")"; _mlstatus=$?; return $_mlstatus }' '_=/usr/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir [...] --exec-args 1 ./a.out [mpiexec@ITC19206] Launch arguments: /usr/bin/hydra_pmi_proxy --control-port ITC19206:39745 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id 0 [proxy:0@ITC19206] Sending upstream hdr.cmd = CMD_PID_LIST [proxy:0@ITC19206] Sending upstream hdr.cmd = CMD_STDOUT [proxy:0@ITC19206] Sending upstream hdr.cmd = CMD_STDOUT Hello, I am 0 of 1 tasks. Hello, I am 0 of 1 tasks. [proxy:0@ITC19206] Sending upstream hdr.cmd = CMD_EXIT_STATUS Am 03.07.24 um 16:25 schrieb Raffenetti, Ken: > Can you confirm which MPI library you binary is linked with using ldd? Adding -v to the mpirun command should also provide some clues in the PMI client/server traffic. > > Ken > > -----Original Message----- > From: Joachim Jenke via discuss <discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> > Reply-To: "discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>" <discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> > Date: Wednesday, July 3, 2024 at 5:16 AM > To: "discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>" <discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> > Cc: Joachim Jenke <jenke(a)itc.rwth-aachen.de <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> > Subject: [mpich-discuss] All processes only find themselfs > > > Hi all, > > > after upgrading my notebook to Ubuntu 24.04, each processes thinks that > MPI_COMM_WORLD only consists of its own process: > > > $ mpirun -np 2 -l ./a.out > [1] Hello, I am 0 of 1 tasks. > [0] Hello, I am 0 of 1 tasks. > > > Any suggestions what could go wrong? > > > > > $ mpirun --version > HYDRA build details: > Version: 4.2.0 > Release Date: Fri Feb 9 12:29:21 CST 2024 > CC: gcc -Wdate-time > -D_FORTIFY_SOURCE=3 -g -O2 -fno-omit-frame-pointer > -mno-omit-leaf-frame-pointer > -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto > -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection > -Wformat -Werror=format-security -fcf-protection > -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3 > -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro > Configure options: '--with-hwloc-prefix=/usr' > '--with-device=ch4:ucx' 'FFLAGS=-O2 -fno-omit-frame-pointer > -mno-omit-leaf-frame-pointer > -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto > -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection > -fcf-protection > -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3 > -fallow-invalid-boz -fallow-argument-mismatch' '--prefix=/usr' > 'CFLAGS=-g -O2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer > -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto > -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection > -Wformat -Werror=format-security -fcf-protection > -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3' > 'LDFLAGS=-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects > -Wl,-z,relro' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=3' > Process Manager: pmi > Launchers available: ssh rsh fork slurm ll lsf > sge manual persist > Topology libraries available: hwloc > Resource management kernels available: user slurm ll lsf sge pbs > cobalt > Demux engines available: poll select > > > Best > Joachim > > -- Dr. rer. nat. Joachim Jenke IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 jenke(a)itc.rwth-aachen.de <mailto:[email protected]> https://urldefense.us/v3/__http://www.itc.rwth-aachen.de__;!!G_uCfscf7eWS!d…

1 0

Re: [mpich-discuss] All processes only find themselfs
by Joachim Jenke 03 Jul '24

03 Jul '24

$ ldd a.out linux-vdso.so.1 (0x00007ffee8145000) libmpich.so.12 => /lib/x86_64-linux-gnu/libmpich.so.12 (0x00007f15d3200000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f15d2e00000) libhwloc.so.15 => /lib/x86_64-linux-gnu/libhwloc.so.15 (0x00007f15d5a13000) libpmix.so.2 => /lib/x86_64-linux-gnu/libpmix.so.2 (0x00007f15d2a00000) libucp.so.0 => /lib/x86_64-linux-gnu/libucp.so.0 (0x00007f15d309d000) libucs.so.0 => /lib/x86_64-linux-gnu/libucs.so.0 (0x00007f15d59ab000) /lib64/ld-linux-x86-64.so.2 (0x00007f15d5aa3000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f15d2d17000) libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f15d5978000) libmunge.so.2 => /lib/x86_64-linux-gnu/libmunge.so.2 (0x00007f15d5970000) libevent_core-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_core-2.1.so.7 (0x00007f15d3068000) libevent_pthreads-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x00007f15d5969000) libuct.so.0 => /lib/x86_64-linux-gnu/libuct.so.0 (0x00007f15d3022000) libucm.so.0 => /lib/x86_64-linux-gnu/libucm.so.0 (0x00007f15d594c000) libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x00007f15d3015000) $ apt-cache policy libmpich-dev libmpich-dev: Installed: 4.2.0-5build3 Candidate: 4.2.0-5build3 Version table: *** 4.2.0-5build3 500 500 http://de.archive.ubuntu.com/ubuntu noble/universe amd64 Packages 100 /var/lib/dpkg/status $ mpirun -np 2 --verbose ./a.out host: ITC19206 [mpiexec@ITC19206] Timeout set to -1 (-1 means infinite) ================================================================================================== mpiexec options: ---------------- Base path: /usr/bin/ Launcher: (null) Debug level: 1 Enable X: -1 Global environment: ------------------- SHELL=/bin/bash SESSION_MANAGER=local/ITC19206:@/tmp/.ICE-unix/3236,unix/ITC19206:/tmp/.ICE-unix/3236 QT_ACCESSIBILITY=1 COLORTERM=truecolor XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg XDG_MENU_PREFIX=gnome- GNOME_DESKTOP_SESSION_ID=this-is-deprecated MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path GNOME_SHELL_SESSION_MODE=ubuntu SSH_AUTH_SOCK=/run/user/1000/keyring/ssh MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA= XMODIFIERS=@im=ibus DESKTOP_SESSION=ubuntu GTK_MODULES=gail:atk-bridge XDG_SESSION_DESKTOP=ubuntu XDG_SESSION_TYPE=x11 MODULESHOME=/usr/share/modules GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1 SYSTEMD_EXEC_PID=3274 XAUTHORITY=/run/user/1000/gdm/Xauthority WINDOWPATH=2 LANG=de_DE.UTF-8 LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90: XDG_CURRENT_DESKTOP=ubuntu:GNOME MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/user.slice/user-1000.slice/[email protected]/session.slice/[email protected]/memory.pressure VTE_VERSION=7600 GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/63974876_8215_47b9_b16e_caeb54580fbc QTWEBENGINE_DICTIONARIES_PATH=/usr/share/hunspell-bdic/ LESSCLOSE=/usr/bin/lesspipe %s %s XDG_SESSION_CLASS=user TERM=xterm-256color DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path LESSOPEN=| /usr/bin/lesspipe %s GNOME_TERMINAL_SERVICE=:1.95 LOADEDMODULES= SHLVL=1 GSM_SKIP_SSH_AGENT_WORKAROUND=true QT_IM_MODULE=ibus XDG_RUNTIME_DIR=/run/user/1000 __MODULES_LMINIT=module use --append /etc/environment-modules/modules:module use --append /usr/share/modules/versions:module use --append /usr/share/modules/$MODULE_VERSION/modulefiles:module use --append /usr/share/modules/modulefiles DEBUGINFOD_URLS=https://debuginfod.ubuntu.com XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop PATH=/usr/lib/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin MODULEPATH=/etc/environment-modules/modules:/usr/share/modules/versions:/usr/share/modules/$MODULE_VERSION/modulefiles:/usr/share/modules/modulefiles GDMSESSION=ubuntu DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus MODULES_CMD=/usr/lib/x86_64-linux-gnu/modulecmd.tcl BASH_FUNC_ml%%=() { module ml "$@" } BASH_FUNC_module%%=() { local _mlredir=1; if [ -n "${MODULES_REDIRECT_OUTPUT+x}" ]; then if [ "$MODULES_REDIRECT_OUTPUT" = '0' ]; then _mlredir=0; else if [ "$MODULES_REDIRECT_OUTPUT" = '1' ]; then _mlredir=1; fi; fi; fi; case " $@ " in *' --no-redirect '*) _mlredir=0 ;; *' --redirect '*) _mlredir=1 ;; esac; if [ $_mlredir -eq 0 ]; then _module_raw "$@"; else _module_raw "$@" 2>&1; fi } BASH_FUNC__module_raw%%=() { eval "$(/usr/bin/tclsh8.6 '/usr/lib/x86_64-linux-gnu/modulecmd.tcl' bash "$@")"; _mlstatus=$?; return $_mlstatus } _=/usr/bin/mpirun Hydra internal environment: --------------------------- GFORTRAN_UNBUFFERED_PRECONNECTED=y Proxy information: ********************* [1] proxy: ITC19206 (1 cores) Exec list: ./a.out (2 processes); ================================================================================================== Proxy launch args: /usr/bin/hydra_pmi_proxy --control-port ITC19206:39745 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id Arguments being passed to proxy 0: --version 4.2.0 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname ITC19206 --global-core-map 0,1,1 --pmi-id-map 0,0 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_718180_0_155951620_ITC19206 --pmi-process-mapping (vector,(0,1,2)) --global-inherited-env 57 'SHELL=/bin/bash' 'SESSION_MANAGER=local/ITC19206:@/tmp/.ICE-unix/3236,unix/ITC19206:/tmp/.ICE-unix/3236' 'QT_ACCESSIBILITY=1' 'COLORTERM=truecolor' 'XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg' 'XDG_MENU_PREFIX=gnome-' 'GNOME_DESKTOP_SESSION_ID=this-is-deprecated' 'MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path' 'GNOME_SHELL_SESSION_MODE=ubuntu' 'SSH_AUTH_SOCK=/run/user/1000/keyring/ssh' 'MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA=' 'XMODIFIERS=@im=ibus' 'DESKTOP_SESSION=ubuntu' 'GTK_MODULES=gail:atk-bridge' 'XDG_SESSION_DESKTOP=ubuntu' 'XDG_SESSION_TYPE=x11' 'MODULESHOME=/usr/share/modules' 'GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1' 'SYSTEMD_EXEC_PID=3274' 'XAUTHORITY=/run/user/1000/gdm/Xauthority' 'WINDOWPATH=2' 'LANG=de_DE.UTF-8' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:' 'XDG_CURRENT_DESKTOP=ubuntu:GNOME' 'MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/user.slice/user-1000.slice/[email protected]/session.slice/[email protected]/memory.pressure' 'VTE_VERSION=7600' 'GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/63974876_8215_47b9_b16e_caeb54580fbc' 'QTWEBENGINE_DICTIONARIES_PATH=/usr/share/hunspell-bdic/' 'LESSCLOSE=/usr/bin/lesspipe %s %s' 'XDG_SESSION_CLASS=user' 'TERM=xterm-256color' 'DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path' 'LESSOPEN=| /usr/bin/lesspipe %s' 'GNOME_TERMINAL_SERVICE=:1.95' 'LOADEDMODULES=' 'SHLVL=1' 'GSM_SKIP_SSH_AGENT_WORKAROUND=true' 'QT_IM_MODULE=ibus' 'XDG_RUNTIME_DIR=/run/user/1000' '__MODULES_LMINIT=module use --append /etc/environment-modules/modules:module use --append /usr/share/modules/versions:module use --append /usr/share/modules/$MODULE_VERSION/modulefiles:module use --append /usr/share/modules/modulefiles' 'DEBUGINFOD_URLS=https://debuginfod.ubuntu.com ' 'XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop' 'PATH=/usr/lib/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin' 'MODULEPATH=/etc/environment-modules/modules:/usr/share/modules/versions:/usr/share/modules/$MODULE_VERSION/modulefiles:/usr/share/modules/modulefiles' 'GDMSESSION=ubuntu' 'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus' 'MODULES_CMD=/usr/lib/x86_64-linux-gnu/modulecmd.tcl' 'BASH_FUNC_ml%%=() { module ml "$@" }' 'BASH_FUNC_module%%=() { local _mlredir=1; if [ -n "${MODULES_REDIRECT_OUTPUT+x}" ]; then if [ "$MODULES_REDIRECT_OUTPUT" = '0' ]; then _mlredir=0; else if [ "$MODULES_REDIRECT_OUTPUT" = '1' ]; then _mlredir=1; fi; fi; fi; case " $@ " in *' --no-redirect '*) _mlredir=0 ;; *' --redirect '*) _mlredir=1 ;; esac; if [ $_mlredir -eq 0 ]; then _module_raw "$@"; else _module_raw "$@" 2>&1; fi }' 'BASH_FUNC__module_raw%%=() { eval "$(/usr/bin/tclsh8.6 '/usr/lib/x86_64-linux-gnu/modulecmd.tcl' bash "$@")"; _mlstatus=$?; return $_mlstatus }' '_=/usr/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir [...] --exec-args 1 ./a.out [mpiexec@ITC19206] Launch arguments: /usr/bin/hydra_pmi_proxy --control-port ITC19206:39745 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id 0 [proxy:0@ITC19206] Sending upstream hdr.cmd = CMD_PID_LIST [proxy:0@ITC19206] Sending upstream hdr.cmd = CMD_STDOUT [proxy:0@ITC19206] Sending upstream hdr.cmd = CMD_STDOUT Hello, I am 0 of 1 tasks. Hello, I am 0 of 1 tasks. [proxy:0@ITC19206] Sending upstream hdr.cmd = CMD_EXIT_STATUS Am 03.07.24 um 16:25 schrieb Raffenetti, Ken: > Can you confirm which MPI library you binary is linked with using ldd? Adding -v to the mpirun command should also provide some clues in the PMI client/server traffic. > > Ken > > -----Original Message----- > From: Joachim Jenke via discuss <discuss(a)mpich.org <mailto:[email protected]>> > Reply-To: "discuss(a)mpich.org <mailto:[email protected]>" <discuss(a)mpich.org <mailto:[email protected]>> > Date: Wednesday, July 3, 2024 at 5:16 AM > To: "discuss(a)mpich.org <mailto:[email protected]>" <discuss(a)mpich.org <mailto:[email protected]>> > Cc: Joachim Jenke <jenke(a)itc.rwth-aachen.de <mailto:[email protected]>> > Subject: [mpich-discuss] All processes only find themselfs > > > Hi all, > > > after upgrading my notebook to Ubuntu 24.04, each processes thinks that > MPI_COMM_WORLD only consists of its own process: > > > $ mpirun -np 2 -l ./a.out > [1] Hello, I am 0 of 1 tasks. > [0] Hello, I am 0 of 1 tasks. > > > Any suggestions what could go wrong? > > > > > $ mpirun --version > HYDRA build details: > Version: 4.2.0 > Release Date: Fri Feb 9 12:29:21 CST 2024 > CC: gcc -Wdate-time > -D_FORTIFY_SOURCE=3 -g -O2 -fno-omit-frame-pointer > -mno-omit-leaf-frame-pointer > -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto > -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection > -Wformat -Werror=format-security -fcf-protection > -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3 > -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro > Configure options: '--with-hwloc-prefix=/usr' > '--with-device=ch4:ucx' 'FFLAGS=-O2 -fno-omit-frame-pointer > -mno-omit-leaf-frame-pointer > -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto > -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection > -fcf-protection > -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3 > -fallow-invalid-boz -fallow-argument-mismatch' '--prefix=/usr' > 'CFLAGS=-g -O2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer > -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto > -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection > -Wformat -Werror=format-security -fcf-protection > -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3' > 'LDFLAGS=-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects > -Wl,-z,relro' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=3' > Process Manager: pmi > Launchers available: ssh rsh fork slurm ll lsf > sge manual persist > Topology libraries available: hwloc > Resource management kernels available: user slurm ll lsf sge pbs > cobalt > Demux engines available: poll select > > > Best > Joachim > > -- Dr. rer. nat. Joachim Jenke IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 jenke(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpich-discuss] All processes only find themselfs
by Raffenetti, Ken 03 Jul '24

03 Jul '24

Can you confirm which MPI library you binary is linked with using ldd? Adding -v to the mpirun command should also provide some clues in the PMI client/server traffic. Ken -----Original Message----- From: Joachim Jenke via discuss <discuss(a)mpich.org <mailto:[email protected]>> Reply-To: "discuss(a)mpich.org <mailto:[email protected]>" <discuss(a)mpich.org <mailto:[email protected]>> Date: Wednesday, July 3, 2024 at 5:16 AM To: "discuss(a)mpich.org <mailto:[email protected]>" <discuss(a)mpich.org <mailto:[email protected]>> Cc: Joachim Jenke <jenke(a)itc.rwth-aachen.de <mailto:[email protected]>> Subject: [mpich-discuss] All processes only find themselfs Hi all, after upgrading my notebook to Ubuntu 24.04, each processes thinks that MPI_COMM_WORLD only consists of its own process: $ mpirun -np 2 -l ./a.out [1] Hello, I am 0 of 1 tasks. [0] Hello, I am 0 of 1 tasks. Any suggestions what could go wrong? $ mpirun --version HYDRA build details: Version: 4.2.0 Release Date: Fri Feb 9 12:29:21 CST 2024 CC: gcc -Wdate-time -D_FORTIFY_SOURCE=3 -g -O2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3 -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro Configure options: '--with-hwloc-prefix=/usr' '--with-device=ch4:ucx' 'FFLAGS=-O2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -fcf-protection -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3 -fallow-invalid-boz -fallow-argument-mismatch' '--prefix=/usr' 'CFLAGS=-g -O2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3' 'LDFLAGS=-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=3' Process Manager: pmi Launchers available: ssh rsh fork slurm ll lsf sge manual persist Topology libraries available: hwloc Resource management kernels available: user slurm ll lsf sge pbs cobalt Demux engines available: poll select Best Joachim -- Dr. rer. nat. Joachim Jenke IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 jenke(a)itc.rwth-aachen.de <mailto:[email protected]> https://urldefense.us/v3/__http://www.itc.rwth-aachen.de__;!!G_uCfscf7eWS!b…

1 0

All processes only find themselfs
by Joachim Jenke 03 Jul '24

03 Jul '24

Hi all, after upgrading my notebook to Ubuntu 24.04, each processes thinks that MPI_COMM_WORLD only consists of its own process: $ mpirun -np 2 -l ./a.out [1] Hello, I am 0 of 1 tasks. [0] Hello, I am 0 of 1 tasks. Any suggestions what could go wrong? $ mpirun --version HYDRA build details: Version: 4.2.0 Release Date: Fri Feb 9 12:29:21 CST 2024 CC: gcc -Wdate-time -D_FORTIFY_SOURCE=3 -g -O2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3 -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro Configure options: '--with-hwloc-prefix=/usr' '--with-device=ch4:ucx' 'FFLAGS=-O2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -fcf-protection -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3 -fallow-invalid-boz -fallow-argument-mismatch' '--prefix=/usr' 'CFLAGS=-g -O2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -ffile-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/build/mpich-C3oteG/mpich-4.2.0=/usr/src/mpich-4.2.0-5build3' 'LDFLAGS=-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=3' Process Manager: pmi Launchers available: ssh rsh fork slurm ll lsf sge manual persist Topology libraries available: hwloc Resource management kernels available: user slurm ll lsf sge pbs cobalt Demux engines available: poll select Best Joachim -- Dr. rer. nat. Joachim Jenke IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 jenke(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpich-discuss] Custom rank for processes
by Niyaz Murshed 02 Jul '24

02 Jul '24

I deleted and tried again. root@ampere-altra-2-1:/# cd /dev/shm/ root@ampere-altra-2-1:/dev/shm# ls -l total 24 -rw------- 1 root root 256 Jul 2 15:07 mpich_shar_tmp7LFvuU -rw------- 1 root root 256 Jul 2 02:01 mpich_shar_tmpDXxqHv -rw------- 1 root root 256 Jul 2 15:07 mpich_shar_tmpEivSSF -rw------- 1 root root 192 Jul 2 15:07 mpich_shar_tmpXaqJui -rw------- 1 root root 192 Jul 2 02:01 mpich_shar_tmpg2Rank -rw------- 1 root root 192 Jul 2 15:07 mpich_shar_tmpxwlNrm root@ampere-altra-2-1:/dev/shm# rm -rf * root@ampere-altra-2-1:/dev/shm# mpirun -np 5 -rankmap 1,1,0,0,0 -hosts 192.168.2.100,192.168.2.200 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Abort(678028815) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory Abort(678028815) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory ^C[mpiexec@ampere-altra-2-1] Sending Ctrl-C to processes as requested [mpiexec@ampere-altra-2-1] Press Ctrl-C again to force abort [mpiexec@ampere-altra-2-1] HYDU_sock_write (lib/utils/sock.c:250): write error (Bad file descriptor) [mpiexec@ampere-altra-2-1] send_hdr_downstream (mpiexec/pmiserv_cb.c:28): sock write error [mpiexec@ampere-altra-2-1] HYD_pmcd_pmiserv_send_signal (mpiexec/pmiserv_cb.c:218): unable to write data to proxy [mpiexec@ampere-altra-2-1] ui_cmd_cb (mpiexec/pmiserv_pmci.c:61): unable to send signal downstream [mpiexec@ampere-altra-2-1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [mpiexec@ampere-altra-2-1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:173): error waiting for event [mpiexec@ampere-altra-2-1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion root@ampere-altra-2-1:/dev/shm# ls -l total 8 -rw------- 1 root root 256 Jul 2 15:46 mpich_shar_tmp0eFIGw -rw------- 1 root root 192 Jul 2 15:46 mpich_shar_tmpd4W7pC From: Zhou, Hui <zhouh(a)anl.gov> Date: Tuesday, July 2, 2024 at 10:20 AM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Not sure why it can't create shared memory. Do you have /dev/shm? Is it full? -- Hui ________________________________ From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Sent: Tuesday, July 2, 2024 10:09 AM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Tried both ways but gives error. root@ ampere-altra-2-1: /# mpirun -np 5 -rankmap '(vector,1,1,0,0,0)' -hosts 192. 168. 2. 100,192. 168. 2. 200 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Abort(678028815) on node 1: Fatal error in internal_Init: ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Tried both ways but gives error. root@ampere-altra-2-1:/# mpirun -np 5 -rankmap '(vector,1,1,0,0,0)' -hosts 192.168.2.100,192.168.2.200 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Abort(678028815) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory Abort(678028815) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory ^C[mpiexec@ampere-altra-2-1] Sending Ctrl-C to processes as requested [mpiexec@ampere-altra-2-1] Press Ctrl-C again to force abort [mpiexec@ampere-altra-2-1] HYDU_sock_write (lib/utils/sock.c:250): write error (Bad file descriptor) [mpiexec@ampere-altra-2-1] send_hdr_downstream (mpiexec/pmiserv_cb.c:28): sock write error [mpiexec@ampere-altra-2-1] HYD_pmcd_pmiserv_send_signal (mpiexec/pmiserv_cb.c:218): unable to write data to proxy [mpiexec@ampere-altra-2-1] ui_cmd_cb (mpiexec/pmiserv_pmci.c:61): unable to send signal downstream [mpiexec@ampere-altra-2-1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [mpiexec@ampere-altra-2-1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:173): error waiting for event [mpiexec@ampere-altra-2-1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion root@ampere-altra-2-1:/# mpirun -np 5 -rankmap 1,1,0,0,0 -hosts 192.168.2.100,192.168.2.200 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Abort(678028815) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory Abort(678028815) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory ^C[mpiexec@ampere-altra-2-1] Sending Ctrl-C to processes as requested [mpiexec@ampere-altra-2-1] Press Ctrl-C again to force abort [mpiexec@ampere-altra-2-1] HYDU_sock_write (lib/utils/sock.c:250): write error (Bad file descriptor) [mpiexec@ampere-altra-2-1] send_hdr_downstream (mpiexec/pmiserv_cb.c:28): sock write error [mpiexec@ampere-altra-2-1] HYD_pmcd_pmiserv_send_signal (mpiexec/pmiserv_cb.c:218): unable to write data to proxy [mpiexec@ampere-altra-2-1] ui_cmd_cb (mpiexec/pmiserv_pmci.c:61): unable to send signal downstream [mpiexec@ampere-altra-2-1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [mpiexec@ampere-altra-2-1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:173): error waiting for event [mpiexec@ampere-altra-2-1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 9:39 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Use quote. Try -rankmap '(vector,1,1,0,0,0)'. Actually, the vector part is optional, so this should also work: -rankmap 1,1,0,0,0. -- Hui ________________________________ From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Sent: Monday, July 1, 2024 9:11 PM To: discuss(a)mpich.org <discuss(a)mpich.org>; Zhou, Hui <zhouh(a)anl.gov>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes I get the below error when trying. . am I trying it worng? root@ ampere-altra-2-1: /# mpirun -np 5 -hosts 192. 168. 2. 100,192. 168. 2. 200 -rankmap (vector,1,1,0,0,0) /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world bash: syntax error near ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I get the below error when trying.. am I trying it worng? root@ampere-altra-2-1:/# mpirun -np 5 -hosts 192.168.2.100,192.168.2.200 -rankmap (vector,1,1,0,0,0) /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world bash: syntax error near unexpected token `(' From: Niyaz Murshed via discuss <discuss(a)mpich.org> Date: Monday, July 1, 2024 at 4:42 PM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: [mpich-discuss] Custom rank for processes Thank you so much Hui. Really appreciate it. I now understand it. From: Zhou, Hui <zhouh@ anl. gov> Date: Monday, July 1, 2024 at 4: 37 PM To: Niyaz Murshed <Niyaz. Murshed@ arm. com>, discuss@ mpich. org <discuss@ mpich. org>, Jenke, ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you so much Hui. Really appreciate it. I now understand it. From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 4:37 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Yes, you can use rankmap to simply list out the node assignment for each rank. "-rankmap (vector,1,1,0,0,0)" is a list of 5 node ids, one for each rank. So rank 0 gets node 1, rank 1 gets node 1, rank 2 gets node 0, and so on. The -hosts options is convenient if you have somewhat uniform assignment. If you want arbitrary assignment, just use -rankmap option. -- Hui ________________________________ From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Sent: Monday, July 1, 2024 4:30 PM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Sorry to bother you again with silly questions. How do I read the below : -rankmap (vector,1,1,0,0,0) I have 5 processes. 1 1 => means node1 will get first 2 processes ? -hosts 192. 168. 2. 100: 2,192. 168. 2. 200: 3 : this will give 2 process in ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Sorry to bother you again with silly questions. How do I read the below : -rankmap (vector,1,1,0,0,0) I have 5 processes. 1 1 => means node1 will get first 2 processes ? -hosts 192.168.2.100:2,192.168.2.200:3 : this will give 2 process in 100 and 3 in 200 .. Still won’t be able to specify which ranks goes to which node. Rank0 will be on node 100 .. what If I want Rank0 on node 200? From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 4:09 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes > root@ampere-altra-2-1:/# mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors Hello world from processor dpr740, rank 0 out of 5 processors Hello world from processor dpr740, rank 4 out of 5 processors Hello world from processor dpr740, rank 2 out of 5 processors > -bind-to user:10,11,12,13 > This would mean on host 192.168.2.100 > P0=>10 , P2=>11 > This would mean on host 192.168.2.200 > P0=>10 , P2=>11, P3=12 > Is this correct understanding ? Is it also possible to say which rank process will be pinned to which core ? Yes, that is correct. The ranks are assigned to hosts as shown in the hello world output. > About the rankmap:, trying to understand if I can select where a particular rank would be from > list of hosts. Currently, the first host in the list always get rank0. > Can I specify the below ranks? > mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world > > Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors => rank0 > Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors. => rank1 > Hello world from processor dpr740, rank 0 out of 5 processors =>rank2 > Hello world from processor dpr740, rank 4 out of 5 processors =>rank3 > Hello world from processor dpr740, rank 2 out of 5 processors =>rank4 Yes. You can use "-rankmap (vector,1,1,0,0,0)". Alternatively, you can use "-hosts 192.168.2.100:2,192.168.2.200:3", the colon syntax specifies how many processes you want to assign to each host. -- Hui

1 0

Re: [mpich-discuss] Custom rank for processes
by Zhou, Hui 02 Jul '24

02 Jul '24

Not sure why it can't create shared memory. Do you have /dev/shm? Is it full? -- Hui ________________________________ From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Sent: Tuesday, July 2, 2024 10:09 AM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Tried both ways but gives error. root@ ampere-altra-2-1: /# mpirun -np 5 -rankmap '(vector,1,1,0,0,0)' -hosts 192. 168. 2. 100,192. 168. 2. 200 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Abort(678028815) on node 1: Fatal error in internal_Init: ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Tried both ways but gives error. root@ampere-altra-2-1:/# mpirun -np 5 -rankmap '(vector,1,1,0,0,0)' -hosts 192.168.2.100,192.168.2.200 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Abort(678028815) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory Abort(678028815) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory ^C[mpiexec@ampere-altra-2-1] Sending Ctrl-C to processes as requested [mpiexec@ampere-altra-2-1] Press Ctrl-C again to force abort [mpiexec@ampere-altra-2-1] HYDU_sock_write (lib/utils/sock.c:250): write error (Bad file descriptor) [mpiexec@ampere-altra-2-1] send_hdr_downstream (mpiexec/pmiserv_cb.c:28): sock write error [mpiexec@ampere-altra-2-1] HYD_pmcd_pmiserv_send_signal (mpiexec/pmiserv_cb.c:218): unable to write data to proxy [mpiexec@ampere-altra-2-1] ui_cmd_cb (mpiexec/pmiserv_pmci.c:61): unable to send signal downstream [mpiexec@ampere-altra-2-1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [mpiexec@ampere-altra-2-1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:173): error waiting for event [mpiexec@ampere-altra-2-1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion root@ampere-altra-2-1:/# mpirun -np 5 -rankmap 1,1,0,0,0 -hosts 192.168.2.100,192.168.2.200 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Abort(678028815) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory Abort(678028815) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory ^C[mpiexec@ampere-altra-2-1] Sending Ctrl-C to processes as requested [mpiexec@ampere-altra-2-1] Press Ctrl-C again to force abort [mpiexec@ampere-altra-2-1] HYDU_sock_write (lib/utils/sock.c:250): write error (Bad file descriptor) [mpiexec@ampere-altra-2-1] send_hdr_downstream (mpiexec/pmiserv_cb.c:28): sock write error [mpiexec@ampere-altra-2-1] HYD_pmcd_pmiserv_send_signal (mpiexec/pmiserv_cb.c:218): unable to write data to proxy [mpiexec@ampere-altra-2-1] ui_cmd_cb (mpiexec/pmiserv_pmci.c:61): unable to send signal downstream [mpiexec@ampere-altra-2-1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [mpiexec@ampere-altra-2-1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:173): error waiting for event [mpiexec@ampere-altra-2-1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 9:39 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Use quote. Try -rankmap '(vector,1,1,0,0,0)'. Actually, the vector part is optional, so this should also work: -rankmap 1,1,0,0,0. -- Hui ________________________________ From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Sent: Monday, July 1, 2024 9:11 PM To: discuss(a)mpich.org <discuss(a)mpich.org>; Zhou, Hui <zhouh(a)anl.gov>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes I get the below error when trying. . am I trying it worng? root@ ampere-altra-2-1: /# mpirun -np 5 -hosts 192. 168. 2. 100,192. 168. 2. 200 -rankmap (vector,1,1,0,0,0) /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world bash: syntax error near ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I get the below error when trying.. am I trying it worng? root@ampere-altra-2-1:/# mpirun -np 5 -hosts 192.168.2.100,192.168.2.200 -rankmap (vector,1,1,0,0,0) /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world bash: syntax error near unexpected token `(' From: Niyaz Murshed via discuss <discuss(a)mpich.org> Date: Monday, July 1, 2024 at 4:42 PM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: [mpich-discuss] Custom rank for processes Thank you so much Hui. Really appreciate it. I now understand it. From: Zhou, Hui <zhouh@ anl. gov> Date: Monday, July 1, 2024 at 4: 37 PM To: Niyaz Murshed <Niyaz. Murshed@ arm. com>, discuss@ mpich. org <discuss@ mpich. org>, Jenke, ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you so much Hui. Really appreciate it. I now understand it. From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 4:37 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Yes, you can use rankmap to simply list out the node assignment for each rank. "-rankmap (vector,1,1,0,0,0)" is a list of 5 node ids, one for each rank. So rank 0 gets node 1, rank 1 gets node 1, rank 2 gets node 0, and so on. The -hosts options is convenient if you have somewhat uniform assignment. If you want arbitrary assignment, just use -rankmap option. -- Hui ________________________________ From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Sent: Monday, July 1, 2024 4:30 PM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Sorry to bother you again with silly questions. How do I read the below : -rankmap (vector,1,1,0,0,0) I have 5 processes. 1 1 => means node1 will get first 2 processes ? -hosts 192. 168. 2. 100: 2,192. 168. 2. 200: 3 : this will give 2 process in ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Sorry to bother you again with silly questions. How do I read the below : -rankmap (vector,1,1,0,0,0) I have 5 processes. 1 1 => means node1 will get first 2 processes ? -hosts 192.168.2.100:2,192.168.2.200:3 : this will give 2 process in 100 and 3 in 200 .. Still won’t be able to specify which ranks goes to which node. Rank0 will be on node 100 .. what If I want Rank0 on node 200? From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 4:09 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes > root@ampere-altra-2-1:/# mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors Hello world from processor dpr740, rank 0 out of 5 processors Hello world from processor dpr740, rank 4 out of 5 processors Hello world from processor dpr740, rank 2 out of 5 processors > -bind-to user:10,11,12,13 > This would mean on host 192.168.2.100 > P0=>10 , P2=>11 > This would mean on host 192.168.2.200 > P0=>10 , P2=>11, P3=12 > Is this correct understanding ? Is it also possible to say which rank process will be pinned to which core ? Yes, that is correct. The ranks are assigned to hosts as shown in the hello world output. > About the rankmap:, trying to understand if I can select where a particular rank would be from > list of hosts. Currently, the first host in the list always get rank0. > Can I specify the below ranks? > mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world > > Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors => rank0 > Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors. => rank1 > Hello world from processor dpr740, rank 0 out of 5 processors =>rank2 > Hello world from processor dpr740, rank 4 out of 5 processors =>rank3 > Hello world from processor dpr740, rank 2 out of 5 processors =>rank4 Yes. You can use "-rankmap (vector,1,1,0,0,0)". Alternatively, you can use "-hosts 192.168.2.100:2,192.168.2.200:3", the colon syntax specifies how many processes you want to assign to each host. -- Hui

1 0

Re: [mpich-discuss] Custom rank for processes
by Niyaz Murshed 02 Jul '24

02 Jul '24

Tried both ways but gives error. root@ampere-altra-2-1:/# mpirun -np 5 -rankmap '(vector,1,1,0,0,0)' -hosts 192.168.2.100,192.168.2.200 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Abort(678028815) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory Abort(678028815) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory ^C[mpiexec@ampere-altra-2-1] Sending Ctrl-C to processes as requested [mpiexec@ampere-altra-2-1] Press Ctrl-C again to force abort [mpiexec@ampere-altra-2-1] HYDU_sock_write (lib/utils/sock.c:250): write error (Bad file descriptor) [mpiexec@ampere-altra-2-1] send_hdr_downstream (mpiexec/pmiserv_cb.c:28): sock write error [mpiexec@ampere-altra-2-1] HYD_pmcd_pmiserv_send_signal (mpiexec/pmiserv_cb.c:218): unable to write data to proxy [mpiexec@ampere-altra-2-1] ui_cmd_cb (mpiexec/pmiserv_pmci.c:61): unable to send signal downstream [mpiexec@ampere-altra-2-1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [mpiexec@ampere-altra-2-1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:173): error waiting for event [mpiexec@ampere-altra-2-1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion root@ampere-altra-2-1:/# mpirun -np 5 -rankmap 1,1,0,0,0 -hosts 192.168.2.100,192.168.2.200 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Abort(678028815) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory Abort(678028815) on node 3: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(70).............: MPI_Init(argc=(nil), argv=(nil)) failed MPII_Init_thread(268).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(800).........: MPIR_Comm_commit_internal(585): MPID_Comm_commit_pre_hook(151): MPIDI_world_pre_init(638).....: MPIDU_Init_shm_init(179)......: unable to allocate shared memory ^C[mpiexec@ampere-altra-2-1] Sending Ctrl-C to processes as requested [mpiexec@ampere-altra-2-1] Press Ctrl-C again to force abort [mpiexec@ampere-altra-2-1] HYDU_sock_write (lib/utils/sock.c:250): write error (Bad file descriptor) [mpiexec@ampere-altra-2-1] send_hdr_downstream (mpiexec/pmiserv_cb.c:28): sock write error [mpiexec@ampere-altra-2-1] HYD_pmcd_pmiserv_send_signal (mpiexec/pmiserv_cb.c:218): unable to write data to proxy [mpiexec@ampere-altra-2-1] ui_cmd_cb (mpiexec/pmiserv_pmci.c:61): unable to send signal downstream [mpiexec@ampere-altra-2-1] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status [mpiexec@ampere-altra-2-1] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:173): error waiting for event [mpiexec@ampere-altra-2-1] main (mpiexec/mpiexec.c:260): process manager error waiting for completion From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 9:39 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Use quote. Try -rankmap '(vector,1,1,0,0,0)'. Actually, the vector part is optional, so this should also work: -rankmap 1,1,0,0,0. -- Hui ________________________________ From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Sent: Monday, July 1, 2024 9:11 PM To: discuss(a)mpich.org <discuss(a)mpich.org>; Zhou, Hui <zhouh(a)anl.gov>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes I get the below error when trying. . am I trying it worng? root@ ampere-altra-2-1: /# mpirun -np 5 -hosts 192. 168. 2. 100,192. 168. 2. 200 -rankmap (vector,1,1,0,0,0) /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world bash: syntax error near ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I get the below error when trying.. am I trying it worng? root@ampere-altra-2-1:/# mpirun -np 5 -hosts 192.168.2.100,192.168.2.200 -rankmap (vector,1,1,0,0,0) /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world bash: syntax error near unexpected token `(' From: Niyaz Murshed via discuss <discuss(a)mpich.org> Date: Monday, July 1, 2024 at 4:42 PM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: [mpich-discuss] Custom rank for processes Thank you so much Hui. Really appreciate it. I now understand it. From: Zhou, Hui <zhouh@ anl. gov> Date: Monday, July 1, 2024 at 4: 37 PM To: Niyaz Murshed <Niyaz. Murshed@ arm. com>, discuss@ mpich. org <discuss@ mpich. org>, Jenke, ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you so much Hui. Really appreciate it. I now understand it. From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 4:37 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Yes, you can use rankmap to simply list out the node assignment for each rank. "-rankmap (vector,1,1,0,0,0)" is a list of 5 node ids, one for each rank. So rank 0 gets node 1, rank 1 gets node 1, rank 2 gets node 0, and so on. The -hosts options is convenient if you have somewhat uniform assignment. If you want arbitrary assignment, just use -rankmap option. -- Hui ________________________________ From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Sent: Monday, July 1, 2024 4:30 PM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Sorry to bother you again with silly questions. How do I read the below : -rankmap (vector,1,1,0,0,0) I have 5 processes. 1 1 => means node1 will get first 2 processes ? -hosts 192. 168. 2. 100: 2,192. 168. 2. 200: 3 : this will give 2 process in ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Sorry to bother you again with silly questions. How do I read the below : -rankmap (vector,1,1,0,0,0) I have 5 processes. 1 1 => means node1 will get first 2 processes ? -hosts 192.168.2.100:2,192.168.2.200:3 : this will give 2 process in 100 and 3 in 200 .. Still won’t be able to specify which ranks goes to which node. Rank0 will be on node 100 .. what If I want Rank0 on node 200? From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 4:09 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes > root@ampere-altra-2-1:/# mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors Hello world from processor dpr740, rank 0 out of 5 processors Hello world from processor dpr740, rank 4 out of 5 processors Hello world from processor dpr740, rank 2 out of 5 processors > -bind-to user:10,11,12,13 > This would mean on host 192.168.2.100 > P0=>10 , P2=>11 > This would mean on host 192.168.2.200 > P0=>10 , P2=>11, P3=12 > Is this correct understanding ? Is it also possible to say which rank process will be pinned to which core ? Yes, that is correct. The ranks are assigned to hosts as shown in the hello world output. > About the rankmap:, trying to understand if I can select where a particular rank would be from > list of hosts. Currently, the first host in the list always get rank0. > Can I specify the below ranks? > mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world > > Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors => rank0 > Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors. => rank1 > Hello world from processor dpr740, rank 0 out of 5 processors =>rank2 > Hello world from processor dpr740, rank 4 out of 5 processors =>rank3 > Hello world from processor dpr740, rank 2 out of 5 processors =>rank4 Yes. You can use "-rankmap (vector,1,1,0,0,0)". Alternatively, you can use "-hosts 192.168.2.100:2,192.168.2.200:3", the colon syntax specifies how many processes you want to assign to each host. -- Hui

1 0

Re: [mpich-discuss] Custom rank for processes
by Zhou, Hui 01 Jul '24

01 Jul '24

Use quote. Try -rankmap '(vector,1,1,0,0,0)'. Actually, the vector part is optional, so this should also work: -rankmap 1,1,0,0,0. -- Hui ________________________________ From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Sent: Monday, July 1, 2024 9:11 PM To: discuss(a)mpich.org <discuss(a)mpich.org>; Zhou, Hui <zhouh(a)anl.gov>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes I get the below error when trying. . am I trying it worng? root@ ampere-altra-2-1: /# mpirun -np 5 -hosts 192. 168. 2. 100,192. 168. 2. 200 -rankmap (vector,1,1,0,0,0) /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world bash: syntax error near ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I get the below error when trying.. am I trying it worng? root@ampere-altra-2-1:/# mpirun -np 5 -hosts 192.168.2.100,192.168.2.200 -rankmap (vector,1,1,0,0,0) /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world bash: syntax error near unexpected token `(' From: Niyaz Murshed via discuss <discuss(a)mpich.org> Date: Monday, July 1, 2024 at 4:42 PM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: [mpich-discuss] Custom rank for processes Thank you so much Hui. Really appreciate it. I now understand it. From: Zhou, Hui <zhouh@ anl. gov> Date: Monday, July 1, 2024 at 4: 37 PM To: Niyaz Murshed <Niyaz. Murshed@ arm. com>, discuss@ mpich. org <discuss@ mpich. org>, Jenke, ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you so much Hui. Really appreciate it. I now understand it. From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 4:37 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Yes, you can use rankmap to simply list out the node assignment for each rank. "-rankmap (vector,1,1,0,0,0)" is a list of 5 node ids, one for each rank. So rank 0 gets node 1, rank 1 gets node 1, rank 2 gets node 0, and so on. The -hosts options is convenient if you have somewhat uniform assignment. If you want arbitrary assignment, just use -rankmap option. -- Hui ________________________________ From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Sent: Monday, July 1, 2024 4:30 PM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Sorry to bother you again with silly questions. How do I read the below : -rankmap (vector,1,1,0,0,0) I have 5 processes. 1 1 => means node1 will get first 2 processes ? -hosts 192. 168. 2. 100: 2,192. 168. 2. 200: 3 : this will give 2 process in ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Sorry to bother you again with silly questions. How do I read the below : -rankmap (vector,1,1,0,0,0) I have 5 processes. 1 1 => means node1 will get first 2 processes ? -hosts 192.168.2.100:2,192.168.2.200:3 : this will give 2 process in 100 and 3 in 200 .. Still won’t be able to specify which ranks goes to which node. Rank0 will be on node 100 .. what If I want Rank0 on node 200? From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 4:09 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes > root@ampere-altra-2-1:/# mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors Hello world from processor dpr740, rank 0 out of 5 processors Hello world from processor dpr740, rank 4 out of 5 processors Hello world from processor dpr740, rank 2 out of 5 processors > -bind-to user:10,11,12,13 > This would mean on host 192.168.2.100 > P0=>10 , P2=>11 > This would mean on host 192.168.2.200 > P0=>10 , P2=>11, P3=12 > Is this correct understanding ? Is it also possible to say which rank process will be pinned to which core ? Yes, that is correct. The ranks are assigned to hosts as shown in the hello world output. > About the rankmap:, trying to understand if I can select where a particular rank would be from > list of hosts. Currently, the first host in the list always get rank0. > Can I specify the below ranks? > mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world > > Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors => rank0 > Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors. => rank1 > Hello world from processor dpr740, rank 0 out of 5 processors =>rank2 > Hello world from processor dpr740, rank 4 out of 5 processors =>rank3 > Hello world from processor dpr740, rank 2 out of 5 processors =>rank4 Yes. You can use "-rankmap (vector,1,1,0,0,0)". Alternatively, you can use "-hosts 192.168.2.100:2,192.168.2.200:3", the colon syntax specifies how many processes you want to assign to each host. -- Hui

1 0

Re: [mpich-discuss] Custom rank for processes
by Niyaz Murshed 01 Jul '24

01 Jul '24

I get the below error when trying.. am I trying it worng? root@ampere-altra-2-1:/# mpirun -np 5 -hosts 192.168.2.100,192.168.2.200 -rankmap (vector,1,1,0,0,0) /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world bash: syntax error near unexpected token `(' From: Niyaz Murshed via discuss <discuss(a)mpich.org> Date: Monday, July 1, 2024 at 4:42 PM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: [mpich-discuss] Custom rank for processes Thank you so much Hui. Really appreciate it. I now understand it. From: Zhou, Hui <zhouh@ anl. gov> Date: Monday, July 1, 2024 at 4: 37 PM To: Niyaz Murshed <Niyaz. Murshed@ arm. com>, discuss@ mpich. org <discuss@ mpich. org>, Jenke, ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you so much Hui. Really appreciate it. I now understand it. From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 4:37 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Yes, you can use rankmap to simply list out the node assignment for each rank. "-rankmap (vector,1,1,0,0,0)" is a list of 5 node ids, one for each rank. So rank 0 gets node 1, rank 1 gets node 1, rank 2 gets node 0, and so on. The -hosts options is convenient if you have somewhat uniform assignment. If you want arbitrary assignment, just use -rankmap option. -- Hui ________________________________ From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Sent: Monday, July 1, 2024 4:30 PM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>; Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes Sorry to bother you again with silly questions. How do I read the below : -rankmap (vector,1,1,0,0,0) I have 5 processes. 1 1 => means node1 will get first 2 processes ? -hosts 192. 168. 2. 100: 2,192. 168. 2. 200: 3 : this will give 2 process in ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Sorry to bother you again with silly questions. How do I read the below : -rankmap (vector,1,1,0,0,0) I have 5 processes. 1 1 => means node1 will get first 2 processes ? -hosts 192.168.2.100:2,192.168.2.200:3 : this will give 2 process in 100 and 3 in 200 .. Still won’t be able to specify which ranks goes to which node. Rank0 will be on node 100 .. what If I want Rank0 on node 200? From: Zhou, Hui <zhouh(a)anl.gov> Date: Monday, July 1, 2024 at 4:09 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>, Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: nd <nd(a)arm.com> Subject: Re: Custom rank for processes > root@ampere-altra-2-1:/# mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors Hello world from processor dpr740, rank 0 out of 5 processors Hello world from processor dpr740, rank 4 out of 5 processors Hello world from processor dpr740, rank 2 out of 5 processors > -bind-to user:10,11,12,13 > This would mean on host 192.168.2.100 > P0=>10 , P2=>11 > This would mean on host 192.168.2.200 > P0=>10 , P2=>11, P3=12 > Is this correct understanding ? Is it also possible to say which rank process will be pinned to which core ? Yes, that is correct. The ranks are assigned to hosts as shown in the hello world output. > About the rankmap:, trying to understand if I can select where a particular rank would be from > list of hosts. Currently, the first host in the list always get rank0. > Can I specify the below ranks? > mpirun -n 5 -bind-to user:10,11,12,13 -hosts 192.168.2.200,192.168.2.100 /mpitutorial/tutorials/mpi-hello-world/code/mpi_hello_world > > Hello world from processor ampere-altra-2-1, rank 1 out of 5 processors => rank0 > Hello world from processor ampere-altra-2-1, rank 3 out of 5 processors. => rank1 > Hello world from processor dpr740, rank 0 out of 5 processors =>rank2 > Hello world from processor dpr740, rank 4 out of 5 processors =>rank3 > Hello world from processor dpr740, rank 2 out of 5 processors =>rank4 Yes. You can use "-rankmap (vector,1,1,0,0,0)". Alternatively, you can use "-hosts 192.168.2.100:2,192.168.2.200:3", the colon syntax specifies how many processes you want to assign to each host. -- Hui

1 0