- discuss - mailman3.cels.anl.gov

MPI_Send over ibverbs
by Niyaz Murshed 07 Oct '24

07 Oct '24

Hi, Is there a way to check the implementation of MPI_Send? As I understand there is an implicit barrier which allows MPI_Send to return only after the buffer is received by the destination. How is this communication done? How does the destination notify the sender that buffer is received? How is this implemented when using rocev2 ? Regards, Niyaz

1 0

Re: [mpich-discuss] Maximum number of communicators
by Palmer, Bruce J 07 Oct '24

07 Oct '24

That seemed to fix the issue. Thanks! Bruce From: Zhou, Hui <zhouh(a)anl.gov> Date: Thursday, October 3, 2024 at 11:01 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: Re: Maximum number of communicators Hi Bruce, Try configure mpich using --with-device=ch4:ofi --enable-extended-context-bits. Hui ________________________________ From: Palmer, Bruce J via discuss <discuss(a)mpich.org> Sent: Thursday, October 3, 2024 11:44 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: [mpich-discuss] Maximum number of communicators Hi, I’m looking at using MPI RMA to support sparse data structures in Global Arrays. I’ve got an application that uses a large number of sparse arrays and it is failing when the number of sparse arrays reaches about 500. Each sparse array is ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, I’m looking at using MPI RMA to support sparse data structures in Global Arrays. I’ve got an application that uses a large number of sparse arrays and it is failing when the number of sparse arrays reaches about 500. Each sparse array is built on top of 4 conventional global arrays and each global array uses one MPI Window. Each Window appears to be creating its own communicator and I’m hitting an internal limit at 2048 communicators. Is there a way to increase the number of communicators? Bruce Palmer

1 0

Re: [mpich-discuss] Maximum number of communicators
by Zhou, Hui 03 Oct '24

03 Oct '24

Hi Bruce, Try configure mpich using --with-device=ch4:ofi --enable-extended-context-bits. Hui ________________________________ From: Palmer, Bruce J via discuss <discuss(a)mpich.org> Sent: Thursday, October 3, 2024 11:44 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: [mpich-discuss] Maximum number of communicators Hi, I’m looking at using MPI RMA to support sparse data structures in Global Arrays. I’ve got an application that uses a large number of sparse arrays and it is failing when the number of sparse arrays reaches about 500. Each sparse array is ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, I’m looking at using MPI RMA to support sparse data structures in Global Arrays. I’ve got an application that uses a large number of sparse arrays and it is failing when the number of sparse arrays reaches about 500. Each sparse array is built on top of 4 conventional global arrays and each global array uses one MPI Window. Each Window appears to be creating its own communicator and I’m hitting an internal limit at 2048 communicators. Is there a way to increase the number of communicators? Bruce Palmer

1 0

Maximum number of communicators
by Palmer, Bruce J 03 Oct '24

03 Oct '24

Hi, I’m looking at using MPI RMA to support sparse data structures in Global Arrays. I’ve got an application that uses a large number of sparse arrays and it is failing when the number of sparse arrays reaches about 500. Each sparse array is built on top of 4 conventional global arrays and each global array uses one MPI Window. Each Window appears to be creating its own communicator and I’m hitting an internal limit at 2048 communicators. Is there a way to increase the number of communicators? Bruce Palmer

1 0

Re: [mpich-discuss] MPICH 4.2.2
by Zhou, Hui 24 Sep '24

24 Sep '24

You need to add the bin path to your PATH variable. Try export PATH=/media/witt/8efd83a8-d3be-4079-b8e3-6fceffff35bb/appdata/mpich4/install/bin:$PATH Hui ________________________________ From: Rich Martorano via discuss <discuss(a)mpich.org> Sent: Tuesday, September 24, 2024 12:29 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Rich Martorano <richmartorano(a)live.com> Subject: [mpich-discuss] MPICH 4.2.2 Good morning, I just subscribed to this however I need some help. I really don't know much about MPICH however in my book Raspberry Pi Sper Cluster I am in a section that calls for this application to be installed. I have an issue when I try ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Good morning, I just subscribed to this however I need some help. I really don't know much about MPICH however in my book Raspberry Pi Sper Cluster I am in a section that calls for this application to be installed. I have an issue when I try to run mpiexec commands, specifically mpiexec -f piprofile hostname. Command not found error. I am not even able to run mpiexec --version without the error. I clearly see them in the below path but I can execute anything. /media/witt/8efd83a8-d3be-4079-b8e3-6fceffff35bb/appdata/mpich4/install/bin# In my book the command they have for install is this; /media/witt/8efd83a8-d3be-4079-b8e3-6fceffff35bb/appdata/mpich4/mpich-4.2.2/configure \-prefix=/media/witt/8efd83a8-d3be-4079-b8e3-6fceffff35bb/appdata/mpich4/install What am I doing wrong. I have tried twice to install it and I get the same result each time command not found. I will day however That after the installation I can run mpiexec -f piprofile hostname and it works, however after about an hour or so I will ge the error message. Also my book references version 3.0.4 but I thought since the book was dated to find the latest. Best Regards Rich Martorano

1 0

MPICH 4.2.2
by Rich Martorano 24 Sep '24

24 Sep '24

Good morning, I just subscribed to this however I need some help. I really don't know much about MPICH however in my book Raspberry Pi Sper Cluster I am in a section that calls for this application to be installed. I have an issue when I try to run mpiexec commands, specifically mpiexec -f piprofile hostname. Command not found error. I am not even able to run mpiexec --version without the error. I clearly see them in the below path but I can execute anything. /media/witt/8efd83a8-d3be-4079-b8e3-6fceffff35bb/appdata/mpich4/install/bin# In my book the command they have for install is this; /media/witt/8efd83a8-d3be-4079-b8e3-6fceffff35bb/appdata/mpich4/mpich-4.2.2/configure \-prefix=/media/witt/8efd83a8-d3be-4079-b8e3-6fceffff35bb/appdata/mpich4/install What am I doing wrong. I have tried twice to install it and I get the same result each time command not found. I will day however That after the installation I can run mpiexec -f piprofile hostname and it works, however after about an hour or so I will ge the error message. Also my book references version 3.0.4 but I thought since the book was dated to find the latest. Best Regards Rich Martorano

1 0

Re: [mpich-discuss] Issue with MPICH + mpi4py on Ubuntu 24.04
by Zhou, Hui 16 Sep '24

16 Sep '24

This is a known ubuntu package issue -https://urldefense.us/v3/__https://bugs.launchpad.net/ubuntu/*source/mpich/*bug/2072338__;Kys!!G_uCfscf7eWS!ZXbM8OZL3oAoMXNbRtfOkqUOlgk7AgJe6H7xvMCPY0O110T3hB1GspOKUNggP0dHHzL542tPtKrL$ Hui ________________________________ From: Jørgen Dokken via discuss <discuss(a)mpich.org> Sent: Monday, September 16, 2024 2:06 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Jørgen Dokken <dokken(a)simula.no> Subject: [mpich-discuss] Issue with MPICH + mpi4py on Ubuntu 24.04 When trying to install MPICH from the apt repository on Ubuntu 24. 04 and then installing mpi4py with pip, I get a non-functional installation of mpi4py. It does not happen if I use openmpi, or downgrade to Ubuntu 22. 04. It has been tested with ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd When trying to install MPICH from the apt repository on Ubuntu 24.04 and then installing mpi4py with pip, I get a non-functional installation of mpi4py. It does not happen if I use openmpi, or downgrade to Ubuntu 22.04. It has been tested with the same MPICH on other systems (See: https://urldefense.us/v3/__https://github.com/mpi4py/mpi4py/issues/547__;!!… <https://urldefense.us/v3/__https://github.com/mpi4py/mpi4py/issues/547__;!!…>) for where it has been tested, leading one to believe that something is wrong with the ubuntu build. Minimal reproducible dockerfile: ``` FROM ubuntu:24.04 ARG MPI="mpich" ENV OPENBLAS_NUM_THREADS=1 \ OPENBLAS_VERBOSE=0 ENV DEB_PYTHON_INSTALL_LAYOUT=deb_system ENV DEBIAN_FRONTEND=noninteractive WORKDIR /tmp RUN apt-get -qq update && \ apt-get -yq upgrade && \ apt-get -y install \ lib${MPI}-dev \ python3-dev \ python3-pip \ python3-setuptools \ python3-venv && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* ENV VIRTUAL_ENV=/test-env ENV PATH=/${VIRTUAL_ENV}/bin:$PATH RUN python3 -m venv ${VIRTUAL_ENV} # Install Python packages (via pip) RUN python3 -m pip install --no-cache-dir mpi4py -v ENV OMPI_ALLOW_RUN_AS_ROOT=1 \ OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1 CMD ["mpiexec", "-v", "--np", "2", "python3", "-c", "from mpi4py import MPI; print(f'{MPI.COMM_WORLD.rank}/{MPI.COMM_WORLD.size}')"] ``` with the output ``` host: 6a2e05040a65 [mpiexec@6a2e05040a65] Timeout set to -1 (-1 means infinite) ================================================================================================== mpiexec options: ---------------- Base path: /usr/bin/ Launcher: (null) Debug level: 1 Enable X: -1 Global environment: ------------------- PATH=//test-env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=6a2e05040a65 TERM=xterm OPENBLAS_NUM_THREADS=1 OPENBLAS_VERBOSE=0 DEB_PYTHON_INSTALL_LAYOUT=deb_system DEBIAN_FRONTEND=noninteractive VIRTUAL_ENV=/test-env OMPI_ALLOW_RUN_AS_ROOT=1 OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1 HOME=/root Hydra internal environment: --------------------------- GFORTRAN_UNBUFFERED_PRECONNECTED=y Proxy information: ********************* [1] proxy: 6a2e05040a65 (1 cores) Exec list: python3 (2 processes); ================================================================================================== Proxy launch args: /usr/bin/hydra_pmi_proxy --control-port 6a2e05040a65:36819 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id Arguments being passed to proxy 0: --version 4.2.0 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 6a2e05040a65 --global-core-map 0,1,1 --pmi-id-map 0,0 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_1_0_1589598326_6a2e05040a65 --pmi-process-mapping (vector,(0,1,2)) --global-inherited-env 11 'PATH=//test-env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' 'HOSTNAME=6a2e05040a65' 'TERM=xterm' 'OPENBLAS_NUM_THREADS=1' 'OPENBLAS_VERBOSE=0' 'DEB_PYTHON_INSTALL_LAYOUT=deb_system' 'DEBIAN_FRONTEND=noninteractive' 'VIRTUAL_ENV=/test-env' 'OMPI_ALLOW_RUN_AS_ROOT=1' 'OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1' 'HOME=/root' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir /tmp --exec-args 3 python3 -c from mpi4py import MPI; print(f'{MPI.COMM_WORLD.rank}/{MPI.COMM_WORLD.size}'); [mpiexec@6a2e05040a65] Launch arguments: /usr/bin/hydra_pmi_proxy --control-port 6a2e05040a65:36819 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id 0 [proxy:0@6a2e05040a65] Sending upstream hdr.cmd = CMD_PID_LIST [proxy:0@6a2e05040a65] Sending upstream hdr.cmd = CMD_STDOUT [proxy:0@6a2e05040a65] Sending upstream hdr.cmd = CMD_STDOUT 0/1 0/1 [proxy:0@6a2e05040a65] Sending upstream hdr.cmd = CMD_EXIT_STATUS ``` Best, Jørgen -- Jørgen S. Dokken, PhD Senior Research Engineer Simula Research Laboratory +47 45286467

1 0

Issue with MPICH + mpi4py on Ubuntu 24.04
by Jørgen Dokken 16 Sep '24

16 Sep '24

When trying to install MPICH from the apt repository on Ubuntu 24.04 and then installing mpi4py with pip, I get a non-functional installation of mpi4py. It does not happen if I use openmpi, or downgrade to Ubuntu 22.04. It has been tested with the same MPICH on other systems (See: https://urldefense.us/v3/__https://github.com/mpi4py/mpi4py/issues/547__;!!… ) for where it has been tested, leading one to believe that something is wrong with the ubuntu build. Minimal reproducible dockerfile: ``` FROM ubuntu:24.04 ARG MPI="mpich" ENV OPENBLAS_NUM_THREADS=1 \ OPENBLAS_VERBOSE=0 ENV DEB_PYTHON_INSTALL_LAYOUT=deb_system ENV DEBIAN_FRONTEND=noninteractive WORKDIR /tmp RUN apt-get -qq update && \ apt-get -yq upgrade && \ apt-get -y install \ lib${MPI}-dev \ python3-dev \ python3-pip \ python3-setuptools \ python3-venv && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* ENV VIRTUAL_ENV=/test-env ENV PATH=/${VIRTUAL_ENV}/bin:$PATH RUN python3 -m venv ${VIRTUAL_ENV} # Install Python packages (via pip) RUN python3 -m pip install --no-cache-dir mpi4py -v ENV OMPI_ALLOW_RUN_AS_ROOT=1 \ OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1 CMD ["mpiexec", "-v", "--np", "2", "python3", "-c", "from mpi4py import MPI; print(f'{MPI.COMM_WORLD.rank}/{MPI.COMM_WORLD.size}')"] ``` with the output ``` host: 6a2e05040a65 [mpiexec@6a2e05040a65] Timeout set to -1 (-1 means infinite) ================================================================================================== mpiexec options: ---------------- Base path: /usr/bin/ Launcher: (null) Debug level: 1 Enable X: -1 Global environment: ------------------- PATH=//test-env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=6a2e05040a65 TERM=xterm OPENBLAS_NUM_THREADS=1 OPENBLAS_VERBOSE=0 DEB_PYTHON_INSTALL_LAYOUT=deb_system DEBIAN_FRONTEND=noninteractive VIRTUAL_ENV=/test-env OMPI_ALLOW_RUN_AS_ROOT=1 OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1 HOME=/root Hydra internal environment: --------------------------- GFORTRAN_UNBUFFERED_PRECONNECTED=y Proxy information: ********************* [1] proxy: 6a2e05040a65 (1 cores) Exec list: python3 (2 processes); ================================================================================================== Proxy launch args: /usr/bin/hydra_pmi_proxy --control-port 6a2e05040a65:36819 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id Arguments being passed to proxy 0: --version 4.2.0 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 6a2e05040a65 --global-core-map 0,1,1 --pmi-id-map 0,0 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_1_0_1589598326_6a2e05040a65 --pmi-process-mapping (vector,(0,1,2)) --global-inherited-env 11 'PATH=//test-env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' 'HOSTNAME=6a2e05040a65' 'TERM=xterm' 'OPENBLAS_NUM_THREADS=1' 'OPENBLAS_VERBOSE=0' 'DEB_PYTHON_INSTALL_LAYOUT=deb_system' 'DEBIAN_FRONTEND=noninteractive' 'VIRTUAL_ENV=/test-env' 'OMPI_ALLOW_RUN_AS_ROOT=1' 'OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1' 'HOME=/root' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir /tmp --exec-args 3 python3 -c from mpi4py import MPI; print(f'{MPI.COMM_WORLD.rank}/{MPI.COMM_WORLD.size}'); [mpiexec@6a2e05040a65] Launch arguments: /usr/bin/hydra_pmi_proxy --control-port 6a2e05040a65:36819 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --gpu-subdevs-per-proc -2 --proxy-id 0 [proxy:0@6a2e05040a65] Sending upstream hdr.cmd = CMD_PID_LIST [proxy:0@6a2e05040a65] Sending upstream hdr.cmd = CMD_STDOUT [proxy:0@6a2e05040a65] Sending upstream hdr.cmd = CMD_STDOUT 0/1 0/1 [proxy:0@6a2e05040a65] Sending upstream hdr.cmd = CMD_EXIT_STATUS ``` Best, Jørgen -- *Jørgen S. Dokken, *PhD Senior Research Engineer Simula Research Laboratory +47 45286467

1 0

Re: [mpich-discuss] How to read --pmi-process-mapping
by Zhou, Hui 07 Aug '24

07 Aug '24

Hi Niyaz, The triplet meanings are (start node id, number of nodes, processes per node), and processes are assigned round-robin fashion. So "(0, 2, 1)" with 2 processes on 2 nodes expands into "0, 1" - first process on node 0 and 2nd process on node 1. Hope that helps. -- Hui ________________________________ From: Niyaz Murshed via discuss <discuss(a)mpich.org> Sent: Wednesday, August 7, 2024 12:18 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>; nd <nd(a)arm.com> Subject: [mpich-discuss] How to read --pmi-process-mapping Hello, I am launching 2 processes on 2 nodes. The processes get rank 0 and 1 --version 4. 3. 0a1 --iface-ip-env-name PMI_HOSTNAME --hostname 192. 168. 2. 100 --global-core-map 0,1,2 --pmi-id-map 0,0 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, I am launching 2 processes on 2 nodes. The processes get rank 0 and 1 --version 4.3.0a1 --iface-ip-env-name PMI_HOSTNAME --hostname 192.168.2.100 --global-core-map 0,1,2 --pmi-id-map 0,0 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_31321_0_150440369_ampere-altra-2-1 --pmi-process-mapping (vector,(0,2,1)) I see once rankmap is generated, its converted to a string “(vector,(0,2,1))”. How do we read this as? Regards, Niyaz

1 0

How to read --pmi-process-mapping
by Niyaz Murshed 07 Aug '24

07 Aug '24

Hello, I am launching 2 processes on 2 nodes. The processes get rank 0 and 1 --version 4.3.0a1 --iface-ip-env-name PMI_HOSTNAME --hostname 192.168.2.100 --global-core-map 0,1,2 --pmi-id-map 0,0 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_31321_0_150440369_ampere-altra-2-1 --pmi-process-mapping (vector,(0,2,1)) I see once rankmap is generated, its converted to a string “(vector,(0,2,1))”. How do we read this as? Regards, Niyaz

1 0