February 2019 - discuss - mailman3.cels.anl.gov

mpiexec Hydra and large numbers of command line options
by Doug Johnson 25 Feb '19

25 Feb '19

Hi, We have encountered a bug with mpiexec when the length of argv is greater than or equal to 1000. This was uncovered in the use of a utility called pbsdcp that uses MPI_Bcast to efficiently copy files to internal disks for parallel jobs. If someone uses a wildcard to pass lists of files for copying on the command line it can result in a large number of arguments being passed to the execution of mpiexec. This results in a seg fault. The problem can be trivially reproduced with the nonsensical command line shown below. owens-rw01:~> mpiexec `seq 999` [proxy:0:[email protected]] HYDU_create_process (utils/launch/launch.c:75): execvp error on file 1 (No such file or directory) owens-rw01:~> mpiexec `seq 1000` Segmentation fault owens-rw01:~> mpiexec --version HYDRA build details: Version: 3.2.1 Release Date: General Availability Release CC: icc CXX: icpc F77: ifort F90: ifort Configure options: '--disable-option-checking' '--prefix=/opt/mvapich2/intel/18.0/2.3' '--enable-shared' '--with-mpe' '--enable-romio' '--enable-mpit-pvars=mv2' '--with-file-system=ufs+nfs+gpfs' '--with-pbs=/opt/torque' '--with-pbs-lib=/opt/torque/lib64' '--with-pbs-include=/opt/torque/include' 'CC=icc' 'CXX=icpc' 'FC=ifort' 'F77=ifort' '--cache-file=/dev/null' '--srcdir=.' 'CFLAGS= -DNDEBUG -DNVALGRIND -O2' 'LDFLAGS=-L/lib -L/lib -L/lib -Wl,-rpath,/lib -L/lib -Wl,-rpath,/lib -L/lib -L/lib' 'LIBS=-libmad -lrdmacm -libumad -libverbs -lrt -lpthread ' 'CPPFLAGS= -I/dev/shm/tmp.PxAZLCzt4P/src/mpl/include -I/dev/shm/tmp.PxAZLCzt4P/src/mpl/include -I/dev/shm/tmp.PxAZLCzt4P/src/openpa/src -I/dev/shm/tmp.PxAZLCzt4P/src/openpa/src -D_REENTRANT -I/dev/shm/tmp.PxAZLCzt4P/src/mpi/romio/include -I/include -I/include -I/include -I/include' 'MPLLIBNAME=mpl' Process Manager: pmi Launchers available: ssh rsh fork slurm ll lsf sge pbs manual persist Topology libraries available: hwloc Resource management kernels available: user slurm ll lsf sge pbs cobalt Checkpointing libraries available: Demux engines available: poll select The root cause seems to be in src/pm/hydra/include/hydra.h, #define HYD_NUM_TMP_STRINGS 1000 This define value is used for static allocation when processing argv in mpiexec. I suppose we could increase this value as a short term workaround that may fix some use cases (could use ARG_MAX, 'getconf ARG_MAX'.) However, a cleaner fix is probably to pass argc along with argv so that dynamic allocations can be used when processing arguments. Thanks, Doug

1 0

Re: [mpich-discuss] Selecting a specific libfabric fabric/domain
by eric＠bit32.net 21 Feb '19

21 Feb '19

Thanks Wesley and Hajime. I tested setting FI_SOCKETS_IFACE=eth1, and it looks to work as one would hope, causing libfabric to open up the chosen interface. Eric On 2/19/19 10:34 AM, Fujita, Hajime via discuss wrote: > As Wesley mentioned there isn't a way to do it in MPICH, but you may try setting FI_SOCKETS_IFACE to control it at the libfabric level. > https://ofiwg.github.io/libfabric/v1.7.0/man/fi_sockets.7.html > > I haven't tried this personally, so I'm not sure how it actually works though. > > > Thanks, > Hajime > > On 2/19/19, 10:30 AM, "Wesley Bland via discuss" <discuss(a)mpich.org> wrote: > > Hi Eric, > > There isn’t a way of doing that at the moment with MPICH (I’m not sure if any particular OFI providers have support for this through environment variables or something else). It will open whichever one OFI gives you first (assuming all other capability bits are the same). > > Thanks, > Wesley > > > On Feb 18, 2019, at 1:40 PM, eric--- via discuss <discuss(a)mpich.org> wrote: > > > > Hi there, > > > > I'm using MPICH with ch4:ofi, and I'm wondering how to select which fabric/domain MPICH will open. I can select the provider with MPIR_CVAR_OFI_USE_PROVIDER, but suppose that the provider has two disjoint fabrics with equivalent capabilities. E.g., suppose I have two ethernet NICs connected to private networks: > > > > provider: sockets > > fabric: 192.168.1.0/24 > > domain: eth0 > > version: 2.0 > > type: FI_EP_RDM > > protocol: FI_PROTO_SOCK_TCP > > provider: sockets > > fabric: 10.0.1.0/24 > > domain: eth1 > > version: 2.0 > > type: FI_EP_RDM > > protocol: FI_PROTO_SOCK_TCP > > > > > > Is there a tunable to pick which will be used? > > > > Thanks, > > Eric > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] mpi_comm_connect hangs in client/server mode runs in multiple process
by Shuwei Zhao 20 Feb '19

20 Feb '19

Hi, Joachim, Thanks a lot for your response. Is that possible that the rank 0 in server can talk to each process in the client? If that's possible, I just need to replace MPI_COMM_WORLD by MPI_COMM_SELF on the server side, right? Thanks, Shuwei On Wed, Feb 20, 2019 at 5:11 PM Protze, Joachim <protze(a)itc.rwth-aachen.de> wrote: > Shuwei, > > Both MPI calls are collective calls. This means all ranks in the provided > communicator must call this function. > If you want to connect both applications so that any rank in server can > communicate with any rank in client, all processes need to call the > function for each side. > If only rank 0 should ever communicate, you can replace MPI_COMM_WORLD by > MPI_COMM_SELF on both sides. The resulting newcomm will be a > intercommunicator with one rank on each side. > > Best > Joachim > > -- > Sent from my mobile phone > > > > > On Wed, Feb 20, 2019 at 10:54 PM +0100, "Shuwei Zhao via discuss" < > discuss(a)mpich.org> wrote: > > Hi, >> >> I was trying to study the usage of mpi client-server model based on the >> example you provided on >> https://www.mcs.anl.gov/research/projects/mpi/mpi-standard/mpi-report-2.0/n…. >> After writing simple separate client and server source file based on the >> samples on your website. I found that: >> 1) when client and server both runs in single process like mpiexec -n 1 >> server, mpiexec -n 1 client. >> it works fine, connection between server and client can successfully be >> built. >> 2) when client runs in single process but server runs in multiple >> processes (2 for example) but only rank 0 in server responsible to call >> mpi_comm_accept, rank 1 of server is just sleeping. Usage is like mpiexec >> -n 1 client, mpiexec -n 2 server. In this case, server blocks in >> mpi_comm_accept, which is expected. However, client blocks in >> mpi_comm_connect which is not expected. >> 3) when client runs in multiple processes and server runs in multiple >> processes, only rank 0 of client responsible for mpi_comm_connect and only >> rank 0 of server responsible for mpi_comm_accept. In this case, same >> observation in 2). >> >> May I ask is the hanging of mpi_comm_connect in 2) & 3) expected? Is >> there a way to build connection successfully when mpi client/server runs >> under multi-process? >> >> Thanks a lot, >> Shuwei >> >

1 0

Re: [mpich-discuss] mpi_comm_connect hangs in client/server mode runs in multiple process
by Protze, Joachim 20 Feb '19

20 Feb '19

Shuwei, Both MPI calls are collective calls. This means all ranks in the provided communicator must call this function. If you want to connect both applications so that any rank in server can communicate with any rank in client, all processes need to call the function for each side. If only rank 0 should ever communicate, you can replace MPI_COMM_WORLD by MPI_COMM_SELF on both sides. The resulting newcomm will be a intercommunicator with one rank on each side. Best Joachim -- Sent from my mobile phone On Wed, Feb 20, 2019 at 10:54 PM +0100, "Shuwei Zhao via discuss" <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi, I was trying to study the usage of mpi client-server model based on the example you provided on https://www.mcs.anl.gov/research/projects/mpi/mpi-standard/mpi-report-2.0/n…. After writing simple separate client and server source file based on the samples on your website. I found that: 1) when client and server both runs in single process like mpiexec -n 1 server, mpiexec -n 1 client. it works fine, connection between server and client can successfully be built. 2) when client runs in single process but server runs in multiple processes (2 for example) but only rank 0 in server responsible to call mpi_comm_accept, rank 1 of server is just sleeping. Usage is like mpiexec -n 1 client, mpiexec -n 2 server. In this case, server blocks in mpi_comm_accept, which is expected. However, client blocks in mpi_comm_connect which is not expected. 3) when client runs in multiple processes and server runs in multiple processes, only rank 0 of client responsible for mpi_comm_connect and only rank 0 of server responsible for mpi_comm_accept. In this case, same observation in 2). May I ask is the hanging of mpi_comm_connect in 2) & 3) expected? Is there a way to build connection successfully when mpi client/server runs under multi-process? Thanks a lot, Shuwei

1 0

mpi_comm_connect hangs in client/server mode runs in multiple process
by Shuwei Zhao 20 Feb '19

20 Feb '19

Hi, I was trying to study the usage of mpi client-server model based on the example you provided on https://www.mcs.anl.gov/research/projects/mpi/mpi-standard/mpi-report-2.0/n…. After writing simple separate client and server source file based on the samples on your website. I found that: 1) when client and server both runs in single process like mpiexec -n 1 server, mpiexec -n 1 client. it works fine, connection between server and client can successfully be built. 2) when client runs in single process but server runs in multiple processes (2 for example) but only rank 0 in server responsible to call mpi_comm_accept, rank 1 of server is just sleeping. Usage is like mpiexec -n 1 client, mpiexec -n 2 server. In this case, server blocks in mpi_comm_accept, which is expected. However, client blocks in mpi_comm_connect which is not expected. 3) when client runs in multiple processes and server runs in multiple processes, only rank 0 of client responsible for mpi_comm_connect and only rank 0 of server responsible for mpi_comm_accept. In this case, same observation in 2). May I ask is the hanging of mpi_comm_connect in 2) & 3) expected? Is there a way to build connection successfully when mpi client/server runs under multi-process? Thanks a lot, Shuwei

1 0

[Request for Cooperation] -- MPI International Survey
by George Bosilca 20 Feb '19

20 Feb '19

Dear colleagues, As part of a wide-ranging effort to understand the current usage of the Message Passing Interface (MPI) in the development of parallel applications and to drive future additions to the MPI standard, an international team is seeking feedback from the largest possible MPI audience (past, current, and potential users on the globe) to obtain a better understanding of their needs and to understand the impact of different MPI capabilities on the development of distributed applications. To obtain representative samples of the MPI community, we have prepared a survey https://docs.google.com/forms/d/e/1FAIpQLSd1bDppVODc8nB0BjIXdqSCO_MuEuNAAbB… that specifically targets all potential MPI users, including those in the public, education, research and engineering domains--from undergraduate and graduate students and postdocs to seasoned researchers and engineers. The information gathered will be used to publish a comprehensive report of the different use cases and potential areas of opportunity. These results will be made freely available, and the raw data, the scripts to manipulate it, as well as the resulting analysis will be, in time, posted on github [1], while the curated results will be available via github pages [2]. For anyone interested in participating in the survey, we sincerely appreciate your feedback. The survey is rather short (about 30 easy questions), and should not take more than 15 minutes to complete. In addition to your participation, we would appreciate if you re-distribute this e-mail to your domestic/local communities. Important Date -- This survey will be closed by the end of February 2019. Questions? -- Please send any queries about this MPI survey to one of the authors of the survey. Thank you on behalf of the International MPI Survey, George Bosilca (UT/ICL) Geoffroy Vallee (ORNL) Emmanuel Jeannot (Inria) Atsushi Hori (RIKEN) Takahiro Ogura (RIKEN) [1] https://github.com/bosilca/MPIsurvey/ [2] https://bosilca.github.io/MPIsurvey/

1 0

Re: [mpich-discuss] Selecting a specific libfabric fabric/domain
by Fujita, Hajime 19 Feb '19

19 Feb '19

As Wesley mentioned there isn't a way to do it in MPICH, but you may try setting FI_SOCKETS_IFACE to control it at the libfabric level. https://ofiwg.github.io/libfabric/v1.7.0/man/fi_sockets.7.html I haven't tried this personally, so I'm not sure how it actually works though. Thanks, Hajime On 2/19/19, 10:30 AM, "Wesley Bland via discuss" <discuss(a)mpich.org> wrote: Hi Eric, There isn’t a way of doing that at the moment with MPICH (I’m not sure if any particular OFI providers have support for this through environment variables or something else). It will open whichever one OFI gives you first (assuming all other capability bits are the same). Thanks, Wesley > On Feb 18, 2019, at 1:40 PM, eric--- via discuss <discuss(a)mpich.org> wrote: > > Hi there, > > I'm using MPICH with ch4:ofi, and I'm wondering how to select which fabric/domain MPICH will open. I can select the provider with MPIR_CVAR_OFI_USE_PROVIDER, but suppose that the provider has two disjoint fabrics with equivalent capabilities. E.g., suppose I have two ethernet NICs connected to private networks: > > provider: sockets > fabric: 192.168.1.0/24 > domain: eth0 > version: 2.0 > type: FI_EP_RDM > protocol: FI_PROTO_SOCK_TCP > provider: sockets > fabric: 10.0.1.0/24 > domain: eth1 > version: 2.0 > type: FI_EP_RDM > protocol: FI_PROTO_SOCK_TCP > > > Is there a tunable to pick which will be used? > > Thanks, > Eric > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Selecting a specific libfabric fabric/domain
by Wesley Bland 19 Feb '19

19 Feb '19

Hi Eric, There isn’t a way of doing that at the moment with MPICH (I’m not sure if any particular OFI providers have support for this through environment variables or something else). It will open whichever one OFI gives you first (assuming all other capability bits are the same). Thanks, Wesley > On Feb 18, 2019, at 1:40 PM, eric--- via discuss <discuss(a)mpich.org> wrote: > > Hi there, > > I'm using MPICH with ch4:ofi, and I'm wondering how to select which fabric/domain MPICH will open. I can select the provider with MPIR_CVAR_OFI_USE_PROVIDER, but suppose that the provider has two disjoint fabrics with equivalent capabilities. E.g., suppose I have two ethernet NICs connected to private networks: > > provider: sockets > fabric: 192.168.1.0/24 > domain: eth0 > version: 2.0 > type: FI_EP_RDM > protocol: FI_PROTO_SOCK_TCP > provider: sockets > fabric: 10.0.1.0/24 > domain: eth1 > version: 2.0 > type: FI_EP_RDM > protocol: FI_PROTO_SOCK_TCP > > > Is there a tunable to pick which will be used? > > Thanks, > Eric > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Selecting a specific libfabric fabric/domain
by eric＠bit32.net 18 Feb '19

18 Feb '19

Hi there, I'm using MPICH with ch4:ofi, and I'm wondering how to select which fabric/domain MPICH will open. I can select the provider with MPIR_CVAR_OFI_USE_PROVIDER, but suppose that the provider has two disjoint fabrics with equivalent capabilities. E.g., suppose I have two ethernet NICs connected to private networks: provider: sockets fabric: 192.168.1.0/24 domain: eth0 version: 2.0 type: FI_EP_RDM protocol: FI_PROTO_SOCK_TCP provider: sockets fabric: 10.0.1.0/24 domain: eth1 version: 2.0 type: FI_EP_RDM protocol: FI_PROTO_SOCK_TCP Is there a tunable to pick which will be used? Thanks, Eric

1 0

Re: [mpich-discuss] A few inconveniences with MPICH 3.3 build
by Balaji, Pavan 13 Feb '19

13 Feb '19

Martin, For the slurm question, no you don't need to configure with slurm to get the nodelist. You only need it if you want to use srun directly. The recommended option is to use mpiexec directly (like you normally would). mpiexec will internally lose srun anyway, and it'll automatically detect the nodes that you allocated in slurm (e.g., through salloc). Adding slurm to configure allows you to use slurm's PMI, which is just internal MPICH-speak. You probably won't need to care about that. -- Pavan > On Feb 12, 2019, at 11:15 PM, Martin Cuma via discuss <discuss(a)mpich.org> wrote: > > Thanks Brice for the tip, after some digging I found the appropriate spot to put these two options (enable_gl=no, enable_opencl=no) to src/pm/hydra/configure. > > MPICH developers - any chance this can be wrapped around the --without-x configure option, or hardcoded? > > Any takers on the SLURM question? > > Thanks, > MC > > -- > Martin Cuma > Center for High Performance Computing > Department of Geology and Geophysics > University of Utah > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0