- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPICH taking MUCH longer to run same job compared to srun
by Anne Hammond 30 Mar '21

30 Mar '21

I built mpich 3.3.2 with ./configure -with-ibverbs=/usr --with-ibverbs-include=/usr/include --with-ibverbs-lib=/usr/lib64 --with-pmi=simple --prefix=/home/admin/hammo nd/mpich Running through the queue with a command line /home/admin/hammond/mpich/bin/mpiexec -n 8 -machinefile ./machinefile myapp -i electron.in I see: srwxr-xr-x 1 root root 0 Mar 30 17:33 sock.pmi2.5442.0 So I guess it is using pmi2. But the slowness compared to srun is the same.

1 0

MPICH taking MUCH longer to run same job compared to srun
by Anne Hammond 30 Mar '21

30 Mar '21

We are running a small cluster using slurm 20.11.2. Our application is built with mpich 3.3.2. If we submit a 2 batch jobs, each on it’s own single node, one using srun and the other using mpiexec to run the application, the mpiexec takes much longer. This is an 8 processor job on a 32 core single node. Comparison times, for the same input file and the same number of processes (8), are srun 0.006366 s mpiexec 22.21 s Something is clearly wrong. From the FAQ for mpich, it says the default build of MPICH will work fine: https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Note_that…. SLURM is an external process manager that uses MPICH's PMI interface as well. Note that the default build of MPICH will work fine in SLURM environments. No extra steps are needed. —————— If I look at the process tree (ps axjf) when the mpiexec job is running: 1 1427683 1427682 1427682 ? -1 Sl 0 0:00 slurmstepd: [5435.batch] 1427683 1427687 1427687 1427682 ? -1 S 0 0:00 \_ /bin/bash /var/spool/slurmd/job05435/slurm_script 1427687 1427702 1427687 1427682 ? -1 S 0 0:00 \_ mpiexec -n 8 -machinefile ./machinefile myapp -i electron.in 1427702 1427703 1427703 1427703 ? -1 Ssl 0 0:00 \_ /bin/srun --nodelist ne04 -N 1 -n 1 --input none /home/common/myapp/myapp-11.0-2021- 1427703 1427704 1427703 1427703 ? -1 S 0 0:00 \_ /bin/srun --nodelist ne04 -N 1 -n 1 --input none /home/common/myapp/myapp-11.0-2 1 1427712 1427711 1427711 ? -1 Sl 0 0:00 slurmstepd: [5435.0] 1427712 1427718 1427718 1427711 ? -1 S 0 0:00 \_ /home/common/myapp/myapp-11.0-2021-03-30-05.02/Contents/engine/bin/hydra_pmi_proxy --control 1427718 1427719 1427719 1427719 ? -1 Rsl 0 0:10 \_ myapp -i electron.in 1427718 1427720 1427720 1427720 ? -1 Rsl 0 0:11 \_ myapp -i electron.in 1427718 1427721 1427721 1427721 ? -1 Rsl 0 0:06 \_ myapp -i electron.in 1427718 1427722 1427722 1427722 ? -1 Rsl 0 0:06 \_ myapp -i electron.in 1427718 1427723 1427723 1427723 ? -1 Rsl 0 0:06 \_ myapp -i electron.in 1427718 1427724 1427724 1427724 ? -1 Rsl 0 0:10 \_ myapp -i electron.in 1427718 1427725 1427725 1427725 ? -1 Rsl 0 0:08 \_ myapp -i electron.in 1427718 1427726 1427726 1427726 ? -1 Rsl 0 0:11 \_ myapp -i electron.in I see that mpiexec is calling /bin/srun with -N 1 -n 1. I don’t know if this matters, because there are 8 myapp processes running under slurmstepd. The problem is that mpiexec is taking so much more time. mpich 3.3.2 says Process Manager: pmi Is pmi2 different from pmi? The issue could also be with slurm. In /etc/slurm/slurm.conf, we define MpiDefault to be pmi2: MpiDefault=pmi2 If anyone can point me where to look to solve this problem, I would be very thankful. Thanks in advance, Anne Hammond

1 0

ECP Community BOF Days: MPICH for Exascale: Supporting MPI-4 and ECP
by Raffenetti, Kenneth J. 24 Mar '21

24 Mar '21

Hi everyone, The MPICH team will host a Birds of a Feather session during next week's ECP Community BoF Days. This virtual event will run similar to our annual MPICH BoF at Supercomputing. Come join us to learn about developments in MPICH and its derivatives. Registration is free! Wednesday, March 31st 1:00pm Eastern Registration link: https://exascaleproject.zoomgov.com/meeting/register/vJItduqrpjMsGGTTf2_YAV… More info: https://github.com/pmodels/mpich/wiki/2021-ECP-Community-BoF Best regards, The MPICH Team

1 0

Re: [mpich-discuss] MPI_Info key host error in SLURM
by Raffenetti, Kenneth J. 24 Mar '21

24 Mar '21

On 3/24/21, 10:44 AM, "Iker Martín Álvarez" <martini(a)uji.es> wrote: Sorry for the late reply, I tried using that option, but the outcome was the same. However, with the info key "hostfile" and creating a file with the the following text, it works as expected. In this example, creating 10 and 5 processes respectively in n00 and n01. n00:10 n01:5 Interesting. I'm glad you could get it working. There must be some bug with the other method. When trying this key, I noticed that if the parent processes try to finish their execution while their sons do some work, the parents are blocked in MPI_Finalize until the sons call this function too. This happens even if for both groups all their processes call MPI_Comm_disconnect. I think this is happening because both groups are still connected and therefore the parents wait until the sons call this function. Could it be that I am missing something? I based my conclusión on what is told here: https://www.mpi-forum.org/docs/mpi-2.2/mpi22-report/node226.htm If both groups call MPI_Comm_disconnect, then theoretically they shouldn't block in MPI_Finalize like you describe. However our implementation may not fully complete the disconnect until finalize is called. We recently began some work to overhaul this part of MPICH. Hopefully a future release will contain a more robust disconnect mechanism. Ken

1 0

Re: [mpich-discuss] MPI_Info key host error in SLURM
by Iker Martín Álvarez 24 Mar '21

24 Mar '21

Sorry for the late reply, I tried using that option, but the outcome was the same. However, with the info key "hostfile" and creating a file with the the following text, it works as expected. In this example, creating 10 and 5 processes respectively in n00 and n01. n00:10 n01:5 When trying this key, I noticed that if the parent processes try to finish their execution while their sons do some work, the parents are blocked in MPI_Finalize until the sons call this function too. This happens even if for both groups all their processes call MPI_Comm_disconnect. I think this is happening because both groups are still connected and therefore the parents wait until the sons call this function. Could it be that I am missing something? I based my conclusión on what is told here: https://www.mpi-forum.org/docs/mpi-2.2/mpi22-report/node226.htm Thanks, Iker El jue, 4 mar 2021 a las 18:31, Raffenetti, Kenneth J. (< raffenet(a)mcs.anl.gov>) escribió: > I notice in the output that the sbatch version uses --launcher slurm while > the interactive version uses --launcher ssh. Can you try adding --launcher > ssh to the mpiexec command of you sbatch script and see if it makes a > difference? > > Ken > > On 3/1/21, 1:38 PM, "Iker Martín Álvarez" <martini(a)uji.es> wrote: > > Hi Ken, > Thanks for your reply. > Here are attached two files with the arg "-v". BatchOutput.txt has the > output which is giving the error message when the code is executed with the > command sbatch, while the file InteractiveOutput.txt is the one executed > interactively in the node and work as expected. Both of them has been > compiled and executed with mpich 3.4.1. > > Thanks, Iker > > > El vie, 26 feb 2021 a las 16:15, Raffenetti, Kenneth J. (< > raffenet(a)mcs.anl.gov>) escribió: > > > Hi, > > Could you add "-v" to your mpiexec command and provide the output? The > "host" info key is handled by the process manager when executing the spawn > command. > > Ken > > On 2/24/21, 11:55 AM, "Iker Martín Álvarez via discuss" < > discuss(a)mpich.org> wrote: > > Hi, > I have been working around the MPI_Comm_spawn function with the > key "host" along a value in a system with the SLURM resource manager > (slurm-wlm 17.11.2). The function works as expected, but when I send to > execute the code with the sbatch command, an error arises. This does not > happen when I execute directly the code in the same machine SLURM decided > to execute it when it was sended with sbatch. In both cases with the key > "host", as when I do not use the key, it works just fine. > > The same code has been tested with MPICH 3.3.2 and 3.4.1, which > gives different errors. Also, I tried it with other implementations > (OpenMPI and Intel MPI), which works as expected creating the processes in > the indicated host. > > I would like to create processes by MPI_Comm_spawn in an assigned > host, so if there are other key values for the Info argument, I could try > them, but I have not found any looking through MPICH documentation. > > Here is the code I have been using: > > int main(int argc, char ** argv) { > > int myId, numP; > MPI_Info info; > MPI_Comm comm; > > MPI_Init(&argc, &argv); > MPI_Comm_rank(MPI_COMM_WORLD, &myId); > MPI_Comm_size(MPI_COMM_WORLD, &numP); > > MPI_Comm comm_par; > MPI_Comm_get_parent(&comm_par); > if(comm_par != MPI_COMM_NULL ) { > if(myId == ROOT) { > printf("SONS\n"); fflush(stdout); > } > } else { > if(myId == ROOT) { > printf("FATHERS\n"); fflush(stdout); > } > MPI_Info_create(&info); > MPI_Info_set(info, "host", "n00"); > MPI_Comm_spawn(argv[0], MPI_ARGV_NULL, numP, info, ROOT, > MPI_COMM_WORLD, &comm, MPI_ERRCODES_IGNORE); > } > MPI_Finalize(); > return 0; > } > > > For MPICH 3.4.1 there is sometimes no error and the code stops > working at MPI_Comm_spawn function, other times this error is shown: > Abort(1615120) on node 0 (rank 0 in comm 0): Fatal error in > PMPI_Init: Internal MPI error!, error stack: > MPIR_Init_thread(152)...: > MPID_Init(562)..........: > MPIDU_Init_shm_init(195): > Init_shm_barrier(94)....: Internal MPI error! barrier not > initialized > > > Also, the error code for MPICH 3.3.2: > > Assertion failed in file > src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank > < pg->size > Assertion failed in file > src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank > < pg->size > Assertion failed in file > src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank > < pg->size > libbacktrace: no debug info in ELF executable > libbacktrace: no debug info in ELF executable > libbacktrace: no debug info in ELF executable > libbacktrace: no debug info in ELF executable > libbacktrace: no debug info in ELF executable > libbacktrace: no debug info in ELF executable > libbacktrace: no debug info in ELF executable > libbacktrace: no debug info in ELF executable > libbacktrace: no debug info in ELF executable > libbacktrace: no debug info in ELF executable > libbacktrace: no debug info in ELF executable > libbacktrace: no debug info in ELF executable > internal ABORT - process 0 > > Thanks, Iker > > > >

1 0

Re: [mpich-discuss] A question about mutex timing
by Zhou, Hui 17 Mar '21

17 Mar '21

Unfortunately, `--enable-mutex-timing` is a leftover option that now does nothing. One option is to enable logging – following this guide https://wiki.mpich.org/mpich/index.php/Debug_Event_Logging . If you set `MPICH_DBG_CLASS=THREAD`, you should see in the log file entries like “enter/exit MPIDU_Thread_mutex_lock xxx”, with timestamps. If that is insufficient for you, you may have manually insert your own timing harness. The file you should modify is `src/mpid/common/thread/mpidu_thread_fallback.h`. Hope that helps. -- Hui Zhou From: Yongyu Chen via discuss <discuss(a)mpich.org> Date: Wednesday, March 17, 2021 at 2:39 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Yongyu Chen <yongyu.chen(a)mail.ru> Subject: [mpich-discuss] A question about mutex timing Dear sir or madam, I enabled a feature on configuring the mpi library, which is --enable-mutex-timing (calculate the time spent waiting on mutexes). But I don’t know how to use it. Could you provide me with some ideas or solutions? Thank you for reading, I appreciate your replies! Sincerely, Yongyu Chen Sincerely, Yongyu Chen

1 0

A question about mutex timing
by Yongyu Chen 17 Mar '21

17 Mar '21

Dear sir or madam, I enabled a feature on configuring the mpi library, which is --enable-mutex-timing (calculate the time spent waiting on mutexes). But I don’t know how to use it. Could you provide me with some ideas or solutions? Thank you for reading, I appreciate your replies! Sincerely, Yongyu Chen Sincerely, Yongyu Chen

1 0

Re: [mpich-discuss] How to set port range used under LSF? [EXT]
by Zhou, Hui 10 Mar '21

10 Mar '21

Hi Sendu, Sorry for not understanding the issue earlier. `blaunch` handles the connection itself so hydra does not need do that. Therefore, port range option does not apply (ignored). It doesn’t look like `blaunch` has a similar port range option. -- Hui Zhou From: Sendu Bala <sb10(a)sanger.ac.uk> Date: Tuesday, March 9, 2021 at 5:36 PM To: Zhou, Hui <zhouh(a)anl.gov> Cc: discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: How to set port range used under LSF? [EXT] Hi, As noted, I had already tried MPIEXEC_PORT_RANGE with no difference. I don’t know if it’s normal for blaunch to just have its own ports and this is nothing to do with mpich or mpiexec at all, but the follow on issue is if I ssh to one of the other nodes, nothing is listening on the control port. I only see a process listening on one of these other ports outside my specified range. Which I think might be related to my fundamental problem of mpiexec failing to do anything under LSF most of the time at high host counts (see my previous thread on this list). On 9 Mar 2021, at 16:21, Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> wrote: Hi Sendu, Could you try `MPIEXEC_PORT_RANGE` in stead? I know it is confusing and the documentation probably need update/correction, but `MPICH_PORT_RANGE` is for `MPICH` rather than `MPIEXEC`. -- Hui Zhou From: Sendu Bala via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Tuesday, March 9, 2021 at 7:16 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Sendu Bala <sb10(a)sanger.ac.uk<mailto:[email protected]>> Subject: [mpich-discuss] How to set port range used under LSF? Via a bsub, I’m doing: MPICH_PORT_RANGE="46107:46140” mpiexec mpich/examples/cpi When I ssh to the controlling node, I see it has spawned a set of blaunch processes with `--control-port node-12-3-2:46107` as expected, but: ss -l -p -n | grep blaunch tcp LISTEN 0 128 0.0.0.0:34361 0.0.0.0:* users:(("blaunch",pid=2823,fd=7)) tcp LISTEN 0 128 0.0.0.0:46107 0.0.0.0:* users:(("cpi",pid=2839,fd=5),("blaunch",pid=2837,fd=5),("blaunch",pid=2836,fd=5),("blaunch",pid=2835,fd=5),("blaunch",pid=2834,fd=5),("blaunch",pid=2833,fd=5),("blaunch",pid=2832,fd=5),("blaunch",pid=2831,fd=5),("blaunch",pid=2830,fd=5),("blaunch",pid=2829,fd=5),("blaunch",pid=2828,fd=5),("blaunch",pid=2827,fd=5),("blaunch",pid=2826,fd=5),("blaunch",pid=2825,fd=5),("blaunch",pid=2824,fd=5),("blaunch",pid=2823,fd=5),("hydra_pmi_proxy",pid=2822,fd=5),("mpiexec",pid=2821,fd=5)) tcp LISTEN 0 128 0.0.0.0:43741 0.0.0.0:* users:(("blaunch",pid=2825,fd=12)) tcp LISTEN 0 128 0.0.0.0:41983 0.0.0.0:* users:(("blaunch",pid=2830,fd=22)) tcp LISTEN 0 128 0.0.0.0:41215 0.0.0.0:* users:(("blaunch",pid=2832,fd=26)) tcp LISTEN 0 128 0.0.0.0:34433 0.0.0.0:* users:(("blaunch",pid=2831,fd=24)) tcp LISTEN 0 128 0.0.0.0:33219 0.0.0.0:* users:(("blaunch",pid=2827,fd=16)) tcp LISTEN 0 128 0.0.0.0:34405 0.0.0.0:* users:(("blaunch",pid=2837,fd=36)) tcp LISTEN 0 128 0.0.0.0:43465 0.0.0.0:* users:(("blaunch",pid=2836,fd=34)) tcp LISTEN 0 128 0.0.0.0:39755 0.0.0.0:* users:(("blaunch",pid=2833,fd=28)) tcp LISTEN 0 128 0.0.0.0:38095 0.0.0.0:* users:(("blaunch",pid=2829,fd=20)) tcp LISTEN 0 128 0.0.0.0:44625 0.0.0.0:* users:(("blaunch",pid=2834,fd=30)) tcp LISTEN 0 128 0.0.0.0:35345 0.0.0.0:* users:(("blaunch",pid=2835,fd=32)) tcp LISTEN 0 128 0.0.0.0:43827 0.0.0.0:* users:(("blaunch",pid=2826,fd=14)) tcp LISTEN 0 128 0.0.0.0:40915 0.0.0.0:* users:(("blaunch",pid=2828,fd=18)) tcp LISTEN 0 128 0.0.0.0:42549 0.0.0.0:* users:(("blaunch",pid=2824,fd=9)) Why are these all listening on ports outside my range? I’ve also tried setting MPIEXEC_PORT_RANGE and MPIR_CVAR_CH3_PORT_RANGE and still have the problem. Is there any way to fully control the ports used? Cheers, Sendu. -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss [lists.mpich.org]<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

1 0

Re: [mpich-discuss] How to set port range used under LSF? [EXT]
by Sendu Bala 09 Mar '21

09 Mar '21

Hi, As noted, I had already tried MPIEXEC_PORT_RANGE with no difference. I don’t know if it’s normal for blaunch to just have its own ports and this is nothing to do with mpich or mpiexec at all, but the follow on issue is if I ssh to one of the other nodes, nothing is listening on the control port. I only see a process listening on one of these other ports outside my specified range. Which I think might be related to my fundamental problem of mpiexec failing to do anything under LSF most of the time at high host counts (see my previous thread on this list). On 9 Mar 2021, at 16:21, Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> wrote: Hi Sendu, Could you try `MPIEXEC_PORT_RANGE` in stead? I know it is confusing and the documentation probably need update/correction, but `MPICH_PORT_RANGE` is for `MPICH` rather than `MPIEXEC`. -- Hui Zhou From: Sendu Bala via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Tuesday, March 9, 2021 at 7:16 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Sendu Bala <sb10(a)sanger.ac.uk<mailto:[email protected]>> Subject: [mpich-discuss] How to set port range used under LSF? Via a bsub, I’m doing: MPICH_PORT_RANGE="46107:46140” mpiexec mpich/examples/cpi When I ssh to the controlling node, I see it has spawned a set of blaunch processes with `--control-port node-12-3-2:46107` as expected, but: ss -l -p -n | grep blaunch tcp LISTEN 0 128 0.0.0.0:34361 0.0.0.0:* users:(("blaunch",pid=2823,fd=7)) tcp LISTEN 0 128 0.0.0.0:46107 0.0.0.0:* users:(("cpi",pid=2839,fd=5),("blaunch",pid=2837,fd=5),("blaunch",pid=2836,fd=5),("blaunch",pid=2835,fd=5),("blaunch",pid=2834,fd=5),("blaunch",pid=2833,fd=5),("blaunch",pid=2832,fd=5),("blaunch",pid=2831,fd=5),("blaunch",pid=2830,fd=5),("blaunch",pid=2829,fd=5),("blaunch",pid=2828,fd=5),("blaunch",pid=2827,fd=5),("blaunch",pid=2826,fd=5),("blaunch",pid=2825,fd=5),("blaunch",pid=2824,fd=5),("blaunch",pid=2823,fd=5),("hydra_pmi_proxy",pid=2822,fd=5),("mpiexec",pid=2821,fd=5)) tcp LISTEN 0 128 0.0.0.0:43741 0.0.0.0:* users:(("blaunch",pid=2825,fd=12)) tcp LISTEN 0 128 0.0.0.0:41983 0.0.0.0:* users:(("blaunch",pid=2830,fd=22)) tcp LISTEN 0 128 0.0.0.0:41215 0.0.0.0:* users:(("blaunch",pid=2832,fd=26)) tcp LISTEN 0 128 0.0.0.0:34433 0.0.0.0:* users:(("blaunch",pid=2831,fd=24)) tcp LISTEN 0 128 0.0.0.0:33219 0.0.0.0:* users:(("blaunch",pid=2827,fd=16)) tcp LISTEN 0 128 0.0.0.0:34405 0.0.0.0:* users:(("blaunch",pid=2837,fd=36)) tcp LISTEN 0 128 0.0.0.0:43465 0.0.0.0:* users:(("blaunch",pid=2836,fd=34)) tcp LISTEN 0 128 0.0.0.0:39755 0.0.0.0:* users:(("blaunch",pid=2833,fd=28)) tcp LISTEN 0 128 0.0.0.0:38095 0.0.0.0:* users:(("blaunch",pid=2829,fd=20)) tcp LISTEN 0 128 0.0.0.0:44625 0.0.0.0:* users:(("blaunch",pid=2834,fd=30)) tcp LISTEN 0 128 0.0.0.0:35345 0.0.0.0:* users:(("blaunch",pid=2835,fd=32)) tcp LISTEN 0 128 0.0.0.0:43827 0.0.0.0:* users:(("blaunch",pid=2826,fd=14)) tcp LISTEN 0 128 0.0.0.0:40915 0.0.0.0:* users:(("blaunch",pid=2828,fd=18)) tcp LISTEN 0 128 0.0.0.0:42549 0.0.0.0:* users:(("blaunch",pid=2824,fd=9)) Why are these all listening on ports outside my range? I’ve also tried setting MPIEXEC_PORT_RANGE and MPIR_CVAR_CH3_PORT_RANGE and still have the problem. Is there any way to fully control the ports used? Cheers, Sendu. -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss [lists.mpich.org]<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

1 0

Re: [mpich-discuss] How to set port range used under LSF?
by Zhou, Hui 09 Mar '21

09 Mar '21

Hi Sendu, Could you try `MPIEXEC_PORT_RANGE` in stead? I know it is confusing and the documentation probably need update/correction, but `MPICH_PORT_RANGE` is for `MPICH` rather than `MPIEXEC`. -- Hui Zhou From: Sendu Bala via discuss <discuss(a)mpich.org> Date: Tuesday, March 9, 2021 at 7:16 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Sendu Bala <sb10(a)sanger.ac.uk> Subject: [mpich-discuss] How to set port range used under LSF? Via a bsub, I’m doing: MPICH_PORT_RANGE="46107:46140” mpiexec mpich/examples/cpi When I ssh to the controlling node, I see it has spawned a set of blaunch processes with `--control-port node-12-3-2:46107` as expected, but: ss -l -p -n | grep blaunch tcp LISTEN 0 128 0.0.0.0:34361 0.0.0.0:* users:(("blaunch",pid=2823,fd=7)) tcp LISTEN 0 128 0.0.0.0:46107 0.0.0.0:* users:(("cpi",pid=2839,fd=5),("blaunch",pid=2837,fd=5),("blaunch",pid=2836,fd=5),("blaunch",pid=2835,fd=5),("blaunch",pid=2834,fd=5),("blaunch",pid=2833,fd=5),("blaunch",pid=2832,fd=5),("blaunch",pid=2831,fd=5),("blaunch",pid=2830,fd=5),("blaunch",pid=2829,fd=5),("blaunch",pid=2828,fd=5),("blaunch",pid=2827,fd=5),("blaunch",pid=2826,fd=5),("blaunch",pid=2825,fd=5),("blaunch",pid=2824,fd=5),("blaunch",pid=2823,fd=5),("hydra_pmi_proxy",pid=2822,fd=5),("mpiexec",pid=2821,fd=5)) tcp LISTEN 0 128 0.0.0.0:43741 0.0.0.0:* users:(("blaunch",pid=2825,fd=12)) tcp LISTEN 0 128 0.0.0.0:41983 0.0.0.0:* users:(("blaunch",pid=2830,fd=22)) tcp LISTEN 0 128 0.0.0.0:41215 0.0.0.0:* users:(("blaunch",pid=2832,fd=26)) tcp LISTEN 0 128 0.0.0.0:34433 0.0.0.0:* users:(("blaunch",pid=2831,fd=24)) tcp LISTEN 0 128 0.0.0.0:33219 0.0.0.0:* users:(("blaunch",pid=2827,fd=16)) tcp LISTEN 0 128 0.0.0.0:34405 0.0.0.0:* users:(("blaunch",pid=2837,fd=36)) tcp LISTEN 0 128 0.0.0.0:43465 0.0.0.0:* users:(("blaunch",pid=2836,fd=34)) tcp LISTEN 0 128 0.0.0.0:39755 0.0.0.0:* users:(("blaunch",pid=2833,fd=28)) tcp LISTEN 0 128 0.0.0.0:38095 0.0.0.0:* users:(("blaunch",pid=2829,fd=20)) tcp LISTEN 0 128 0.0.0.0:44625 0.0.0.0:* users:(("blaunch",pid=2834,fd=30)) tcp LISTEN 0 128 0.0.0.0:35345 0.0.0.0:* users:(("blaunch",pid=2835,fd=32)) tcp LISTEN 0 128 0.0.0.0:43827 0.0.0.0:* users:(("blaunch",pid=2826,fd=14)) tcp LISTEN 0 128 0.0.0.0:40915 0.0.0.0:* users:(("blaunch",pid=2828,fd=18)) tcp LISTEN 0 128 0.0.0.0:42549 0.0.0.0:* users:(("blaunch",pid=2824,fd=9)) Why are these all listening on ports outside my range? I’ve also tried setting MPIEXEC_PORT_RANGE and MPIR_CVAR_CH3_PORT_RANGE and still have the problem. Is there any way to fully control the ports used? Cheers, Sendu. -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0