May 2020 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] troubles with 3.3.2 and libslurm
by Per Berg 29 May '20

29 May '20

Thank you for the quick answer; it is much appreciated. Your suggestion seems to do the trick. Thanks a lot! ciao Per ________________________________ From: Zhou, Hui <zhouh(a)anl.gov> Sent: 28 May 2020 22:12 To: discuss(a)mpich.org Cc: Per Berg Subject: Re: [mpich-discuss] troubles with 3.3.2 and libslurm Unfortunately, there isn’t an easy way not to link with some optional libraries that is available on the building host. Probably at some point we’ll clean up our configure scripts to approach optional libraries more consistently. For now, all I can suggest is to hack it: modify `src/pm/hydra/configure.ac`, search for `have_libslurm`, it should land you somewhere that has `AC_CHECK_HEADERS` and `AC_CHECK_LIB` for slurm, delete both. Re-run `autogen.sh` follow by the usual `configure` and `make`. -- Hui Zhou From: Per Berg via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, May 28, 2020 at 3:13 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: Per Berg <peb(a)fcoo.dk> Subject: [mpich-discuss] troubles with 3.3.2 and libslurm Hi, I have version 3.2 running smoothly. Now I want to upgrade to vers 3.3.2, but I get a problem that the libslurm.so is required by the mpiexec in v3.3.2. This is a pitty since I don't have that on the compute nodes (and I don't use slurm). In vers3.2 this was not a problem. I use the same compiler suite and the same configure arguments for both versions (except prefix path, of course). >From ldd on the mpiexec from v3.3.2 I get: libslurm.so.29 => /usr/lib64/libslurm.so.29 (0x00007fc91ebf8000) which will eventually lead to runtime errors when running across nodes that don't have it. As said, vers3.2 doesn't have the slurm lib path burned into the mpiexec (and I don't need it). How do I get rid of the libslurm in mpiexec? Which combination of configure args should I use? I have tried a tons of different args, but without success. Please, help. Regards, Per

1 0

Re: [mpich-discuss] Problems in SGE submission when upgrade from 3.2.1 to 3.3.2
by Shuwei Zhao 28 May '20

28 May '20

Corrected error message as below: mt Dp Mpich 3.2.1 pass Pass Mpich 3.3.2 Fail error: executing task of job 490150 failed: execution daemon on host "host1" didn't accept task Fail [proxy:0:0@host1] HYDU_sock_write (utils/sock/sock.c:289): write error ( Bad file descriptor) 2 [proxy:0:0@host1] main (pm/pmiserv/pmip.c:189): unable to send the proxyID to the server On Thu, May 28, 2020 at 9:40 PM Shuwei Zhao <shuweizhao1991(a)gmail.com> wrote: > Hi > > > > I was trying to upgrade our mpich version from 3.2.1 to 3.3.2 to consume > the latest stable version of mpich, however the new mpich version run get > failed. > > > > We have 2 parallel environment configuration as below: > > mt > > dp > > pe_name mt > > slots 2000000 > > used_slots 1181 > > bound_slots 0 > > user_lists NONE > > xuser_lists NONE > > start_proc_args /bin/true > > stop_proc_args /bin/true > > per_pe_task_prolog NONE > > per_pe_task_epilog NONE > > allocation_rule $pe_slots > > control_slaves FALSE > > job_is_first_task FALSE > > urgency_slots min > > accounting_summary FALSE > > daemon_forks_slaves FALSE > > master_forks_slaves TRUE > > pe_name dp > > slots 10000 > > used_slots 0 > > bound_slots 0 > > user_lists NONE > > xuser_lists NONE > > start_proc_args /bin/true > > stop_proc_args /bin/true > > per_pe_task_prolog NONE > > per_pe_task_epilog NONE > > allocation_rule $round_robin > > control_slaves TRUE > > job_is_first_task FALSE > > urgency_slots min > > accounting_summary FALSE > > daemon_forks_slaves FALSE > > master_forks_slaves FALSE > > > > Running command: > > We are using qsub to submit workers and master-worker connection will be > established using MPI_COMM_ACCEPT and MPI_COMM_CONNECT > > qsub -P bnormal -pe mt 1 -e sge_err -o sge_out mpiexec -n 1 > /path/to/my/binary binary_arguments > > qsub -P bnormal -pe dp 1 -e sge_err -o sge_out mpiexec -n 1 > /path/to/my/binary binary_arguments > > > > Running result: > > > > mt > > Dp > > Mpich 3.2.1 > > pass > > Pass > > Mpich 3.3.2 > > Fail > > [proxy:0:0@host1] HYDU_sock_write (utils/sock/sock.c:289): write error > ( Bad file descriptor) > > 2 [proxy:0:0@host1] main (pm/pmiserv/pmip.c:189): unable to send the > proxyID to the server > > Fail > > error: executing task of job 490150 failed: execution daemon on host > "host1" didn't accept task > > > Could you please explain why the run get failed in mpich-3.3 and any > solution we can use to get the run pass with mpich-3.3.2? > > Appreciate any help. > > > > Thanks >

1 0

Problems in SGE submission when upgrade from 3.2.1 to 3.3.2
by Shuwei Zhao 28 May '20

28 May '20

Hi I was trying to upgrade our mpich version from 3.2.1 to 3.3.2 to consume the latest stable version of mpich, however the new mpich version run get failed. We have 2 parallel environment configuration as below: mt dp pe_name mt slots 2000000 used_slots 1181 bound_slots 0 user_lists NONE xuser_lists NONE start_proc_args /bin/true stop_proc_args /bin/true per_pe_task_prolog NONE per_pe_task_epilog NONE allocation_rule $pe_slots control_slaves FALSE job_is_first_task FALSE urgency_slots min accounting_summary FALSE daemon_forks_slaves FALSE master_forks_slaves TRUE pe_name dp slots 10000 used_slots 0 bound_slots 0 user_lists NONE xuser_lists NONE start_proc_args /bin/true stop_proc_args /bin/true per_pe_task_prolog NONE per_pe_task_epilog NONE allocation_rule $round_robin control_slaves TRUE job_is_first_task FALSE urgency_slots min accounting_summary FALSE daemon_forks_slaves FALSE master_forks_slaves FALSE Running command: We are using qsub to submit workers and master-worker connection will be established using MPI_COMM_ACCEPT and MPI_COMM_CONNECT qsub -P bnormal -pe mt 1 -e sge_err -o sge_out mpiexec -n 1 /path/to/my/binary binary_arguments qsub -P bnormal -pe dp 1 -e sge_err -o sge_out mpiexec -n 1 /path/to/my/binary binary_arguments Running result: mt Dp Mpich 3.2.1 pass Pass Mpich 3.3.2 Fail [proxy:0:0@host1] HYDU_sock_write (utils/sock/sock.c:289): write error ( Bad file descriptor) 2 [proxy:0:0@host1] main (pm/pmiserv/pmip.c:189): unable to send the proxyID to the server Fail error: executing task of job 490150 failed: execution daemon on host "host1" didn't accept task Could you please explain why the run get failed in mpich-3.3 and any solution we can use to get the run pass with mpich-3.3.2? Appreciate any help. Thanks

1 0

Re: [mpich-discuss] troubles with 3.3.2 and libslurm
by Zhou, Hui 28 May '20

28 May '20

Unfortunately, there isn’t an easy way not to link with some optional libraries that is available on the building host. Probably at some point we’ll clean up our configure scripts to approach optional libraries more consistently. For now, all I can suggest is to hack it: modify `src/pm/hydra/configure.ac`, search for `have_libslurm`, it should land you somewhere that has `AC_CHECK_HEADERS` and `AC_CHECK_LIB` for slurm, delete both. Re-run `autogen.sh` follow by the usual `configure` and `make`. -- Hui Zhou From: Per Berg via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, May 28, 2020 at 3:13 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: Per Berg <peb(a)fcoo.dk> Subject: [mpich-discuss] troubles with 3.3.2 and libslurm Hi, I have version 3.2 running smoothly. Now I want to upgrade to vers 3.3.2, but I get a problem that the libslurm.so is required by the mpiexec in v3.3.2. This is a pitty since I don't have that on the compute nodes (and I don't use slurm). In vers3.2 this was not a problem. I use the same compiler suite and the same configure arguments for both versions (except prefix path, of course). From ldd on the mpiexec from v3.3.2 I get: libslurm.so.29 => /usr/lib64/libslurm.so.29 (0x00007fc91ebf8000) which will eventually lead to runtime errors when running across nodes that don't have it. As said, vers3.2 doesn't have the slurm lib path burned into the mpiexec (and I don't need it). How do I get rid of the libslurm in mpiexec? Which combination of configure args should I use? I have tried a tons of different args, but without success. Please, help. Regards, Per

1 0

troubles with 3.3.2 and libslurm
by Per Berg 28 May '20

28 May '20

Hi, I have version 3.2 running smoothly. Now I want to upgrade to vers 3.3.2, but I get a problem that the libslurm.so is required by the mpiexec in v3.3.2. This is a pitty since I don't have that on the compute nodes (and I don't use slurm). In vers3.2 this was not a problem. I use the same compiler suite and the same configure arguments for both versions (except prefix path, of course). >From ldd on the mpiexec from v3.3.2 I get: libslurm.so.29 => /usr/lib64/libslurm.so.29 (0x00007fc91ebf8000) which will eventually lead to runtime errors when running across nodes that don't have it. As said, vers3.2 doesn't have the slurm lib path burned into the mpiexec (and I don't need it). How do I get rid of the libslurm in mpiexec? Which combination of configure args should I use? I have tried a tons of different args, but without success. Please, help. Regards, Per

1 0

Using MPICH in Python breaks Fortran MPI_IN_PLACE
by Patrick McNally 27 May '20

27 May '20

Our application consists primarily of a Python head calling into Fortran routines to do the heavy lifting. We have never been able to successfully use MPI_IN_PLACE in Fortran but weren't sure why. Recently, we discovered that it works fine in standalone Fortran code and is only broken when the Fortran code is run through our Python modules. The issue appears to be related to having code that only links to the C libmpi library loaded first and with RTLD_LOCAL, as happens when we load mpi4py. It works if you load something linked to libmpifort first or load everything with RTLD_GLOBAL. I'm assuming this has something to do with how MPICH tests the address of MPIR_F08_MPI_IN_PLACE but I don't understand SO loading well enough to fully grasp the issue. Below is some standalone code to show the issue. I'd appreciate any insight you can provide into why this is happening. Relevant system details: RHEL 7.8 Python 2.7 GCC 7.3.0 MPICH 3.3.2 (and 3.2) The below files are also available towards the end of the bug report at the following link: https://bitbucket.org/mpi4py/mpi4py/issues/162/mpi4py-initialization-breaks… Thanks, Patrick makefile ----------- libs = testc.so testf.so all: $(libs) testc.so: testc.c mpicc -shared -fPIC $< -o $@ testf.so: testf.f90 mpifort -shared -fPIC $< -o $@ clean: $(RM) $(libs) testc.c --------- #include <stddef.h> #include <stdio.h> #include <mpi.h> extern void initc(void); extern void testc(void); void initc(void) { MPI_Init(NULL,NULL); } void testc(void) { int val = 1; MPI_Allreduce(MPI_IN_PLACE, &val, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD); printf("C val: %2d\n",val); } testf.f90 ----------- subroutine initf() bind(C) use mpi integer ierr call MPI_Init(ierr) end subroutine initf subroutine testf() bind(C) use mpi integer ierr integer val val = 1 call MPI_Allreduce(MPI_IN_PLACE, val, 1, MPI_INTEGER, MPI_SUM, MPI_COMM_WORLD, ierr) print '(A,I2)', 'F val: ', val end subroutine testf test.py --------- from ctypes import CDLL, RTLD_LOCAL, RTLD_GLOBAL mode = RTLD_LOCAL cfirst = True if cfirst: # it does not work! libc = CDLL("./testc.so", mode) libf = CDLL("./testf.so", mode) else: # it works! libf = CDLL("./testf.so", mode) libc = CDLL("./testc.so", mode) libc.initc.restype = None libc.testc.argtypes = [] libf.initf.restype = None libf.testf.argtypes = [] libc.initc() libc.testc() libf.testf()

1 0

Re: [mpich-discuss] MPI_Barrier(comm=MPI_COMM_WORLD) failed
by Sudhakar D 17 May '20

17 May '20

Hui Zhou, Fatal error in PMPI_Barrier: Other MPI error, error stack: PMPI_Barrier(289)...................: MPI_Barrier(comm=MPI_COMM_WORLD) failed PMPI_Barrier(275)...................: MPIR_Barrier_impl(175)..............: MPIR_Barrier_intra_auto(110)........: MPIR_Barrier_intra_dissemination(49): MPIC_Sendrecv(344)..................: (unknown)(): Other MPI error regards, Sudhakar On Sun, May 17, 2020 at 6:04 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > Hi Sudhakar, > > > > Was that the completed error message? Seems missing a few lines. > > > > -- > Hui Zhou > > > > > > *From: *Sudhakar D via discuss <discuss(a)mpich.org> > *Reply-To: *"discuss(a)mpich.org" <discuss(a)mpich.org> > *Date: *Sunday, May 17, 2020 at 3:10 PM > *To: *"discuss(a)mpich.org" <discuss(a)mpich.org> > *Cc: *Sudhakar D <home.sudhakar(a)gmail.com> > *Subject: *[mpich-discuss] MPI_Barrier(comm=MPI_COMM_WORLD) failed > > > > Hello, > > Some times I see the below error when running my python app across two nodes. Just wondering under what conditions we get the below error? > > I am using 3.3.1 version > > regards, > > Sudhakar > > PMPI_Barrier(289)...................: MPI_Barrier(comm=MPI_COMM_WORLD) failed > PMPI_Barrier(275)...................: > MPIR_Barrier_impl(175)..............: > MPIR_Barrier_intra_auto(110)........: > MPIR_Barrier_intra_dissemination(49): > MPIC_Sendrecv(344)..................: > >

1 0

Re: [mpich-discuss] MPI_Barrier(comm=MPI_COMM_WORLD) failed
by Zhou, Hui 17 May '20

17 May '20

Hi Sudhakar, Was that the completed error message? Seems missing a few lines. -- Hui Zhou From: Sudhakar D via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Sunday, May 17, 2020 at 3:10 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: Sudhakar D <home.sudhakar(a)gmail.com> Subject: [mpich-discuss] MPI_Barrier(comm=MPI_COMM_WORLD) failed Hello, Some times I see the below error when running my python app across two nodes. Just wondering under what conditions we get the below error? I am using 3.3.1 version regards, Sudhakar PMPI_Barrier(289)...................: MPI_Barrier(comm=MPI_COMM_WORLD) failed PMPI_Barrier(275)...................: MPIR_Barrier_impl(175)..............: MPIR_Barrier_intra_auto(110)........: MPIR_Barrier_intra_dissemination(49): MPIC_Sendrecv(344)..................:

1 0

MPI_Barrier(comm=MPI_COMM_WORLD) failed
by Sudhakar D 17 May '20

17 May '20

Hello, Some times I see the below error when running my python app across two nodes. Just wondering under what conditions we get the below error? I am using 3.3.1 version regards, Sudhakar PMPI_Barrier(289)...................: MPI_Barrier(comm=MPI_COMM_WORLD) failed PMPI_Barrier(275)...................: MPIR_Barrier_impl(175)..............: MPIR_Barrier_intra_auto(110)........: MPIR_Barrier_intra_dissemination(49): MPIC_Sendrecv(344)..................:

1 0

Re: [mpich-discuss] discuss Digest, Vol 91, Issue 21
by hritikesh semwal 16 May '20

16 May '20

On Sat, 16 May, 2020, 10:30 PM , <discuss-request(a)mpich.org> wrote: > Send discuss mailing list submissions to > discuss(a)mpich.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.mpich.org/mailman/listinfo/discuss > or, via email, send a message with subject or body 'help' to > discuss-request(a)mpich.org > > You can reach the person managing the list at > discuss-owner(a)mpich.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of discuss digest..." > > > Today's Topics: > > 1. Understanding process bindings in MPICH (hritikesh semwal) > 2. Re: Understanding process bindings in MPICH (Benson Muite) > 3. Re: Understanding process bindings in MPICH (Benson Muite) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 15 May 2020 23:22:35 +0530 > From: hritikesh semwal <hritikesh.semwal(a)gmail.com> > To: Benson Muite via discuss <discuss(a)mpich.org> > Subject: [mpich-discuss] Understanding process bindings in MPICH > Message-ID: > < > CAA+35d2JQ25SYkqST9A5aPN-KuqHSkK7oC6OThsB9KRQvP6d9g(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hello, > > I am working on a parallel CFD solver with MPI and I am using an account on > a cluster to run my executable. The hardware structure of my account is as > follows; > > Architecture: x86_64 > CPU op-mode(s): 32-bit, 64-bit > Byte Order: Little Endian > CPU(s): 32 > On-line CPU(s) list: 0-31 > Thread(s) per core: 2 > Core(s) per socket: 8 > CPU socket(s): 2 > NUMA node(s): 2 > Vendor ID: GenuineIntel > CPU family: 6 > Model: 62 > Stepping: 4 > CPU MHz: 2600.079 > BogoMIPS: 5199.25 > Virtualization: VT-x > L1d cache: 32K > L1i cache: 32K > L2 cache: 256K > L3 cache: 20480K > NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30 > NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31 > > Initially, I was running my executable with any binding options and in that > case, whenever I was switching from 2 to 4 processors my computation time > was also increasing along with communication time inside some iterative > loop. > > Today, somewhere I read about binding options in MPI through which I can > manage the allocation of processors. Initially, I used the "-bind-to core" > option and the results were different and I got time reduction up to 16 > processors and after that with 24 and 32 processors, it has started > increasing. Results of timing are as follows; > 2 procs- 160 seconds, 4 procs- 84 seconds, 8 procs- 45 seconds, 16 procs- > 28 seconds, 24 procs- 38 seconds, 32 procs- 34 seconds. > > After that, I used some other combinations of binding option but did not > get better timing results compared to -bind-to core option. So, I back > edited the bind to option to core but now I am getting different timing > results with the same executable which are as follows, > 2 procs- 164 seconds, 4 procs- 85 seconds, 8 procs- 45 seconds, 16 procs- > 48 seconds, 24 procs- 52 seconds, 32 seconds- 98 seconds. > > I have following two questions for which I am seeking your help, > > 1. Can anyone please suggest me is it possible an optimum binding and > mapping options based on my cluster account hardware topology?? If yes > then please tell me. > 2. Why I am getting such an irregular pattern of jump in timing without > binding option and why with a binding option, my timings are varying for > each run? Is it my cluster network problem or my MPI code problem? > > If you need further details about my iterative loop then please tell me. As > this message got too long I can share it later if you think above data is > not sufficient. > > Thank you. >

1 0