April 2017 - discuss - mailman3.cels.anl.gov

mpicc -V broken
by Sobel-Sorenson, Chandler - (scar) 14 Apr '17

14 Apr '17

I have just installed mpich 3.2 on our CentOS 6.8 machine with gcc 4.4.7. It worked fine with maker but now am having a strange issue when trying to install mpiblast. I noticed a problem when "mpicc -V" is run, for example: $ mpicc -V gcc: error trying to exec 'x86_64-redhat-linux-gcc--I/opt/mpich-install/include': execvp: No such file or directory It seems that a space is missing, because when I run "x86_64-redhat-linux-gcc --I/opt/mpich-install/include" it says "x86_64-redhat-linux-gcc: no input files" I'm not sure how this happened, maybe it is a bug? I did a standard installation with "./configure --prefix=/opt/mpich-install; make; make install" If you know how I could fix this or if you need more info about our system/installation then please let me know! Thanks Chandler / Systems Administrator Arizona Genomics Institute www.genome.arizona.edu _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Process Group Collision for multiple clients from different host machines having same pid with MPI_Comm_accept
by Roy, Hirak 14 Apr '17

14 Apr '17

Dear MPICH team, We use MPICH for a server-client application. We use MPICH-3.0.4 with sock channel. In this application there is one server and 100 clients. Each client is launched independently in different host-machines using individual-wrapper scripts. (we explicitly use : mpiexec -n 1 ) The server is multithreaded and it uses MPI_Comm_accept (on MPI_COMM_SELF) and clients use MPI_Comm_connect to connect. We have observed the following issue after all the clients connect to server : if we send message to a client (lets say 'm'), it reaches unexpectedly to some other client (lets say 'n'). { server sends the message using the communicator returned by accept call }. This happens randomly in one out of 5-6 runs. On further looking into MPICH code, we found that 1) There is a collsion of pg (process-group) of two processes (m and n) after mpi-comm-accept 2) As a result of (1), comm->vc are same (for m and n, although comm are different). It seems that the unique string (something like kva_<int>_int) is not unique for such two processes. 'm' and 'n' processes are running in different host-machine and they have the same pid. The kva string looked like kva_pid_rank. We have the following questions : 1) Have we built MPICH with some kind of incorrect configuration (hydra configuration at the end of the email) ? 2) Are we using incorrect process-manager or configuration and that is why there is a possible collision of process-groups? 3) What is the purpose of process group sharing/uniquifying? If there is no real reason for this, could it be disabled or will something else rely on the id string being unique? 4) If there are no other work-around, what could be done to make the id string unique? Add the host-name? Will everything else be ok with this? It would be good if you can let us know if there is any workaround for this issue or not. Thanks, Hirak Roy HYDRA build details: CXX: no -O3 -fPIC F77: no F90: no Configure options: '--disable-option-checking' '--prefix=/home/hroy/local/mpich-3.0.4/linux_x86_64' '--disable-f77' '--disable-fc' '--disable-f90modules' '--disable-cxx' '--enable-fast=nochkmsg' '--enable-fast=notiming' '--enable-fast=ndebug' '--enable-fast=O3' '--with-device=ch3:sock' 'CFLAGS=-O3 -fPIC -O3' 'CXXFLAGS=-O3 -fPIC ' 'CC=/u/prod/gnu/gcc/20121129/gcc-4.5.0-linux_x86_64/bin/gcc' 'LDFLAGS= ' '--cache-file=/dev/null' '--srcdir=.' 'LIBS=-lrt -lpthread ' 'CPPFLAGS= -I/home/hroy/tools/mpich/mpich-3.0.4/src/mpl/include -I/home/hroy/tools/mpich/mpich-3.0.4/src/mpl/include -I/home/hroy/tools/mpich/mpich-3.0.4/src/openpa/src -I/home/hroy/tools/mpich/mpich-3.0.4/src/openpa/src -I/home/hroy/tools/mpich/mpich-3.0.4/src/mpi/romio/include' Process Manager: pmi Launchers available: ssh rsh fork slurm ll lsf sge manual persist Topology libraries available: hwloc Resource management kernels available: user slurm ll lsf sge pbs cobalt Checkpointing libraries available: Demux engines available: poll select _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] scheduling to real hw cores, not using hyperthreading (mpich-3.2)
by Heinz-Ado Arnolds 13 Apr '17

13 Apr '17

Dear Ken, thanks a lot for your fast reply! See my comments/questions inline too: On 12.04.2017 21:38, Kenneth Raffenetti wrote: > Hi, > > Some answers inline: > > On 04/12/2017 10:02 AM, Heinz-Ado Arnolds wrote: >> Dear MPIch users and developers, >> >> first of all many thanks for all the great work you have done for MPIch! >> >> I'd like to have 4 MPI jobs scheduled by SGE starting 1 OpenMP job each with 10 threads, running on 2 nodes, each having 2 sockets, with 10 cores & 10 hwthreads. Only 10 cores (no hwthreads) should be used on each socket. >> >> 4 MPI: 1 OpenMP with 10 thread (i.e. 4x10 threads) >> 2 nodes, 2 sockets each, 10 cores & 10 hwthreads each >> >> lscpu -a -e >> >> CPU NODE SOCKET CORE L1d:L1i:L2:L3 >> 0 0 0 0 0:0:0:0 >> 1 1 1 1 1:1:1:1 >> 2 0 0 2 2:2:2:0 >> 3 1 1 3 3:3:3:1 >> 4 0 0 4 4:4:4:0 >> 5 1 1 5 5:5:5:1 >> 6 0 0 6 6:6:6:0 >> 7 1 1 7 7:7:7:1 >> 8 0 0 8 8:8:8:0 >> 9 1 1 9 9:9:9:1 >> 10 0 0 10 10:10:10:0 >> 11 1 1 11 11:11:11:1 >> 12 0 0 12 12:12:12:0 >> 13 1 1 13 13:13:13:1 >> 14 0 0 14 14:14:14:0 >> 15 1 1 15 15:15:15:1 >> 16 0 0 16 16:16:16:0 >> 17 1 1 17 17:17:17:1 >> 18 0 0 18 18:18:18:0 >> 19 1 1 19 19:19:19:1 >> 20 0 0 0 0:0:0:0 >> 21 1 1 1 1:1:1:1 >> 22 0 0 2 2:2:2:0 >> 23 1 1 3 3:3:3:1 >> 24 0 0 4 4:4:4:0 >> 25 1 1 5 5:5:5:1 >> 26 0 0 6 6:6:6:0 >> 27 1 1 7 7:7:7:1 >> 28 0 0 8 8:8:8:0 >> 29 1 1 9 9:9:9:1 >> 30 0 0 10 10:10:10:0 >> 31 1 1 11 11:11:11:1 >> 32 0 0 12 12:12:12:0 >> 33 1 1 13 13:13:13:1 >> 34 0 0 14 14:14:14:0 >> 35 1 1 15 15:15:15:1 >> 36 0 0 16 16:16:16:0 >> 37 1 1 17 17:17:17:1 >> 38 0 0 18 18:18:18:0 >> 39 1 1 19 19:19:19:1 >> >> When I try to submit the job by using >> >> HYDRA_TOPO_DEBUG=1 mpirun -np 4 -bind-to socket:1 -map-by socket ./myid >> >> the distribution is done to 2 sockets on 2 nodes correctly, but all 10 cores + 10 hwthreads are used on each node: >> >> process 0 binding: 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 >> process 1 binding: 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 >> process 2 binding: 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 >> process 3 binding: 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 > > The process binding code treats the machine as a hierarchy. If a core contains 2 hwthreads, then binding to that core effectively binds to both hwthreads. If you bind to one socket on your machine, you should see that as being bound to 20 hwthreads like above. You can view the system hierarchy if you have hwloc installed on your system using the 'lstopo' command. Hopefully that is answering your question. Yes, I know the architecture of the machine and the hwloc & lstopo tools too and I see that the binding is done to both hwthreads of a core. But that's not what I like to get. Can you give me a hint how I could achieve a binding like this process 0 binding: 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 process 1 binding: 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 i.e. binding only to one thread of a core, not both. That would be of great help to me. In OpenMPI that's done with "-use-hwthread-cpus". >> >> Additionally it seems, that the CPU masks & Cpus_allowed_list are not set for the called processes: >> >> MPI Instance 0001 of 0004 is on pascal-1-04, 0x000000ff,0xffffffff, Cpus_allowed_list: 0-39 >> MPI Instance 0002 of 0004 is on pascal-1-04, 0x000000ff,0xffffffff, Cpus_allowed_list: 0-39 >> MPI Instance 0003 of 0004 is on pascal-3-06, 0x000000ff,0xffffffff, Cpus_allowed_list: 0-39 >> MPI Instance 0004 of 0004 is on pascal-3-06, 0x000000ff,0xffffffff, Cpus_allowed_list: 0-39 > > This is true only if HYDRA_TOPO_DEBUG=1 is set. We changed that behavior based on user feedback, and future versions will actually set the binding when debug is enabled. https://github.com/pmodels/mpich/commit/a145fd73a1bb7dd4f06c8c6b7dff5f6c369… Thanks for that helpful hint! I didn't know that I have to choose between a working *OR* a verbose arrangement. >> >> >> Another way to achive my task would be to use "-bind-to user" >> >> HYDRA_TOPO_DEBUG=1 mpirun -np $nmpi -ppn $nmpipn -bind-to user:0+2+4+6+8+10+12+14+16+18,1+3+5+7+9+11+13+15+17+19 ./myid >> >> This works great up to specifying cores 9 cores on each socket ("0+2+4+6+8+10+12+14+16,1+3+5+7+9+11+13+15+17"). As soon as I add ...+18,...+19, the job crashes with these messages: >> > > Probably overrunning a buffer. Thanks for the bug report :). > > Ken > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss I'd appreciate an advise how to bind to cores w/o using all their hwthreads very much! Have nice Easter days Ado

1 0

Re: [mpich-discuss] scheduling to real hw cores, not using hyperthreading (mpich-3.2)
by Kenneth Raffenetti 12 Apr '17

12 Apr '17

Hi, Some answers inline: On 04/12/2017 10:02 AM, Heinz-Ado Arnolds wrote: > Dear MPIch users and developers, > > first of all many thanks for all the great work you have done for MPIch! > > I'd like to have 4 MPI jobs scheduled by SGE starting 1 OpenMP job each with 10 threads, running on 2 nodes, each having 2 sockets, with 10 cores & 10 hwthreads. Only 10 cores (no hwthreads) should be used on each socket. > > 4 MPI: 1 OpenMP with 10 thread (i.e. 4x10 threads) > 2 nodes, 2 sockets each, 10 cores & 10 hwthreads each > > lscpu -a -e > > CPU NODE SOCKET CORE L1d:L1i:L2:L3 > 0 0 0 0 0:0:0:0 > 1 1 1 1 1:1:1:1 > 2 0 0 2 2:2:2:0 > 3 1 1 3 3:3:3:1 > 4 0 0 4 4:4:4:0 > 5 1 1 5 5:5:5:1 > 6 0 0 6 6:6:6:0 > 7 1 1 7 7:7:7:1 > 8 0 0 8 8:8:8:0 > 9 1 1 9 9:9:9:1 > 10 0 0 10 10:10:10:0 > 11 1 1 11 11:11:11:1 > 12 0 0 12 12:12:12:0 > 13 1 1 13 13:13:13:1 > 14 0 0 14 14:14:14:0 > 15 1 1 15 15:15:15:1 > 16 0 0 16 16:16:16:0 > 17 1 1 17 17:17:17:1 > 18 0 0 18 18:18:18:0 > 19 1 1 19 19:19:19:1 > 20 0 0 0 0:0:0:0 > 21 1 1 1 1:1:1:1 > 22 0 0 2 2:2:2:0 > 23 1 1 3 3:3:3:1 > 24 0 0 4 4:4:4:0 > 25 1 1 5 5:5:5:1 > 26 0 0 6 6:6:6:0 > 27 1 1 7 7:7:7:1 > 28 0 0 8 8:8:8:0 > 29 1 1 9 9:9:9:1 > 30 0 0 10 10:10:10:0 > 31 1 1 11 11:11:11:1 > 32 0 0 12 12:12:12:0 > 33 1 1 13 13:13:13:1 > 34 0 0 14 14:14:14:0 > 35 1 1 15 15:15:15:1 > 36 0 0 16 16:16:16:0 > 37 1 1 17 17:17:17:1 > 38 0 0 18 18:18:18:0 > 39 1 1 19 19:19:19:1 > > When I try to submit the job by using > > HYDRA_TOPO_DEBUG=1 mpirun -np 4 -bind-to socket:1 -map-by socket ./myid > > the distribution is done to 2 sockets on 2 nodes correctly, but all 10 cores + 10 hwthreads are used on each node: > > process 0 binding: 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 > process 1 binding: 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 > process 2 binding: 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 > process 3 binding: 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 The process binding code treats the machine as a hierarchy. If a core contains 2 hwthreads, then binding to that core effectively binds to both hwthreads. If you bind to one socket on your machine, you should see that as being bound to 20 hwthreads like above. You can view the system hierarchy if you have hwloc installed on your system using the 'lstopo' command. Hopefully that is answering your question. > > Additionally it seems, that the CPU masks & Cpus_allowed_list are not set for the called processes: > > MPI Instance 0001 of 0004 is on pascal-1-04, 0x000000ff,0xffffffff, Cpus_allowed_list: 0-39 > MPI Instance 0002 of 0004 is on pascal-1-04, 0x000000ff,0xffffffff, Cpus_allowed_list: 0-39 > MPI Instance 0003 of 0004 is on pascal-3-06, 0x000000ff,0xffffffff, Cpus_allowed_list: 0-39 > MPI Instance 0004 of 0004 is on pascal-3-06, 0x000000ff,0xffffffff, Cpus_allowed_list: 0-39 This is true only if HYDRA_TOPO_DEBUG=1 is set. We changed that behavior based on user feedback, and future versions will actually set the binding when debug is enabled. https://github.com/pmodels/mpich/commit/a145fd73a1bb7dd4f06c8c6b7dff5f6c369… > > > Another way to achive my task would be to use "-bind-to user" > > HYDRA_TOPO_DEBUG=1 mpirun -np $nmpi -ppn $nmpipn -bind-to user:0+2+4+6+8+10+12+14+16+18,1+3+5+7+9+11+13+15+17+19 ./myid > > This works great up to specifying cores 9 cores on each socket ("0+2+4+6+8+10+12+14+16,1+3+5+7+9+11+13+15+17"). As soon as I add ...+18,...+19, the job crashes with these messages: > Probably overrunning a buffer. Thanks for the bug report :). Ken _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Fatal error in MPI_Finalize: Other MPI error, error stack:
by Jeff Hammond 12 Apr '17

12 Apr '17

I observe this crash during MPI_Finalize in an ARMCI-MPI unit test (test_malloc) that was not failing until I switched to using OFI-based MPICH on my Mac laptop. Any suggestions on how to debug? Jeff Abort(806972175) on node 0 (rank 0 in comm 0): Fatal error in MPI_Finalize: Other MPI error, error stack: MPI_Finalize(317)..........: MPI_Finalize failed MPI_Finalize(239)..........: MPID_Finalize(318).........: MPIDI_NM_mpi_init_hook(948): OFI av close failed (ofi_init.h:948:MPIDI_NM_mpi_init_hook:Resource busy) Abort(806972175) on node 1 (rank 1 in comm 0): Fatal error in MPI_Finalize: Other MPI error, error stack: MPI_Finalize(317)..........: MPI_Finalize failed MPI_Finalize(239)..........: MPID_Finalize(318).........: MPIDI_NM_mpi_init_hook(948): OFI av close failed (ofi_init.h:948:MPIDI_NM_mpi_init_hook:Resource busy) $ mpicc -show icc -g -Wl,-commons,use_dylibs -I/opt/mpich/dev/intel/default/include -L/opt/mpich/dev/intel/default/lib -lmpi -lpmpi $ mpichversion MPICH Version: 3.3a2 MPICH Release date: unreleased development copy MPICH Device: ch4:ofi MPICH configure: CC=icc CXX=icpc FC=ifort F77=ifort --enable-cxx --enable-fortran --enable-threads=runtime --enable-g=dbg --with-pm=hydra --prefix=/opt/mpich/dev/intel/default --enable-wrapper-rpath --enable-two-level-namespace --disable-static --enable-shared --with-device=ch4:ofi --with-libfabric=/usr/local/Cellar/libfabric/1.4.1 MPICH CC: icc -g -O2 MPICH CXX: icpc -g MPICH F77: ifort -g -O2 MPICH FC: ifort -g -O2 MPICH Custom Information: -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

scheduling to real hw cores, not using hyperthreading (mpich-3.2)
by Heinz-Ado Arnolds 12 Apr '17

12 Apr '17

Dear MPIch users and developers, first of all many thanks for all the great work you have done for MPIch! I'd like to have 4 MPI jobs scheduled by SGE starting 1 OpenMP job each with 10 threads, running on 2 nodes, each having 2 sockets, with 10 cores & 10 hwthreads. Only 10 cores (no hwthreads) should be used on each socket. 4 MPI: 1 OpenMP with 10 thread (i.e. 4x10 threads) 2 nodes, 2 sockets each, 10 cores & 10 hwthreads each lscpu -a -e CPU NODE SOCKET CORE L1d:L1i:L2:L3 0 0 0 0 0:0:0:0 1 1 1 1 1:1:1:1 2 0 0 2 2:2:2:0 3 1 1 3 3:3:3:1 4 0 0 4 4:4:4:0 5 1 1 5 5:5:5:1 6 0 0 6 6:6:6:0 7 1 1 7 7:7:7:1 8 0 0 8 8:8:8:0 9 1 1 9 9:9:9:1 10 0 0 10 10:10:10:0 11 1 1 11 11:11:11:1 12 0 0 12 12:12:12:0 13 1 1 13 13:13:13:1 14 0 0 14 14:14:14:0 15 1 1 15 15:15:15:1 16 0 0 16 16:16:16:0 17 1 1 17 17:17:17:1 18 0 0 18 18:18:18:0 19 1 1 19 19:19:19:1 20 0 0 0 0:0:0:0 21 1 1 1 1:1:1:1 22 0 0 2 2:2:2:0 23 1 1 3 3:3:3:1 24 0 0 4 4:4:4:0 25 1 1 5 5:5:5:1 26 0 0 6 6:6:6:0 27 1 1 7 7:7:7:1 28 0 0 8 8:8:8:0 29 1 1 9 9:9:9:1 30 0 0 10 10:10:10:0 31 1 1 11 11:11:11:1 32 0 0 12 12:12:12:0 33 1 1 13 13:13:13:1 34 0 0 14 14:14:14:0 35 1 1 15 15:15:15:1 36 0 0 16 16:16:16:0 37 1 1 17 17:17:17:1 38 0 0 18 18:18:18:0 39 1 1 19 19:19:19:1 When I try to submit the job by using HYDRA_TOPO_DEBUG=1 mpirun -np 4 -bind-to socket:1 -map-by socket ./myid the distribution is done to 2 sockets on 2 nodes correctly, but all 10 cores + 10 hwthreads are used on each node: process 0 binding: 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 process 1 binding: 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 process 2 binding: 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 process 3 binding: 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 Additionally it seems, that the CPU masks & Cpus_allowed_list are not set for the called processes: MPI Instance 0001 of 0004 is on pascal-1-04, 0x000000ff,0xffffffff, Cpus_allowed_list: 0-39 MPI Instance 0002 of 0004 is on pascal-1-04, 0x000000ff,0xffffffff, Cpus_allowed_list: 0-39 MPI Instance 0003 of 0004 is on pascal-3-06, 0x000000ff,0xffffffff, Cpus_allowed_list: 0-39 MPI Instance 0004 of 0004 is on pascal-3-06, 0x000000ff,0xffffffff, Cpus_allowed_list: 0-39 Another way to achive my task would be to use "-bind-to user" HYDRA_TOPO_DEBUG=1 mpirun -np $nmpi -ppn $nmpipn -bind-to user:0+2+4+6+8+10+12+14+16+18,1+3+5+7+9+11+13+15+17+19 ./myid This works great up to specifying cores 9 cores on each socket ("0+2+4+6+8+10+12+14+16,1+3+5+7+9+11+13+15+17"). As soon as I add ...+18,...+19, the job crashes with these messages: *** Error in `/usr/bin/hydra_pmi_proxy': free(): invalid next size (fast): 0x0000000000f9f0c0 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x704fb)[0x2b3f62f5c4fb] /lib64/libc.so.6(+0x76976)[0x2b3f62f62976] /lib64/libc.so.6(+0x7716e)[0x2b3f62f6316e] /usr/bin/hydra_pmi_proxy[0x41aa27] /usr/bin/hydra_pmi_proxy[0x418aea] /usr/bin/hydra_pmi_proxy[0x407c0c] /usr/bin/hydra_pmi_proxy[0x420bb0] /usr/bin/hydra_pmi_proxy[0x403e49] /lib64/libc.so.6(__libc_start_main+0xf1)[0x2b3f62f0c4e1] /usr/bin/hydra_pmi_proxy[0x404cca] ======= Memory map: ======== 00400000-0045c000 r-xp 00000000 00:10 2142949266 /afs/mpa/(a)sys/system/MPA-8.03/usr/bin/hydra_pmi_proxy 0065c000-0065d000 r--p 0005c000 00:10 2142949266 /afs/mpa/(a)sys/system/MPA-8.03/usr/bin/hydra_pmi_proxy 0065d000-0065f000 rw-p 0005d000 00:10 2142949266 /afs/mpa/(a)sys/system/MPA-8.03/usr/bin/hydra_pmi_proxy 0065f000-006ff000 rw-p 00000000 00:00 0 00f98000-00fda000 rw-p 00000000 00:00 0 [heap] 2b3f619c5000-2b3f619e8000 r-xp 00000000 00:02 1229 /lib64/ld-2.25.so 2b3f619e8000-2b3f619eb000 rw-p 00000000 00:00 0 2b3f61a09000-2b3f61a0d000 rw-p 00000000 00:00 0 2b3f61be7000-2b3f61be8000 r--p 00022000 00:02 1229 /lib64/ld-2.25.so 2b3f61be8000-2b3f61be9000 rw-p 00023000 00:02 1229 /lib64/ld-2.25.so 2b3f61be9000-2b3f61bea000 rw-p 00000000 00:00 0 2b3f61bea000-2b3f61c03000 r-xp 00000000 00:02 1417 /lib64/libpthread-2.25.so 2b3f61c03000-2b3f61e02000 ---p 00019000 00:02 1417 /lib64/libpthread-2.25.so 2b3f61e02000-2b3f61e03000 r--p 00018000 00:02 1417 /lib64/libpthread-2.25.so 2b3f61e03000-2b3f61e04000 rw-p 00019000 00:02 1417 /lib64/libpthread-2.25.so 2b3f61e04000-2b3f61e08000 rw-p 00000000 00:00 0 2b3f61e08000-2b3f61e2c000 r-xp 00000000 00:02 1227 /lib64/libudev.so.1.6.3 2b3f61e2c000-2b3f6202b000 ---p 00024000 00:02 1227 /lib64/libudev.so.1.6.3 2b3f6202b000-2b3f6202c000 r--p 00023000 00:02 1227 /lib64/libudev.so.1.6.3 2b3f6202c000-2b3f6202d000 rw-p 00024000 00:02 1227 /lib64/libudev.so.1.6.3 2b3f6202d000-2b3f62035000 r-xp 00000000 00:10 2142711818 /afs/mpa/(a)sys/system/MPA-8.03/usr/X11/lib64/libpciaccess.so.0.11.1 2b3f62035000-2b3f62234000 ---p 00008000 00:10 2142711818 /afs/mpa/(a)sys/system/MPA-8.03/usr/X11/lib64/libpciaccess.so.0.11.1 2b3f62234000-2b3f62235000 r--p 00007000 00:10 2142711818 /afs/mpa/(a)sys/system/MPA-8.03/usr/X11/lib64/libpciaccess.so.0.11.1 2b3f62235000-2b3f62236000 rw-p 00008000 00:10 2142711818 /afs/mpa/(a)sys/system/MPA-8.03/usr/X11/lib64/libpciaccess.so.0.11.1 2b3f62236000-2b3f62390000 r-xp 00000000 00:10 2142836572 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/libxml2.so.2.9.4 2b3f62390000-2b3f62590000 ---p 0015a000 00:10 2142836572 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/libxml2.so.2.9.4 2b3f62590000-2b3f62598000 r--p 0015a000 00:10 2142836572 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/libxml2.so.2.9.4 2b3f62598000-2b3f6259a000 rw-p 00162000 00:10 2142836572 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/libxml2.so.2.9.4 2b3f6259a000-2b3f6259b000 rw-p 00000000 00:00 0 2b3f6259b000-2b3f6259e000 r-xp 00000000 00:02 1433 /lib64/libdl-2.25.so 2b3f6259e000-2b3f6279d000 ---p 00003000 00:02 1433 /lib64/libdl-2.25.so 2b3f6279d000-2b3f6279e000 r--p 00002000 00:02 1433 /lib64/libdl-2.25.so 2b3f6279e000-2b3f6279f000 rw-p 00003000 00:02 1433 /lib64/libdl-2.25.so 2b3f6279f000-2b3f627b4000 r-xp 00000000 00:10 2142836590 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/libz.so.1.2.6 2b3f627b4000-2b3f629b3000 ---p 00015000 00:10 2142836590 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/libz.so.1.2.6 2b3f629b3000-2b3f629b4000 r--p 00014000 00:10 2142836590 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/libz.so.1.2.6 2b3f629b4000-2b3f629b5000 rw-p 00015000 00:10 2142836590 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/libz.so.1.2.6 2b3f629b5000-2b3f629d9000 r-xp 00000000 00:10 2142835680 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/liblzma.so.5.2.3 2b3f629d9000-2b3f62bd9000 ---p 00024000 00:10 2142835680 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/liblzma.so.5.2.3 2b3f62bd9000-2b3f62bda000 r--p 00024000 00:10 2142835680 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/liblzma.so.5.2.3 2b3f62bda000-2b3f62bdb000 rw-p 00025000 00:10 2142835680 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/liblzma.so.5.2.3 2b3f62bdb000-2b3f62ceb000 r-xp 00000000 00:02 1401 /lib64/libm-2.25.so 2b3f62ceb000-2b3f62eea000 ---p 00110000 00:02 1401 /lib64/libm-2.25.so 2b3f62eea000-2b3f62eeb000 r--p 0010f000 00:02 1401 /lib64/libm-2.25.so 2b3f62eeb000-2b3f62eec000 rw-p 00110000 00:02 1401 /lib64/libm-2.25.so 2b3f62eec000-2b3f63081000 r-xp 00000000 00:02 1233 /lib64/libc-2.25.so 2b3f63081000-2b3f63280000 ---p 00195000 00:02 1233 /lib64/libc-2.25.so 2b3f63280000-2b3f63284000 r--p 00194000 00:02 1233 /lib64/libc-2.25.so 2b3f63284000-2b3f63286000 rw-p 00198000 00:02 1233 /lib64/libc-2.25.so 2b3f63286000-2b3f6328a000 rw-p 00000000 00:00 0 2b3f6328a000-2b3f632a0000 r-xp 00000000 00:10 2142835216 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/libgcc_s.so.1 2b3f632a0000-2b3f6349f000 ---p 00016000 00:10 2142835216 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/libgcc_s.so.1 2b3f6349f000-2b3f634a0000 r--p 00015000 00:10 2142835216 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/libgcc_s.so.1 2b3f634a0000-2b3f634a1000 rw-p 00016000 00:10 2142835216 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/libgcc_s.so.1 2b3f64000000-2b3f64021000 rw-p 00000000 00:00 0 2b3f64021000-2b3f68000000 ---p 00000000 00:00 0 7ffc37219000-7ffc3723a000 rw-p 00000000 00:00 0 [stack] 7ffc37355000-7ffc37357000 r--p 00000000 00:00 0 [vvar] 7ffc37357000-7ffc37359000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] *** Error in `/usr/bin/hydra_pmi_proxy': free(): invalid next size (fast): 0x0000000001c734c0 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x704fb)[0x2b4dc54914fb] /lib64/libc.so.6(+0x76976)[0x2b4dc5497976] /lib64/libc.so.6(+0x7716e)[0x2b4dc549816e] /usr/bin/hydra_pmi_proxy[0x41aa27] /usr/bin/hydra_pmi_proxy[0x418aea] /usr/bin/hydra_pmi_proxy[0x407c0c] /usr/bin/hydra_pmi_proxy[0x420bb0] /usr/bin/hydra_pmi_proxy[0x403e49] /lib64/libc.so.6(__libc_start_main+0xf1)[0x2b4dc54414e1] /usr/bin/hydra_pmi_proxy[0x404cca] ======= Memory map: ======== 00400000-0045c000 r-xp 00000000 00:10 2142949266 /afs/mpa/(a)sys/system/MPA-8.03/usr/bin/hydra_pmi_proxy 0065c000-0065d000 r--p 0005c000 00:10 2142949266 /afs/mpa/(a)sys/system/MPA-8.03/usr/bin/hydra_pmi_proxy 0065d000-0065f000 rw-p 0005d000 00:10 2142949266 /afs/mpa/(a)sys/system/MPA-8.03/usr/bin/hydra_pmi_proxy 0065f000-006ff000 rw-p 00000000 00:00 0 01c6b000-01cad000 rw-p 00000000 00:00 0 [heap] 2b4dc3efa000-2b4dc3f1d000 r-xp 00000000 00:02 4297 /lib64/ld-2.25.so 2b4dc3f1d000-2b4dc3f20000 rw-p 00000000 00:00 0 2b4dc3f3e000-2b4dc3f42000 rw-p 00000000 00:00 0 2b4dc411c000-2b4dc411d000 r--p 00022000 00:02 4297 /lib64/ld-2.25.so 2b4dc411d000-2b4dc411e000 rw-p 00023000 00:02 4297 /lib64/ld-2.25.so 2b4dc411e000-2b4dc411f000 rw-p 00000000 00:00 0 2b4dc411f000-2b4dc4138000 r-xp 00000000 00:02 5136 /lib64/libpthread-2.25.so 2b4dc4138000-2b4dc4337000 ---p 00019000 00:02 5136 /lib64/libpthread-2.25.so 2b4dc4337000-2b4dc4338000 r--p 00018000 00:02 5136 /lib64/libpthread-2.25.so 2b4dc4338000-2b4dc4339000 rw-p 00019000 00:02 5136 /lib64/libpthread-2.25.so 2b4dc4339000-2b4dc433d000 rw-p 00000000 00:00 0 2b4dc433d000-2b4dc4361000 r-xp 00000000 00:02 4295 /lib64/libudev.so.1.6.3 2b4dc4361000-2b4dc4560000 ---p 00024000 00:02 4295 /lib64/libudev.so.1.6.3 2b4dc4560000-2b4dc4561000 r--p 00023000 00:02 4295 /lib64/libudev.so.1.6.3 2b4dc4561000-2b4dc4562000 rw-p 00024000 00:02 4295 /lib64/libudev.so.1.6.3 2b4dc4562000-2b4dc456a000 r-xp 00000000 00:10 2142711818 /afs/mpa/(a)sys/system/MPA-8.03/usr/X11/lib64/libpciaccess.so.0.11.1 2b4dc456a000-2b4dc4769000 ---p 00008000 00:10 2142711818 /afs/mpa/(a)sys/system/MPA-8.03/usr/X11/lib64/libpciaccess.so.0.11.1 2b4dc4769000-2b4dc476a000 r--p 00007000 00:10 2142711818 /afs/mpa/(a)sys/system/MPA-8.03/usr/X11/lib64/libpciaccess.so.0.11.1 2b4dc476a000-2b4dc476b000 rw-p 00008000 00:10 2142711818 /afs/mpa/(a)sys/system/MPA-8.03/usr/X11/lib64/libpciaccess.so.0.11.1 2b4dc476b000-2b4dc48c5000 r-xp 00000000 00:10 2142836572 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/libxml2.so.2.9.4 2b4dc48c5000-2b4dc4ac5000 ---p 0015a000 00:10 2142836572 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/libxml2.so.2.9.4 2b4dc4ac5000-2b4dc4acd000 r--p 0015a000 00:10 2142836572 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/libxml2.so.2.9.4 2b4dc4acd000-2b4dc4acf000 rw-p 00162000 00:10 2142836572 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/libxml2.so.2.9.4 2b4dc4acf000-2b4dc4ad0000 rw-p 00000000 00:00 0 2b4dc4ad0000-2b4dc4ad3000 r-xp 00000000 00:02 5152 /lib64/libdl-2.25.so 2b4dc4ad3000-2b4dc4cd2000 ---p 00003000 00:02 5152 /lib64/libdl-2.25.so 2b4dc4cd2000-2b4dc4cd3000 r--p 00002000 00:02 5152 /lib64/libdl-2.25.so 2b4dc4cd3000-2b4dc4cd4000 rw-p 00003000 00:02 5152 /lib64/libdl-2.25.so 2b4dc4cd4000-2b4dc4ce9000 r-xp 00000000 00:10 2142836590 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/libz.so.1.2.6 2b4dc4ce9000-2b4dc4ee8000 ---p 00015000 00:10 2142836590 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/libz.so.1.2.6 2b4dc4ee8000-2b4dc4ee9000 r--p 00014000 00:10 2142836590 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/libz.so.1.2.6 2b4dc4ee9000-2b4dc4eea000 rw-p 00015000 00:10 2142836590 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/libz.so.1.2.6 2b4dc4eea000-2b4dc4f0e000 r-xp 00000000 00:10 2142835680 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/liblzma.so.5.2.3 2b4dc4f0e000-2b4dc510e000 ---p 00024000 00:10 2142835680 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/liblzma.so.5.2.3 2b4dc510e000-2b4dc510f000 r--p 00024000 00:10 2142835680 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/liblzma.so.5.2.3 2b4dc510f000-2b4dc5110000 rw-p 00025000 00:10 2142835680 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/liblzma.so.5.2.3 2b4dc5110000-2b4dc5220000 r-xp 00000000 00:02 4469 /lib64/libm-2.25.so 2b4dc5220000-2b4dc541f000 ---p 00110000 00:02 4469 /lib64/libm-2.25.so 2b4dc541f000-2b4dc5420000 r--p 0010f000 00:02 4469 /lib64/libm-2.25.so 2b4dc5420000-2b4dc5421000 rw-p 00110000 00:02 4469 /lib64/libm-2.25.so 2b4dc5421000-2b4dc55b6000 r-xp 00000000 00:02 4301 /lib64/libc-2.25.so 2b4dc55b6000-2b4dc57b5000 ---p 00195000 00:02 4301 /lib64/libc-2.25.so 2b4dc57b5000-2b4dc57b9000 r--p 00194000 00:02 4301 /lib64/libc-2.25.so 2b4dc57b9000-2b4dc57bb000 rw-p 00198000 00:02 4301 /lib64/libc-2.25.so 2b4dc57bb000-2b4dc57bf000 rw-p 00000000 00:00 0 2b4dc57bf000-2b4dc57c4000 r-xp 00000000 00:02 4321 /lib64/libnss_dns-2.25.so 2b4dc57c4000-2b4dc59c3000 ---p 00005000 00:02 4321 /lib64/libnss_dns-2.25.so 2b4dc59c3000-2b4dc59c4000 r--p 00004000 00:02 4321 /lib64/libnss_dns-2.25.so 2b4dc59c4000-2b4dc59c5000 rw-p 00005000 00:02 4321 /lib64/libnss_dns-2.25.so 2b4dc59c5000-2b4dc59d7000 r-xp 00000000 00:02 4457 /lib64/libresolv-2.25.so 2b4dc59d7000-2b4dc5bd7000 ---p 00012000 00:02 4457 /lib64/libresolv-2.25.so 2b4dc5bd7000-2b4dc5bd8000 r--p 00012000 00:02 4457 /lib64/libresolv-2.25.so 2b4dc5bd8000-2b4dc5bd9000 rw-p 00013000 00:02 4457 /lib64/libresolv-2.25.so 2b4dc5bd9000-2b4dc5bdb000 rw-p 00000000 00:00 0 2b4dc5bdb000-2b4dc5bf1000 r-xp 00000000 00:10 2142835216 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/libgcc_s.so.1 2b4dc5bf1000-2b4dc5df0000 ---p 00016000 00:10 2142835216 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/libgcc_s.so.1 2b4dc5df0000-2b4dc5df1000 r--p 00015000 00:10 2142835216 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/libgcc_s.so.1 2b4dc5df1000-2b4dc5df2000 rw-p 00016000 00:10 2142835216 /afs/mpa/(a)sys/system/MPA-8.03/usr/lib64/libgcc_s.so.1 2b4dc8000000-2b4dc8021000 rw-p 00000000 00:00 0 2b4dc8021000-2b4dcc000000 ---p 00000000 00:00 0 7fff6a554000-7fff6a575000 rw-p 00000000 00:00 0 [stack] 7fff6a58e000-7fff6a590000 r--p 00000000 00:00 0 [vvar] 7fff6a590000-7fff6a592000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] [mpiexec@pascal-3-06] control_cb (/tmp/S/mpich-3.2/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:200): assert (!closed) failed [mpiexec@pascal-3-06] HYDT_dmxu_poll_wait_for_event (/tmp/S/mpich-3.2/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [mpiexec@pascal-3-06] HYD_pmci_wait_for_completion (/tmp/S/mpich-3.2/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec@pascal-3-06] main (/tmp/S/mpich-3.2/src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion Do you have any hint which option & parameters of mpirun I have to choose to achieve my task? Kind regards, Ado Arnolds _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH process freezing without any error message or warning
by CEAM Meteorología 12 Apr '17

12 Apr '17

Hi MPICH 3.0.4 was installed in the system because other applications have been tested with this implementation. And specifically, RAMS model developers recommend mpich2-1.4.1 as they have tested before and run without any problem. How can I debug/save logs for an mpich application? Although I'm not an expert with MPICH and parallel computing I would like (I need) to solve this problem. Thanks El mié., 12 abr. 2017 a las 2:36, Guo, Yanfei (<yguo(a)anl.gov>) escribió: > Hi, > > MPICH 3.0.4 is a quite old version. Can you try your application with the > latest MPICH (3.2) http://www.mpich.org/downloads/ ? > > Yanfei Guo > Postdoctoral Researcher > MCS Division, ANL > > > On 4/11/17, 5:57 AM, "CEAM Meteorología" <ceamet(a)gmail.com> wrote: > > Hi MPICH users > > > > I am trying to run a meteorological simulation with RAMS model ( > http://vandenheever.atmos.colostate.edu/vdhpage/rams.php) < > http://vandenheever.atmos.colostate.edu/vdhpage/rams.php%29> in a new > cluster with CentOs7 in all nodes. We have other > applications running fine in the cluster, sending processes to each > node,... everything seemd to run fine. > > > > But the RAMS model freezes at its first stage on the parallel run. > RAMS developers recommend using mpich2-1.4.1 as they have successfully > tested. If I try to run a parallel simulation in the master node it runs > fine and starts the required number of processes. > If I try to use the other nodes the simulation freezes, do not stop > with any error message; RAMS model processes appear in all used nodes and > in the master node but any output is created and the usual status messages > from RAMS do not appear on screen. > > > > The commnad line to run the model is > > [paco@Llamp RUN]$ time ../misc/mpich2-1.4.1/bin/mpirun -verbose > -machinefile mpd.hosts -n 20 ./rams-6.2.03 -f RAMSIN > > > > I have also compiled and tried to run with mpich 3.0.4 but it behaves > exactly the same way. > > > > Log messages with -verbose option for both mpich2-1.4.1 and mpich3.0.4 > can be found at > > > > MPICH2-1.4.1: > https://www.dropbox.com/s/6sgkarmsi5vrdfd/RAMS-mpich2-1.4.1.log?dl=0 < > https://www.dropbox.com/s/6sgkarmsi5vrdfd/RAMS-mpich2-1.4.1.log?dl=0> > > MPICH3.0.4: > https://www.dropbox.com/s/bxvl5q6dy03pgew/RAMS-mpich2-3.0.4.log?dl=0 < > https://www.dropbox.com/s/bxvl5q6dy03pgew/RAMS-mpich2-3.0.4.log?dl=0> > > > > cpi example runs fine with both mpich instances > > > > Thanks in advance for your help and best regards > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH process freezing without any error message or warning
by Guo, Yanfei 11 Apr '17

11 Apr '17

Hi, MPICH 3.0.4 is a quite old version. Can you try your application with the latest MPICH (3.2) http://www.mpich.org/downloads/ ? Yanfei Guo Postdoctoral Researcher MCS Division, ANL On 4/11/17, 5:57 AM, "CEAM Meteorología" <ceamet(a)gmail.com> wrote: Hi MPICH users I am trying to run a meteorological simulation with RAMS model (http://vandenheever.atmos.colostate.edu/vdhpage/rams.php) <http://vandenheever.atmos.colostate.edu/vdhpage/rams.php%29> in a new cluster with CentOs7 in all nodes. We have other applications running fine in the cluster, sending processes to each node,... everything seemd to run fine. But the RAMS model freezes at its first stage on the parallel run. RAMS developers recommend using mpich2-1.4.1 as they have successfully tested. If I try to run a parallel simulation in the master node it runs fine and starts the required number of processes. If I try to use the other nodes the simulation freezes, do not stop with any error message; RAMS model processes appear in all used nodes and in the master node but any output is created and the usual status messages from RAMS do not appear on screen. The commnad line to run the model is [paco@Llamp RUN]$ time ../misc/mpich2-1.4.1/bin/mpirun -verbose -machinefile mpd.hosts -n 20 ./rams-6.2.03 -f RAMSIN I have also compiled and tried to run with mpich 3.0.4 but it behaves exactly the same way. Log messages with -verbose option for both mpich2-1.4.1 and mpich3.0.4 can be found at MPICH2-1.4.1: https://www.dropbox.com/s/6sgkarmsi5vrdfd/RAMS-mpich2-1.4.1.log?dl=0 <https://www.dropbox.com/s/6sgkarmsi5vrdfd/RAMS-mpich2-1.4.1.log?dl=0> MPICH3.0.4: https://www.dropbox.com/s/bxvl5q6dy03pgew/RAMS-mpich2-3.0.4.log?dl=0 <https://www.dropbox.com/s/bxvl5q6dy03pgew/RAMS-mpich2-3.0.4.log?dl=0> cpi example runs fine with both mpich instances Thanks in advance for your help and best regards _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPICH process freezing without any error message or warning
by CEAM Meteorología 11 Apr '17

11 Apr '17

Hi MPICH users I am trying to run a meteorological simulation with RAMS model ( http://vandenheever.atmos.colostate.edu/vdhpage/rams.php) in a new cluster with CentOs7 in all nodes. We have other applications running fine in the cluster, sending processes to each node,... everything seemd to run fine. But the RAMS model freezes at its first stage on the parallel run. RAMS developers recommend using mpich2-1.4.1 as they have successfully tested. If I try to run a parallel simulation in the master node it runs fine and starts the required number of processes. If I try to use the other nodes the simulation freezes, do not stop with any error message; RAMS model processes appear in all used nodes and in the master node but any output is created and the usual status messages from RAMS do not appear on screen. The commnad line to run the model is [paco@Llamp RUN]$ time ../misc/mpich2-1.4.1/bin/mpirun -verbose -machinefile mpd.hosts -n 20 ./rams-6.2.03 -f RAMSIN I have also compiled and tried to run with mpich 3.0.4 but it behaves exactly the same way. Log messages with -verbose option for both mpich2-1.4.1 and mpich3.0.4 can be found at MPICH2-1.4.1: https://www.dropbox.com/s/6sgkarmsi5vrdfd/RAMS-mpich2-1.4.1.log?dl=0 MPICH3.0.4: https://www.dropbox.com/s/bxvl5q6dy03pgew/RAMS-mpich2-3.0.4.log?dl=0 cpi example runs fine with both mpich instances Thanks in advance for your help and best regards _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPICH with IPv6
by Santiago Lambrisca 03 Apr '17

03 Apr '17

Hello, I am trying to use MPICH to connect several hosts through IPv6, and got some problems. I am not sure if it's supported, and could not find any documentation talking about that. I would like to know if its supported, and if any configuration is required to get it working. Thanks in advance, Santiago. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0