May 2016 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] How do I use the HYDRA_BINDING environmental variable?
by Kenneth Raffenetti 24 May '16

24 May '16

John, Just pushed a patch to mpich.git to fix this particular issue. http://git.mpich.org/mpich.git/commit/a131e534d26ea79ba028435b2a162e14b5c77… Ken On 05/24/2016 09:04 AM, Kenneth Raffenetti wrote: > Hi John, > > This looks like a bug. Thanks for reporting. I'll put together a fix > shortly. > > Ken > > On 05/24/2016 08:05 AM, Low, John J. wrote: >> Mpich discussion group, >> >> I would like to use hydra environment variables to bind threads to >> cores. I am using hydra 3.2 with mvapich2 version 2.2b_psm built with >> intel 15.0. >> >> I have a simple program to print out the cpu affinity for each process >> which I have used to test process binding. I have included the output >> of this program below. >> >> [jlow@blogin2 hybrid_hello]$ HYDRA_BINDING=socket mpirun -np 4 a.out >> Hello from node blogin2, core 0-15; AKA rank 3, thread 0 >> Hello from node blogin2, core 0-15; AKA rank 0, thread 0 >> Hello from node blogin2, core 0-15; AKA rank 2, thread 0 >> Hello from node blogin2, core 0-15; AKA rank 1, thread 0 >> [jlow@blogin2 hybrid_hello]$ mpirun -bind-to core:2 -np 4 a.out >> Hello from node blogin2, core 4,5; AKA rank 2, thread 0 >> Hello from node blogin2, core 6,7; AKA rank 3, thread 0 >> Hello from node blogin2, core 0,1; AKA rank 0, thread 0 >> Hello from node blogin2, core 2,3; AKA rank 1, thread 0 >> [jlow@blogin2 hybrid_hello]$ mpirun -bind-to socket -np 4 a.out >> Hello from node blogin2, core 8-15; AKA rank 3, thread 0 >> Hello from node blogin2, core 0-7; AKA rank 0, thread 0 >> Hello from node blogin2, core 0-7; AKA rank 2, thread 0 >> Hello from node blogin2, core 8-15; AKA rank 1, thread 0 >> [jlow@blogin2 hybrid_hello]$ HYDRA_BINDING=socket mpirun -np 4 a.out >> Hello from node blogin2, core 0-15; AKA rank 3, thread 0 >> Hello from node blogin2, core 0-15; AKA rank 1, thread 0 >> Hello from node blogin2, core 0-15; AKA rank 2, thread 0 >> Hello from node blogin2, core 0-15; AKA rank 0, thread 0 >> >> It appears that hydra is ignoring my environmental variables. How can I >> use environmental variables to bind processes to cores with hydra? >> >> Thanks, >> >> John J. Low >> Principal Computational Science Specialist >> Computing, Environment and Life Sciences >> Building 240, 2143 >> 9700 South Cass Avenue >> Argonne National Laboratory >> Argonne, IL 60439. >> 630-252-0045 >> www.linkedin.com/pub/john-low/15/8b0/5aa/ >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] How do I use the HYDRA_BINDING environmental variable?
by Kenneth Raffenetti 24 May '16

24 May '16

Hi John, This looks like a bug. Thanks for reporting. I'll put together a fix shortly. Ken On 05/24/2016 08:05 AM, Low, John J. wrote: > Mpich discussion group, > > I would like to use hydra environment variables to bind threads to > cores. I am using hydra 3.2 with mvapich2 version 2.2b_psm built with > intel 15.0. > > I have a simple program to print out the cpu affinity for each process > which I have used to test process binding. I have included the output > of this program below. > > [jlow@blogin2 hybrid_hello]$ HYDRA_BINDING=socket mpirun -np 4 a.out > Hello from node blogin2, core 0-15; AKA rank 3, thread 0 > Hello from node blogin2, core 0-15; AKA rank 0, thread 0 > Hello from node blogin2, core 0-15; AKA rank 2, thread 0 > Hello from node blogin2, core 0-15; AKA rank 1, thread 0 > [jlow@blogin2 hybrid_hello]$ mpirun -bind-to core:2 -np 4 a.out > Hello from node blogin2, core 4,5; AKA rank 2, thread 0 > Hello from node blogin2, core 6,7; AKA rank 3, thread 0 > Hello from node blogin2, core 0,1; AKA rank 0, thread 0 > Hello from node blogin2, core 2,3; AKA rank 1, thread 0 > [jlow@blogin2 hybrid_hello]$ mpirun -bind-to socket -np 4 a.out > Hello from node blogin2, core 8-15; AKA rank 3, thread 0 > Hello from node blogin2, core 0-7; AKA rank 0, thread 0 > Hello from node blogin2, core 0-7; AKA rank 2, thread 0 > Hello from node blogin2, core 8-15; AKA rank 1, thread 0 > [jlow@blogin2 hybrid_hello]$ HYDRA_BINDING=socket mpirun -np 4 a.out > Hello from node blogin2, core 0-15; AKA rank 3, thread 0 > Hello from node blogin2, core 0-15; AKA rank 1, thread 0 > Hello from node blogin2, core 0-15; AKA rank 2, thread 0 > Hello from node blogin2, core 0-15; AKA rank 0, thread 0 > > It appears that hydra is ignoring my environmental variables. How can I > use environmental variables to bind processes to cores with hydra? > > Thanks, > > John J. Low > Principal Computational Science Specialist > Computing, Environment and Life Sciences > Building 240, 2143 > 9700 South Cass Avenue > Argonne National Laboratory > Argonne, IL 60439. > 630-252-0045 > www.linkedin.com/pub/john-low/15/8b0/5aa/ > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

How do I use the HYDRA_BINDING environmental variable?
by Low, John J. 24 May '16

24 May '16

Mpich discussion group, I would like to use hydra environment variables to bind threads to cores. I am using hydra 3.2 with mvapich2 version 2.2b_psm built with intel 15.0. I have a simple program to print out the cpu affinity for each process which I have used to test process binding. I have included the output of this program below. [jlow@blogin2 hybrid_hello]$ HYDRA_BINDING=socket mpirun -np 4 a.out Hello from node blogin2, core 0-15; AKA rank 3, thread 0 Hello from node blogin2, core 0-15; AKA rank 0, thread 0 Hello from node blogin2, core 0-15; AKA rank 2, thread 0 Hello from node blogin2, core 0-15; AKA rank 1, thread 0 [jlow@blogin2 hybrid_hello]$ mpirun -bind-to core:2 -np 4 a.out Hello from node blogin2, core 4,5; AKA rank 2, thread 0 Hello from node blogin2, core 6,7; AKA rank 3, thread 0 Hello from node blogin2, core 0,1; AKA rank 0, thread 0 Hello from node blogin2, core 2,3; AKA rank 1, thread 0 [jlow@blogin2 hybrid_hello]$ mpirun -bind-to socket -np 4 a.out Hello from node blogin2, core 8-15; AKA rank 3, thread 0 Hello from node blogin2, core 0-7; AKA rank 0, thread 0 Hello from node blogin2, core 0-7; AKA rank 2, thread 0 Hello from node blogin2, core 8-15; AKA rank 1, thread 0 [jlow@blogin2 hybrid_hello]$ HYDRA_BINDING=socket mpirun -np 4 a.out Hello from node blogin2, core 0-15; AKA rank 3, thread 0 Hello from node blogin2, core 0-15; AKA rank 1, thread 0 Hello from node blogin2, core 0-15; AKA rank 2, thread 0 Hello from node blogin2, core 0-15; AKA rank 0, thread 0 It appears that hydra is ignoring my environmental variables. How can I use environmental variables to bind processes to cores with hydra? Thanks, John J. Low Principal Computational Science Specialist Computing, Environment and Life Sciences Building 240, 2143 9700 South Cass Avenue Argonne National Laboratory Argonne, IL 60439. 630-252-0045 www.linkedin.com/pub/john-low/15/8b0/5aa/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Open MPI does not work when MPICH or intel MPI are installed
by Jeff Squyres (jsquyres) 23 May '16

23 May '16

FWIW, this question was answered on the Open MPI list: https://www.open-mpi.org/community/lists/users/2016/05/29280.php Hope that helps. > On May 23, 2016, at 7:58 AM, Megdich Islem <megdich_islem(a)yahoo.fr> wrote: > > Hi, > > I am using 2 software, one is called Open Foam and the other called EMPIRE that need to run together at the same time. > Open Foam uses Open MPI implementation and EMPIRE uses either MPICH or intel mpi. > The version of Open MPI that comes with Open Foam is 1.6.5. > I am using Intel (R) MPI Library for linux * OS, version 5.1.3 and MPICH 3.0.4. > > My problem is when I have the environment variables of either mpich or Intel MPI sourced to bashrc, I fail to run a case of Open Foam with parallel processing ( You find attached a picture of the error I got ) > This is an example of a command line I use to run Open Foam > mpirun -np 4 interFoam -parallel > > Once I keep the environment variable of OpenFoam only, the parallel processing works without any problem, so I won't be able to run EMPIRE. > > I am sourcing the environment variables in this way: > > For Open Foam: > source /opt/openfoam30/etc/bashrc > > For MPICH 3.0.4 > > export PATH=/home/islem/Desktop/mpich/bin:$PATH > export LD_LIBRARY_PATH="/home/islem/Desktop/mpich/lib/:$LD_LIBRARY_PATH" > export MPICH_F90=gfortran > export MPICH_CC=/opt/intel/bin/icc > export MPICH_CXX=/opt/intel/bin/icpc > export MPICH-LINK_CXX="-L/home/islem/Desktop/mpich/lib/ -Wl,-rpath -Wl,/home/islem/Desktop/mpich/lib -lmpichcxx -lmpich -lopa -lmpl -lrt -lpthread" > > For intel > > export PATH=$PATH:/opt/intel/bin/ > LD_LIBRARY_PATH="/opt/intel/lib/intel64:$LD_LIBRARY_PATH" > export LD_LIBRARY_PATH > source /opt/intel/compilers_and_libraries_2016.3.210/linux/mpi/intel64/bin/mpivars.sh intel64 > > If Only Open Foam is sourced, mpirun --version gives OPEN MPI (1.6.5) > If Open Foam and MPICH are sourced, mpirun --version gives mpich 3.0.1 > If Open Foam and intel MPI are sourced, mpirun --version gives intel (R) MPI libarary for linux, version 5.1.3 > > My question is why I can't have two MPI implementation installed and sourced together. How can I solve the problem ? > > Regards, > Islem Megdiche > <OF-error.png>_______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Open MPI does not work when MPICH or intel MPI are installed
by Megdich Islem 23 May '16

23 May '16

Hi, I am using 2 software, one is called Open Foam and the other called EMPIRE that need to run together at the same time.Open Foam uses Open MPI implementation and EMPIRE uses either MPICH or intel mpi.The version of Open MPI that comes with Open Foam is 1.6.5.I am using Intel (R) MPI Library for linux * OS, version 5.1.3 and MPICH 3.0.4. My problem is when I have the environment variables of either mpich or Intel MPI sourced to bashrc, I fail to run a case of Open Foam with parallel processing ( You find attached a picture of the error I got ) This is an example of a command line I use to run Open Foammpirun -np 4 interFoam -parallel Once I keep the environment variable of OpenFoam only, the parallel processing works without any problem, so I won't be able to run EMPIRE. I am sourcing the environment variables in this way: For Open Foam:source /opt/openfoam30/etc/bashrc For MPICH 3.0.4 export PATH=/home/islem/Desktop/mpich/bin:$PATHexport LD_LIBRARY_PATH="/home/islem/Desktop/mpich/lib/:$LD_LIBRARY_PATH"export MPICH_F90=gfortranexport MPICH_CC=/opt/intel/bin/iccexport MPICH_CXX=/opt/intel/bin/icpcexport MPICH-LINK_CXX="-L/home/islem/Desktop/mpich/lib/ -Wl,-rpath -Wl,/home/islem/Desktop/mpich/lib -lmpichcxx -lmpich -lopa -lmpl -lrt -lpthread" For intel export PATH=$PATH:/opt/intel/bin/LD_LIBRARY_PATH="/opt/intel/lib/intel64:$LD_LIBRARY_PATH"export LD_LIBRARY_PATHsource /opt/intel/compilers_and_libraries_2016.3.210/linux/mpi/intel64/bin/mpivars.sh intel64 If Only Open Foam is sourced, mpirun --version gives OPEN MPI (1.6.5)If Open Foam and MPICH are sourced, mpirun --version gives mpich 3.0.1If Open Foam and intel MPI are sourced, mpirun --version gives intel (R) MPI libarary for linux, version 5.1.3 My question is why I can't have two MPI implementation installed and sourced together. How can I solve the problem ? Regards,Islem Megdiche _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] High school project on distributed computing
by Oden, Lena 19 May '16

19 May '16

Hi Alex. one Question - additional to Yankees question: Did you ensure to use the same overall problem size? In the example code, the problem sizes increases linear with the number of processes - in this case, a single process still has to do the same amount of work - additional to the communication. Than, of course, you will not see any speedup. Lena On May 19, 2016, at 12:33 PM, Guo, Yanfei <yguo(a)anl.gov<mailto:[email protected]>> wrote: Hi Alex, On 5/18/16, 6:26 PM, "Alexandre Vieira" <nullpt(a)gmail.com<mailto:[email protected]>> wrote: Hello list, I'm currently writing a paper on distributed computing and also elaborate a simple use case to include as a proof of concept. My setup: - 3 VMs running on different machines (same HW) with 1 CPU and 2GB RAM running FreeBSD 10.3 - Connected through a 1Gbps LAN and communicate passwordless through SSH. - MPICH 3.2_1 installed and working on the three nodes I've tried to compile and run some code samples that are available for mpich (i.e: http://people.sc.fsu.edu/~jburkardt/c_src/mpi/search_mpi.c) but I always end up getting worst times running on the 3 nodes that I get running on 1 node alone, even when I can get all 3 nodes to get up to a 100% CPU usage for a couple of minutes. How much slower? When you say running on 1 node, do you mean all three VMs are on the same physical machine, or just running a serial implementation of the program? How many CPU cores does the physical machine have? I know there is network overhead and SSH isn't helping either, so without investing too much time I unfortunately won't have, I'd like to ask you what kind of simple program could I write just for the sake of a proof of concept that running a program with 3 nodes would be faster than running on one. Network latency could be a problem because the communication goes through memory when all processes/VMs are on the same physical machine, which is way faster then Ethernet. Maybe dealing with slower operations like disk I/O do counter the inter-node communication? It is unlikely that disk I/O has something to do with this. Many thanks for your kind help BR Alex Yanfei Guo Postdoctoral Researcher MCS Division, ANL _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] High school project on distributed computing
by Guo, Yanfei 19 May '16

19 May '16

Hi Alex, On 5/18/16, 6:26 PM, "Alexandre Vieira" <nullpt(a)gmail.com> wrote: >Hello list, > > >I'm currently writing a paper on distributed computing and also elaborate a simple use case to include as a proof of concept. > > >My setup: > > >- 3 VMs running on different machines (same HW) with 1 CPU and 2GB RAM running FreeBSD 10.3 >- Connected through a 1Gbps LAN and communicate passwordless through SSH. > >- MPICH 3.2_1 installed and working on the three nodes > > >I've tried to compile and run some code samples that are available for mpich (i.e: >http://people.sc.fsu.edu/~jburkardt/c_src/mpi/search_mpi.c) but I always end up getting worst times running on the 3 nodes that I get running on 1 node alone, even when I can get all > 3 nodes to get up to a 100% CPU usage for a couple of minutes. > How much slower? When you say running on 1 node, do you mean all three VMs are on the same physical machine, or just running a serial implementation of the program? How many CPU cores does the physical machine have? > >I know there is network overhead and SSH isn't helping either, so without investing too much time I unfortunately won't have, I'd like to ask you what kind of simple program could I write just for the sake of a proof of concept > that running a program with 3 nodes would be faster than running on one. > Network latency could be a problem because the communication goes through memory when all processes/VMs are on the same physical machine, which is way faster then Ethernet. > >Maybe dealing with slower operations like disk I/O do counter the inter-node communication? > It is unlikely that disk I/O has something to do with this. > >Many thanks for your kind help > > >BR >Alex > > Yanfei Guo Postdoctoral Researcher MCS Division, ANL _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

High school project on distributed computing
by Alexandre Vieira 18 May '16

18 May '16

Hello list, I'm currently writing a paper on distributed computing and also elaborate a simple use case to include as a proof of concept. My setup: - 3 VMs running on different machines (same HW) with 1 CPU and 2GB RAM running FreeBSD 10.3 - Connected through a 1Gbps LAN and communicate passwordless through SSH. - MPICH 3.2_1 installed and working on the three nodes I've tried to compile and run some code samples that are available for mpich (i.e: http://people.sc.fsu.edu/~jburkardt/c_src/mpi/search_mpi.c) but I always end up getting worst times running on the 3 nodes that I get running on 1 node alone, even when I can get all 3 nodes to get up to a 100% CPU usage for a couple of minutes. I know there is network overhead and SSH isn't helping either, so without investing too much time I unfortunately won't have, I'd like to ask you what kind of simple program could I write just for the sake of a proof of concept that running a program with 3 nodes would be faster than running on one. Maybe dealing with slower operations like disk I/O do counter the inter-node communication? Many thanks for your kind help BR Alex _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_Comm_connect issue
by Min Si 12 May '16

12 May '16

Hi Hirak, In your scenario, the hanging on the client side is not a bug (but yes, it is not user-friendly). Here is the description about MPI_Comm_connect in the MPI 3.0 standard. > > If the named port does not exist (or has been closed), > MPI_COMM_CONNECT raises an error of class MPI_ERR_PORT. > > run_good in your example. > > If the port exists, but does not have a pending MPI_COMM_ACCEPT, the > connection attempt will eventually time out after an > implementation-defined time, or succeed when the server calls > MPI_COMM_ACCEPT. In the case of a time out, MPI_COMM_CONNECT raises an > error of class MPI_ERR_PORT. > MPI: A Message-Passing Interface Stand > This is the run_bad case, the "eventually time out" is implementation dependent : A Message-Passing Interface Standard . However, I just glanced over the code, it seems MPICH currently does not handle user-specified timeout through the info argument. I will look into code and keep you updated. Min On 5/11/16 11:39 PM, Roy, Hirak wrote: > > Hi mpich team, > > I am using MPICH 3.0.4 sock with dynamic connection (accept/connect). > > I am facing issues with the following scenario, when there is a race > condition in MPI_Comm_connect. > > Here is the case > > 1>Server opens a port, writes the port info in a file, calls > MPI_Comm_accept > > 2>Client1 & Client2 reads the port information from file and each > client calls MPI_Comm_connect > > 3>Server accepts one client, disconnects, closes the port, exit > > 4>One of the client successfully connects, disconnects, exit > > 5>The other client hangs in MPI_Comm_connect > > If you do the following steps, you can reproduce the issue > > 1>Please set compiler and installation path of MPICH in makefile > > 2>make : compiles > > 3>make run_bad > > I have noticed that in case client calls MPI_Comm_connect after the > port is closed, the connect calls successfully terminates with error > (make run_good). > > Please let me know whether it is a bug or not. > > Please also let me know if the connect call can be configured with a > timeout or not. > > Thanks, > > Hirak > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPI_Comm_connect issue
by Roy, Hirak 11 May '16

11 May '16

Hi mpich team, I am using MPICH 3.0.4 sock with dynamic connection (accept/connect). I am facing issues with the following scenario, when there is a race condition in MPI_Comm_connect. Here is the case 1> Server opens a port, writes the port info in a file, calls MPI_Comm_accept 2> Client1 & Client2 reads the port information from file and each client calls MPI_Comm_connect 3> Server accepts one client, disconnects, closes the port, exit 4> One of the client successfully connects, disconnects, exit 5> The other client hangs in MPI_Comm_connect If you do the following steps, you can reproduce the issue 1> Please set compiler and installation path of MPICH in makefile 2> make : compiles 3> make run_bad I have noticed that in case client calls MPI_Comm_connect after the port is closed, the connect calls successfully terminates with error (make run_good). Please let me know whether it is a bug or not. Please also let me know if the connect call can be configured with a timeout or not. Thanks, Hirak _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0