September 2013 - discuss - mailman3.cels.anl.gov

affinity problems with mpiexec.hydra 3.0.4
by Bill Ryder 16 Sep '13

16 Sep '13

Greetings all, I'm trying set affinity for mybrid MPI/OpenMP tasks. I want to run two processes on a host, and give one socket to one rank, and the other socket to the other rank. I have two types of hardware - one works perfectly - the other doesn't. I first saw the problem using mpiexec.hydra with slurm but I've moved to using ssh to remove some possible variables I have a trivial script which just greps out /proc/$$/status for the Cpus_allowed mask and Cpus_allowed_list It's just: echo "`hostname` $PMI_RANK `grep Cpus_allowed /proc/$$/status`" On the machine that is doing what I want I get: mpiexec.hydra -ppn 2 -hosts abrams201a --bind-to socket -launcher ssh ./get_mapping abrams201a 0 Cpus_allowed: 00555555 Cpus_allowed_list: 0,2,4,6,8,10,12,14,16,18,20,22 abrams201a 1 Cpus_allowed: 00aaaaaa Cpus_allowed_list: 1,3,5,7,9,11,13,15,17,19,21,23 rank 0 gets one socket, rank 1 gets the other socket. This is what I want But on my other machine with a different topology I get this: mpiexec.hydra -ppn 2 -hosts jericho101 --bind-to socket -launcher ssh ./get_mapping jericho101 0 Cpus_allowed: 00000000,00ff00ff Cpus_allowed_list: 0-7,16-23 jericho101 1 Cpus_allowed: 00000000,00ff00ff Cpus_allowed_list: 0-7,16-23 So each rank is trying to use the same socket. Similarly if I try to bind to a numanode mpiexec.hydra -ppn 2 -hosts jericho101 --bind-to numa -launcher ssh ./get_mapping jericho101 1 Cpus_allowed: 00000000,00ff00ff Cpus_allowed_list: 0-7,16-23 jericho101 0 Cpus_allowed: 00000000,00ff00ff Cpus_allowed_list: 0-7,16-23 Or even if I send numa:2 mpiexec.hydra -ppn 2 -hosts jericho101 --bind-to numa:2 -launcher ssh ./get_mapping jericho101 0 Cpus_allowed: 00000000,00ff00ff Cpus_allowed_list: 0-7,16-23 jericho101 1 Cpus_allowed: 00000000,00ff00ff Cpus_allowed_list: 0-7,16-23 So once again instead of handing a numa node to each process - it's handing the same node to both. How would I start debugging this? Or am I missing something really obvious Thanks! --------- Bill Ryder Weta Digital A bit more data: mpiexec.hydra --info HYDRA build details: Version: 3.0.4 Release Date: Wed Apr 24 10:08:10 CDT 2013 CC: cc CXX: F77: F90: Configure options: '--prefix=/tech/apps/mpich/hydra' Process Manager: pmi Launchers available: ssh rsh fork slurm ll lsf sge manual persist Topology libraries available: hwloc Resource management kernels available: user slurm ll lsf sge pbs cobalt Checkpointing libraries available: Demux engines available: poll select I have hwloc 1.3.1 installed locally on each machine abrams201a looks like: Machine (48GB) NUMANode L#0 (P#1 24GB) + Socket L#0 + L3 L#0 (12MB) L2 L#0 (256KB) + L1 L#0 (32KB) + Core L#0 PU L#0 (P#0) PU L#1 (P#12) L2 L#1 (256KB) + L1 L#1 (32KB) + Core L#1 PU L#2 (P#2) PU L#3 (P#14) L2 L#2 (256KB) + L1 L#2 (32KB) + Core L#2 PU L#4 (P#4) PU L#5 (P#16) L2 L#3 (256KB) + L1 L#3 (32KB) + Core L#3 PU L#6 (P#6) PU L#7 (P#18) L2 L#4 (256KB) + L1 L#4 (32KB) + Core L#4 PU L#8 (P#8) PU L#9 (P#20) L2 L#5 (256KB) + L1 L#5 (32KB) + Core L#5 PU L#10 (P#10) PU L#11 (P#22) NUMANode L#1 (P#0 24GB) + Socket L#1 + L3 L#1 (12MB) L2 L#6 (256KB) + L1 L#6 (32KB) + Core L#6 PU L#12 (P#1) PU L#13 (P#13) L2 L#7 (256KB) + L1 L#7 (32KB) + Core L#7 PU L#14 (P#3) PU L#15 (P#15) L2 L#8 (256KB) + L1 L#8 (32KB) + Core L#8 PU L#16 (P#5) PU L#17 (P#17) L2 L#9 (256KB) + L1 L#9 (32KB) + Core L#9 PU L#18 (P#7) PU L#19 (P#19) L2 L#10 (256KB) + L1 L#10 (32KB) + Core L#10 PU L#20 (P#9) PU L#21 (P#21) L2 L#11 (256KB) + L1 L#11 (32KB) + Core L#11 PU L#22 (P#11) PU L#23 (P#23) HostBridge L#0 PCIBridge PCI 8086:10e7 Net L#0 "eth0" PCI 8086:10e7 Net L#1 "eth1" PCIBridge PCI 15b3:6746 Net L#2 "eth2" OpenFabrics L#3 "mlx4_0" PCI 15b3:6746 PCI 15b3:6746 PCI 15b3:6746 PCIBridge PCI 102b:0533 PCI 8086:3a20 Block L#4 "sda" And jericho101 looks like: achine (96GB) NUMANode L#0 (P#0 48GB) Socket L#0 + L3 L#0 (20MB) L2 L#0 (256KB) + L1 L#0 (32KB) + Core L#0 PU L#0 (P#0) PU L#1 (P#16) L2 L#1 (256KB) + L1 L#1 (32KB) + Core L#1 PU L#2 (P#1) PU L#3 (P#17) L2 L#2 (256KB) + L1 L#2 (32KB) + Core L#2 PU L#4 (P#2) PU L#5 (P#18) L2 L#3 (256KB) + L1 L#3 (32KB) + Core L#3 PU L#6 (P#3) PU L#7 (P#19) L2 L#4 (256KB) + L1 L#4 (32KB) + Core L#4 PU L#8 (P#4) PU L#9 (P#20) L2 L#5 (256KB) + L1 L#5 (32KB) + Core L#5 PU L#10 (P#5) PU L#11 (P#21) L2 L#6 (256KB) + L1 L#6 (32KB) + Core L#6 PU L#12 (P#6) PU L#13 (P#22) L2 L#7 (256KB) + L1 L#7 (32KB) + Core L#7 PU L#14 (P#7) PU L#15 (P#23) HostBridge L#0 PCIBridge PCI 14e4:168e Net L#0 "eth0" PCI 14e4:168e Net L#1 "eth1" PCI 14e4:168e Net L#2 "eth2" PCI 14e4:168e Net L#3 "eth3" PCI 14e4:168e Net L#4 "eth4" PCI 14e4:168e Net L#5 "eth5" PCI 14e4:168e Net L#6 "eth6" PCI 14e4:168e Net L#7 "eth7" PCIBridge PCI 103c:323b Block L#8 "sda" PCIBridge PCI 102b:0533 NUMANode L#1 (P#1 48GB) + Socket L#1 + L3 L#1 (20MB) L2 L#8 (256KB) + L1 L#8 (32KB) + Core L#8 PU L#16 (P#8) PU L#17 (P#24) L2 L#9 (256KB) + L1 L#9 (32KB) + Core L#9 PU L#18 (P#9) PU L#19 (P#25) L2 L#10 (256KB) + L1 L#10 (32KB) + Core L#10 PU L#20 (P#10) PU L#21 (P#26) L2 L#11 (256KB) + L1 L#11 (32KB) + Core L#11 PU L#22 (P#11) PU L#23 (P#27) L2 L#12 (256KB) + L1 L#12 (32KB) + Core L#12 PU L#24 (P#12) PU L#25 (P#28) L2 L#13 (256KB) + L1 L#13 (32KB) + Core L#13 PU L#26 (P#13) PU L#27 (P#29) L2 L#14 (256KB) + L1 L#14 (32KB) + Core L#14 PU L#28 (P#14) PU L#29 (P#30) L2 L#15 (256KB) + L1 L#15 (32KB) + Core L#15 PU L#30 (P#15) PU L#31 (P#31)

1 0

Re: [mpich-discuss] In MPI for multiple process scanf taking input only once
by Pavan Balaji 15 Sep '13

15 Sep '13

stdin is only forwarded to rank 0. In any case, reading from stdin is a bad idea with a capital VERY, and is not supported by the standard. We only support that for legacy applications. -- Pavan On Sep 15, 2013, at 4:48 AM, Mahesh Doijade wrote: > I am trying to write MPI code with scanf which will take input for all process individually, but only one process taking the input from user and others assign garbage value to that variable. The program is as below > > #include <stdlib.h> > #include <stdio.h> > #include "mpi.h" > #include<string.h> > > > > int main(int argc, char* argv[]) > > > { > int i, size, rank; > int arr; > > MPI_Init > (&argc, &argv); > > MPI_Comm_size > (MPI_COMM_WORLD, &size); > > MPI_Comm_rank > (MPI_COMM_WORLD, &rank); > > printf > ("Enter the number\n"); > > scanf > ("%d",&i); > > > printf > ("%d\n",i); > > MPI_Finalize > (); > > exit > (0); > } > > > -- > > Regards, > -- Mahesh Doijade > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] In MPI for multiple process scanf taking input only once
by Jeff Hammond 15 Sep '13

15 Sep '13

Just read all variables from a file. I don't see how you can read from stdin on every proc anyways. Jeff Sent from my iPhone On Sep 15, 2013, at 4:48 AM, Mahesh Doijade <maheshdoijade(a)gmail.com> wrote: I am trying to write MPI code with scanf which will take input for all process individually, but only one process taking the input from user and others assign garbage value to that variable. The program is as below #include <stdlib.h>#include <stdio.h>#include "mpi.h"#include<string.h> int main(int argc, char* argv[]) {int i, size, rank;int arr; MPI_Init(&argc, &argv); MPI_Comm_size(MPI_COMM_WORLD, &size); MPI_Comm_rank(MPI_COMM_WORLD, &rank); printf("Enter the number\n"); scanf("%d",&i); printf("%d\n",i); MPI_Finalize(); exit(0);} -- Regards, -- Mahesh Doijade _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

In MPI for multiple process scanf taking input only once
by Mahesh Doijade 15 Sep '13

15 Sep '13

I am trying to write MPI code with scanf which will take input for all process individually, but only one process taking the input from user and others assign garbage value to that variable. The program is as below #include <stdlib.h>#include <stdio.h>#include "mpi.h"#include<string.h> int main(int argc, char* argv[]) {int i, size, rank;int arr; MPI_Init(&argc, &argv); MPI_Comm_size(MPI_COMM_WORLD, &size); MPI_Comm_rank(MPI_COMM_WORLD, &rank); printf("Enter the number\n"); scanf("%d",&i); printf("%d\n",i); MPI_Finalize(); exit(0);} -- Regards, -- Mahesh Doijade

1 0

Re: [mpich-discuss] Fortran/C configuration issue with MPICH2
by Pavan Balaji 14 Sep '13

14 Sep '13

Your gcc is 32-bit and ifort is 64-bit. They don't work well together. Install a 64-bit gcc or use a different C compiler (e.g., icc). -- Pavan On Sep 14, 2013, at 8:33 PM, sunyeping wrote: > Dear all, > > I am trying to install mpich2-1.4.1p1 onto my centos 5.9 ssystem. I > > used the following command: > > > > ./configure CFLAGS="-fPIC" CXXFLAGS="-fPIC" FFLAGS="-fPIC" > > --enable-shared--prefix=/opt/mpich2 > > > > However, I receive error message: > > > > configure: error: **** Incompatible Fortran and C Object File > > Types! ****F77 Object File Type produced by "ifort -fPIC " is : : > > ERROR: cannot open `' (No such file or directory). C Object File > > Type produced by "gcc -fPIC -O2" is : : ELF 64-bit LSB > > relocatable, AMD x86-64, version 1 (SYSV), not stripped.What does > > that mean and how to resolve the problem? > > Institute of Microbiology, Chinese Academy of Sciences > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] source code query
by Rayson Ho 14 Sep '13

14 Sep '13

Hi Volkan, The header file has documentation on MPIR_Err_create_code: https://svn.mcs.anl.gov/repos/mpi/mpich2/trunk/src/include/mpierror.h Rayson ================================================== Open Grid Scheduler - The Official Open Source Grid Engine http://gridscheduler.sourceforge.net/ http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html On Sat, Sep 14, 2013 at 8:39 PM, Huiwei Lu <huiweilu(a)mcs.anl.gov> wrote: > Hi, Volkan, > > What version are you looking at? If there is no specific reason, we would > recommend you to use the newest mpich-3.0.4 which can be downloaded from > http://www.mpich.org/downloads/ > > The following are some documents to help you understand the function > MPIR_Err_create_code. Basically what this function does is to return a error > code when you do something wrong in MPI_ calls. > http://wiki.mpich.org/mpich/index.php/Coding_Standards#Returning_Error_Codes > http://wiki.mpich.org/mpich/index.php/Reporting_And_Returning_Error_Codes > http://wiki.mpich.org/mpich/index.php/Adding_New_Error_Messages > > To help you improve your code reading skills to solve future problems like > this one. I would like to share with you one of my favorite books: Code > Reading: The Open Source Perspective. http://www.spinellis.gr/codereading/ > > Best, > Huiwei Lu > http://www.mcs.anl.gov/~huiweilu/ > > > On Fri, Sep 13, 2013 at 3:48 AM, 陈涛 <volkanchen(a)gmail.com> wrote: >> >> To whom it may concern, >> >> I am trying to analyzing the source code of MPICH2. Currently I am >> studying the function MPI_Send(const void *buf, int count, MPI_Datatype >> datatype, int dest, int tag,MPI_Comm comm) in the file send.c. But it >> involves a commonly function MPIR_Err_create_code. It is really beyond my >> ability to understand this one and it drives me crazy. Could you help me >> with that or recommend me some materials analyzing that? >> >> Looking forward to your early reply. >> >> Kind regards! >> >> Volkan >> 2013-09-13 >> >> 你好！ >> 我正分析MPICH2的源代码。目前在看send.c文件中的函数MPI_Send(const void *buf, int count, >> MPI_Datatype datatype, int dest, int tag,MPI_Comm comm) >> 。其中涉及一个常用的函数MPIR_Err_create_code。我真的是搞不清楚它的作用是什么。请问高手能不能帮我分析一下或推荐相关的资料！不胜感激！ >> 祝好！ >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Fortran/C configuration issue with MPICH2
by sunyeping 14 Sep '13

14 Sep '13

Dear all, I am trying to install mpich2-1.4.1p1 onto my centos 5.9 ssystem. I> used the following command:> > ./configure CFLAGS="-fPIC" CXXFLAGS="-fPIC" FFLAGS="-fPIC"> --enable-shared--prefix=/opt/mpich2> > However, I receive error message:> > configure: error: **** Incompatible Fortran and C Object File> Types! ****F77 Object File Type produced by "ifort -fPIC " is : :> ERROR: cannot open `' (No such file or directory). C Object File> Type produced by "gcc -fPIC -O2" is : : ELF 64-bit LSB> relocatable, AMD x86-64, version 1 (SYSV), not stripped.What does> that mean and how to resolve the problem? Institute of Microbiology, Chinese Academy of Sciences

1 0

Re: [mpich-discuss] source code query
by Huiwei Lu 14 Sep '13

14 Sep '13

Hi, Volkan, What version are you looking at? If there is no specific reason, we would recommend you to use the newest mpich-3.0.4 which can be downloaded from http://www.mpich.org/downloads/ The following are some documents to help you understand the function MPIR_Err_create_code. Basically what this function does is to return a error code when you do something wrong in MPI_ calls. http://wiki.mpich.org/mpich/index.php/Coding_Standards#Returning_Error_Codes http://wiki.mpich.org/mpich/index.php/Reporting_And_Returning_Error_Codes http://wiki.mpich.org/mpich/index.php/Adding_New_Error_Messages To help you improve your code reading skills to solve future problems like this one. I would like to share with you one of my favorite books: Code Reading: The Open Source Perspective. http://www.spinellis.gr/codereading/ Best, Huiwei Lu http://www.mcs.anl.gov/~huiweilu/ On Fri, Sep 13, 2013 at 3:48 AM, 陈涛 <volkanchen(a)gmail.com> wrote: > To whom it may concern, > > I am trying to analyzing the source code of MPICH2. Currently I am > studying the function MPI_Send(const void *buf, int count, MPI_Datatype > datatype, int dest, int tag,MPI_Comm comm) in the file send.c. But it > involves a commonly function MPIR_Err_create_code. It is really beyond my > ability to understand this one and it drives me crazy. Could you help me > with that or recommend me some materials analyzing that? > > Looking forward to your early reply. > > Kind regards! > > Volkan > 2013-09-13 > > 你好！ > 我正分析MPICH2的源代码。目前在看send.c文件中的函数MPI_Send(const void *buf, int count, > MPI_Datatype datatype, int dest, int tag,MPI_Comm comm) > 。其中涉及一个常用的函数MPIR_Err_create_code。我真的是搞不清楚它的作用是什么。请问高手能不能帮我分析一下或推荐相关的资料！不胜感激！ > 祝好！ > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Implementation of MPICH collectives
by Pavan Balaji 14 Sep '13

14 Sep '13

It's strange that the environment variable didn't work. We'll look into it. -- Pavan On Sep 14, 2013, at 12:33 PM, Jiri Simsa wrote: > Using the configure option does the right thing. Thank you. > > --Jiri > > > On Fri, Sep 13, 2013 at 10:23 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > Or try configuring with the option --enable-nemesis-dbg-nolocal. It should disable shared memory communication in the build. > > On Sep 13, 2013, at 8:17 PM, Jiri Simsa wrote: > > > Yes. To verify the behavior I wrote a simple test program: > > > > #include "mpi.h" > > #include <stdlib.h> > > #include <string.h> > > > > int main(int argc, char **argv) { > > char message[256]; > > int rank; > > if (getenv("MPIR_PARAM_CH3_NO_LOCAL") != NULL) { > > printf("MPIR_PARAM_CH3_NO_LOCAL = %s\n", getenv("MPIR_PARAM_CH3_NO_LOCAL")); > > } > > MPI_Init(&argc, &argv); > > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > > if (rank == 0) { strncpy(message, "Hello!", strlen("Hello!")); } > > MPI_Bcast(message, strlen("Hello!"), MPI_CHAR, 0, MPI_COMM_WORLD); > > MPI_Finalize(); > > printf("%d: %s\n", rank, message); > > return 0; > > } > > > > When I run it with "mpiexec -n 2 ./simple" I get the following output: > > > > MPIR_PARAM_CH3_NO_LOCAL = 1 > > MPIR_PARAM_CH3_NO_LOCAL = 1 > > 0: Hello! > > 1: Hello! > > > > I have compiled mpich-3.0.4 with --enable-g=dbg,log and set the MPICH_DBG environment variable to FILE and the MPICH_DBG_LEVEL environment variable to VERBOSE. I am attaching the log file for the process 0, which shows (to the best of my understanding) that the broadcast uses fbox and memcpy to transfer the data. > > > > > > On Fri, Sep 13, 2013 at 8:56 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > > > Not really. It shouldn't be using the nemesis fast box. Are you setting the environment correctly? > > > > -- Pavan > > > > On Sep 13, 2013, at 7:08 PM, Jiri Simsa wrote: > > > > > To be more precise, I don't see any such call before MPI_Bcast() returns in the root. Is MPICH buffering the data to be broadcasted to some later point? > > > > > > --Jiri > > > > > > > > > On Fri, Sep 13, 2013 at 7:55 PM, Jiri Simsa <jsimsa(a)cs.cmu.edu> wrote: > > > Well, it seems like it is copying data from "nemesis fastbox". More importantly, I don't see any call to socket(), connect(), and send(), sendto(), or sendmsg() that I would expect to be part of the data transfer. > > > > > > --Jiri > > > > > > > > > On Fri, Sep 13, 2013 at 5:44 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > > > > > Depends on what the memcpy is doing. It might be some internal data manipulation. > > > > > > -- Pavan > > > > > > On Sep 13, 2013, at 4:34 PM, Jiri Simsa wrote: > > > > > > > Hm, I have set that variable and then I have stepped through a program that calls MPI_Bcast (using mpiexec -n 2 <program> on a single node). The MPI_Bcast still seems to use memcpy() while I would expect it to use the sockets interface. Is the memcpy() to be expected? > > > > > > > > --Jiri > > > > > > > > > > > > On Fri, Sep 13, 2013 at 10:25 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > > > > > > > Yes, you can set the environment variable MPIR_PARAM_CH3_NOLOCAL=1. > > > > > > > > -- Pavan > > > > > > > > On Sep 13, 2013, at 7:53 AM, Jiri Simsa wrote: > > > > > > > > > Pavan, > > > > > > > > > > Thank you for your answer. That's precisely what I was looking for. Any chance there is a way to force the intranode communication to use tcp? > > > > > > > > > > --Jiri > > > > > > > > > > Within the node, it uses shared memory. Outside the node, it depends on the netmod you configured with. tcp is the default netmod. > > > > > -- Pavan > > > > > On Sep 12, 2013, at 2:24 PM, Jiri Simsa wrote: > > > > > > The high-order bit of my question is: What OS interface(s) does MPICH use to transfer data from one MPI process to another? > > > > > > > > > > > > > > > > > > On Thu, Sep 12, 2013 at 1:36 PM, Jiri Simsa <jsimsa(a)cs.cmu.edu> wrote: > > > > > > Hello, > > > > > > > > > > > > I have been trying to understand how MPICH implements collective operations. To do so, I have been reading the MPICH source code and stepping through mpiexec executions. > > > > > > > > > > > > For the sake of this discussion, let's assume that all MPI processes are executed on the same computer using: mpiexec -n <n> <mpi_program> > > > > > > > > > > > > This is my current abstract understanding of MPICH: > > > > > > > > > > > > - mpiexec spawns a hydra_pmi_proxy process, which in turn spawns <n> instances of <mpi_program> > > > > > > - hydra_pmi_proxy process uses socket pairs to communicate with the instances of <mpi_program> > > > > > > > > > > > > I am not quite sure though what happens under the hoods when a collective operation, such as MPI_Allreduce, is executed. I have noticed that instances of <mpi_program> create and listen on a socket in the course of executing MPI_Allreduce but I am not sure who connects to these sockets. Any chance someone could describe the data flow inside of MPICH when a collective operation, such as MPI_Allreduce, is executed? Thanks! > > > > > > > > > > > > Best, > > > > > > > > > > > > --Jiri Simsa > > > > > > > > > > > > _______________________________________________ > > > > > > discuss mailing list discuss(a)mpich.org > > > > > > To manage subscription options or unsubscribe: > > > > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > -- > > > > > Pavan Balaji > > > > > http://www.mcs.anl.gov/~balaji > > > > > > > > -- > > > > Pavan Balaji > > > > http://www.mcs.anl.gov/~balaji > > > > > > > > > > > > > > -- > > > Pavan Balaji > > > http://www.mcs.anl.gov/~balaji > > > > > > > > > > > > > -- > > Pavan Balaji > > http://www.mcs.anl.gov/~balaji > > > > > > <dbg0-7f997b82eb40.log>_______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Implementation of MPICH collectives
by Jiri Simsa 14 Sep '13

14 Sep '13

Using the configure option does the right thing. Thank you. --Jiri On Fri, Sep 13, 2013 at 10:23 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > Or try configuring with the option --enable-nemesis-dbg-nolocal. It should > disable shared memory communication in the build. > > On Sep 13, 2013, at 8:17 PM, Jiri Simsa wrote: > > > Yes. To verify the behavior I wrote a simple test program: > > > > #include "mpi.h" > > #include <stdlib.h> > > #include <string.h> > > > > int main(int argc, char **argv) { > > char message[256]; > > int rank; > > if (getenv("MPIR_PARAM_CH3_NO_LOCAL") != NULL) { > > printf("MPIR_PARAM_CH3_NO_LOCAL = %s\n", > getenv("MPIR_PARAM_CH3_NO_LOCAL")); > > } > > MPI_Init(&argc, &argv); > > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > > if (rank == 0) { strncpy(message, "Hello!", strlen("Hello!")); } > > MPI_Bcast(message, strlen("Hello!"), MPI_CHAR, 0, MPI_COMM_WORLD); > > MPI_Finalize(); > > printf("%d: %s\n", rank, message); > > return 0; > > } > > > > When I run it with "mpiexec -n 2 ./simple" I get the following output: > > > > MPIR_PARAM_CH3_NO_LOCAL = 1 > > MPIR_PARAM_CH3_NO_LOCAL = 1 > > 0: Hello! > > 1: Hello! > > > > I have compiled mpich-3.0.4 with --enable-g=dbg,log and set the > MPICH_DBG environment variable to FILE and the MPICH_DBG_LEVEL environment > variable to VERBOSE. I am attaching the log file for the process 0, which > shows (to the best of my understanding) that the broadcast uses fbox and > memcpy to transfer the data. > > > > > > On Fri, Sep 13, 2013 at 8:56 PM, Pavan Balaji <balaji(a)mcs.anl.gov> > wrote: > > > > Not really. It shouldn't be using the nemesis fast box. Are you > setting the environment correctly? > > > > -- Pavan > > > > On Sep 13, 2013, at 7:08 PM, Jiri Simsa wrote: > > > > > To be more precise, I don't see any such call before MPI_Bcast() > returns in the root. Is MPICH buffering the data to be broadcasted to some > later point? > > > > > > --Jiri > > > > > > > > > On Fri, Sep 13, 2013 at 7:55 PM, Jiri Simsa <jsimsa(a)cs.cmu.edu> wrote: > > > Well, it seems like it is copying data from "nemesis fastbox". More > importantly, I don't see any call to socket(), connect(), and send(), > sendto(), or sendmsg() that I would expect to be part of the data transfer. > > > > > > --Jiri > > > > > > > > > On Fri, Sep 13, 2013 at 5:44 PM, Pavan Balaji <balaji(a)mcs.anl.gov> > wrote: > > > > > > Depends on what the memcpy is doing. It might be some internal data > manipulation. > > > > > > -- Pavan > > > > > > On Sep 13, 2013, at 4:34 PM, Jiri Simsa wrote: > > > > > > > Hm, I have set that variable and then I have stepped through a > program that calls MPI_Bcast (using mpiexec -n 2 <program> on a single > node). The MPI_Bcast still seems to use memcpy() while I would expect it to > use the sockets interface. Is the memcpy() to be expected? > > > > > > > > --Jiri > > > > > > > > > > > > On Fri, Sep 13, 2013 at 10:25 AM, Pavan Balaji <balaji(a)mcs.anl.gov> > wrote: > > > > > > > > Yes, you can set the environment variable MPIR_PARAM_CH3_NOLOCAL=1. > > > > > > > > -- Pavan > > > > > > > > On Sep 13, 2013, at 7:53 AM, Jiri Simsa wrote: > > > > > > > > > Pavan, > > > > > > > > > > Thank you for your answer. That's precisely what I was looking > for. Any chance there is a way to force the intranode communication to use > tcp? > > > > > > > > > > --Jiri > > > > > > > > > > Within the node, it uses shared memory. Outside the node, it > depends on the netmod you configured with. tcp is the default netmod. > > > > > -- Pavan > > > > > On Sep 12, 2013, at 2:24 PM, Jiri Simsa wrote: > > > > > > The high-order bit of my question is: What OS interface(s) does > MPICH use to transfer data from one MPI process to another? > > > > > > > > > > > > > > > > > > On Thu, Sep 12, 2013 at 1:36 PM, Jiri Simsa <jsimsa(a)cs.cmu.edu> > wrote: > > > > > > Hello, > > > > > > > > > > > > I have been trying to understand how MPICH implements collective > operations. To do so, I have been reading the MPICH source code and > stepping through mpiexec executions. > > > > > > > > > > > > For the sake of this discussion, let's assume that all MPI > processes are executed on the same computer using: mpiexec -n <n> > <mpi_program> > > > > > > > > > > > > This is my current abstract understanding of MPICH: > > > > > > > > > > > > - mpiexec spawns a hydra_pmi_proxy process, which in turn spawns > <n> instances of <mpi_program> > > > > > > - hydra_pmi_proxy process uses socket pairs to communicate with > the instances of <mpi_program> > > > > > > > > > > > > I am not quite sure though what happens under the hoods when a > collective operation, such as MPI_Allreduce, is executed. I have noticed > that instances of <mpi_program> create and listen on a socket in the course > of executing MPI_Allreduce but I am not sure who connects to these sockets. > Any chance someone could describe the data flow inside of MPICH when a > collective operation, such as MPI_Allreduce, is executed? Thanks! > > > > > > > > > > > > Best, > > > > > > > > > > > > --Jiri Simsa > > > > > > > > > > > > _______________________________________________ > > > > > > discuss mailing list discuss(a)mpich.org > > > > > > To manage subscription options or unsubscribe: > > > > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > -- > > > > > Pavan Balaji > > > > > http://www.mcs.anl.gov/~balaji > > > > > > > > -- > > > > Pavan Balaji > > > > http://www.mcs.anl.gov/~balaji > > > > > > > > > > > > > > -- > > > Pavan Balaji > > > http://www.mcs.anl.gov/~balaji > > > > > > > > > > > > > -- > > Pavan Balaji > > http://www.mcs.anl.gov/~balaji > > > > > > <dbg0-7f997b82eb40.log>_______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > >

1 0