February 2013 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] shared memory mmap file
by Dustin Sorge 04 Feb '13

04 Feb '13

I can get MPICH working without it, however I am doing some NFS testing and having that file mounted over NFS would be a very interesting use case for me. I would like to avoid mounting /tmp over NFS so i wanted to see if there was a way to specifying another path for the file. thanks very much, -Dustin On Mon, Feb 4, 2013 at 11:54 AM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: > Hi Dustin, > > AFAICT, we don't provide a way to change the filename, apart from editing > src/util/wrappers/mpiu_shm_**wrappers.h:590. Is this something that you > need to get MPICH working on your system? Would an environment variable > work, or do you need to set this up at compile time? > > Cheers, > ~Jim. > > > On 2/4/13 9:52 AM, Dustin Sorge wrote: > >> Hello everyone, >> I was wondering if it is possible to have the shared memory mmap'ed >> file be in a different location other than /dev/shm or /tmp. Is there >> an environment variable I can specify? >> >> I am using MPICH2 and have configured it like this: >> >> ./configure --prefix=/opt/mpich2 --with-device=ch3:nemesis >> --with-shared-memory=mmap --enable-nemesis-shm-**collectives >> >> Is this configuration sufficient for MPICH2 to use the shared memory file? >> >> thanks very much, >> -Dustin >> > ______________________________**_________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/**mailman/listinfo/discuss<https://lists.mpich.org/mailman/listinfo/discuss> >

1 0

Re: [mpich-discuss] shared memory mmap file
by Jim Dinan 04 Feb '13

04 Feb '13

Hi Dustin, AFAICT, we don't provide a way to change the filename, apart from editing src/util/wrappers/mpiu_shm_wrappers.h:590. Is this something that you need to get MPICH working on your system? Would an environment variable work, or do you need to set this up at compile time? Cheers, ~Jim. On 2/4/13 9:52 AM, Dustin Sorge wrote: > Hello everyone, > I was wondering if it is possible to have the shared memory mmap'ed > file be in a different location other than /dev/shm or /tmp. Is there > an environment variable I can specify? > > I am using MPICH2 and have configured it like this: > > ./configure --prefix=/opt/mpich2 --with-device=ch3:nemesis > --with-shared-memory=mmap --enable-nemesis-shm-collectives > > Is this configuration sufficient for MPICH2 to use the shared memory file? > > thanks very much, > -Dustin

1 0

shared memory mmap file
by Dustin Sorge 04 Feb '13

04 Feb '13

Hello everyone, I was wondering if it is possible to have the shared memory mmap'ed file be in a different location other than /dev/shm or /tmp. Is there an environment variable I can specify? I am using MPICH2 and have configured it like this: ./configure --prefix=/opt/mpich2 --with-device=ch3:nemesis --with-shared-memory=mmap --enable-nemesis-shm-collectives Is this configuration sufficient for MPICH2 to use the shared memory file? thanks very much, -Dustin

1 0

srun env passing
by Biddiscombe, John A. 04 Feb '13

04 Feb '13

Hello, I'm working on someone else's cluster and the mpi is not well setup (i.e. not working). I compiled my own mpich2 using ./configure --with-pmi=slurm --with-pm=no --prefix=/home/biddisco/apps/mpich2-1.5 --enable-shared --disable-static And a simple hello world executed using srun, works as expected and I get the ranks from all nodes responding correctly. When I used mpiexec previously, I ran an MPMD command like this mpiexec -n 4 -env DISPLAY :0.0 /home/biddisco/build/pv-38/bin/pvserver -rc -ch=128.178.187.20 -sp=11111 : -n 4 -env DISPLAY :0.1 /home/biddisco/build/pv-38/bin/pvserver -rc -ch=128.178.187.20 -sp=11111 : -n 4 -env DISPLAY :0.2 /home/biddisco/build/pv-38/bin/pvserver -rc -ch=128.178.187.20 -sp=11111 I need each of the 3 sets of mpi ranks to have a different DISPLAY environment variable to correctly utilise the GPUs. Unfortunately, the mpiexec on the machine does not work with slurm and I'd like to use srun -n XXX -env XXX /program : -n XXX -env YYY /program ... etc etc but srun doesn't have an option to pass command line environment vars to the mpi processes. I just can't find a way of launching an MPMD job using srun with different env vars. Is there another way? Thanks JB -- John Biddiscombe, email:biddisco @.at.@ cscs.ch http://www.cscs.ch/ CSCS, Swiss National Supercomputing Centre | Tel: +41 (91) 610.82.07 Via Trevano 131, 6900 Lugano, Switzerland | Fax: +41 (91) 610.82.82

1 0

Re: [mpich-discuss] Fortran/C string passing convention
by Dave Goodell 04 Feb '13

04 Feb '13

Hi Nils, It looks like Bill is tracking this issue in http://trac.mpich.org/projects/mpich/ticket/1790, in case you want to follow along at home. -Dave On Feb 1, 2013, at 1:44 PM CST, William Gropp wrote: > Thanks - when this code was written, all of the Fortran compilers used ints, but that was a long time ago. I'll look into both a robust configure check and updating the code. > > Bill > > William Gropp > Director, Parallel Computing Institute > Deputy Director for Research > Institute for Advanced Computing Applications and Technologies > Paul and Cynthia Saylor Professor of Computer Science > University of Illinois Urbana-Champaign > > > > On Feb 1, 2013, at 9:20 AM, Nils Smeds wrote: > >> Hi all, >> >> I notice in the mpich source code that Fortran string length arguments are supposed to be int. However, in many compilers this hidden argument is size_t. Below are excerpts from documentation of Intel ifort, GFortran and PathScale. (The last one claims to use int as the string length). In practice using an int here "just works" - at least for strings shorter than 2GB. Since the string length argument is passed by value there is little risk for any errors to occur as it is most likely to be passed in a register so any "additional upper bits" get cleared in the process of the call. Possibly it could be a problem with routines with vary many arguments where arguments are passed over the stack. >> >> ./src/mpi/romio/adio/include/adio.h >> ./src/binding/f77/mpi_fortimpl.h >> >> #ifdef USE_FORT_MIXED_STR_LEN >> #define FORT_MIXED_LEN_DECL , int >> #define FORT_END_LEN_DECL >> #define FORT_MIXED_LEN(a) , int a >> #define FORT_END_LEN(a) >> #else >> #define FORT_MIXED_LEN_DECL >> #define FORT_END_LEN_DECL , int >> #define FORT_MIXED_LEN(a) >> #define FORT_END_LEN(a) , int a >> #endif >> >> >> Intel Fortran >> >> http://software.intel.com/sites/products/documentation/doclib/stdxe/2013/co… >> (Not a perfect example, but the best I could find) I've seen more explicit formulations in earlier Intel manuals. >> >> http://software.intel.com/en-us/forums/topic/270023 >> Is more to the point >> >> GFortran >> >> http://www.xgear.eu/callfortranfromc.html >> >> Fortran functions that receive string variables from C >> C >> This is slightly more complicated. The Fortran function must be declared at the beginning of the C calling function (e.g. main) like this: >> extern void load_(char *string, size_t *len_string); >> >> Note that we have to pass all variables to Fortran as pointers. Our string variable is already a pointer, but we have to pass the string's length as a pointer too. Although the function name is not case sensitive in Fortran, it gains an underscore in the C declaration and when it is called: >> len_string = strlen(string); >> >> load_(string, &len_string); >> Note that in passing the string from C to Fortran we also need to pass the length of the string, hence the use of strlen beforehand (not forgetting to declare the string length variable as a size_trather than an int). >> >> [ NOTE: The above is likely incorrect in that I believe the declaration should be size_t len_string and the call load_(string, len_string). >> The actual arguments are passed by reference but the hidden length arguments are likely passed by value. >> (My comment) >> ] >> >> >> Pathscale >> >> http://www.pathscale.com/EKOPath-User-Guide#htoc66 >> >> Like most Fortran compilers, we represent character strings passed to subprograms with a character pointer, and add an integer length parameter to the end of the call list. >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_Datatype for a structure containing pointers
by Jed Brown 04 Feb '13

04 Feb '13

On Mon, Feb 4, 2013 at 3:45 AM, Aya Aya <aya_tounsi(a)hotmail.fr> wrote: > I want to send (scatter) an array of N element of the structure below, > with sizeof(Tab)=T > > struct bloc > { > int T; > double *Tab; > }; > typedef struct bloc BLOC; > > I wonder if it is possible to create a new MPI_Datatype for this structure > (the size of Tab depends on T), or the only way is to sent it as an array > (using Scatterv). > You can create a datatype for one particular instance of this structure (fixed location of the array and fixed size T), but not a "dynamic datatype" that sends any instance of this struct. There is not an MPI_Scatterw, however, so you have to pack a homogeneous buffer yourself. Since you can't allocate on the receiver without knowing the size T, do an MPI_Scatter of the sizes T (if you don't somehow know this already), allocate, and do an MPI_Scatterv out of a single packed array (containing the concatenation of all the Tabs).

1 0

MPI_Datatype for a structure containing pointers
by Aya Aya 04 Feb '13

04 Feb '13

Good morning, I want to send (scatter) an array of N element of the structure below, with sizeof(Tab)=T struct bloc { int T; double *Tab; }; typedef struct bloc BLOC; I wonder if it is possible to create a new MPI_Datatype for this structure (the size of Tab depends on T), or the only way is to sent it as an array (using Scatterv). Thanks for your help.

1 0

Re: [mpich-discuss] process-core binding doesn't work on newer multisocket opteron systems with mpich 3.0.x
by Róbert Špir 02 Feb '13

02 Feb '13

We are using version 3.0.2 $ mpichversion MPICH Version: 3.0.2 MPICH Release date: Wed Jan 30 17:38:15 CST 2013 MPICH Device: ch3:sock MPICH configure: --prefix=/opt/mpich --enable-shared --enable-sharedlibs=gcc --enable-error-checking=runtime --enable-error-messages=all --enable-timer-type=clock_gettime --with-device=ch3:sock --with-python=python2 MPICH CC: gcc -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector --param=ssp-buffer-size=4 -D_FORTIFY_SOURCE=2 -O2 MPICH CXX: c++ -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector --param=ssp-buffer-size=4 -D_FORTIFY_SOURCE=2 -O2 MPICH F77: gfortran -O2 MPICH FC: gfortran -O2 and binding is not working since 1.5. Only binding that works is user:0,1,2,3.... Robert > -----Original Message----- > From: Pavan Balaji [mailto:[email protected]] > Sent: Saturday, February 2, 2013 3:54 PM > To: discuss(a)mpich.org > Cc: Róbert Špir > Subject: Re: [mpich-discuss] process-core binding doesn't work on newer > multisocket opteron systems with mpich 3.0.x > > Hi Robert, > > Which version of mpich are you using? We did have some problem with this > in mpich-3.0.1, which should be fixed in the just released mpich-3.0.2. Can > you try that? > > -- Pavan > > On 02/02/2013 03:35 AM US Central Time, Róbert Špir wrote: > > Hello, > > > > we have multiple quad-socket AMD opteron systems with linux operating > > system. Since these systems are NUMA architecture, we are using > > binding of MPI processes to single NUMA nodes for optimal local memory > > usage. With our older systems, with 4 quadcore processors the binding > > in mpich 3 works as expected, so when we use "mpiexec ... -bind-to > > numa ..." the MPI processes are bound to NUMA nodes. > > The output of hwloc lstopo is: > > Machine (128GB) > > NUMANode L#0 (P#0 32GB) > > Socket L#0 + L3 L#0 (2048KB) > > L2 L#0 (512KB) + L1d L#0 (64KB) + L1i L#0 (64KB) + Core L#0 + PU > > L#0 > > (P#0) > > L2 L#1 (512KB) + L1d L#1 (64KB) + L1i L#1 (64KB) + Core L#1 + PU > > L#1 > > (P#1) > > L2 L#2 (512KB) + L1d L#2 (64KB) + L1i L#2 (64KB) + Core L#2 + PU > > L#2 > > (P#2) > > L2 L#3 (512KB) + L1d L#3 (64KB) + L1i L#3 (64KB) + Core L#3 + PU > > L#3 > > (P#3) > > HostBridge L#0 > > PCI 10de:036e > > Block L#0 "sr0" > > PCI 10de:037f > > PCI 10de:037f > > PCI 10de:037f > > PCIBridge > > PCI 1002:515e > > GPU L#1 "card0" > > PCIBridge > > PCI 1000:0058 > > Block L#2 "sda" > > NUMANode L#1 (P#1 32GB) + Socket L#1 + L3 L#1 (2048KB) > > L2 L#4 (512KB) + L1d L#4 (64KB) + L1i L#4 (64KB) + Core L#4 + PU > > L#4 > > (P#4) > > L2 L#5 (512KB) + L1d L#5 (64KB) + L1i L#5 (64KB) + Core L#5 + PU > > L#5 > > (P#5) > > L2 L#6 (512KB) + L1d L#6 (64KB) + L1i L#6 (64KB) + Core L#6 + PU > > L#6 > > (P#6) > > L2 L#7 (512KB) + L1d L#7 (64KB) + L1i L#7 (64KB) + Core L#7 + PU > > L#7 > > (P#7) > > NUMANode L#2 (P#2 32GB) + Socket L#2 + L3 L#2 (2048KB) > > L2 L#8 (512KB) + L1d L#8 (64KB) + L1i L#8 (64KB) + Core L#8 + PU > > L#8 > > (P#8) > > L2 L#9 (512KB) + L1d L#9 (64KB) + L1i L#9 (64KB) + Core L#9 + PU > > L#9 > > (P#9) > > L2 L#10 (512KB) + L1d L#10 (64KB) + L1i L#10 (64KB) + Core L#10 + > > PU > > L#10 (P#10) > > L2 L#11 (512KB) + L1d L#11 (64KB) + L1i L#11 (64KB) + Core L#11 + > > PU > > L#11 (P#11) > > NUMANode L#3 (P#3 32GB) + Socket L#3 + L3 L#3 (2048KB) > > L2 L#12 (512KB) + L1d L#12 (64KB) + L1i L#12 (64KB) + Core L#12 + > > PU > > L#12 (P#12) > > L2 L#13 (512KB) + L1d L#13 (64KB) + L1i L#13 (64KB) + Core L#13 + > > PU > > L#13 (P#13) > > L2 L#14 (512KB) + L1d L#14 (64KB) + L1i L#14 (64KB) + Core L#14 + > > PU > > L#14 (P#14) > > L2 L#15 (512KB) + L1d L#15 (64KB) + L1i L#15 (64KB) + Core L#15 + > > PU > > L#15 (P#15) > > > > Now when we try this binding on our newer systems, where we have four > > 8-core AMD opterons, we get errors (no form of binding works, even if > > we use binding to socket or core) > > > > [proxy:0:0@embryo09] get_nbobjs_by_type > > (../../../../src/pm/hydra/tools/topo/hwloc/topo_hwloc.c:189): assert > > (nb % x == 0) failed [proxy:0:0@embryo09] handle_bitmap_binding > > (../../../../src/pm/hydra/tools/topo/hwloc/topo_hwloc.c:450): unable > > to get number of objects [proxy:0:0@embryo09] HYDT_topo_hwloc_init > > (../../../../src/pm/hydra/tools/topo/hwloc/topo_hwloc.c:527): error > > binding with bind "numa" and map "(null)" > > [proxy:0:0@embryo09] HYDT_topo_init > > (../../../../src/pm/hydra/tools/topo/topo.c:60): unable to initialize > > hwloc [proxy:0:0@embryo09] launch_procs > > (../../../../src/pm/hydra/pm/pmiserv/pmip_cb.c:520): unable to > > initialize process topology [proxy:0:0@embryo09] > > HYD_pmcd_pmip_control_cmd_cb > > (../../../../src/pm/hydra/pm/pmiserv/pmip_cb.c:893): launch_procs > > returned error [proxy:0:0@embryo09] HYDT_dmxu_poll_wait_for_event > > (../../../../src/pm/hydra/tools/demux/demux_poll.c:77): callback > > returned error status [proxy:0:0@embryo09] main > > (../../../../src/pm/hydra/pm/pmiserv/pmip.c:206): > > demux engine error waiting for event > > [mpiexec@embryo09] control_cb > > (../../../../src/pm/hydra/pm/pmiserv/pmiserv_cb.c:202): assert > > (!closed) failed [mpiexec@embryo09] HYDT_dmxu_poll_wait_for_event > > (../../../../src/pm/hydra/tools/demux/demux_poll.c:77): callback > > returned error status [mpiexec@embryo09] > HYD_pmci_wait_for_completion > > (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:197): error > > waiting for event [mpiexec@embryo09] main > > (../../../../src/pm/hydra/ui/mpich/mpiexec.c:330): > > process manager error waiting for completion > > > > lstopo output is: > > Machine (256GB) > > Socket L#0 (64GB) > > NUMANode L#0 (P#0 32GB) > > L3 L#0 (5118KB) > > L2 L#0 (512KB) + L1d L#0 (64KB) + L1i L#0 (64KB) + Core L#0 + > > PU L#0 > > (P#0) > > L2 L#1 (512KB) + L1d L#1 (64KB) + L1i L#1 (64KB) + Core L#1 + > > PU L#1 > > (P#1) > > L2 L#2 (512KB) + L1d L#2 (64KB) + L1i L#2 (64KB) + Core L#2 + > > PU L#2 > > (P#2) > > L2 L#3 (512KB) + L1d L#3 (64KB) + L1i L#3 (64KB) + Core L#3 + > > PU L#3 > > (P#3) > > HostBridge L#0 > > PCIBridge > > PCI 8086:10c9 > > Net L#0 "eth0" > > PCI 8086:10c9 > > Net L#1 "eth1" > > PCI 1002:4390 > > Block L#2 "sda" > > PCI 1002:439c > > PCIBridge > > PCI 102b:0532 > > NUMANode L#1 (P#1 32GB) + L3 L#1 (5118KB) > > L2 L#4 (512KB) + L1d L#4 (64KB) + L1i L#4 (64KB) + Core L#4 + PU > > L#4 > > (P#4) > > L2 L#5 (512KB) + L1d L#5 (64KB) + L1i L#5 (64KB) + Core L#5 + PU > > L#5 > > (P#5) > > L2 L#6 (512KB) + L1d L#6 (64KB) + L1i L#6 (64KB) + Core L#6 + PU > > L#6 > > (P#6) > > L2 L#7 (512KB) + L1d L#7 (64KB) + L1i L#7 (64KB) + Core L#7 + PU > > L#7 > > (P#7) > > Socket L#1 (64GB) > > NUMANode L#2 (P#2 32GB) + L3 L#2 (5118KB) > > L2 L#8 (512KB) + L1d L#8 (64KB) + L1i L#8 (64KB) + Core L#8 + PU > > L#8 > > (P#8) > > L2 L#9 (512KB) + L1d L#9 (64KB) + L1i L#9 (64KB) + Core L#9 + PU > > L#9 > > (P#9) > > L2 L#10 (512KB) + L1d L#10 (64KB) + L1i L#10 (64KB) + Core L#10 > > + PU > > L#10 (P#10) > > L2 L#11 (512KB) + L1d L#11 (64KB) + L1i L#11 (64KB) + Core L#11 > > + PU > > L#11 (P#11) > > NUMANode L#3 (P#3 32GB) + L3 L#3 (5118KB) > > L2 L#12 (512KB) + L1d L#12 (64KB) + L1i L#12 (64KB) + Core L#12 > > + PU > > L#12 (P#12) > > L2 L#13 (512KB) + L1d L#13 (64KB) + L1i L#13 (64KB) + Core L#13 > > + PU > > L#13 (P#13) > > L2 L#14 (512KB) + L1d L#14 (64KB) + L1i L#14 (64KB) + Core L#14 > > + PU > > L#14 (P#14) > > L2 L#15 (512KB) + L1d L#15 (64KB) + L1i L#15 (64KB) + Core L#15 > > + PU > > L#15 (P#15) > > Socket L#2 (64GB) > > NUMANode L#4 (P#4 32GB) + L3 L#4 (5118KB) > > L2 L#16 (512KB) + L1d L#16 (64KB) + L1i L#16 (64KB) + Core L#16 > > + PU > > L#16 (P#16) > > L2 L#17 (512KB) + L1d L#17 (64KB) + L1i L#17 (64KB) + Core L#17 > > + PU > > L#17 (P#17) > > L2 L#18 (512KB) + L1d L#18 (64KB) + L1i L#18 (64KB) + Core L#18 > > + PU > > L#18 (P#18) > > L2 L#19 (512KB) + L1d L#19 (64KB) + L1i L#19 (64KB) + Core L#19 > > + PU > > L#19 (P#19) > > NUMANode L#5 (P#5 32GB) + L3 L#5 (5118KB) > > L2 L#20 (512KB) + L1d L#20 (64KB) + L1i L#20 (64KB) + Core L#20 > > + PU > > L#20 (P#20) > > L2 L#21 (512KB) + L1d L#21 (64KB) + L1i L#21 (64KB) + Core L#21 > > + PU > > L#21 (P#21) > > L2 L#22 (512KB) + L1d L#22 (64KB) + L1i L#22 (64KB) + Core L#22 > > + PU > > L#22 (P#22) > > L2 L#23 (512KB) + L1d L#23 (64KB) + L1i L#23 (64KB) + Core L#23 > > + PU > > L#23 (P#23) > > Socket L#3 (64GB) > > NUMANode L#6 (P#6 32GB) + L3 L#6 (5118KB) > > L2 L#24 (512KB) + L1d L#24 (64KB) + L1i L#24 (64KB) + Core L#24 > > + PU > > L#24 (P#24) > > L2 L#25 (512KB) + L1d L#25 (64KB) + L1i L#25 (64KB) + Core L#25 > > + PU > > L#25 (P#25) > > L2 L#26 (512KB) + L1d L#26 (64KB) + L1i L#26 (64KB) + Core L#26 > > + PU > > L#26 (P#26) > > L2 L#27 (512KB) + L1d L#27 (64KB) + L1i L#27 (64KB) + Core L#27 > > + PU > > L#27 (P#27) > > NUMANode L#7 (P#7 32GB) + L3 L#7 (5118KB) > > L2 L#28 (512KB) + L1d L#28 (64KB) + L1i L#28 (64KB) + Core L#28 > > + PU > > L#28 (P#28) > > L2 L#29 (512KB) + L1d L#29 (64KB) + L1i L#29 (64KB) + Core L#29 > > + PU > > L#29 (P#29) > > L2 L#30 (512KB) + L1d L#30 (64KB) + L1i L#30 (64KB) + Core L#30 > > + PU > > L#30 (P#30) > > L2 L#31 (512KB) + L1d L#31 (64KB) + L1i L#31 (64KB) + Core L#31 > > + PU > > L#31 (P#31) > > > > the last mpich version with working binding on new systems was mpich2 > > version 1.4.1 > > > > Robert Spir > > -- > > Department of Mathematics > > Faculty of Civil Engineering > > Slovak University of Technology Bratislava Radlinskeho 11 > > 813 68 Bratislava > > Slovakia > > http://www.math.sk > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] process-core binding doesn't work on newer multisocket opteron systems with mpich 3.0.x
by Pavan Balaji 02 Feb '13

02 Feb '13

Hi Robert, Which version of mpich are you using? We did have some problem with this in mpich-3.0.1, which should be fixed in the just released mpich-3.0.2. Can you try that? -- Pavan On 02/02/2013 03:35 AM US Central Time, Rï¿½bert ï¿½pir wrote: > Hello, > > we have multiple quad-socket AMD opteron systems with linux operating > system. Since these systems are NUMA architecture, we are using binding of > MPI processes to single NUMA nodes for optimal local memory usage. With our > older systems, with 4 quadcore processors the binding in mpich 3 works as > expected, so when we use "mpiexec ... -bind-to numa ..." the MPI processes > are bound to NUMA nodes. > The output of hwloc lstopo is: > Machine (128GB) > NUMANode L#0 (P#0 32GB) > Socket L#0 + L3 L#0 (2048KB) > L2 L#0 (512KB) + L1d L#0 (64KB) + L1i L#0 (64KB) + Core L#0 + PU L#0 > (P#0) > L2 L#1 (512KB) + L1d L#1 (64KB) + L1i L#1 (64KB) + Core L#1 + PU L#1 > (P#1) > L2 L#2 (512KB) + L1d L#2 (64KB) + L1i L#2 (64KB) + Core L#2 + PU L#2 > (P#2) > L2 L#3 (512KB) + L1d L#3 (64KB) + L1i L#3 (64KB) + Core L#3 + PU L#3 > (P#3) > HostBridge L#0 > PCI 10de:036e > Block L#0 "sr0" > PCI 10de:037f > PCI 10de:037f > PCI 10de:037f > PCIBridge > PCI 1002:515e > GPU L#1 "card0" > PCIBridge > PCI 1000:0058 > Block L#2 "sda" > NUMANode L#1 (P#1 32GB) + Socket L#1 + L3 L#1 (2048KB) > L2 L#4 (512KB) + L1d L#4 (64KB) + L1i L#4 (64KB) + Core L#4 + PU L#4 > (P#4) > L2 L#5 (512KB) + L1d L#5 (64KB) + L1i L#5 (64KB) + Core L#5 + PU L#5 > (P#5) > L2 L#6 (512KB) + L1d L#6 (64KB) + L1i L#6 (64KB) + Core L#6 + PU L#6 > (P#6) > L2 L#7 (512KB) + L1d L#7 (64KB) + L1i L#7 (64KB) + Core L#7 + PU L#7 > (P#7) > NUMANode L#2 (P#2 32GB) + Socket L#2 + L3 L#2 (2048KB) > L2 L#8 (512KB) + L1d L#8 (64KB) + L1i L#8 (64KB) + Core L#8 + PU L#8 > (P#8) > L2 L#9 (512KB) + L1d L#9 (64KB) + L1i L#9 (64KB) + Core L#9 + PU L#9 > (P#9) > L2 L#10 (512KB) + L1d L#10 (64KB) + L1i L#10 (64KB) + Core L#10 + PU > L#10 (P#10) > L2 L#11 (512KB) + L1d L#11 (64KB) + L1i L#11 (64KB) + Core L#11 + PU > L#11 (P#11) > NUMANode L#3 (P#3 32GB) + Socket L#3 + L3 L#3 (2048KB) > L2 L#12 (512KB) + L1d L#12 (64KB) + L1i L#12 (64KB) + Core L#12 + PU > L#12 (P#12) > L2 L#13 (512KB) + L1d L#13 (64KB) + L1i L#13 (64KB) + Core L#13 + PU > L#13 (P#13) > L2 L#14 (512KB) + L1d L#14 (64KB) + L1i L#14 (64KB) + Core L#14 + PU > L#14 (P#14) > L2 L#15 (512KB) + L1d L#15 (64KB) + L1i L#15 (64KB) + Core L#15 + PU > L#15 (P#15) > > Now when we try this binding on our newer systems, where we have four 8-core > AMD opterons, we get errors (no form of binding works, even if we use > binding to socket or core) > > [proxy:0:0@embryo09] get_nbobjs_by_type > (../../../../src/pm/hydra/tools/topo/hwloc/topo_hwloc.c:189): assert (nb % x > == 0) failed > [proxy:0:0@embryo09] handle_bitmap_binding > (../../../../src/pm/hydra/tools/topo/hwloc/topo_hwloc.c:450): unable to get > number of objects > [proxy:0:0@embryo09] HYDT_topo_hwloc_init > (../../../../src/pm/hydra/tools/topo/hwloc/topo_hwloc.c:527): error binding > with bind "numa" and map "(null)" > [proxy:0:0@embryo09] HYDT_topo_init > (../../../../src/pm/hydra/tools/topo/topo.c:60): unable to initialize hwloc > [proxy:0:0@embryo09] launch_procs > (../../../../src/pm/hydra/pm/pmiserv/pmip_cb.c:520): unable to initialize > process topology > [proxy:0:0@embryo09] HYD_pmcd_pmip_control_cmd_cb > (../../../../src/pm/hydra/pm/pmiserv/pmip_cb.c:893): launch_procs returned > error > [proxy:0:0@embryo09] HYDT_dmxu_poll_wait_for_event > (../../../../src/pm/hydra/tools/demux/demux_poll.c:77): callback returned > error status > [proxy:0:0@embryo09] main (../../../../src/pm/hydra/pm/pmiserv/pmip.c:206): > demux engine error waiting for event > [mpiexec@embryo09] control_cb > (../../../../src/pm/hydra/pm/pmiserv/pmiserv_cb.c:202): assert (!closed) > failed > [mpiexec@embryo09] HYDT_dmxu_poll_wait_for_event > (../../../../src/pm/hydra/tools/demux/demux_poll.c:77): callback returned > error status > [mpiexec@embryo09] HYD_pmci_wait_for_completion > (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:197): error waiting for > event > [mpiexec@embryo09] main (../../../../src/pm/hydra/ui/mpich/mpiexec.c:330): > process manager error waiting for completion > > lstopo output is: > Machine (256GB) > Socket L#0 (64GB) > NUMANode L#0 (P#0 32GB) > L3 L#0 (5118KB) > L2 L#0 (512KB) + L1d L#0 (64KB) + L1i L#0 (64KB) + Core L#0 + PU L#0 > (P#0) > L2 L#1 (512KB) + L1d L#1 (64KB) + L1i L#1 (64KB) + Core L#1 + PU L#1 > (P#1) > L2 L#2 (512KB) + L1d L#2 (64KB) + L1i L#2 (64KB) + Core L#2 + PU L#2 > (P#2) > L2 L#3 (512KB) + L1d L#3 (64KB) + L1i L#3 (64KB) + Core L#3 + PU L#3 > (P#3) > HostBridge L#0 > PCIBridge > PCI 8086:10c9 > Net L#0 "eth0" > PCI 8086:10c9 > Net L#1 "eth1" > PCI 1002:4390 > Block L#2 "sda" > PCI 1002:439c > PCIBridge > PCI 102b:0532 > NUMANode L#1 (P#1 32GB) + L3 L#1 (5118KB) > L2 L#4 (512KB) + L1d L#4 (64KB) + L1i L#4 (64KB) + Core L#4 + PU L#4 > (P#4) > L2 L#5 (512KB) + L1d L#5 (64KB) + L1i L#5 (64KB) + Core L#5 + PU L#5 > (P#5) > L2 L#6 (512KB) + L1d L#6 (64KB) + L1i L#6 (64KB) + Core L#6 + PU L#6 > (P#6) > L2 L#7 (512KB) + L1d L#7 (64KB) + L1i L#7 (64KB) + Core L#7 + PU L#7 > (P#7) > Socket L#1 (64GB) > NUMANode L#2 (P#2 32GB) + L3 L#2 (5118KB) > L2 L#8 (512KB) + L1d L#8 (64KB) + L1i L#8 (64KB) + Core L#8 + PU L#8 > (P#8) > L2 L#9 (512KB) + L1d L#9 (64KB) + L1i L#9 (64KB) + Core L#9 + PU L#9 > (P#9) > L2 L#10 (512KB) + L1d L#10 (64KB) + L1i L#10 (64KB) + Core L#10 + PU > L#10 (P#10) > L2 L#11 (512KB) + L1d L#11 (64KB) + L1i L#11 (64KB) + Core L#11 + PU > L#11 (P#11) > NUMANode L#3 (P#3 32GB) + L3 L#3 (5118KB) > L2 L#12 (512KB) + L1d L#12 (64KB) + L1i L#12 (64KB) + Core L#12 + PU > L#12 (P#12) > L2 L#13 (512KB) + L1d L#13 (64KB) + L1i L#13 (64KB) + Core L#13 + PU > L#13 (P#13) > L2 L#14 (512KB) + L1d L#14 (64KB) + L1i L#14 (64KB) + Core L#14 + PU > L#14 (P#14) > L2 L#15 (512KB) + L1d L#15 (64KB) + L1i L#15 (64KB) + Core L#15 + PU > L#15 (P#15) > Socket L#2 (64GB) > NUMANode L#4 (P#4 32GB) + L3 L#4 (5118KB) > L2 L#16 (512KB) + L1d L#16 (64KB) + L1i L#16 (64KB) + Core L#16 + PU > L#16 (P#16) > L2 L#17 (512KB) + L1d L#17 (64KB) + L1i L#17 (64KB) + Core L#17 + PU > L#17 (P#17) > L2 L#18 (512KB) + L1d L#18 (64KB) + L1i L#18 (64KB) + Core L#18 + PU > L#18 (P#18) > L2 L#19 (512KB) + L1d L#19 (64KB) + L1i L#19 (64KB) + Core L#19 + PU > L#19 (P#19) > NUMANode L#5 (P#5 32GB) + L3 L#5 (5118KB) > L2 L#20 (512KB) + L1d L#20 (64KB) + L1i L#20 (64KB) + Core L#20 + PU > L#20 (P#20) > L2 L#21 (512KB) + L1d L#21 (64KB) + L1i L#21 (64KB) + Core L#21 + PU > L#21 (P#21) > L2 L#22 (512KB) + L1d L#22 (64KB) + L1i L#22 (64KB) + Core L#22 + PU > L#22 (P#22) > L2 L#23 (512KB) + L1d L#23 (64KB) + L1i L#23 (64KB) + Core L#23 + PU > L#23 (P#23) > Socket L#3 (64GB) > NUMANode L#6 (P#6 32GB) + L3 L#6 (5118KB) > L2 L#24 (512KB) + L1d L#24 (64KB) + L1i L#24 (64KB) + Core L#24 + PU > L#24 (P#24) > L2 L#25 (512KB) + L1d L#25 (64KB) + L1i L#25 (64KB) + Core L#25 + PU > L#25 (P#25) > L2 L#26 (512KB) + L1d L#26 (64KB) + L1i L#26 (64KB) + Core L#26 + PU > L#26 (P#26) > L2 L#27 (512KB) + L1d L#27 (64KB) + L1i L#27 (64KB) + Core L#27 + PU > L#27 (P#27) > NUMANode L#7 (P#7 32GB) + L3 L#7 (5118KB) > L2 L#28 (512KB) + L1d L#28 (64KB) + L1i L#28 (64KB) + Core L#28 + PU > L#28 (P#28) > L2 L#29 (512KB) + L1d L#29 (64KB) + L1i L#29 (64KB) + Core L#29 + PU > L#29 (P#29) > L2 L#30 (512KB) + L1d L#30 (64KB) + L1i L#30 (64KB) + Core L#30 + PU > L#30 (P#30) > L2 L#31 (512KB) + L1d L#31 (64KB) + L1i L#31 (64KB) + Core L#31 + PU > L#31 (P#31) > > the last mpich version with working binding on new systems was mpich2 > version 1.4.1 > > Robert Spir > -- > Department of Mathematics > Faculty of Civil Engineering > Slovak University of Technology Bratislava > Radlinskeho 11 > 813 68 Bratislava > Slovakia > http://www.math.sk > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

process-core binding doesn't work on newer multisocket opteron systems with mpich 3.0.x
by Róbert Špir 02 Feb '13

02 Feb '13

Hello, we have multiple quad-socket AMD opteron systems with linux operating system. Since these systems are NUMA architecture, we are using binding of MPI processes to single NUMA nodes for optimal local memory usage. With our older systems, with 4 quadcore processors the binding in mpich 3 works as expected, so when we use "mpiexec ... -bind-to numa ..." the MPI processes are bound to NUMA nodes. The output of hwloc lstopo is: Machine (128GB) NUMANode L#0 (P#0 32GB) Socket L#0 + L3 L#0 (2048KB) L2 L#0 (512KB) + L1d L#0 (64KB) + L1i L#0 (64KB) + Core L#0 + PU L#0 (P#0) L2 L#1 (512KB) + L1d L#1 (64KB) + L1i L#1 (64KB) + Core L#1 + PU L#1 (P#1) L2 L#2 (512KB) + L1d L#2 (64KB) + L1i L#2 (64KB) + Core L#2 + PU L#2 (P#2) L2 L#3 (512KB) + L1d L#3 (64KB) + L1i L#3 (64KB) + Core L#3 + PU L#3 (P#3) HostBridge L#0 PCI 10de:036e Block L#0 "sr0" PCI 10de:037f PCI 10de:037f PCI 10de:037f PCIBridge PCI 1002:515e GPU L#1 "card0" PCIBridge PCI 1000:0058 Block L#2 "sda" NUMANode L#1 (P#1 32GB) + Socket L#1 + L3 L#1 (2048KB) L2 L#4 (512KB) + L1d L#4 (64KB) + L1i L#4 (64KB) + Core L#4 + PU L#4 (P#4) L2 L#5 (512KB) + L1d L#5 (64KB) + L1i L#5 (64KB) + Core L#5 + PU L#5 (P#5) L2 L#6 (512KB) + L1d L#6 (64KB) + L1i L#6 (64KB) + Core L#6 + PU L#6 (P#6) L2 L#7 (512KB) + L1d L#7 (64KB) + L1i L#7 (64KB) + Core L#7 + PU L#7 (P#7) NUMANode L#2 (P#2 32GB) + Socket L#2 + L3 L#2 (2048KB) L2 L#8 (512KB) + L1d L#8 (64KB) + L1i L#8 (64KB) + Core L#8 + PU L#8 (P#8) L2 L#9 (512KB) + L1d L#9 (64KB) + L1i L#9 (64KB) + Core L#9 + PU L#9 (P#9) L2 L#10 (512KB) + L1d L#10 (64KB) + L1i L#10 (64KB) + Core L#10 + PU L#10 (P#10) L2 L#11 (512KB) + L1d L#11 (64KB) + L1i L#11 (64KB) + Core L#11 + PU L#11 (P#11) NUMANode L#3 (P#3 32GB) + Socket L#3 + L3 L#3 (2048KB) L2 L#12 (512KB) + L1d L#12 (64KB) + L1i L#12 (64KB) + Core L#12 + PU L#12 (P#12) L2 L#13 (512KB) + L1d L#13 (64KB) + L1i L#13 (64KB) + Core L#13 + PU L#13 (P#13) L2 L#14 (512KB) + L1d L#14 (64KB) + L1i L#14 (64KB) + Core L#14 + PU L#14 (P#14) L2 L#15 (512KB) + L1d L#15 (64KB) + L1i L#15 (64KB) + Core L#15 + PU L#15 (P#15) Now when we try this binding on our newer systems, where we have four 8-core AMD opterons, we get errors (no form of binding works, even if we use binding to socket or core) [proxy:0:0@embryo09] get_nbobjs_by_type (../../../../src/pm/hydra/tools/topo/hwloc/topo_hwloc.c:189): assert (nb % x == 0) failed [proxy:0:0@embryo09] handle_bitmap_binding (../../../../src/pm/hydra/tools/topo/hwloc/topo_hwloc.c:450): unable to get number of objects [proxy:0:0@embryo09] HYDT_topo_hwloc_init (../../../../src/pm/hydra/tools/topo/hwloc/topo_hwloc.c:527): error binding with bind "numa" and map "(null)" [proxy:0:0@embryo09] HYDT_topo_init (../../../../src/pm/hydra/tools/topo/topo.c:60): unable to initialize hwloc [proxy:0:0@embryo09] launch_procs (../../../../src/pm/hydra/pm/pmiserv/pmip_cb.c:520): unable to initialize process topology [proxy:0:0@embryo09] HYD_pmcd_pmip_control_cmd_cb (../../../../src/pm/hydra/pm/pmiserv/pmip_cb.c:893): launch_procs returned error [proxy:0:0@embryo09] HYDT_dmxu_poll_wait_for_event (../../../../src/pm/hydra/tools/demux/demux_poll.c:77): callback returned error status [proxy:0:0@embryo09] main (../../../../src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@embryo09] control_cb (../../../../src/pm/hydra/pm/pmiserv/pmiserv_cb.c:202): assert (!closed) failed [mpiexec@embryo09] HYDT_dmxu_poll_wait_for_event (../../../../src/pm/hydra/tools/demux/demux_poll.c:77): callback returned error status [mpiexec@embryo09] HYD_pmci_wait_for_completion (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:197): error waiting for event [mpiexec@embryo09] main (../../../../src/pm/hydra/ui/mpich/mpiexec.c:330): process manager error waiting for completion lstopo output is: Machine (256GB) Socket L#0 (64GB) NUMANode L#0 (P#0 32GB) L3 L#0 (5118KB) L2 L#0 (512KB) + L1d L#0 (64KB) + L1i L#0 (64KB) + Core L#0 + PU L#0 (P#0) L2 L#1 (512KB) + L1d L#1 (64KB) + L1i L#1 (64KB) + Core L#1 + PU L#1 (P#1) L2 L#2 (512KB) + L1d L#2 (64KB) + L1i L#2 (64KB) + Core L#2 + PU L#2 (P#2) L2 L#3 (512KB) + L1d L#3 (64KB) + L1i L#3 (64KB) + Core L#3 + PU L#3 (P#3) HostBridge L#0 PCIBridge PCI 8086:10c9 Net L#0 "eth0" PCI 8086:10c9 Net L#1 "eth1" PCI 1002:4390 Block L#2 "sda" PCI 1002:439c PCIBridge PCI 102b:0532 NUMANode L#1 (P#1 32GB) + L3 L#1 (5118KB) L2 L#4 (512KB) + L1d L#4 (64KB) + L1i L#4 (64KB) + Core L#4 + PU L#4 (P#4) L2 L#5 (512KB) + L1d L#5 (64KB) + L1i L#5 (64KB) + Core L#5 + PU L#5 (P#5) L2 L#6 (512KB) + L1d L#6 (64KB) + L1i L#6 (64KB) + Core L#6 + PU L#6 (P#6) L2 L#7 (512KB) + L1d L#7 (64KB) + L1i L#7 (64KB) + Core L#7 + PU L#7 (P#7) Socket L#1 (64GB) NUMANode L#2 (P#2 32GB) + L3 L#2 (5118KB) L2 L#8 (512KB) + L1d L#8 (64KB) + L1i L#8 (64KB) + Core L#8 + PU L#8 (P#8) L2 L#9 (512KB) + L1d L#9 (64KB) + L1i L#9 (64KB) + Core L#9 + PU L#9 (P#9) L2 L#10 (512KB) + L1d L#10 (64KB) + L1i L#10 (64KB) + Core L#10 + PU L#10 (P#10) L2 L#11 (512KB) + L1d L#11 (64KB) + L1i L#11 (64KB) + Core L#11 + PU L#11 (P#11) NUMANode L#3 (P#3 32GB) + L3 L#3 (5118KB) L2 L#12 (512KB) + L1d L#12 (64KB) + L1i L#12 (64KB) + Core L#12 + PU L#12 (P#12) L2 L#13 (512KB) + L1d L#13 (64KB) + L1i L#13 (64KB) + Core L#13 + PU L#13 (P#13) L2 L#14 (512KB) + L1d L#14 (64KB) + L1i L#14 (64KB) + Core L#14 + PU L#14 (P#14) L2 L#15 (512KB) + L1d L#15 (64KB) + L1i L#15 (64KB) + Core L#15 + PU L#15 (P#15) Socket L#2 (64GB) NUMANode L#4 (P#4 32GB) + L3 L#4 (5118KB) L2 L#16 (512KB) + L1d L#16 (64KB) + L1i L#16 (64KB) + Core L#16 + PU L#16 (P#16) L2 L#17 (512KB) + L1d L#17 (64KB) + L1i L#17 (64KB) + Core L#17 + PU L#17 (P#17) L2 L#18 (512KB) + L1d L#18 (64KB) + L1i L#18 (64KB) + Core L#18 + PU L#18 (P#18) L2 L#19 (512KB) + L1d L#19 (64KB) + L1i L#19 (64KB) + Core L#19 + PU L#19 (P#19) NUMANode L#5 (P#5 32GB) + L3 L#5 (5118KB) L2 L#20 (512KB) + L1d L#20 (64KB) + L1i L#20 (64KB) + Core L#20 + PU L#20 (P#20) L2 L#21 (512KB) + L1d L#21 (64KB) + L1i L#21 (64KB) + Core L#21 + PU L#21 (P#21) L2 L#22 (512KB) + L1d L#22 (64KB) + L1i L#22 (64KB) + Core L#22 + PU L#22 (P#22) L2 L#23 (512KB) + L1d L#23 (64KB) + L1i L#23 (64KB) + Core L#23 + PU L#23 (P#23) Socket L#3 (64GB) NUMANode L#6 (P#6 32GB) + L3 L#6 (5118KB) L2 L#24 (512KB) + L1d L#24 (64KB) + L1i L#24 (64KB) + Core L#24 + PU L#24 (P#24) L2 L#25 (512KB) + L1d L#25 (64KB) + L1i L#25 (64KB) + Core L#25 + PU L#25 (P#25) L2 L#26 (512KB) + L1d L#26 (64KB) + L1i L#26 (64KB) + Core L#26 + PU L#26 (P#26) L2 L#27 (512KB) + L1d L#27 (64KB) + L1i L#27 (64KB) + Core L#27 + PU L#27 (P#27) NUMANode L#7 (P#7 32GB) + L3 L#7 (5118KB) L2 L#28 (512KB) + L1d L#28 (64KB) + L1i L#28 (64KB) + Core L#28 + PU L#28 (P#28) L2 L#29 (512KB) + L1d L#29 (64KB) + L1i L#29 (64KB) + Core L#29 + PU L#29 (P#29) L2 L#30 (512KB) + L1d L#30 (64KB) + L1i L#30 (64KB) + Core L#30 + PU L#30 (P#30) L2 L#31 (512KB) + L1d L#31 (64KB) + L1i L#31 (64KB) + Core L#31 + PU L#31 (P#31) the last mpich version with working binding on new systems was mpich2 version 1.4.1 Robert Spir -- Department of Mathematics Faculty of Civil Engineering Slovak University of Technology Bratislava Radlinskeho 11 813 68 Bratislava Slovakia http://www.math.sk

1 0