- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Affinity with MPICH_ASYNC_PROGRESS
by Jeff Hammond 23 Feb '13

23 Feb '13

>> >> For example, on an 8-core node, I was hoping to get async progress on >> >> 7 processes by pinning 7 comm threads to the 8th core. >> > >> > Did this work at all? >> >> What is your definition of work? > > Does it make better async progress than the default? There's absolutely no reason why it should. I am not trying to maximize progress; I am trying to maximize computational performance. Pinning the 7 comm threads to one core is going to be terrible for them, but I am assuming that I don't need that much progress, whereas I do need to the computation to run at top speed. DGEMM on 7 cores and 1 core of MPI should be much better than DGEMM on 8 cores where each core is time-shared with a comm thread. >> As far as I know, there is no API in MPICH for controlling thread >> affinity. The right way to improve this could would be to move it >> inside of Nemesis and then add support for hwloc for comm threads. I >> assume you can move the parent processes around such that your 7 >> comm-intensive procs are closer to the NIC though. You should look at >> hwloc though. > > I'm thinking about how to interact with other threads _because_ I'm writing > hwloc-based affinity management now. The behavior that I think people will > want is to use MPI to set process affinity and then tell me to use some > affinity policy to _restrict_ the process affinity to each thread. I prefer > using MPI to set process affinity because I think it's more messy for > software at a higher level than MPI to "figure out" what other processes are > running on the same node and agree among themselves how to divvy up > resources. If there was an important configuration that could not be > supported this way, however, then I suppose I could do it. You should run Intel MPI, MKL, OpenMP and TBB and only compile your code with Intel compilers. That's your best chance to have everything work together. I really don't see how Hydra is supposed to know what GOMP is doing and try to deal with it. Maybe I am reading too much into your statement, but it sounds a lot like "give me a low calorie stout that tastes fantastic." :-) >> My diff w.r.t. the SVN trunk (I did all of this before the SVN->Git >> conversion) is below. It is clearly a hack and I don't care. It only >> works on Linux or other systems that support CPU_SET. It does not >> work on my Mac, for example. >> >> I have not done very much experimenting with this code other than to >> verify that it works (as in "does not crash and gives the same result >> for cpi"). Eventually, I am going to see how it works with ARMCI-MPI. > > Cool, I'll be curious. What testing I have done of MPICH_NEMESIS_ASYNC_PROGRESS=1 on Cray XC30 indicates that NWChem is better off without comm threads since it communicates and computes at fine enough granularity such that the lack of progress is less than the overhead of internal locking (because ASYNC_PROGRESS implies MPI_THREAD_MULTIPLE) and competition for execution resources (even though XC30 has Intel SNB with HT enabled). Jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-discuss] Affinity with MPICH_ASYNC_PROGRESS
by Jed Brown 23 Feb '13

23 Feb '13

On Sat, Feb 23, 2013 at 7:01 PM, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: > >> For example, on an 8-core node, I was hoping to get async progress on > >> 7 processes by pinning 7 comm threads to the 8th core. > > > > Did this work at all? > > What is your definition of work? Does it make better async progress than the default? > As far as I know, there is no API in MPICH for controlling thread > affinity. The right way to improve this could would be to move it > inside of Nemesis and then add support for hwloc for comm threads. I > assume you can move the parent processes around such that your 7 > comm-intensive procs are closer to the NIC though. You should look at > hwloc though. > I'm thinking about how to interact with other threads _because_ I'm writing hwloc-based affinity management now. The behavior that I think people will want is to use MPI to set process affinity and then tell me to use some affinity policy to _restrict_ the process affinity to each thread. I prefer using MPI to set process affinity because I think it's more messy for software at a higher level than MPI to "figure out" what other processes are running on the same node and agree among themselves how to divvy up resources. If there was an important configuration that could not be supported this way, however, then I suppose I could do it. > My diff w.r.t. the SVN trunk (I did all of this before the SVN->Git > conversion) is below. It is clearly a hack and I don't care. It only > works on Linux or other systems that support CPU_SET. It does not > work on my Mac, for example. > > I have not done very much experimenting with this code other than to > verify that it works (as in "does not crash and gives the same result > for cpi"). Eventually, I am going to see how it works with ARMCI-MPI. Cool, I'll be curious.

1 0

Re: [mpich-discuss] Maximum number of communicators
by Jeff Hammond 23 Feb '13

23 Feb '13

It's 2048 in all MPICH2-derived implementations that I know about, including the ones used on BGP and BGQ. Since I know what your issue is already, I think the problem has something to do with context id leaking. You should post your test code. Jeff On Sat, Feb 23, 2013 at 6:24 PM, Nichols A. Romero <naromero(a)alcf.anl.gov> wrote: > Hi, > > What is the maximum number of communicators in the MPICH that is used on BG/P vs. that used on BG/Q? > > Sent from my iPhone > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-discuss] Affinity with MPICH_ASYNC_PROGRESS
by Jeff Hammond 23 Feb '13

23 Feb '13

>> My understanding, having looked at the code for this, is that no >> thread binding is set. The OS will hopefully do the right thing when >> the cores are undersubscribed, but I see no thread affinity code in >> the MPICH source for the comm threads. MVAPICH has some additional >> (i.e. non-MPICH) affinity code but I think it is mostly for >> process-binding. >> >> I have a modified version of MPICH that allows the user to set the >> affinity of the comm threads explicitly. I was interested in pinning >> all of the comm threads to one core and letting them fight for time. >> For example, on an 8-core node, I was hoping to get async progress on >> 7 processes by pinning 7 comm threads to the 8th core. > > Did this work at all? What is your definition of work? I have done no performance tests yet because I the systems I care about the most have their own async mechanisms. >> My patches for setting comm thread affinity are pretty simple. I >> assume you want me to share them? > > Sure, but I'm more interested in whether there will be an API for > determining this information. That is, suppose I have a node running 4 MPI > ranks consisting of 15 user threads each . Those 15 threads are split > between two dies and will be running in two groups: one group of 8 doing > entirely local work and one group of 7 that communicates frequently. If I > could detect the affinity of the comm thread, I could choose how to set > affinity for the application threads to get the layout I wanted. As far as I know, there is no API in MPICH for controlling thread affinity. The right way to improve this could would be to move it inside of Nemesis and then add support for hwloc for comm threads. I assume you can move the parent processes around such that your 7 comm-intensive procs are closer to the NIC though. You should look at hwloc though. My diff w.r.t. the SVN trunk (I did all of this before the SVN->Git conversion) is below. It is clearly a hack and I don't care. It only works on Linux or other systems that support CPU_SET. It does not work on my Mac, for example. I have not done very much experimenting with this code other than to verify that it works (as in "does not crash and gives the same result for cpi"). Eventually, I am going to see how it works with ARMCI-MPI. [jhammond@flogin1 init]$ svn info Path: . URL: https://svn.mcs.anl.gov/repos/mpi/mpich2/trunk/src/mpi/init Repository Root: https://svn.mcs.anl.gov/repos/mpi Repository UUID: a5d90c62-d51d-0410-9f91-bf5351168976 Revision: 10808 Node Kind: directory Schedule: normal Last Changed Author: balaji Last Changed Rev: 10804 Last Changed Date: 2012-12-25 22:48:46 -0600 (Tue, 25 Dec 2012) [jhammond@flogin1 init]$ svn diff *.c Index: async.c =================================================================== --- async.c (revision 10808) +++ async.c (working copy) @@ -4,6 +4,17 @@ * See COPYRIGHT in top-level directory. */ +#ifndef REMOVE_HACKING +/*these have to go before the MPICH headers since they include these without the GNU macro set */ +# define _GNU_SOURCE +# define __USE_GNU +# include <unistd.h> +# include <sched.h> +# if !defined(CPU_ZERO) || !defined(CPU_SET) +# error CPU_ZERO and/or CPU_SET not defined +# endif +#endif + #include "mpiimpl.h" #include "mpi_init.h" #include "mpiu_thread.h" @@ -19,6 +30,25 @@ #define WAKE_TAG 100 +/* Jeff: from http://stackoverflow.com/questions/1407786/how-to-set-cpu-affinity-of-a-par… */ +static void bind_thread_to_core(void) { + int num_cores = (int) sysconf(_SC_NPROCESSORS_ONLN); + int core_id = num_cores-1; + + cpu_set_t cpuset; + CPU_ZERO(&cpuset); + CPU_SET(core_id, &cpuset); + + pthread_t current_thread = pthread_self(); + int rc = pthread_setaffinity_np(current_thread, sizeof(cpu_set_t), &cpuset); + if (rc!=0) fprintf(stderr, "pthread_setaffinity_np rc = %d \n", rc); + else fprintf(stderr,"bound async thread to core %d \n", core_id); + + return; +} + +/**************************/ + #undef FUNCNAME #define FUNCNAME progress_fn #undef FCNAME @@ -29,12 +59,20 @@ MPID_Request *request_ptr = NULL; MPI_Request request; MPI_Status status; + int same = 0; /* Explicitly add CS_ENTER/EXIT since this thread is created from * within an internal function and will call NMPI functions * directly. */ MPIU_THREAD_CS_ENTER(ALLFUNC,); + /* JEFF: this is where to put the affinity code to pin async CHT + to core N-1 on an N-core system */ + + MPL_env2bool("MPICH_ASYNC_PLACE_SAME", &same); + if (same>0) + bind_thread_to_core(); + /* FIXME: We assume that waiting on some request forces progress * on all requests. With fine-grained threads, will this still * work as expected? We can imagine an approach where a request on -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-discuss] Affinity with MPICH_ASYNC_PROGRESS
by Jed Brown 23 Feb '13

23 Feb '13

On Sat, Feb 23, 2013 at 6:49 PM, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: > My understanding, having looked at the code for this, is that no > thread binding is set. The OS will hopefully do the right thing when > the cores are undersubscribed, but I see no thread affinity code in > the MPICH source for the comm threads. MVAPICH has some additional > (i.e. non-MPICH) affinity code but I think it is mostly for > process-binding. > > I have a modified version of MPICH that allows the user to set the > affinity of the comm threads explicitly. I was interested in pinning > all of the comm threads to one core and letting them fight for time. > For example, on an 8-core node, I was hoping to get async progress on > 7 processes by pinning 7 comm threads to the 8th core. > Did this work at all? > My patches for setting comm thread affinity are pretty simple. I > assume you want me to share them? > Sure, but I'm more interested in whether there will be an API for determining this information. That is, suppose I have a node running 4 MPI ranks consisting of 15 user threads each . Those 15 threads are split between two dies and will be running in two groups: one group of 8 doing entirely local work and one group of 7 that communicates frequently. If I could detect the affinity of the comm thread, I could choose how to set affinity for the application threads to get the layout I wanted.

1 0

Re: [mpich-discuss] Affinity with MPICH_ASYNC_PROGRESS
by Jeff Hammond 23 Feb '13

23 Feb '13

My understanding, having looked at the code for this, is that no thread binding is set. The OS will hopefully do the right thing when the cores are undersubscribed, but I see no thread affinity code in the MPICH source for the comm threads. MVAPICH has some additional (i.e. non-MPICH) affinity code but I think it is mostly for process-binding. I have a modified version of MPICH that allows the user to set the affinity of the comm threads explicitly. I was interested in pinning all of the comm threads to one core and letting them fight for time. For example, on an 8-core node, I was hoping to get async progress on 7 processes by pinning 7 comm threads to the 8th core. My patches for setting comm thread affinity are pretty simple. I assume you want me to share them? Jeff On Sat, Feb 23, 2013 at 6:23 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > Do the Hydra affinity settings apply to the internal comm thread created > using MPICH_ASYNC_PROGRESS? Is there a way to set affinity for the comm > threads so that they do not compete with application threads? > > http://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager#Proce… > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Maximum number of communicators
by Nichols A. Romero 23 Feb '13

23 Feb '13

Hi, What is the maximum number of communicators in the MPICH that is used on BG/P vs. that used on BG/Q? Sent from my iPhone

1 0

Affinity with MPICH_ASYNC_PROGRESS
by Jed Brown 23 Feb '13

23 Feb '13

Do the Hydra affinity settings apply to the internal comm thread created using MPICH_ASYNC_PROGRESS? Is there a way to set affinity for the comm threads so that they do not compete with application threads? http://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager#Proce…

1 0

Re: [mpich-discuss] stupid question about release names
by Jeff Hammond 23 Feb '13

23 Feb '13

On Sat, Feb 23, 2013 at 10:46 AM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: > What are you looking at? SVN? Git? The downloads directory on the web? (the last one is my guess) Downloads directory. > 1.5 was the last release of MPICH2. You have correctly decoded the numbering scheme. Thanks for confirming this. > Is there a reason you want MPICH2-1.5 instead of MPICH-3.0.2? The latter should be almost strictly better... I want to verify that one can build the official MPICH2 1.5 release on Blue Gene/Q. The MPICH2 1.5 source that one can get from IBM (via ALCF) should be the same, but I like to be sure rather than assume. It's also easy to provide directions relative to the MPICH download site rather than ask people to extract the source from the BGQ driver tarballs (two are required because IBM insists on putting all the GPL m4 code in a separate GPL-only tarball from the rest of MPICH). I have been able to compile MPICH 3.0.2 as well, but there are things missing in PAMID that cause it to unusable for RMA, for example. I will eventually try to reconcile these issues, but it isn't going to happen any time soon. Jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-discuss] Problem with SystemStubs under Mac OS X 10.7.5.
by Dave Goodell 23 Feb '13

23 Feb '13

Glad to hear you sorted it out. Note that the newer install of MPICH put mpif90 in a different place because you specifically asked it to do so (with "--prefix=/Users/dougdommermuth/mpich2/mpich-install"). It should still be defaulting to "/usr/local" as the prefix in most cases when "--prefix=BLAH" is not specified. -Dave On Feb 23, 2013, at 12:41 PM CST, Douglas Dommermuth wrote: > Hello Dave, > > It was a silly mistake on my part. I had an old mpif90 in /usr/local/bin/. The most recent release of mpich puts mpif90 in another area. I did not realize. I have been using mpich for so long, I did not realize the change. Thanks for pointing me in the right direction. > > Thank you, Doug. > > On Feb 23, 2013, at 10:31 AM, Douglas Dommermuth wrote: > >> Hello Dave, >> >> Did I correctly configure the build: >> >> ./configure --prefix=/Users/dougdommermuth/mpich2/mpich-install CC=/usr/bin/icc FC=/usr/bin/ifort | & tee c.txt >> >> When I look at the commands that you just sent me, it seems to be pointing to an older version of Fortran, whereas my logical links are as follows: >> >> [dgd]% ls -al /usr/bin/ifort >> lrwxr-xr-x 1 root wheel 19 Feb 23 08:21 /usr/bin/ifort -> /usr/bin/ifort-13.0 >> [dgd]% ls -al /usr/bin/ifort-13.0 >> lrwxr-xr-x 1 root wheel 42 Feb 23 08:21 /usr/bin/ifort-13.0 -> /usr/bin/ifort-13.0-base/bin/intel64/ifort >> >> Thank you, Doug. >> >> >> On Feb 23, 2013, at 10:13 AM, Dave Goodell wrote: >> >>> On Feb 23, 2013, at 11:52 AM CST, Douglas Dommermuth wrote: >>> >>>> I have attached the files that you requested (files.tar.gz). I could not find this file: mpich-3.0.2/src/pm/hydra/tools/topo/hwloc/hwloc/config.log. The output for compling pi3f90.f90 is below: >>>> >>>> mpif90 pi3f90.f90 >>>> ld: library not found for -lSystemStubs >>> >>> Did you upgrade to 10.7 (Lion) without reinstalling the Intel Fortran compiler? Most of the google hits I'm getting about this indicate that it can happen when: >>> >>> 1) you have an ifort version <12.1; or >>> >>> 2) you installed ifort, then upgraded from 10.6 (or earlier) to 10.7 (or later). >>> >>> Obviously, #1 does not appear to apply to you. So maybe #2 does? >>> >>> We do have some logic to add "-lSystemStubs" to the link line in certain circumstances, but your config.log file is showing that we are not doing that in your case: >>> >>> ----8<---- >>> pac_cv_prog_f77_and_c_stdio_libs=none >>> […] >>> pac_cv_prog_fc_and_c_stdio_libs=none >>> ----8<---- >>> >>> Further debug might be accomplished by inspecting the output of these two commands: >>> >>> ----8<---- >>> mpif90 -show pi3f90.f90 >>> mpif90 -v pi3f90.f90 >>> ----8<---- >>> >>> The first will show the ifort command that MPICH intends to execute. The second will show the commands that the ifort compiler driver executes as it attempts to compile your program. This should show us which part of the stack is incorrectly adding "-lSystemStubs" to your link line. My guess is that it's your ifort install, based on the other information that you have sent so far. >>> >>> -Dave >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0