- devel - mailman3.cels.anl.gov

Re: [mpich-devel] Fix for documentation ready for review
by Kenneth Raffenetti 28 Jul '16

28 Jul '16

I'm backed up reviewing changes for the upcoming 3.3a1 release. If nobody else jumps to review this, I can do it in a couple weeks. Ken On 07/27/2016 02:36 PM, William Gropp wrote: > I have a fix for #2340, missing documentation. This restores incorrectly deleted files and updates them to include MPI 3.1 constants. Who can review this? > > Bill > > William Gropp > Director, Parallel Computing Institute > Thomas M. Siebel Chair in Computer Science > Chief Scientist, NCSA > University of Illinois Urbana-Champaign > > > > > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel >

1 0

Re: [mpich-devel] mpi process wireup and apache yarn
by Kenneth Raffenetti 28 Jul '16

28 Jul '16

On 07/27/2016 12:31 AM, Ryan Lewis wrote: > However, assuming that this is all I need, it seems that essentially > each YARN container needs to execute these command lines: > > /usr/lib64/mpich/bin/hydra_pmi_proxy --control-port skynet03:58584 > --rmk user --launcher manual --demux poll --pgid 0 --retries 10 --usize > -2 --proxy-id 0 > > Which they can get from starting the mpi control process on the machine > which runs the YARN Application Master. > > And then they will all just work. Is this accurate? Is this a > "supported" mode of operation? this certainly is an extremely easy way > to get MPI to run on top of YARN, with zero code change necessary to > the MPICH codebase. I'm not sure how portable (across MPI > implementations) this is though, but, for now I don't care. This is correct. In a typical launch mode, the PMI server (mpiexec) takes care of starting the proxies for you, but the proxy startup commands are the same as the manual ones you discovered. Ken

1 0

Fix for documentation ready for review
by William Gropp 27 Jul '16

27 Jul '16

I have a fix for #2340, missing documentation. This restores incorrectly deleted files and updates them to include MPI 3.1 constants. Who can review this? Bill William Gropp Director, Parallel Computing Institute Thomas M. Siebel Chair in Computer Science Chief Scientist, NCSA University of Illinois Urbana-Champaign

1 0

mpi process wireup and apache yarn
by Ryan Lewis 27 Jul '16

27 Jul '16

Hi, I am doing a bit of experimentation with the goal of getting MPI to run on top of Apache YARN. I know that a few others have written here looking for help with mpich2-yarn, and the strangely unreleased hamster project on the hadoop JIRA. I'm not interested in those things. I am writing this note to document my progress so far, and get some confirmation that what I am doing is considered a "supported" mode of operation. For context, within YARN, a java based process called a "YARN Application Master: submits requests for resources to the YARN ResourceManager and launches "YARN Containers" via its own process launcher. There can be many AppMasters and each of them may do different things. As a proof of concept, I want to make a given Application Master request N containers, and within each of them start the individual mpi processes. I'm used to well, honestly, never dealing with any of this, and showing up at some cluster where SLURM (or whatever) already exists, and all I need to do is just write code compile it and 'qsub.' So all of this is a learning experience. Looking at Hydra, it seems that the intention is for Hydra to start processes, and oddly (and surprising to me) it is designed to need to _ask_ a RM for resources. With different logic for different RM. There is not a huge amount of documentation here, and so I was largely flying blind. I was expecting that a RM just starts processes on machines, and wire up just happens via some set of environment variable commands and shell commands, and perhaps black magic. After some googling, and private discussion with Jeff Hammond, he pointed me and the -launcher manual flag for mpirun. By issuing: 1. rlewis@skynet03 build]$ mpirun -np 2 -launcher manual -hosts skynet01,skynet02 a.out I was able to get these two hydra_pmi_proxy command lines, which after running both on the two machines, roughly seem to make my mpi program execute normally. 1. HYDRA_LAUNCH: /usr/lib64/mpich/bin/hydra_pmi_proxy --control-port skynet03:58584 --rmk user --launcher manual --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 0 2. HYDRA_LAUNCH: /usr/lib64/mpich/bin/hydra_pmi_proxy --control-port skynet03:58584 --rmk user --launcher manual --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 1 3. HYDRA_LAUNCH_END However, using MPI_Send, I would then see this occur: 1. Fatal error in MPI_Send: A process has failed, error stack: 2. MPI_Send(171)..............: MPI_Send(buf=0x7fff2e38b044, count=1, MPI_INT, dest=1, tag=0, MPI_COMM_WORLD) failed 3. MPID_nem_tcp_connpoll(1833): Communication error with rank 1: Connection refused 4. 5. ===================================================================== It seems that when I add the option: `-disable-hostname-propagation` the underlying code seems to work. I'm not exactly sure if this is an accident. However, assuming that this is all I need, it seems that essentially each YARN container needs to execute these command lines: /usr/lib64/mpich/bin/hydra_pmi_proxy --control-port skynet03:58584 --rmk user --launcher manual --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 0 Which they can get from starting the mpi control process on the machine which runs the YARN Application Master. And then they will all just work. Is this accurate? Is this a "supported" mode of operation? this certainly is an extremely easy way to get MPI to run on top of YARN, with zero code change necessary to the MPICH codebase. I'm not sure how portable (across MPI implementations) this is though, but, for now I don't care. Best, -rhl

1 0

Re: [mpich-devel] ignoring status cost
by Jeff Hammond 27 May '16

27 May '16

Thanks. src/mpi/pt2pt/waitall.c has both code and comments indicating that it matters. The ideas for that code came from IBM (14f8143312e0dfd5eafcaedda8a874f66b912d1a), which presumably means that it helped on Blue Gene. It is easy enough to measure message rate with and without an array of status objects to understand this quantitatively. Best, Jeff On Fri, May 27, 2016 at 6:40 AM, William Gropp <wgropp(a)illinois.edu> wrote: > > This was the motivation, more than avoiding declaring a variable that was never read in the user’s program. There was evidence for this from Intel in the early ‘90s, though I don’t know if anyone has looked into it recently. > > Bill > > William Gropp > Director, Parallel Computing Institute > Thomas M. Siebel Chair in Computer Science > Chief Scientist, NCSA > University of Illinois Urbana-Champaign > > > > > > On May 26, 2016, at 6:30 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > Is it correct to assume that it is cheaper to check for MPI_STATUS_IGNORE and MPI_STATUSES_IGNORE than to fill in the status object, in the case where the user does not need the status? > > Thanks, > > Jeff > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel > > > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-devel] ignoring status cost
by William Gropp 27 May '16

27 May '16

This was the motivation, more than avoiding declaring a variable that was never read in the user’s program. There was evidence for this from Intel in the early ‘90s, though I don’t know if anyone has looked into it recently. Bill William Gropp Director, Parallel Computing Institute Thomas M. Siebel Chair in Computer Science Chief Scientist, NCSA University of Illinois Urbana-Champaign On May 26, 2016, at 6:30 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > Is it correct to assume that it is cheaper to check for MPI_STATUS_IGNORE and MPI_STATUSES_IGNORE than to fill in the status object, in the case where the user does not need the status? > > Thanks, > > Jeff > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel

1 0

ignoring status cost
by Jeff Hammond 26 May '16

26 May '16

Is it correct to assume that it is cheaper to check for MPI_STATUS_IGNORE and MPI_STATUSES_IGNORE than to fill in the status object, in the case where the user does not need the status? Thanks, Jeff -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-devel] romio build failure on OpenBSD >= 5.7
by Rob Latham 25 May '16

25 May '16

On 05/15/2016 08:38 PM, Gilles Gouaillardet wrote: > Folks, > > currently, mpich (master and previous releases) cannot be built on > OpenBSD 5.7 and later. > the root cause is 'struct statfs' has no f_type field. > > here is attached a patch that fixes that, and add some extra #ifdef > protections > > this issue was initially reported at > https://www.open-mpi.org/community/lists/devel/2015/05/17449.php and > more recently at https://github.com/open-mpi/ompi/issues/1635 this has been committed to master. thanks for catching and reporting the build error and making ROMIO's configure checks more robust. ==rob

1 0

Re: [mpich-devel] small MPI_Win_allocate
by Balaji, Pavan 19 May '16

19 May '16

Yes, TLB would be a concern, particularly because shared memory (mmap backed by a file) uses regular-sized pages, not large pages. The shared memory pool is a bit messy because we have to remember how much memory in each page was already given away and deal with fragmentation, etc. A simpler "optimization" might be to turn off noncontiguity after some number of window allocations. -- Pavan > On May 17, 2016, at 2:31 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > I always ask for noncontig window allocations. I use as many RMA info keys as possible :-) > > I am not so worried about running out of VA bits, although 48 bits don't go as far as they used to, as I am about TLB slots. > > However, this problem can be solved above MPI, it just requires me to do it in every MPI-3 code I write. > > Jeff > > On Tue, May 17, 2016 at 12:20 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > It does allocate from shared memory irrespective of the size. It's not page-aligned unless you ask for noncontig allocations. With respect to wasted VM space, you'll run out of window contexts much before you run out of VM space, so I'm not sure this is a worthwhile optimization. > > -- Pavan > > > On May 12, 2016, at 4:54 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > > > If I call MPI_Win_allocate for e.g. 8 bytes, does it allocate shared memory every time, and thus potentially allocate with page alignment, which could cause a lot of empty VM space? > > > > Perhaps I do not understand POSIX shared memory well enough, but it strikes me that it might be a worthwhile optimization to have a memory pool for small-size MPI_Win_allocate(_shared) calls. > > > > Thanks, > > > > Jeff > > > > -- > > Jeff Hammond > > jeff.science(a)gmail.com > > http://jeffhammond.github.io/ > > _______________________________________________ > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/devel > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel

1 0

Re: [mpich-devel] small MPI_Win_allocate
by Jeff Hammond 17 May '16

17 May '16

I always ask for noncontig window allocations. I use as many RMA info keys as possible :-) I am not so worried about running out of VA bits, although 48 bits don't go as far as they used to, as I am about TLB slots. However, this problem can be solved above MPI, it just requires me to do it in every MPI-3 code I write. Jeff On Tue, May 17, 2016 at 12:20 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > It does allocate from shared memory irrespective of the size. It's not > page-aligned unless you ask for noncontig allocations. With respect to > wasted VM space, you'll run out of window contexts much before you run out > of VM space, so I'm not sure this is a worthwhile optimization. > > -- Pavan > > > On May 12, 2016, at 4:54 PM, Jeff Hammond <jeff.science(a)gmail.com> > wrote: > > > > If I call MPI_Win_allocate for e.g. 8 bytes, does it allocate shared > memory every time, and thus potentially allocate with page alignment, which > could cause a lot of empty VM space? > > > > Perhaps I do not understand POSIX shared memory well enough, but it > strikes me that it might be a worthwhile optimization to have a memory pool > for small-size MPI_Win_allocate(_shared) calls. > > > > Thanks, > > > > Jeff > > > > -- > > Jeff Hammond > > jeff.science(a)gmail.com > > http://jeffhammond.github.io/ > > _______________________________________________ > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/devel > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0