- devel - mailman3.cels.anl.gov

Re: [mpich-devel] when is MPICH going to be count-safe?
by Rob Latham 14 Oct '13

14 Oct '13

On Mon, Oct 14, 2013 at 02:25:49PM -0500, Rob Latham wrote: > er, maybe I should say "qualified to review, once I pass your tests": > > % ./test_type_contig.x 15 > count = 32768 > ocount = 32768 > % ./test_type_contig.x 25 > count = 33554432 > ocount = 33554432 > % ./test_type_contig.x 30 > count = 1073741824 > ocount = 1073741824 > % ./test_type_contig.x 31 > count = 2147483648 > ocount = -2147483648 > > Heh. silent error vs violent explosion. I'll let you consider which > mode is preferable while I go fix my branch. good news is that I think everything "important" is happening correctly. It's just that when the status object is updated it's picking up an overflowed integer. status.count is MPI_Count big, so that's not the source of the overflow. will look once more at my mountain of clang -Wshorten-64-to-32 related warnings in the morning. ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-devel] when is MPICH going to be count-safe?
by Rob Latham 14 Oct '13

14 Oct '13

On Sun, Oct 13, 2013 at 09:13:31PM -0500, Jeff Hammond wrote: > I'd like to be able to use MPI_Datatypes to send e.g. 4 GiB > messages. Right now, even 2 GiB fails. Right on, man. My "large count large type" branch has been stuck in review purgatory. Think you are you qualified to review mpich-review/large_count_large_typesize_combo ? > Code is here https://github.com/jeffhammond/HPCGloves/tree/master/examples/mpi64. Test machine is in MCS. er, maybe I should say "qualified to review, once I pass your tests": % ./test_type_contig.x 15 count = 32768 ocount = 32768 % ./test_type_contig.x 25 count = 33554432 ocount = 33554432 % ./test_type_contig.x 30 count = 1073741824 ocount = 1073741824 % ./test_type_contig.x 31 count = 2147483648 ocount = -2147483648 Heh. silent error vs violent explosion. I'll let you consider which mode is preferable while I go fix my branch. ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-devel] when is MPICH going to be count-safe?
by Jeff Hammond 13 Oct '13

13 Oct '13

Side note: the OpenMPI trunk gives the correct answers for this test. Jeff ----- Original Message ----- > From: "Jeff Hammond" <jhammond(a)alcf.anl.gov> > To: devel(a)mpich.org > Sent: Sunday, October 13, 2013 9:13:31 PM > Subject: when is MPICH going to be count-safe? > > I'd like to be able to use MPI_Datatypes to send e.g. 4 GiB messages. > Right now, even 2 GiB fails. > > Code is here > https://github.com/jeffhammond/HPCGloves/tree/master/examples/mpi64. > Test machine is in MCS. > > Thanks, > > Jeff > > [jhammond@protos mpi64]$ ./test_type_contig.x 15 > count = 32768 > ocount = 32768 > [jhammond@protos mpi64]$ ./test_type_contig.x 20 > count = 1048576 > ocount = 1048576 > [jhammond@protos mpi64]$ ./test_type_contig.x 25 > count = 33554432 > ocount = 33554432 > [jhammond@protos mpi64]$ ./test_type_contig.x 30 > count = 1073741824 > ocount = 1073741824 > [jhammond@protos mpi64]$ ./test_type_contig.x 31 > count = 2147483648 > Assertion failed in file > ../src/mpid/common/datatype/dataloop/segment_packunpack.c at line > 133: FALSE > memcpy argument memory ranges overlap, dst_=0x254b310 > src_=0x7f6ad81f3010 len_=-2147483648 > > internal ABORT - process 0 > [jhammond@protos mpi64]$ ./test_type_contig.x 32 > count = 4294967296 > ocount = 0 > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > ALCF docs: http://www.alcf.anl.gov/user-guides > -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

when is MPICH going to be count-safe?
by Jeff Hammond 13 Oct '13

13 Oct '13

I'd like to be able to use MPI_Datatypes to send e.g. 4 GiB messages. Right now, even 2 GiB fails. Code is here https://github.com/jeffhammond/HPCGloves/tree/master/examples/mpi64. Test machine is in MCS. Thanks, Jeff [jhammond@protos mpi64]$ ./test_type_contig.x 15 count = 32768 ocount = 32768 [jhammond@protos mpi64]$ ./test_type_contig.x 20 count = 1048576 ocount = 1048576 [jhammond@protos mpi64]$ ./test_type_contig.x 25 count = 33554432 ocount = 33554432 [jhammond@protos mpi64]$ ./test_type_contig.x 30 count = 1073741824 ocount = 1073741824 [jhammond@protos mpi64]$ ./test_type_contig.x 31 count = 2147483648 Assertion failed in file ../src/mpid/common/datatype/dataloop/segment_packunpack.c at line 133: FALSE memcpy argument memory ranges overlap, dst_=0x254b310 src_=0x7f6ad81f3010 len_=-2147483648 internal ABORT - process 0 [jhammond@protos mpi64]$ ./test_type_contig.x 32 count = 4294967296 ocount = 0 -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

Re: [mpich-devel] ROMIO for Lustre
by Rob Latham 23 Sep '13

23 Sep '13

On Sun, Sep 22, 2013 at 08:41:21PM -0500, Jaln wrote: > Interesting, Thanks Rob, > So I can assume the Hopper( a cray XE6 with MPT 3.2) contains the > lustre-specific optimizations? Hopper does. Just to be clear, don't ask for 3.2: run the newest possible MPT. You'll have to experiment a bit with the settings described in the intro_mpi man page. In my experience, things that are documented as the default are not actually the default. It's quite frustrating, but the hopper staff is quite good at answering site-specific MPI-IO questions. > Does it work both for read and write? I don't know for sure (it's a Cray, and I have not see the source code), but most of the focus was on writes since in the write path there are lustre-level locks to deal with. In the read path, lustre-level locks don't really come into play. ==rob > Jailin > > > On Sun, Sep 22, 2013 at 2:00 PM, Rob Latham <robl(a)mcs.anl.gov> wrote: > > > On Sat, Sep 21, 2013 at 11:21:19PM -0500, Jaln wrote: > > > Hi everyone, > > > > > > I'm not sure, whether the lustre or the MPI forum is the right place for > > my > > > question. > > > > both, i guess :> > > > > > The question is about the ROMIO optimization on Lustre, > > > In one SC'08 paper, > > > http://users.eecs.northwestern.edu/~wkliao/PAPERS/fd_sc08_revised.pdf< > > https://mail.ttu.edu/owa/redir.aspx?C=yUmbVUH4hUWLFEWFA2GcoiKOEhnhitAIatZfG… > > > > > > , it's said that the way ROMIO assigns the file domains to I/O > > aggregators > > > will not make two aggregators access the same OST. > > > > > > In my understanding, this means, the data locality on Lustre layer has > > been > > > taken care of in the ROMIO, such that the aggregators will not > > > have competition on the same OST. > > > > > > My question is "is this optimization used in all current lustre system, > > > e.g., Hopper at NERSC?" > > > > Wei-keng never contributed the specific ROMIO optimizations he discussed in > > the SC 08 paper, but his work did spur a lot of community discussion > > and contributions. > > > > Emoly Lu contributed a bunch of Lustre ADIO driver work, which Pascal > > Deveze and Martin Pokorny improved upon. MPICH-1.3 and newer contain > > these improvements. > > > > David Knaak from Cray implemented his own improvements. Cray's MPI-IO > > is based on ROMIO but the cray modifications are proprietary. MPT-3.2 > > and newer contain lustre-specific optimizations. > > > > The community has been quiet with respect to Lustre MPI-IO work since > > then. I hope that's because everything "just works". > > > > ==rob > > > > -- > > Rob Latham > > Mathematics and Computer Science Division > > Argonne National Lab, IL USA > > > > > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-devel] ROMIO for Lustre
by Jaln 22 Sep '13

22 Sep '13

Interesting, Thanks Rob, So I can assume the Hopper( a cray XE6 with MPT 3.2) contains the lustre-specific optimizations? Does it work both for read and write? Jailin On Sun, Sep 22, 2013 at 2:00 PM, Rob Latham <robl(a)mcs.anl.gov> wrote: > On Sat, Sep 21, 2013 at 11:21:19PM -0500, Jaln wrote: > > Hi everyone, > > > > I'm not sure, whether the lustre or the MPI forum is the right place for > my > > question. > > both, i guess :> > > > The question is about the ROMIO optimization on Lustre, > > In one SC'08 paper, > > http://users.eecs.northwestern.edu/~wkliao/PAPERS/fd_sc08_revised.pdf< > https://mail.ttu.edu/owa/redir.aspx?C=yUmbVUH4hUWLFEWFA2GcoiKOEhnhitAIatZfG… > > > > , it's said that the way ROMIO assigns the file domains to I/O > aggregators > > will not make two aggregators access the same OST. > > > > In my understanding, this means, the data locality on Lustre layer has > been > > taken care of in the ROMIO, such that the aggregators will not > > have competition on the same OST. > > > > My question is "is this optimization used in all current lustre system, > > e.g., Hopper at NERSC?" > > Wei-keng never contributed the specific ROMIO optimizations he discussed in > the SC 08 paper, but his work did spur a lot of community discussion > and contributions. > > Emoly Lu contributed a bunch of Lustre ADIO driver work, which Pascal > Deveze and Martin Pokorny improved upon. MPICH-1.3 and newer contain > these improvements. > > David Knaak from Cray implemented his own improvements. Cray's MPI-IO > is based on ROMIO but the cray modifications are proprietary. MPT-3.2 > and newer contain lustre-specific optimizations. > > The community has been quiet with respect to Lustre MPI-IO work since > then. I hope that's because everything "just works". > > ==rob > > -- > Rob Latham > Mathematics and Computer Science Division > Argonne National Lab, IL USA > -- Genius only means hard-working all one's life

1 0

Re: [mpich-devel] ROMIO for Lustre
by Rob Latham 22 Sep '13

22 Sep '13

On Sat, Sep 21, 2013 at 11:21:19PM -0500, Jaln wrote: > Hi everyone, > > I'm not sure, whether the lustre or the MPI forum is the right place for my > question. both, i guess :> > The question is about the ROMIO optimization on Lustre, > In one SC'08 paper, > http://users.eecs.northwestern.edu/~wkliao/PAPERS/fd_sc08_revised.pdf<https://mail.ttu.edu/owa/redir.aspx?C=yUmbVUH4hUWLFEWFA2GcoiKOEhnhitAIatZfG…> > , it's said that the way ROMIO assigns the file domains to I/O aggregators > will not make two aggregators access the same OST. > > In my understanding, this means, the data locality on Lustre layer has been > taken care of in the ROMIO, such that the aggregators will not > have competition on the same OST. > > My question is "is this optimization used in all current lustre system, > e.g., Hopper at NERSC?" Wei-keng never contributed the specific ROMIO optimizations he discussed in the SC 08 paper, but his work did spur a lot of community discussion and contributions. Emoly Lu contributed a bunch of Lustre ADIO driver work, which Pascal Deveze and Martin Pokorny improved upon. MPICH-1.3 and newer contain these improvements. David Knaak from Cray implemented his own improvements. Cray's MPI-IO is based on ROMIO but the cray modifications are proprietary. MPT-3.2 and newer contain lustre-specific optimizations. The community has been quiet with respect to Lustre MPI-IO work since then. I hope that's because everything "just works". ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

ROMIO for Lustre
by Jaln 21 Sep '13

21 Sep '13

Hi everyone, I'm not sure, whether the lustre or the MPI forum is the right place for my question. The question is about the ROMIO optimization on Lustre, In one SC'08 paper, http://users.eecs.northwestern.edu/~wkliao/PAPERS/fd_sc08_revised.pdf<https://mail.ttu.edu/owa/redir.aspx?C=yUmbVUH4hUWLFEWFA2GcoiKOEhnhitAIatZfG…> , it's said that the way ROMIO assigns the file domains to I/O aggregators will not make two aggregators access the same OST. In my understanding, this means, the data locality on Lustre layer has been taken care of in the ROMIO, such that the aggregators will not have competition on the same OST. My question is "is this optimization used in all current lustre system, e.g., Hopper at NERSC?" Thanks, Jailin Liu Ph. D student Texas Tech University

1 0

Re: [mpich-devel] Message Passing models for Hybrid Parallelism
by Pavan Balaji 05 Sep '13

05 Sep '13

You can create shared memory in regular MPI as well. I'm not sure you need HMPI extensions for this. -- Pavan On Aug 29, 2013, at 1:59 PM, Huiwei Lu wrote: > Thank you Shigang. Your links are very helpful. > > Sorry Muhammad I didn't answer your question at the first time. I have just read about HMPI briefly several days ago. > > > Huiwei Lu > http://www.mcs.anl.gov/~huiweilu/ > > > On Thu, Aug 29, 2013 at 1:53 PM, Shigang Li <shigangli.cs(a)gmail.com> wrote: > Hi Muhammad, Huiwei, > > There is a Hybrid MPI(HMPI) library which aims at implementing MPI efficiently on both inter-node network and shared memory architecture. Here are some papers talking about this, which may give you some info. > > Ownership Passing: Efficient Distributed Memory Programming on Multi-core Systems > NUMA-Aware Shared Memory Collective Communication for MPI > Hybrid MPI: Efﬁcient Message Passing for Multi-core Systems > > Best Regards, > Shigang > > > On Thu, Aug 29, 2013 at 12:31 PM, Muhammad Ansar Javed <muhammad.ansar(a)seecs.edu.pk> wrote: > Hi Huiwei, > Thank you so much for suggesting reading material. These are really useful papers but I am looking for models that provide seamless integration of two devices (Network device and Shared Memory device) into one hybrid device. Message passing mechanism should be exactly same for end user at MPI layer. > Idea is to use to use Network Device for establishing network communication between nodes and Shared memory device should be responsible for Intra-node communication on each multi-core machine. Shared Memory device is implemented using Multi-threading approach. Both Network and Shared Memory device are in working state independently. > > I hope it explains what I am really looking for. Please suggest papers that can provide guidelines for achieving this goal. > > > > On Wed, Aug 28, 2013 at 7:40 PM, Huiwei Lu <huiweilu(a)mcs.anl.gov> wrote: > Hi, Muhammad, > > As I am also starting to exploit this area, I would like to share with you some of my readings. > > Following are a few papers to help you start: > 1. Early Experiments with the OpenMP/MPI Hybrid Programming Model, http://www.mcs.anl.gov/~lusk/hybridpaper.pdf > 2. Test Suite for Evaluating Performance of MPI Implementations That Support MPI THREAD MULTIPLE, http://www.mcs.anl.gov/~thakur/papers/thread-tests.pdf > > In the second paper, you can also find some code examples for using hybrid programming model. > > Best, > > Huiwei Lu > http://www.mcs.anl.gov/~huiweilu/ > > > On Wed, Aug 28, 2013 at 12:54 AM, Muhammad Ansar Javed <muhammad.ansar(a)seecs.edu.pk> wrote: > Hi, > I am looking for message passing models for Hybrid Parallelism that are being used in MPICH implementations for the Cluster of Multi-core machines. Can you please suggest me some papers or study material that can explain communication models that incorporate network communication as well as shared memory communication. > > Thanks, > Ansar > > > > > -- > > Regards: > > Muhammad Ansar Javed > NUST SEECS > Contact: +92 334 438 9394 > Email: muhammad.ansar(a)seecs.edu.pk > > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-devel] Message Passing models for Hybrid Parallelism
by Huiwei Lu 29 Aug '13

29 Aug '13

Thank you Shigang. Your links are very helpful. Sorry Muhammad I didn't answer your question at the first time. I have just read about HMPI briefly several days ago. Huiwei Lu http://www.mcs.anl.gov/~huiweilu/ On Thu, Aug 29, 2013 at 1:53 PM, Shigang Li <shigangli.cs(a)gmail.com> wrote: > Hi Muhammad, Huiwei, > > There is a Hybrid MPI(HMPI) library which aims at implementing MPI efficiently > on both inter-node network and shared memory architecture. Here are some > papers talking about this, which may give you some info. > > Ownership Passing: Efficient Distributed Memory Programming on Multi-core > Systems<http://htor.inf.ethz.ch/publications//img/friedley-ownership-passing.pdf> > NUMA-Aware Shared Memory Collective Communication for MPI<http://dl.acm.org/citation.cfm?id=2462903> > Hybrid MPI: Efﬁcient Message Passing for Multi-core Systems<http://htor.inf.ethz.ch/publications//img/friedley-hmpi-sc13.pdf> > > Best Regards, > Shigang > > > On Thu, Aug 29, 2013 at 12:31 PM, Muhammad Ansar Javed < > muhammad.ansar(a)seecs.edu.pk> wrote: > >> Hi Huiwei, >> Thank you so much for suggesting reading material. These are really >> useful papers but I am looking for models that provide seamless integration >> of two devices (Network device and Shared Memory device) into one hybrid >> device. Message passing mechanism should be exactly same for end user at >> MPI layer. >> Idea is to use to use Network Device for establishing network >> communication between nodes and Shared memory device should >> be responsible for Intra-node communication on each multi-core machine. >> Shared Memory device is implemented using Multi-threading approach. Both >> Network and Shared Memory device are in working state independently. >> >> I hope it explains what I am really looking for. Please suggest papers >> that can provide guidelines for achieving this goal. >> >> >> >> On Wed, Aug 28, 2013 at 7:40 PM, Huiwei Lu <huiweilu(a)mcs.anl.gov> wrote: >> >>> Hi, Muhammad, >>> >>> As I am also starting to exploit this area, I would like to share with >>> you some of my readings. >>> >>> Following are a few papers to help you start: >>> 1. Early Experiments with the OpenMP/MPI Hybrid Programming Model, >>> http://www.mcs.anl.gov/~lusk/hybridpaper.pdf >>> 2. Test Suite for Evaluating Performance of MPI Implementations That >>> Support MPI THREAD MULTIPLE, >>> http://www.mcs.anl.gov/~thakur/papers/thread-tests.pdf >>> >>> In the second paper, you can also find some code examples for using >>> hybrid programming model. >>> >>> Best, >>> >>> Huiwei Lu >>> http://www.mcs.anl.gov/~huiweilu/ >>> >>> >>> On Wed, Aug 28, 2013 at 12:54 AM, Muhammad Ansar Javed < >>> muhammad.ansar(a)seecs.edu.pk> wrote: >>> >>>> Hi, >>>> I am looking for message passing models for Hybrid Parallelism that are >>>> being used in MPICH implementations for the Cluster of Multi-core machines. >>>> Can you please suggest me some papers or study material that can explain >>>> communication models that incorporate network communication as well as >>>> shared memory communication. >>>> >>>> Thanks, >>>> Ansar >>>> >>> >>> >> >> >> -- >> >> Regards: >> * >> *Muhammad Ansar Javed >> NUST SEECS >> Contact: +92 334 438 9394 >> Email: muhammad.ansar(a)seecs.edu.pk >> > >

1 0