April 2017 - discuss - mailman3.cels.anl.gov

Does mpich support ipv6
by Shuwei Zhao 24 Apr '17

24 Apr '17

Hi all ， I have a question right as the topic. Does the newest mpich support IPv6, I didn't find it on the user manual. Best _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Hanging behavior with derived types in a 'user-defined gatherv'
by Jeff Hammond 21 Apr '17

21 Apr '17

On Fri, Apr 21, 2017 at 1:31 PM, Latham, Robert J. <robl(a)mcs.anl.gov> wrote: > > On Fri, 2017-04-21 at 16:56 +0000, Sewall, Jason wrote: > > Folks, > > > > I have been working on some code that does something akin to a > > 'manual' implementation of gatherv with derived types. I've run into > > a case where some requests never complete. I think it's a bug in > > MPICH.: > > good news, maybe? > > I can't reproduce this with today's MPICH . I get some debug-logging > warnings that you didn't free two of your types, but it doesn't hang on > my laptop. Those datatype-related allocations are the only valgrind > error I see when I run "mpiexec -np 3 ./mpi-gather 256 256" > I was able to reproduce it with d8e9213647e782b77a1e72fde6a7638198ac3be5 on Mac, but only when I ran with MPIR_CVAR_NOLOCAL=1. Jeff > > Is it possible a 256 by 256 grids could overflow an integer anywhere? > I think master has some integer overflow fixes in the gather path that > might not have made it into an MPICH release. Aside from that, I'd > have to dig through the history to figure out what might be different. > > ==rob > > > > > My minimal code example is still ~300 lines long, so I've stuck it, > > as well as a run script and some example output here: > > > > https://gist.github.com/jasonsewall/0d1fb12a93e157795786e560733cbf0c > > > > I'm new to this list, so if there's a better place/practice for > > sharing code, please let me know. > > > > The code mimics a decomposition of a multi-field grid in 2 > > dimensions. Each grid has a halo region allocated, and this gather is > > only interested in transmitting the interior of each grid. > > > > The conditions, as far as I can tell, are related to: > > > > 1. nranks > 2 > > 2. 'larger' messages: I haven't fully bisected it, but 256x128 grids > > work, and 256x256 don't. > > 3. Having halos. With the global integer halow = 2, the strides for > > the grids (and the resulting hvector derived types) exceed the > > lengths, and that seems to be important; if you set halow = 0 and > > run, it completes, independent of the other factors. > > > > The test program uses Waitsome to figure out which requests are > > completing, and it looks like rank 0 fails to complete a recv from > > rank 2, despite rank 2 completing its send. > > > > You can fiddle with the hacky 'mask' parameter to have a subset of > > the ranks send (line 326). -1ULL means all ranks are allowed in, but > > you get interesting results with 6ULL, which has just ranks 1 and 2 > > participate in sending, but rank 0 still hangs on the recv from rank > > 2. (Obviously the correctness test will fail here, since it expects > > results from all ranks). > > > > I discovered this on Intel MPI, and thanks to my colleague Jeff > > Hammond's analysis, was able to reproduce the behavior on MPICH 3.2. > > OpenMPI 2.1.0 successfully completes every case I have tried. > > > > The obvious caveat here is that there could very well be a bug in how > > I'm using the derived datatypes, but I have tried to carefully check > > that I'm not trashing random memory. Things seem correct. > > > > Please let me know what I can do help pinpoint the problem. I'm not > > familiar with the implementation details of any MPI libraries. > > > > Cheers, > > Jason > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Hanging behavior with derived types in a 'user-defined gatherv'
by Sewall, Jason 21 Apr '17

21 Apr '17

> From: Latham, Robert J. [mailto:[email protected]] > Sent: Friday, April 21, 2017 4:32 PM > > > good news, maybe? > > I can't reproduce this with today's MPICH . I get some debug-logging > warnings that you didn't free two of your types, but it doesn't hang on > my laptop. Those datatype-related allocations are the only valgrind > error I see when I run "mpiexec -np 3 ./mpi-gather 256 256" That *is* good news. Thanks! The leak is probably because I didn't call MPI_Type_free on the tmp arrays I use to build up the compound types. They are never committed, and it wasn't clear to me if they needed to be freed or not. > Is it possible a 256 by 256 grids could overflow an integer anywhere? > I think master has some integer overflow fixes in the gather path that > might not have made it into an MPICH release. Aside from that, I'd > have to dig through the history to figure out what might be different. It's tricky to bisect when the error is 'this thing doesn't return at all'! I don't see how that size grid could get in trouble with overflow. It should use (256+4)^2 * 4 * 8 bytes = ~2Mb. The fact that it only shows up with 3 or more ranks suggest that it might be more than just the grid size, but maybe I'm wrong. Now I need to see about getting the Intel MPI fixed... Cheers, Jason _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Hanging behavior with derived types in a 'user-defined gatherv'
by Latham, Robert J. 21 Apr '17

21 Apr '17

On Fri, 2017-04-21 at 16:56 +0000, Sewall, Jason wrote: > Folks, > > I have been working on some code that does something akin to a > 'manual' implementation of gatherv with derived types. I've run into > a case where some requests never complete. I think it's a bug in > MPICH.: good news, maybe? I can't reproduce this with today's MPICH . I get some debug-logging warnings that you didn't free two of your types, but it doesn't hang on my laptop. Those datatype-related allocations are the only valgrind error I see when I run "mpiexec -np 3 ./mpi-gather 256 256" Is it possible a 256 by 256 grids could overflow an integer anywhere? I think master has some integer overflow fixes in the gather path that might not have made it into an MPICH release. Aside from that, I'd have to dig through the history to figure out what might be different. ==rob > > My minimal code example is still ~300 lines long, so I've stuck it, > as well as a run script and some example output here: > > https://gist.github.com/jasonsewall/0d1fb12a93e157795786e560733cbf0c > > I'm new to this list, so if there's a better place/practice for > sharing code, please let me know. > > The code mimics a decomposition of a multi-field grid in 2 > dimensions. Each grid has a halo region allocated, and this gather is > only interested in transmitting the interior of each grid. > > The conditions, as far as I can tell, are related to: > > 1. nranks > 2 > 2. 'larger' messages: I haven't fully bisected it, but 256x128 grids > work, and 256x256 don't. > 3. Having halos. With the global integer halow = 2, the strides for > the grids (and the resulting hvector derived types) exceed the > lengths, and that seems to be important; if you set halow = 0 and > run, it completes, independent of the other factors. > > The test program uses Waitsome to figure out which requests are > completing, and it looks like rank 0 fails to complete a recv from > rank 2, despite rank 2 completing its send. > > You can fiddle with the hacky 'mask' parameter to have a subset of > the ranks send (line 326). -1ULL means all ranks are allowed in, but > you get interesting results with 6ULL, which has just ranks 1 and 2 > participate in sending, but rank 0 still hangs on the recv from rank > 2. (Obviously the correctness test will fail here, since it expects > results from all ranks). > > I discovered this on Intel MPI, and thanks to my colleague Jeff > Hammond's analysis, was able to reproduce the behavior on MPICH 3.2. > OpenMPI 2.1.0 successfully completes every case I have tried. > > The obvious caveat here is that there could very well be a bug in how > I'm using the derived datatypes, but I have tried to carefully check > that I'm not trashing random memory. Things seem correct. > > Please let me know what I can do help pinpoint the problem. I'm not > familiar with the implementation details of any MPI libraries. > > Cheers, > Jason > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] About mpi i/o, mpi hints and collective i/o memory usage
by pramod kumbhar 21 Apr '17

21 Apr '17

Hi Rob, Are you the same Pramod Kumbhar that works at EPFL? Yes. After seeing failure on 8-rack, I started debugging/profiling on our local 4-rack bg-q system. I planned to send an email to support team with more detailed information (for which job is currently in queue). 1: collective I/O does consume some memory. not only is there an > internal "collective buffer" maintained by MPI-IO itself, but the data > exchange copies data as well before calling ALLTOALL. > Just wondering if there any way to print or query some internal statistics about this. > Paul Coffman has done a one-sided based two-phase implementation that > should be lower memory overhead. But here we should take the > discussion off-list. > Perfect ! Thanks! Regards, Pramod ==rob > > > > > Quick Summary : > > > > 1. On bg-q I see cb_buffer_size as 16MB when we query on file handle > > using MPI_File_get_info. > > An application that we are looking at has code section like: > > > > …. > > MPI_File_set_view( fh, position_to_write, MPI_FLOAT, mappingType, > > _native_, MPI_INFO_NULL ); > > max_mb_on_any_rank_using_Kernel_GetMemorySize () => 275 MB > > MPI_File_write_all( fh, mappingBuffer, .................... > > MPI_FLOAT, &status); > > max_mb_on_any_rank_using_Kernel_GetMemorySize () => 373 MB > > …… > > > > Why we see that spike in memory usage? (see Detail section for size > > information) > > > > I have seen “Kernel_GetMemorySize(KERNEL_MEMSIZE_HEAP….)” not > > returning accurate memory footprint but I am not sure if that is the > > case here. > > Darshan screenshot attached shows the access sizes while running on 4 > > rack. > > > > 2. Is romio_cb_alltoall ignored on bg-q? Even if I disable it, I see > > “automatic” in the output. > > > > (I am looking at > > srcV1R2M4/comm/lib/dev/mpich2/src/mpi/romio/adio/ad_bg/ad_bg_hints.c > > and see the code section is commented.) > > > > More Details : > > > > We are debugging an application on MIRA which runs on 1,2,4 racks but > > fails at 8 racks while dumping a custom checkpoint. These are strong > > scaling runs and the size of checkpoint remains same (~172GB). 32 > > ranks per mode. Max memory usage before start of checkpoint (i.e. > > before single write_all call) > > for 8 rack is ~ 300 MB. The checkpoint size from each rank is between > > Kbs to few MBs (as shown by darshan). Once application call > > checkpoint, we see below error : > > > > Out of memory in file > > /bgsys/source/srcV1R2M2.15270/comm/lib/dev/mpich2/src/mpi/romio/adio/ > > ad_bg/ad_bg_wrcoll.c, line 500 > > > > And hence I am confused about behaviour mentioned in question 1. > > If someone has any insight, it will be great help! > > > > Regards, > > Pramod > > > > p.s. > > > > Default values of all hints > > > > cb_buffer_size, value = 16777216 > > romio_cb_read, value = enable > > romio_cb_write, value = enable > > cb_nodes, value = 8320 (change based on partition size) > > romio_no_indep_rw, value = false > > romio_cb_pfr, value = disable > > romio_cb_fr_types, value = aar > > romio_cb_fr_alignment, value = 1 > > romio_cb_ds_threshold, value = 0 > > romio_cb_alltoall, value = automatic > > ind_rd_buffer_size, value = 4194304 > > romio_ds_read, value = automatic > > romio_ds_write, value = disable > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Hanging behavior with derived types in a 'user-defined gatherv'
by Sewall, Jason 21 Apr '17

21 Apr '17

Folks, I have been working on some code that does something akin to a 'manual' implementation of gatherv with derived types. I've run into a case where some requests never complete. I think it's a bug in MPICH. My minimal code example is still ~300 lines long, so I've stuck it, as well as a run script and some example output here: https://gist.github.com/jasonsewall/0d1fb12a93e157795786e560733cbf0c I'm new to this list, so if there's a better place/practice for sharing code, please let me know. The code mimics a decomposition of a multi-field grid in 2 dimensions. Each grid has a halo region allocated, and this gather is only interested in transmitting the interior of each grid. The conditions, as far as I can tell, are related to: 1. nranks > 2 2. 'larger' messages: I haven't fully bisected it, but 256x128 grids work, and 256x256 don't. 3. Having halos. With the global integer halow = 2, the strides for the grids (and the resulting hvector derived types) exceed the lengths, and that seems to be important; if you set halow = 0 and run, it completes, independent of the other factors. The test program uses Waitsome to figure out which requests are completing, and it looks like rank 0 fails to complete a recv from rank 2, despite rank 2 completing its send. You can fiddle with the hacky 'mask' parameter to have a subset of the ranks send (line 326). -1ULL means all ranks are allowed in, but you get interesting results with 6ULL, which has just ranks 1 and 2 participate in sending, but rank 0 still hangs on the recv from rank 2. (Obviously the correctness test will fail here, since it expects results from all ranks). I discovered this on Intel MPI, and thanks to my colleague Jeff Hammond's analysis, was able to reproduce the behavior on MPICH 3.2. OpenMPI 2.1.0 successfully completes every case I have tried. The obvious caveat here is that there could very well be a bug in how I'm using the derived datatypes, but I have tried to carefully check that I'm not trashing random memory. Things seem correct. Please let me know what I can do help pinpoint the problem. I'm not familiar with the implementation details of any MPI libraries. Cheers, Jason _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] About mpi i/o, mpi hints and collective i/o memory usage
by Latham, Robert J. 21 Apr '17

21 Apr '17

On Fri, 2017-04-21 at 16:13 +0200, pramod kumbhar wrote: > > Dear All, > > I would like to understand some details about MPI I/O hints on bg-q > and out of memory error while doing collective i/o. Are you the same Pramod Kumbhar that works at EPFL? I only ask because it does not seem you have asked ALCF support (who run Mira) about your problem. We're going to quickly get off-topic for this general MPICH mailing list, but let me give some on-topic answers before we go discuss this on a more machine-specific venue. 1: collective I/O does consume some memory. not only is there an internal "collective buffer" maintained by MPI-IO itself, but the data exchange copies data as well before calling ALLTOALL. 1a: your darshan summary looks like it's doing the right stuff. Your tiny MPI-IO operations are getting transformed into more GPFS-friendly multi-megabyte requests. Good. Paul Coffman has done a one-sided based two-phase implementation that should be lower memory overhead. But here we should take the discussion off-list. ==rob > > Quick Summary : > > 1. On bg-q I see cb_buffer_size as 16MB when we query on file handle > using MPI_File_get_info. > An application that we are looking at has code section like: > > …. > MPI_File_set_view( fh, position_to_write, MPI_FLOAT, mappingType, > _native_, MPI_INFO_NULL ); > max_mb_on_any_rank_using_Kernel_GetMemorySize () => 275 MB > MPI_File_write_all( fh, mappingBuffer, .................... > MPI_FLOAT, &status); > max_mb_on_any_rank_using_Kernel_GetMemorySize () => 373 MB > …… > > Why we see that spike in memory usage? (see Detail section for size > information) > > I have seen “Kernel_GetMemorySize(KERNEL_MEMSIZE_HEAP….)” not > returning accurate memory footprint but I am not sure if that is the > case here. > Darshan screenshot attached shows the access sizes while running on 4 > rack. > > 2. Is romio_cb_alltoall ignored on bg-q? Even if I disable it, I see > “automatic” in the output. > > (I am looking at > srcV1R2M4/comm/lib/dev/mpich2/src/mpi/romio/adio/ad_bg/ad_bg_hints.c > and see the code section is commented.) > > More Details : > > We are debugging an application on MIRA which runs on 1,2,4 racks but > fails at 8 racks while dumping a custom checkpoint. These are strong > scaling runs and the size of checkpoint remains same (~172GB). 32 > ranks per mode. Max memory usage before start of checkpoint (i.e. > before single write_all call) > for 8 rack is ~ 300 MB. The checkpoint size from each rank is between > Kbs to few MBs (as shown by darshan). Once application call > checkpoint, we see below error : > > Out of memory in file > /bgsys/source/srcV1R2M2.15270/comm/lib/dev/mpich2/src/mpi/romio/adio/ > ad_bg/ad_bg_wrcoll.c, line 500 > > And hence I am confused about behaviour mentioned in question 1. > If someone has any insight, it will be great help! > > Regards, > Pramod > > p.s. > > Default values of all hints > > cb_buffer_size, value = 16777216 > romio_cb_read, value = enable > romio_cb_write, value = enable > cb_nodes, value = 8320 (change based on partition size) > romio_no_indep_rw, value = false > romio_cb_pfr, value = disable > romio_cb_fr_types, value = aar > romio_cb_fr_alignment, value = 1 > romio_cb_ds_threshold, value = 0 > romio_cb_alltoall, value = automatic > ind_rd_buffer_size, value = 4194304 > romio_ds_read, value = automatic > romio_ds_write, value = disable > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

About mpi i/o, mpi hints and collective i/o memory usage
by pramod kumbhar 21 Apr '17

21 Apr '17

Dear All, I would like to understand some details about MPI I/O hints on bg-q and out of memory error while doing collective i/o. Quick Summary : 1. On bg-q I see cb_buffer_size as 16MB when we query on file handle using MPI_File_get_info. An application that we are looking at has code section like: …. MPI_File_set_view( fh, position_to_write, MPI_FLOAT, mappingType, _native_, MPI_INFO_NULL ); max_mb_on_any_rank_using_Kernel_GetMemorySize () => 275 MB MPI_File_write_all( fh, mappingBuffer, .................... MPI_FLOAT, &status); max_mb_on_any_rank_using_Kernel_GetMemorySize () => 373 MB …… Why we see that spike in memory usage? (see Detail section for size information) I have seen “Kernel_GetMemorySize(KERNEL_MEMSIZE_HEAP….)” not returning accurate memory footprint but I am not sure if that is the case here. Darshan screenshot attached shows the access sizes while running on 4 rack. 2. Is romio_cb_alltoall ignored on bg-q? Even if I disable it, I see “automatic” in the output. (I am looking at srcV1R2M4/comm/lib/dev/mpich2/src/mpi/romio/adio/ad_bg/ad_bg_hints.c and see the code section is commented.) More Details : We are debugging an application on MIRA which runs on 1,2,4 racks but fails at 8 racks while dumping a custom checkpoint. These are strong scaling runs and the size of checkpoint remains same (~172GB). 32 ranks per mode. Max memory usage before start of checkpoint (i.e. before single write_all call) for 8 rack is ~ 300 MB. The checkpoint size from each rank is between Kbs to few MBs (as shown by darshan). Once application call checkpoint, we see below error : Out of memory in file /bgsys/source/srcV1R2M2.15270/comm/lib/dev/mpich2/src/mpi/romio/adio/ad_bg/ad_bg_wrcoll.c, line 500 And hence I am confused about behaviour mentioned in question 1. If someone has any insight, it will be great help! Regards, Pramod p.s. Default values of all hints cb_buffer_size, value = 16777216 romio_cb_read, value = enable romio_cb_write, value = enable cb_nodes, value = 8320 (change based on partition size) romio_no_indep_rw, value = false romio_cb_pfr, value = disable romio_cb_fr_types, value = aar romio_cb_fr_alignment, value = 1 romio_cb_ds_threshold, value = 0 romio_cb_alltoall, value = automatic ind_rd_buffer_size, value = 4194304 romio_ds_read, value = automatic romio_ds_write, value = disable _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Limitation of open files on Mira
by jt.meng＠siat.ac.cn 20 Apr '17

20 Apr '17

I notice that after sending that email. Sorry for that. jt.meng(a)siat.ac.cn From: Jeff Hammond Date: 2017-04-20 23:05 To: MPICH Subject: Re: [mpich-discuss] Limitation of open files on Mira This is the wrong email list for such a question. Please contact ALCF support regarding Mira. Jeff On Thu, Apr 20, 2017 at 1:35 AM, jt.meng(a)siat.ac.cn <jt.meng(a)siat.ac.cn> wrote: Hi Rob, Can we increase the number of open files on Mira to 1 million? Currently the number of open files is set to be 65536, and users can increase it manually to its hard limitation of 81920. Can we increase this hard limitation to 1 million for Mira for a one week test on MPI IO in Mira? Thanks. --------------- /etc/security/limits.conf--------------------------------- root soft nofile 65536 root hard nofile 81920 <-- Best, jt.meng(a)siat.ac.cn From: Latham, Robert J. Date: 2017-04-19 22:35 To: discuss(a)mpich.org Subject: Re: [mpich-discuss] Installation on Blue Gene/Q On Wed, 2017-04-19 at 13:58 +0200, pramod kumbhar wrote: > Hi Rob, > > Is this already installed/avaialble on MIRA? I would like to try this > to check some mpi i/o hints. Yes, there are several versions available: manual progress optimized: /projects/aurora_app/mpich3-ch4-ofi/install/gnu/bin auto progress optimized: /projects/aurora_app/mpich3-ch4-ofi/install-auto/gnu/bin manual progress debug: /projects/aurora_app/mpich3-ch4-ofi/install/gnu.debug/bin auto progress debug: /projects/aurora_app/mpich3-ch4-ofi/install-auto/gnu.debug/bin There are some performance considerations (both positive and negative) to consider: Current performance improvements versus [IBM's vendor-supplied MPICH]: 1.) Pt2pt latency - better than 2x speedup for small messages, large ones with few ranks-per-node less so due to detriment #2 listed below. 2.) RMA latency - exponentially better for small messages, then becomes gradually even for large messages in the case of put and get. Current performance detriments versus [IBM's vendor-supplied MPICH]: 1.) No optimized collectives -- all collectives are pt2pt algorithms in MPICH CH4. There is a plan to develop optimized collectives via OFI triggered operations but that work wouldn't begin until the April-May timeframe at the earliest. There is also a plan to make optimized pt2pt collectives available sooner. 2.) Only 1 rget injection fifo utilized per rank to pt2pt communication --- pami can utilize up all 11 to maximize internode bandwidth, you would mainly see this have an impact if you are running < 8 ranks per node and passing large messages. ==rob _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Limitation of open files on Mira
by Jeff Hammond 20 Apr '17

20 Apr '17

This is the wrong email list for such a question. Please contact ALCF support regarding Mira. Jeff On Thu, Apr 20, 2017 at 1:35 AM, jt.meng(a)siat.ac.cn <jt.meng(a)siat.ac.cn> wrote: > Hi Rob, > Can we increase the number of open files on Mira to 1 million? Currently > the number of open files is set to be 65536, and users can increase it > manually to its hard limitation of 81920. > Can we increase this hard limitation to 1 million for Mira for a one > week test on MPI IO in Mira? Thanks. > > --------------- /etc/security/limits.conf--------------------------------- > root soft nofile 65536 > root hard nofile 81920 <-- > > Best, > ------------------------------ > jt.meng(a)siat.ac.cn > > > *From:* Latham, Robert J. <robl(a)mcs.anl.gov> > *Date:* 2017-04-19 22:35 > *To:* discuss(a)mpich.org > *Subject:* Re: [mpich-discuss] Installation on Blue Gene/Q > On Wed, 2017-04-19 at 13:58 +0200, pramod kumbhar wrote: > > Hi Rob, > > > > Is this already installed/avaialble on MIRA? I would like to try this > > to check some mpi i/o hints. > > Yes, there are several versions available: > > manual progress optimized: > /projects/aurora_app/mpich3-ch4-ofi/install/gnu/bin > > auto progress optimized: > /projects/aurora_app/mpich3-ch4-ofi/install-auto/gnu/bin > > manual progress debug: > /projects/aurora_app/mpich3-ch4-ofi/install/gnu.debug/bin > > auto progress debug: > /projects/aurora_app/mpich3-ch4-ofi/install-auto/gnu.debug/bin > > There are some performance considerations (both positive and negative) > to consider: > > Current performance improvements versus [IBM's vendor-supplied MPICH]: > > 1.) Pt2pt latency - better than 2x speedup for small messages, large > ones with few ranks-per-node less so due to detriment #2 listed below. > > 2.) RMA latency - exponentially better for small messages, then becomes > gradually even for large messages in the case of put and get. > > Current performance detriments versus [IBM's vendor-supplied MPICH]: > > 1.) No optimized collectives -- all collectives are pt2pt algorithms in > MPICH CH4. There is a plan to develop optimized collectives via OFI > triggered operations but that work wouldn't begin until the April-May > timeframe at the earliest. There is also a plan to make optimized > pt2pt collectives available sooner. > > 2.) Only 1 rget injection fifo utilized per rank to pt2pt communication > --- pami can utilize up all 11 to maximize internode bandwidth, you > would mainly see this have an impact if you are running < 8 ranks per > node and passing large messages. > > ==rob > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0