- devel - mailman3.cels.anl.gov

Re: [mpich-devel] ROMIO collective i/o memory use
by Rob Latham 07 May '13

07 May '13

On Mon, May 06, 2013 at 03:05:10PM -0500, Bob Cernohous wrote: > Ouch. So you've seen the scaling failures at 8k... ranks? racks? > Kevin is failing at... 16 racks x 16 ranks per node... I think ... > so 256k ranks. MOAB in particular has fairly straightforward workloads that will exhaust memory (by attempting a 400 MiB memory allocation) at 2048 MPI processes. ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-devel] ROMIO collective i/o memory use
by Rob Latham 07 May '13

07 May '13

On Mon, May 06, 2013 at 01:41:07PM -0500, Bob Cernohous wrote: > I agree and suggested: > --------------------- > It appears they don't have enough memory for an alltoallv exchange. Try > '1'... > > * - BGMPIO_COMM - Define how data is exchanged on collective > > * reads and writes. Possible values: > > * - 0 - Use MPI_Alltoallv. > > * - 1 - Use MPI_Isend/MPI_Irecv. > > * - Default is 0. > > --------------------- > > but they didn't want a work around they wanted a 'fix for o(p) > allocations'. There are o(p) allocations all over collective i/o from a > quick glance. Just wanted some input from the experts about scaling > romio. I haven't heard if the suggestion worked. MOAB, a framework for "doing stuff to meshed data": simple experiment of reading and writing a mesh with 8 million "tetrahedron" elements. Setting BGMPIO_COMM to 1 serves to push the memory allocation problem only a little bit down the road. I could not run this test case until our Blue Gene /P came out of maintenance With default BG /P settings, MOAB cannot even read in the initial 8 million tet dataset with 2048 MPI processes. Set BGPMPIO_COMM to 1 and 2048 MPI processes work, but 4192 does not: I'm still working off of our /P but the code for /Q is the same: stderr[3260] Abort(-1) on node 3260: Unable to allocate non-contiguous buffer (Not enough space for file ) (that error message is misleading: it's a malloc that has returned NULL) 0x01a7e0ec raise ../nptl/sysdeps/unix/sysv/linux/raise.c:67 0x01a3ce40 abort /bgsys/drivers/V1R4M2_200_2010-100508P/ppc/toolchain/gnu/glibc-2.4/stdlib/abort.c:73 0x018a9df4 MPID_Abort /gpfs/home/robl/src/dcmf/BGP/IBM_V1R4M3/bgp/comm/lib/dev/mpich2/src/mpid/dcmfd/src/misc/mp id_abort.c:81 0x018ad4e8 MPIDI_DCMF_StartMsg /gpfs/home/robl/src/dcmf/BGP/IBM_V1R4M3/bgp/comm/lib/dev/mpich2/src/mpid/dcmfd/src/pt2pt/m pidi_startmessage.c:173 0x018abcb0 MPID_Isend /gpfs/home/robl/src/dcmf/BGP/IBM_V1R4M3/bgp/comm/lib/dev/mpich2/src/mpid/dcmfd/src/pt2pt/m pid_isend.c:105 0x01878c34 PMPI_Isend /gpfs/home/robl/src/dcmf/BGP/IBM_V1R4M3/bgp/comm/lib/dev/mpich2/src/mpi/pt2pt/isend.c:124 0x01890818 ADIOI_R_Exchange_data /gpfs/home/robl/src/dcmf/BGP/IBM_V1R4M3/bgp/comm/lib/dev/mpich2/src/mpi/romio/adio/ad_bgl/ ad_bgl_rdcoll.c:828 0x018929e0 ADIOI_Read_and_exch /gpfs/home/robl/src/dcmf/BGP/IBM_V1R4M3/bgp/comm/lib/dev/mpich2/src/mpi/romio/adio/ad_bgl/ ad_bgl_rdcoll.c:660 So, the suggestion to turn off the alltoallv optimization doesn't really work if we can't even run on one rack of /P I would say. ==rob > Bob Cernohous: (T/L 553) 507-253-6093 > > BobC(a)us.ibm.com > IBM Rochester, Building 030-2(C335), Department 61L > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > Chaos reigns within. > > Reflect, repent, and reboot. > > Order shall return. > > > devel-bounces(a)mpich.org wrote on 05/04/2013 09:43:10 PM: > > > From: "Rob Latham" <robl(a)mcs.anl.gov> > > To: devel(a)mpich.org, > > Cc: mpich2-dev(a)mcs.anl.gov > > Date: 05/04/2013 09:48 PM > > Subject: Re: [mpich-devel] ROMIO collective i/o memory use > > Sent by: devel-bounces(a)mpich.org > > > > On Mon, Apr 29, 2013 at 10:28:01AM -0500, Bob Cernohous wrote: > > > A customer (Argonne ;) is complaining about O(p) allocations in > collective > > > i/o. A collective read is failing at larger scale. > > > > > > Any thoughts or comments or advice? There appears to be lots of O(p) > in > > > ROMIO collective I/O. Plus a lot of (possibly large) aggregated data > > > buffers. A quick search shows > > > > The O(p) allocations are a concern, sure. For two-phase, though, the > > real problem lies in ADIOI_R_Exchange_data_alltoallv and > > ADIOI_W_Exchange_data_alltoallv . The O(p) allocations are the least > > of our worries! > > > > around line 1063 of ad_bg_rdcoll.c > > > > all_recv_buf = (char *) ADIOI_Malloc( rtail ); > > > > all_send_buf = (char *) ADIOI_Malloc( stail ); > > > > (rtail and stail are the sum of the receive and send arrays) > > > > ==rob > > > > > The common ROMIO read collective code: > > > > > > Find all "ADIOI_Malloc", Match case, Regular expression (UNIX) > > > > > > File > Z:\bgq\comm\lib\dev\mpich2\src\mpi\romio\adio\common\ad_read_coll.c > > > > > > 124 38: st_offsets = (ADIO_Offset *) > > > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > > > > > > 125 39: end_offsets = (ADIO_Offset *) > > > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > > > > > > 317 44: *offset_list_ptr = (ADIO_Offset *) > > > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > > > > > > 318 41: *len_list_ptr = (ADIO_Offset *) > > > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > > > > > > 334 44: *offset_list_ptr = (ADIO_Offset *) > > > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > > > > > > 335 41: *len_list_ptr = (ADIO_Offset *) > > > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > > > > > > 436 18: ADIOI_Malloc((contig_access_count+1)*sizeof(ADIO_Offset)); > > > > > > 437 41: *len_list_ptr = (ADIO_Offset *) > > > ADIOI_Malloc((contig_access_count+1)*sizeof(ADIO_Offset)); > > > > > > 573 37: if (ntimes) read_buf = (char *) > ADIOI_Malloc(coll_bufsize); > > > > > > 578 21: count = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > > > 587 25: send_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > > > 590 25: recv_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > > > 598 25: start_pos = (int *) ADIOI_Malloc(nprocs*sizeof(int)); > > > > > > 739 32: tmp_buf = (char *) ADIOI_Malloc(for_next_iter); > > > > > > 744 33: read_buf = (char *) > > > ADIOI_Malloc(for_next_iter+coll_bufsize); > > > > > > 805 9: > ADIOI_Malloc((nprocs_send+nprocs_recv+1)*sizeof(MPI_Request)); > > > > > > 827 30: recv_buf = (char **) ADIOI_Malloc(nprocs * > sizeof(char*)); > > > > > > 830 44: (char *) > > > ADIOI_Malloc(recv_size[i]); > > > > > > 870 31: statuses = (MPI_Status *) > > > ADIOI_Malloc((nprocs_send+nprocs_recv+1) * \ > > > > > > 988 35: curr_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > > > sizeof(unsigned)); > > > > > > 989 35: done_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > > > sizeof(unsigned)); > > > > > > 990 35: recv_buf_idx = (unsigned *) ADIOI_Malloc(nprocs * > > > sizeof(unsigned)); > > > > > > Total found: 22 > > > > > > > > > Our BG version of read collective: > > > > > > File > Z:\bgq\comm\lib\dev\mpich2\src\mpi\romio\adio\ad_bg\ad_bg_rdcoll.c > > > > > > 179 40: st_offsets = (ADIO_Offset *) > > > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > > > > > > 180 40: end_offsets = (ADIO_Offset *) > > > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > > > > > > 183 43: bg_offsets0 = (ADIO_Offset *) > > > ADIOI_Malloc(2*nprocs*sizeof(ADIO_Offset)); > > > > > > 184 43: bg_offsets = (ADIO_Offset *) > > > ADIOI_Malloc(2*nprocs*sizeof(ADIO_Offset)); > > > > > > 475 37: if (ntimes) read_buf = (char *) > ADIOI_Malloc(coll_bufsize); > > > > > > 480 21: count = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > > > 489 25: send_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > > > 492 25: recv_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > > > 500 25: start_pos = (int *) ADIOI_Malloc(nprocs*sizeof(int)); > > > > > > 676 32: tmp_buf = (char *) ADIOI_Malloc(for_next_iter); > > > > > > 681 33: read_buf = (char *) > > > ADIOI_Malloc(for_next_iter+coll_bufsize); > > > > > > 761 9: > ADIOI_Malloc((nprocs_send+nprocs_recv+1)*sizeof(MPI_Request)); > > > > > > 783 30: recv_buf = (char **) ADIOI_Malloc(nprocs * > sizeof(char*)); > > > > > > 786 44: (char *) > > > ADIOI_Malloc(recv_size[i]); > > > > > > 826 31: statuses = (MPI_Status *) > > > ADIOI_Malloc((nprocs_send+nprocs_recv+1) * \ > > > > > > 944 35: curr_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > > > sizeof(unsigned)); > > > > > > 945 35: done_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > > > sizeof(unsigned)); > > > > > > 946 35: recv_buf_idx = (unsigned *) ADIOI_Malloc(nprocs * > > > sizeof(unsigned)); > > > > > > 1058 23: rdispls = (int *) ADIOI_Malloc( nprocs * sizeof(int) ); > > > > > > 1063 29: all_recv_buf = (char *) ADIOI_Malloc( rtail ); > > > > > > 1064 26: recv_buf = (char **) ADIOI_Malloc(nprocs * sizeof(char > *)); > > > > > > 1068 23: sdispls = (int *) ADIOI_Malloc( nprocs * sizeof(int) ); > > > > > > 1073 29: all_send_buf = (char *) ADIOI_Malloc( stail ); > > > > > > Total found: 23 > > > > > > > > > Bob Cernohous: (T/L 553) 507-253-6093 > > > > > > BobC(a)us.ibm.com > > > IBM Rochester, Building 030-2(C335), Department 61L > > > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > > > > > Chaos reigns within. > > > > Reflect, repent, and reboot. > > > > Order shall return. > > > > -- > > Rob Latham > > Mathematics and Computer Science Division > > Argonne National Lab, IL USA > > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

unsubscribe
by Paul, Brent S 06 May '13

06 May '13

==================================================== Brent S. Paul Principal Engineer, Hydrodynamics and Acoustics R&D Department Alion Science and Technology (860) 462-1038 <mailto:[email protected]> bpaul(a)alionscience.com ====================================================

1 0

Re: [mpich-devel] ROMIO collective i/o memory use
by Bob Cernohous 06 May '13

06 May '13

I think my memory was off. Jeff actually put in a different alltoall involving... --------------------------------------------------------------------- The performance problem has been isolated to a single function in the MPI-IO ROMIO common directory, ADIOI_Calc_others_req(). This function was consuming 90% of the time between I/O syscalls. It was doing Isend/Irecv between all of the nodes, twice. I copied that function to a BG/L-specific module, rewrote it to use MPI_Alltoallv(), and changed MPI-IO collective write and read to call it by default, or when -env BGLMPIO_TUNEBLOCKING=1 is specified. The new function only consumed 16% of the time between I/O syscalls, and brought the total time in line with using MPI_Gather() and POSIX I/O. ... Both the performance fix and the memory leak fix went into the V1R3M2 DRV140_2007-070417. I compiled both testcases against that driver and ran them on a full rack. The performance was good (see below), and the memory leak was gone. Performance fix: [0] using 5120 blocks per task, and ntasks = 1024 ... [0] Time to write the file with MPI_File_write_all = 61.602 seconds, bandwidth = 0.340 MB/sec [0] file block size = 87600 [0] Time to write the file with posix write = 84.563 seconds, bandwidth = 0.248 MB/sec ------------------------------------------------------------------- Anyway, this is somewhat off the original topic of o(p) allocations except that it often seems to be a trade off between performance and memory. Bob Cernohous: (T/L 553) 507-253-6093 BobC(a)us.ibm.com IBM Rochester, Building 030-2(C335), Department 61L 3605 Hwy 52 North, Rochester, MN 55901-7829 > Chaos reigns within. > Reflect, repent, and reboot. > Order shall return. devel-bounces(a)mpich.org wrote on 05/06/2013 04:54:10 PM: > From: Bob Cernohous/Rochester/IBM@IBMUS > To: mpich2-dev(a)mcs.anl.gov, > Date: 05/06/2013 04:55 PM > Subject: Re: [mpich-devel] ROMIO collective i/o memory use > Sent by: devel-bounces(a)mpich.org > > I did a couple quick searches and this goes back to bg/l, not bg/p. > Jeff once investigated a problem and said > ---------------- > In general, for smaller blocks, MPI-IO performed better than POSIX > IO. For a midplane, they are about equal. For a rack, MPI-IO is > noticably slower. I am now suspecting the collective phase of MPI- > IO may be taking the time (#2 above). > > Specifying -env BGLMPIO_TUNEGATHER=0 did not significantly change > the 1 rack result. This controls using allgather (0) vs allreduce > (1 - the default) to communicate start and end offsets among the nodes. > > **Specifying -env BGLMPIO_COMM=1 made the 1 rack result twice as > slow. This controls using alltoallv (0 - the default) vs send/recv > (1) to do the consolidation phase. > > Specifying -env BGLMPIO_TUNEBLOCKING=0 made the 1 rack result so > slow that I cancelled the job. This controls whether to take psets > and GPFS into account (1 - the default) or not (0). > --------------- > I can't find the issue where he first implemented BGLMPIO_COMM. I > seem to remember it performed MUCH better on small scattered i/o > than send/recv. He had tables and numbers which I can't find right now. > > > Bob Cernohous: (T/L 553) 507-253-6093 > > BobC(a)us.ibm.com > IBM Rochester, Building 030-2(C335), Department 61L > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > Chaos reigns within. > > Reflect, repent, and reboot. > > Order shall return. > > > devel-bounces(a)mpich.org wrote on 05/06/2013 04:35:11 PM: > > > From: Jeff Hammond <jhammond(a)alcf.anl.gov> > > To: devel(a)mpich.org, > > Cc: mpich2-dev(a)mcs.anl.gov, devel-bounces(a)mpich.org > > Date: 05/06/2013 04:36 PM > > Subject: Re: [mpich-devel] ROMIO collective i/o memory use > > Sent by: devel-bounces(a)mpich.org > > > > Do alltoallv actually run faster than send-recv for the MPIO use case? > > For >1MB messages, is alltoallv noticeably faster than a well-written > > send-recv implantation? > > > > At least on BGQ, send-recv turns into a receiver-side PAMI_Rget for > > large messages; I would guess the optimized alltoallv implementation > > is rput-based at the SPI level. Other than overhead, they should run > > at the same speed, no? If execution overhead is not significant, then > > the implementation that minimizes memory usage should be the default. > > > > I suppose I should just write alltoallv using send-recv and see what > > the difference is... > > > > Jeff > > > > On Mon, May 6, 2013 at 3:05 PM, Bob Cernohous <bobc(a)us.ibm.com> wrote: > > > > > >> From: "Rob Latham" <robl(a)mcs.anl.gov> > > >> > > >> On Mon, May 06, 2013 at 02:30:15PM -0500, Bob Cernohous wrote: > > >> > > From: Rob Ross <rross(a)mcs.anl.gov> > > >> > > > > >> > > Should we consider this as interest in working on this problem on > > >> > > the IBM side :)? -- Rob > > >> > > > >> > Say what?! ;) > > >> > > >> RobR's excited that IBM's looking at the ROMIO piece of DCMF. We > > >> thought we were on our own with that one. > > >> > > >> > > >> > I was looking more for agreement that collective i/o is 'what it > > >> > is'... and maybe some idea if we just have some known limitations on > > >> > scaling it. Yes, that BG alltoallv is a bigger problem that we can > > >> > avoid > > >> > with an env var -- is that just going to have to be 'good enough'? (I > > >> > think that Jeff P wrote that on BG/P and got good performancewith that > > >> > alltoallv. Trading memory for performance, not unusual, and at least > > >> > it's > > >> > selectable.) > > >> > > >> I can't test while our Blue Gene is under maintenance. I know the > > >> environment variable selection helps only a little bit (like improves > > >> scaling from 4k to 8k maybe? don't have the notes offhand). > > > > > > Ouch. So you've seen the scaling failures at 8k... ranks? > racks? Kevin is > > > failing at... 16 racks x 16 ranks per node... I think ... so 256k ranks. > > > > > > > > -- > > Jeff Hammond > > Argonne Leadership Computing Facility > > University of Chicago Computation Institute > > jhammond(a)alcf.anl.gov / (630) 252-5381 > > http://www.linkedin.com/in/jeffhammond > > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > > ALCF docs: http://www.alcf.anl.gov/user-guides > >

1 0

Re: [mpich-devel] ROMIO collective i/o memory use
by Jeff Hammond 06 May '13

06 May '13

There are at least a few cases where BGQ collective performance is _better_ with the MPICH built-in algorithms than the PAMI_Collective implementation, which suggests to me that the assumption that the BG MPIDI collectives are always faster no longer holds. This is partly because PAMI collectives are not necessarily written for BGQ but could also be because BGQ is less sensitive to certain types of contention than previous iterations were. I'm not that "post all isend/irecv pairs and waitall" is the right way to do this is reducing memory is the goal. There are a variety of alternatives but I'd have to run experiments and compare to alltoallv to know if anything of them are better. Jeff On Mon, May 6, 2013 at 4:44 PM, Rob Latham <robl(a)mcs.anl.gov> wrote: > On Mon, May 06, 2013 at 04:35:11PM -0500, Jeff Hammond wrote: >> Do alltoallv actually run faster than send-recv for the MPIO use case? >> For >1MB messages, is alltoallv noticeably faster than a well-written >> send-recv implantation? > > We only have data from Blue Gene /L: Hao Yu's paper only studied > scalability up to 1k mpi processes. Alltoallv got the implementation > an extra 100 MiB /sec or so. > > That paper cites almassi's ICS 2005 paper "MPI collective communication on > BlueGene/L systems", but surely there has been more recent work in > this area? > > What is a "well written send-recv implementation"? One that posts a > bunch of nonblocking sends/receives and lets the MPICH progress engine > figure it out? One that tries to schedule the send/recieves in some > way? > > ==rob > > -- > Rob Latham > Mathematics and Computer Science Division > Argonne National Lab, IL USA -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

Re: [mpich-devel] ROMIO collective i/o memory use
by Bob Cernohous 06 May '13

06 May '13

I did a couple quick searches and this goes back to bg/l, not bg/p. Jeff once investigated a problem and said ---------------- In general, for smaller blocks, MPI-IO performed better than POSIX IO. For a midplane, they are about equal. For a rack, MPI-IO is noticably slower. I am now suspecting the collective phase of MPI-IO may be taking the time (#2 above). Specifying -env BGLMPIO_TUNEGATHER=0 did not significantly change the 1 rack result. This controls using allgather (0) vs allreduce (1 - the default) to communicate start and end offsets among the nodes. **Specifying -env BGLMPIO_COMM=1 made the 1 rack result twice as slow. This controls using alltoallv (0 - the default) vs send/recv (1) to do the consolidation phase. Specifying -env BGLMPIO_TUNEBLOCKING=0 made the 1 rack result so slow that I cancelled the job. This controls whether to take psets and GPFS into account (1 - the default) or not (0). --------------- I can't find the issue where he first implemented BGLMPIO_COMM. I seem to remember it performed MUCH better on small scattered i/o than send/recv. He had tables and numbers which I can't find right now. Bob Cernohous: (T/L 553) 507-253-6093 BobC(a)us.ibm.com IBM Rochester, Building 030-2(C335), Department 61L 3605 Hwy 52 North, Rochester, MN 55901-7829 > Chaos reigns within. > Reflect, repent, and reboot. > Order shall return. devel-bounces(a)mpich.org wrote on 05/06/2013 04:35:11 PM: > From: Jeff Hammond <jhammond(a)alcf.anl.gov> > To: devel(a)mpich.org, > Cc: mpich2-dev(a)mcs.anl.gov, devel-bounces(a)mpich.org > Date: 05/06/2013 04:36 PM > Subject: Re: [mpich-devel] ROMIO collective i/o memory use > Sent by: devel-bounces(a)mpich.org > > Do alltoallv actually run faster than send-recv for the MPIO use case? > For >1MB messages, is alltoallv noticeably faster than a well-written > send-recv implantation? > > At least on BGQ, send-recv turns into a receiver-side PAMI_Rget for > large messages; I would guess the optimized alltoallv implementation > is rput-based at the SPI level. Other than overhead, they should run > at the same speed, no? If execution overhead is not significant, then > the implementation that minimizes memory usage should be the default. > > I suppose I should just write alltoallv using send-recv and see what > the difference is... > > Jeff > > On Mon, May 6, 2013 at 3:05 PM, Bob Cernohous <bobc(a)us.ibm.com> wrote: > > > >> From: "Rob Latham" <robl(a)mcs.anl.gov> > >> > >> On Mon, May 06, 2013 at 02:30:15PM -0500, Bob Cernohous wrote: > >> > > From: Rob Ross <rross(a)mcs.anl.gov> > >> > > > >> > > Should we consider this as interest in working on this problem on > >> > > the IBM side :)? -- Rob > >> > > >> > Say what?! ;) > >> > >> RobR's excited that IBM's looking at the ROMIO piece of DCMF. We > >> thought we were on our own with that one. > >> > >> > >> > I was looking more for agreement that collective i/o is 'what it > >> > is'... and maybe some idea if we just have some known limitations on > >> > scaling it. Yes, that BG alltoallv is a bigger problem that we can > >> > avoid > >> > with an env var -- is that just going to have to be 'good enough'? (I > >> > think that Jeff P wrote that on BG/P and got good performance with that > >> > alltoallv. Trading memory for performance, not unusual, and at least > >> > it's > >> > selectable.) > >> > >> I can't test while our Blue Gene is under maintenance. I know the > >> environment variable selection helps only a little bit (like improves > >> scaling from 4k to 8k maybe? don't have the notes offhand). > > > > Ouch. So you've seen the scaling failures at 8k... ranks? racks? Kevin is > > failing at... 16 racks x 16 ranks per node... I think ... so 256k ranks. > > > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > ALCF docs: http://www.alcf.anl.gov/user-guides >

1 0

Re: [mpich-devel] ROMIO collective i/o memory use
by Rob Latham 06 May '13

06 May '13

On Mon, May 06, 2013 at 04:35:11PM -0500, Jeff Hammond wrote: > Do alltoallv actually run faster than send-recv for the MPIO use case? > For >1MB messages, is alltoallv noticeably faster than a well-written > send-recv implantation? We only have data from Blue Gene /L: Hao Yu's paper only studied scalability up to 1k mpi processes. Alltoallv got the implementation an extra 100 MiB /sec or so. That paper cites almassi's ICS 2005 paper "MPI collective communication on BlueGene/L systems", but surely there has been more recent work in this area? What is a "well written send-recv implementation"? One that posts a bunch of nonblocking sends/receives and lets the MPICH progress engine figure it out? One that tries to schedule the send/recieves in some way? ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-devel] ROMIO collective i/o memory use
by Jeff Hammond 06 May '13

06 May '13

Do alltoallv actually run faster than send-recv for the MPIO use case? For >1MB messages, is alltoallv noticeably faster than a well-written send-recv implantation? At least on BGQ, send-recv turns into a receiver-side PAMI_Rget for large messages; I would guess the optimized alltoallv implementation is rput-based at the SPI level. Other than overhead, they should run at the same speed, no? If execution overhead is not significant, then the implementation that minimizes memory usage should be the default. I suppose I should just write alltoallv using send-recv and see what the difference is... Jeff On Mon, May 6, 2013 at 3:05 PM, Bob Cernohous <bobc(a)us.ibm.com> wrote: > >> From: "Rob Latham" <robl(a)mcs.anl.gov> >> >> On Mon, May 06, 2013 at 02:30:15PM -0500, Bob Cernohous wrote: >> > > From: Rob Ross <rross(a)mcs.anl.gov> >> > > >> > > Should we consider this as interest in working on this problem on >> > > the IBM side :)? -- Rob >> > >> > Say what?! ;) >> >> RobR's excited that IBM's looking at the ROMIO piece of DCMF. We >> thought we were on our own with that one. >> >> >> > I was looking more for agreement that collective i/o is 'what it >> > is'... and maybe some idea if we just have some known limitations on >> > scaling it. Yes, that BG alltoallv is a bigger problem that we can >> > avoid >> > with an env var -- is that just going to have to be 'good enough'? (I >> > think that Jeff P wrote that on BG/P and got good performance with that >> > alltoallv. Trading memory for performance, not unusual, and at least >> > it's >> > selectable.) >> >> I can't test while our Blue Gene is under maintenance. I know the >> environment variable selection helps only a little bit (like improves >> scaling from 4k to 8k maybe? don't have the notes offhand). > > Ouch. So you've seen the scaling failures at 8k... ranks? racks? Kevin is > failing at... 16 racks x 16 ranks per node... I think ... so 256k ranks. -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

Re: [mpich-devel] ROMIO collective i/o memory use
by Bob Cernohous 06 May '13

06 May '13

> From: "Rob Latham" <robl(a)mcs.anl.gov> > > On Mon, May 06, 2013 at 02:30:15PM -0500, Bob Cernohous wrote: > > > From: Rob Ross <rross(a)mcs.anl.gov> > > > > > > Should we consider this as interest in working on this problem on > > > the IBM side :)? -- Rob > > > > Say what?! ;) > > RobR's excited that IBM's looking at the ROMIO piece of DCMF. We > thought we were on our own with that one. > > > > I was looking more for agreement that collective i/o is 'what it > > is'... and maybe some idea if we just have some known limitations on > > scaling it. Yes, that BG alltoallv is a bigger problem that we can avoid > > with an env var -- is that just going to have to be 'good enough'? (I > > think that Jeff P wrote that on BG/P and got good performance with that > > alltoallv. Trading memory for performance, not unusual, and at least it's > > selectable.) > > I can't test while our Blue Gene is under maintenance. I know the > environment variable selection helps only a little bit (like improves > scaling from 4k to 8k maybe? don't have the notes offhand). Ouch. So you've seen the scaling failures at 8k... ranks? racks? Kevin is failing at... 16 racks x 16 ranks per node... I think ... so 256k ranks.

1 0

Re: [mpich-devel] ROMIO collective i/o memory use
by Rob Ross 06 May '13

06 May '13

Hi, Let's separate collective I/O the concept from collective I/O the implementation in ROMIO. Lots of things we *could* do: - switch to a different internal representation of noncontiguous regions, perhaps dramatically decreasing memory allocation when exchanging regions - switch to a different collective I/O algorithm that uses some hierarchy to reduce number of processes for which space must be allocated - don't globally coordinate, but rather coordinate more loosely to reduce communication and memory allocation requirements - … I'm sure we can come up with more ideas, some of which would be new research and some of which would be mostly engineering. All relatively major activities. Thus my question :). Rob On May 6, 2013, at 2:30 PM, Bob Cernohous wrote: > > > From: Rob Ross <rross(a)mcs.anl.gov> > > > > Should we consider this as interest in working on this problem on > > the IBM side :)? -- Rob > > Say what?! ;) > > Meaning we can get rid of all o(p) allocations? Not sure how you do internal collectives on behalf of the app/mpi-io without at least some of those. I was looking more for agreement that collective i/o is 'what it is'... and maybe some idea if we just have some known limitations on scaling it. Yes, that BG alltoallv is a bigger problem that we can avoid with an env var -- is that just going to have to be 'good enough'? (I think that Jeff P wrote that on BG/P and got good performance with that alltoallv. Trading memory for performance, not unusual, and at least it's selectable.)

1 0