November 2012 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Problem with MPICH 3.0 rc1 compilation on RHEL 5
by Dave Goodell 20 Nov '12

20 Nov '12

I'm not sure I follow you on the fstat64/stat64 double-definition issue. Is this just some sort of bug in the RHEL6 headers? Is there something that we (MPICH) are doing incorrectly w.r.t. this? -Dave On Nov 14, 2012, at 5:39 PM CST, Yauheni Zelenko wrote: > Hi, Dave! > > Build failed on first compilation because fstat64 and stat64 defined twice. Likely because of these functions inlining in /usr/include/sys/stat.h. > > Also GCC 4.7.2 warnings on Red Hat Enterprise Linux 6: > > src/mpltrmem.c: In function 'MPL_trdump': > src/mpltrmem.c:536:14: warning: variable 'old_head' set but not used [-Wunused-but-set-variable] > src/mpi/coll/opminloc.c: In function 'MPIR_MINLOC': > src/mpi/coll/opminloc.c:85:24: warning: variable 'flen' set but not used [-Wunused-but-set-variable] > src/mpi/coll/opmaxloc.c: In function 'MPIR_MAXLOC': > src/mpi/coll/opmaxloc.c:86:24: warning: variable 'flen' set but not used [-Wunused-but-set-variable] > src/mpid/ch3/src/ch3u_rma_sync.c: In function 'MPIDI_CH3I_Recv_rma_msg': > src/mpid/ch3/src/ch3u_rma_sync.c:1251:5: warning: 'mpiu_chkpmem_stk_[0]' may be used uninitialized in this function [-Wmaybe-uninitialized] > src/mpid/ch3/src/mpidi_pg.c: In function 'connToString': > src/mpid/ch3/src/mpidi_pg.c:972:5: warning: 'mpiu_chkpmem_stk_[0]' may be used uninitialized in this function [-Wmaybe-uninitialized] > src/mpid/ch3/channels/nemesis/src/ch3_progress.c: In function 'MPIDI_CH3I_Progress': > src/mpid/ch3/channels/nemesis/src/ch3_progress.c:394:21: warning: comparison between 'MPIDI_CH3_Pkt_type_t' and 'enum MPID_nem_pkt_type' [-Wenum-compare] > src/mpid/ch3/channels/nemesis/src/ch3_progress.c: In function 'MPID_nem_handle_pkt': > src/mpid/ch3/channels/nemesis/src/ch3_progress.c:636:17: warning: comparison between 'MPIDI_CH3_Pkt_type_t' and 'enum MPID_nem_pkt_type' [-Wenum-compare] > src/mpid/ch3/channels/nemesis/src/ch3_progress.c:684:13: warning: comparison between 'MPIDI_CH3_Pkt_type_t' and 'enum MPID_nem_pkt_type' [-Wenum-compare] > src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c: In function 'alloc_sc_plfd_tbls': > src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c:183:5: warning: the comparison will always evaluate as 'true' for the address of 'MPID_nem_tcp_g_lstn_sc' will never be NULL [-Waddress] > src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c:184:5: warning: the comparison will always evaluate as 'true' for the address of 'MPID_nem_tcp_g_lstn_plfd' will never be NULL [-Waddress] > src/mpid/common/datatype/mpid_type_debug.c: In function 'MPIDI_Datatype_printf': > src/mpid/common/datatype/mpid_type_debug.c:238:59: warning: variable 'sticky_ub' set but not used [-Wunused-but-set-variable] > src/mpid/common/datatype/mpid_type_debug.c:238:48: warning: variable 'sticky_lb' set but not used [-Wunused-but-set-variable] > src/mpid/common/datatype/mpid_type_debug.c:238:44: warning: variable 'ub' set but not used [-Wunused-but-set-variable] > src/mpid/common/datatype/mpid_type_debug.c:238:31: warning: variable 'true_ub' set but not used [-Wunused-but-set-variable] > src/mpid/common/datatype/mpid_type_debug.c:236:11: warning: variable 'string' set but not used [-Wunused-but-set-variable] > src/mpi/coll/iallgather.c: In function 'MPIR_Iallgather_bruck': > src/mpi/coll/iallgather.c:329:5: warning: 'mpir_sched_chkpmem_stk_[0]' may be used uninitialized in this function [-Wmaybe-uninitialized] > src/mpi/coll/iallgather.c: In function 'MPIR_Iallgather_inter': > src/mpi/coll/iallgather.c:580:5: warning: 'mpir_sched_chkpmem_stk_[0]' may be used uninitialized in this function [-Wmaybe-uninitialized] > src/mpi/coll/iallgatherv.c: In function 'MPIR_Iallgatherv_rec_dbl': > src/mpi/coll/iallgatherv.c:259:5: warning: 'mpir_sched_chkpmem_stk_[0]' may be used uninitialized in this function [-Wmaybe-uninitialized] > src/mpi/coll/iallgatherv.c: In function 'MPIR_Iallgatherv_bruck': > src/mpi/coll/iallgatherv.c:398:5: warning: 'mpir_sched_chkpmem_stk_[0]' may be used uninitialized in this function [-Wmaybe-uninitialized] > src/mpi/coll/iallreduce.c: In function 'MPIR_Iallreduce_redscat_allgather': > src/mpi/coll/iallreduce.c:282:5: warning: 'mpir_sched_chkpmem_stk_[0]' may be used uninitialized in this function [-Wmaybe-uninitialized] > src/mpi/coll/iallreduce.c: In function 'MPIR_Iallreduce_rec_dbl': > src/mpi/coll/iallreduce.c:425:5: warning: 'mpir_sched_chkpmem_stk_[0]' may be used uninitialized in this function [-Wmaybe-uninitialized] > src/mpi/coll/ired_scat_block.c: In function 'MPIR_Ireduce_scatter_block_inter': > src/mpi/coll/ired_scat_block.c:899:5: warning: 'mpir_sched_chkpmem_stk_[0]' may be used uninitialized in this function [-Wmaybe-uninitialized] > src/mpi/coll/ireduce.c: In function 'MPIR_Ireduce_SMP': > src/mpi/coll/ireduce.c:677:5: warning: 'mpir_sched_chkpmem_stk_[0]' may be used uninitialized in this function [-Wmaybe-uninitialized] > > Eugene. > ________________________________________ > From: discuss-bounces(a)mpich.org [discuss-bounces(a)mpich.org] On Behalf Of Dave Goodell [goodell(a)mcs.anl.gov] > Sent: Wednesday, November 14, 2012 2:46 PM > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] Problem with MPICH 3.0 rc1 compilation on RHEL 5 > > This is because you have selected a bare "--enable-strict" option to configure, which defaults to c89. I'll see if there's a more pleasant way around this, but in the meantime is it acceptable for you to pass "--enable-strict=c99" instead? > > The problem occurs because your compiler still responds to the "long long" type but disables the "LLONG_MAX" constant when asked for c89 mode. > > -Dave > > On Nov 14, 2012, at 4:14 PM CST, Yauheni Zelenko wrote: > >> Hi! >> >> I'm trying to compile MPICH 3.0 rc1 on Red Hat Enterprise Linux 5 and compilation failed because of undefined LLONG_MAX. >> >> LLONG_MAX definition in /usr/include/limits.h is depend on C99 condition. -std=c89 is passed to GCC. >> >> My configure options: >> >> ./configure --prefix=${InstallDir} \ >> --with-pm=hydra --with-device=ch3:nemesis \ >> --disable-chekpointing \ >> --enable-strict --enable-fast=O3 \ >> --disable-f77 --disable-fc \ >> CC=/grid/common/pkgs/gcc/v4.4.5/bin/gcc CFLAGS="-m32 -fPIC" \ >> CPP=/grid/common/pkgs/gcc/v4.4.5/bin/cpp \ >> CXX=/grid/common/pkgs/gcc/v4.4.5/bin/g++ CXXFLAGS="-m32 -fPIC" \ >> CXXCPP=/grid/common/pkgs/gcc/v4.4.5/bin/cpp >> >> GCC 4.4.5 is also produce next warnings: >> >> mpi-io/write.c: In function 'MPIOI_File_write': >> mpi-io/write.c:126: warning: passing argument 1 of 'MPIU_external32_buffer_setup' discards qualifiers from pointer target type >> ./adio/include/mpiu_external32.h:20: note: expected 'void *' but argument is of type 'const void *' >> mpi-io/write_all.c: In function 'MPIOI_File_write_all': >> mpi-io/write_all.c:109: warning: passing argument 1 of 'MPIU_external32_buffer_setup' discards qualifiers from pointer target type >> ./adio/include/mpiu_external32.h:20: note: expected 'void *' but argument is of type 'const void *' >> mpi-io/write_allb.c: In function 'MPIOI_File_write_all_begin': >> mpi-io/write_allb.c:105: warning: passing argument 1 of 'MPIU_external32_buffer_setup' discards qualifiers from pointer target type >> ./adio/include/mpiu_external32.h:20: note: expected 'void *' but argument is of type 'const void *' >> mpi-io/write_ord.c: In function 'PMPI_File_write_ordered': >> mpi-io/write_ord.c:98: warning: assignment discards qualifiers from pointer target type >> mpi-io/write_ord.c:100: warning: passing argument 1 of 'MPIU_external32_buffer_setup' discards qualifiers from pointer target type >> ./adio/include/mpiu_external32.h:20: note: expected 'void *' but argument is of type 'const void *' >> mpi-io/write_ordb.c: In function 'PMPI_File_write_ordered_begin': >> mpi-io/write_ordb.c:104: warning: assignment discards qualifiers from pointer target type >> mpi-io/write_ordb.c:106: warning: passing argument 1 of 'MPIU_external32_buffer_setup' discards qualifiers from pointer target type >> ./adio/include/mpiu_external32.h:20: note: expected 'void *' but argument is of type 'const void *' >> mpi-io/write_sh.c: In function 'PMPI_File_write_shared': >> mpi-io/write_sh.c:100: warning: assignment discards qualifiers from pointer target type >> mpi-io/write_sh.c:102: warning: passing argument 1 of 'MPIU_external32_buffer_setup' discards qualifiers from pointer target type >> ./adio/include/mpiu_external32.h:20: note: expected 'void *' but argument is of type 'const void *' >> mpi-io/mpiu_external32.c: In function 'MPIU_write_external32_conversion_fn': >> mpi-io/mpiu_external32.c:24: warning: implicit declaration of function 'MPIR_Datatype_iscontig' >> mpi-io/mpiu_external32.c:24: warning: nested extern declaration of 'MPIR_Datatype_iscontig' >> mpi-io/mpiu_external32.c: In function 'MPIU_datatype_full_size': >> mpi-io/mpiu_external32.c:123: warning: unused variable 'extent' >> ./pm/pmiserv/pmip_cb.c: In function 'parse_exec_params': >> ./pm/pmiserv/pmip_cb.c:808: warning: null argument where non-null required (argument 1) >> ./pm/pmiserv/pmip_cb.c:808: warning: null argument where non-null required (argument 1) >> ./pm/pmiserv/pmip_cb.c:808: warning: null argument where non-null required (argument 2) >> ./pm/pmiserv/pmip_cb.c:808: warning: null argument where non-null required (argument 2) >> src/mpi/datatype/status_set_elements.c: In function 'PMPI_Status_set_elements': >> src/mpi/datatype/status_set_elements.c:54: warning: unused variable 'size' >> src/mpi/datatype/status_set_elements_x.c: In function 'MPIR_Status_set_elements_x_impl': >> src/mpi/datatype/status_set_elements_x.c:41: error: 'LLONG_MAX' undeclared (first use in this function) >> src/mpi/datatype/status_set_elements_x.c:41: error: (Each undeclared identifier is reported only once >> src/mpi/datatype/status_set_elements_x.c:41: error: for each function it appears in.) >> >> Eugene. >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Fwd: Windows and mpicc
by Jayesh Krishna 20 Nov '12

20 Nov '12

Hi, A google search suggests that your setting for LANG environment variable may be incorrect. Can you check what it is (Can you set it to say "LANG=C" and see if it works)? Regards, Jayesh ----- Original Message ----- From: "John Mole" <j.mole(a)imperial.ac.uk> To: "Jayesh Krishna" <jayesh(a)mcs.anl.gov> Sent: Monday, November 19, 2012 9:37:49 AM Subject: RE: [mpich-discuss] Fwd: Windows and mpicc Hi Jayesh, I have checked dbxerr.c in hex and there are no DOS characters - make output attached. I don't want to disable MPE so I need to solve this problem somehow. I will post on the Cygwin forum about the memory issue as you suggested. Regards, John Mole -----Original Message----- From: Jayesh Krishna [mailto:[email protected]] Sent: 12 November 2012 16:32 To: mpich-discuss(a)mcs.anl.gov Cc: Mole, John Subject: Re: [mpich-discuss] Fwd: Windows and mpicc Hi, Looks like you have compile time errors for mpe because of DOS characters. If you are not planning to use MPE you can disable it (Specify "--disable-mpe" during configure time). The error that stopped the build looks like a Cygwin error (Unable to allocate enough memory for the linker). I have seen these errors when packages are updated (A "rebase" sometimes solves the problem). I would recommend posting your issue on a Cygwin forum. Can you try running the build again (after disabling MPE at compile time) after you fix the Cygwin issue? Regards, Jayesh ----- Original Message ----- From: "Jayesh Krishna" <jayesh(a)mcs.anl.gov> To: mpich-discuss(a)mcs.anl.gov Sent: Friday, November 9, 2012 2:58:06 PM Subject: [mpich-discuss] Fwd: Windows and mpicc FYI ----- Forwarded Message ----- From: "John Mole" <j.mole(a)imperial.ac.uk> To: "Jayesh Krishna" <jayesh(a)mcs.anl.gov> Sent: Friday, November 9, 2012 10:25:19 AM Subject: RE: [mpich-discuss] Windows and mpicc Hi Jayesh, Thanks. I did as you suggested. On the configure stage, I had to convert all the configure-related files to change the end of line characters to UNIX rather than DOS. The Cygwin version of BASH seems to be very picky about this. Eventually I managed to persuade the ./configure to work (output attached). However, the make stage failed with various errors (output attached). I have also included a list of the environmental variables, which may be relevant. Any advice on how to proceed would be much appreciated. Regards, John Mole -----Original Message----- From: Jayesh Krishna [mailto:[email protected]] Sent: 07 November 2012 15:50 To: Mole, John Cc: mpich-discuss(a)mcs.anl.gov Subject: Re: [mpich-discuss] Windows and mpicc Hi, There are two ways to install MPICH2 on Windows - (1) Using the Windows installers (2) Downloading MPICH2 source and compiling it under Cygwin. If you install MPICH2 using the Windows msi installers mpicc is not installed (Users typically compile their programs using Visual Studio, Intel Windows compilers etc). Since you are working in the Cygwin environment please uninstall MPICH2 and download the MPICH2 source from the link below and install (configure/make/make install - like you would install software on a unix-like/cygwin system) it. Regards, Jayesh ----- Original Message ----- From: "John Mole" <j.mole(a)imperial.ac.uk> To: "Jayesh Krishna" <jayesh(a)mcs.anl.gov> Sent: Wednesday, November 7, 2012 6:49:59 AM Subject: RE: [mpich-discuss] Windows and mpicc Hi Jayesh, I have already downloaded and installed the Windows version but there is no sign of mpicc (or the Fortran equivalent) in the /bin folder hence my original query. Regards, John Mole -----Original Message----- From: Jayesh Krishna [mailto:[email protected]] Sent: 06 November 2012 15:27 To: mpich-discuss(a)mcs.anl.gov Cc: Mole, John Subject: Re: [mpich-discuss] Windows and mpicc Hi, mpicc is the compiler wrapper that is installed with MPICH2 (or an MPI library suite). You can download the source and install MPICH2 from the MPICH2 website (http://www.mcs.anl.gov/research/projects/mpich2/downloads/index.php?s=downl…) Regards, Jayesh ----- Original Message ----- From: "John Mole" <j.mole(a)imperial.ac.uk> To: mpich-discuss(a)mcs.anl.gov Sent: Monday, November 5, 2012 4:40:58 PM Subject: [mpich-discuss] Windows and mpicc I am currently porting a mathematical modelling application from Ubuntu to Windows. I am fairly new to Linux. I am mostly working in Cygwin which emulates the Linux environment in Windows. I have installed MPICH2 on Windows and have set the necessary paths. In trying to make Parmetis in Cygwin I get the error ‘mpicc not found’. So I am trying to track down mpicc. I don’t know where it comes from. Is it from GNU or MPICH or where? How do I get hold of it? I would be most grateful if someone could provide guidance. Thanks, John Mole _______________________________________________ mpich-discuss mailing list mpich-discuss(a)mcs.anl.gov To manage subscription options or unsubscribe: https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss _______________________________________________ mpich-discuss mailing list mpich-discuss(a)mcs.anl.gov To manage subscription options or unsubscribe: https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss

1 0

Re: [mpich-discuss] Use of MPI derived data types / MPI file IO
by Wei-keng Liao 19 Nov '12

19 Nov '12

Hi, John, I think the question you raised is important and can improve the MPI and MPICH documentation. Based on what you observed from testing the latest MPICH and the comments from Dave Goodell (thanks Dave), I suggest at least the MPICH man page should be revised to clarify. Wei-keng On Nov 19, 2012, at 9:25 AM, <jgrime(a)uchicago.edu> <jgrime(a)uchicago.edu> wrote: > Hi Wei-keng, > > I checked this out with the latest mpich2 installation I have on my desktop, and I > can indeed call MPI_Type_free() on a derived type which has not been committed > with no error code returned. > > Maybe this could/should be clarified in the MPI standard at some point, although > I'm probably one of only 5 people who ever cared! > > J. > > ---- Original message ---- >> Date: Mon, 19 Nov 2012 09:17:36 -0600 >> From: discuss-bounces(a)mpich.org (on behalf of Wei-keng Liao > <wkliao(a)ece.northwestern.edu>) >> Subject: Re: [mpich-discuss] Use of MPI derived data types / MPI file IO >> To: discuss(a)mpich.org >> >> Hi, John, >> >> I did not find from the MPI standard specific about this question. However, >> if you check the man page of MPI_Type_free, one of the errors is >> MPI_ERR_TYPE >> Invalid datatype argument. May be an uncommitted MPI_Datatype (see > MPI_Type_commit). >> >> So, it seems to imply one should only free the data types that have been > committed. >> But, I think the MPICH development team should confirm if this is the case. >> >> >> Wei-keng >> >> >> On Nov 19, 2012, at 8:46 AM, <jgrime(a)uchicago.edu> > <jgrime(a)uchicago.edu> wrote: >> >>> Hi Wei-keng, >>> >>> It now works! Thanks for the help! >>> >>> One last question to the list: >>> >>> I've looked at the MPI 2 standards documents, and I'm still a little confused > as to >>> the precise semantics of MPI_Type_free(); as "MPI_Datatype" is an opaque > type, >>> I'm assuming that there is a certain amount of background allocation going > on >>> inside the MPI runtime when I call something like MPI_Type_create_struct() or >>> similar routines. >>> >>> Am I right in assuming that I should call MPI_Type_free() on *all* derived > data >>> types I generate, even if they are not subsequently registered using >>> MPI_Type_commit()? I would imagine that any other behaviour is likely to > lead to >>> memory leaks! >>> >>> Cheers, >>> >>> J. >>> >>> ---- Original message ---- >>>> Date: Sun, 18 Nov 2012 18:58:37 -0600 >>>> From: discuss-bounces(a)mpich.org (on behalf of Wei-keng Liao >>> <wkliao(a)ece.northwestern.edu>) >>>> Subject: Re: [mpich-discuss] Use of MPI derived data types / MPI file IO >>>> To: discuss(a)mpich.org >>>> >>>> Hi, John, >>>> >>>> You certainly are on the right track to achieve that. Your code is almost >>>> there, only the call to MPI_File_set_view is incorrect. In fact, you don't need > it. >>>> >>>> Try remove the call to MPI_File_set_view and replace the MPI_File_write_all > with: >>>> MPI_File_write_at_all(f, offset, &atoms[0], (int)atoms.size(), >>> mpi_atom_type_resized, &stat); >>>> >>>> On the reader side, you need to set the offset based on the new struct. > Other >>> than >>>> that, it is the same as the writer case. (no need of MPI_File_set_view either). >>>> >>>> As for the portability issue, I would suggest you to use high-level I/O > libraries, >>>> such as PnetCDF. >>>> >>>> Wei-keng >>>> >>>> On Nov 18, 2012, at 12:38 PM, <jgrime(a)uchicago.edu> >>> <jgrime(a)uchicago.edu> wrote: >>>> >>>>> Hi Wei-keng, >>>>> >>>>> That's a good point, thanks! >>>>> >>>>> However, I actually only want to save certain parts of the "atom" structure > to >>> file, >>>>> and saving the whole array as a raw dump could waste a lot of disk > space. >>>>> >>>>> For example, the "atom" structure I used in the example code in reality >>> contains >>>>> not only an integer and three contiguous doubles, but also at least > another >>> two >>>>> double[3] entries which I may not want to save to disk. As the full data > set >>> can >>>>> be hundreds of millions (or even billions) of "atom" structures, using a >>> derived >>>>> data type with only a restricted subset of the data in each "atom" > structure >>> will >>>>> produce considerably smaller file sizes! >>>>> >>>>> There's also the problem of making the resultant file "portable" - raw >>> memory >>>>> dumps could make life difficult in trying to use output files on machines > with >>>>> different processor architectures. Once I get the derived data types > working, >>> I >>>>> can then switch from the "native" representation to something else >>> ("external32" >>>>> etc), which should allow me to create portable output files, provided I'm >>> careful >>>>> with using MPIs file offset routines etc if the file is larger than plain old > 32 >>> bit >>>>> offsets can handle. >>>>> >>>>> Cheers, >>>>> >>>>> J. >>>>> >>>>> ---- Original message ---- >>>>>> Date: Sun, 18 Nov 2012 12:27:04 -0600 >>>>>> From: discuss-bounces(a)mpich.org (on behalf of Wei-keng Liao >>>>> <wkliao(a)ece.northwestern.edu>) >>>>>> Subject: Re: [mpich-discuss] Use of MPI derived data types / MPI file IO >>>>>> To: discuss(a)mpich.org >>>>>> >>>>>> Hi, John >>>>>> >>>>>> If your I/O is simply appending one process's data after another and the >>> I/O >>>>> buffers in memory >>>>>> are all contiguous, then you can simply do the following without > defining >>> MPI >>>>>> derived data types or setting the file view. >>>>>> >>>>>> MPI_File_write_at_all(f, offset, &atoms[0], (int)atoms.size() * sizeof(struct >>> atom), >>>>> MPI_BYTE, &stat); >>>>>> >>>>>> Using derived data types is usually when you have noncontiguous > buffer in >>>>> memory or >>>>>> want to access non-contiguous data in files. >>>>>> >>>>>> >>>>>> Wei-keng >>>>>> >>>>>> On Nov 18, 2012, at 11:52 AM, <jgrime(a)uchicago.edu> >>>>> <jgrime(a)uchicago.edu> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I'm having some problems with using derived data types and MPI > parallel >>> IO, >>>>> and >>>>>>> was wondering if anyone could help. I tried to search the archives in > case >>>>> this >>>>>>> was covered earlier, but that just gave me "ht://Dig error" messages. >>>>>>> >>>>>>> Outline: I have written a C++ program where each MPI rank acts on > data >>>>> stored >>>>>>> in a local array of structures. The arrays are typically of different > lengths >>> on >>>>> each >>>>>>> rank. I wish to write and read the contents of these arrays to disk > using >>> MPI's >>>>>>> parallel IO routines. The file format is simply an initial integer which >>>>> describes >>>>>>> how many "structures" are in the file, followed by the data which >>> represents >>>>> the >>>>>>> "structure information" from all ranks (ie the total data set). >>>>>>> >>>>>>> So far, I've tried two different approaches: the first consists of each > rank >>>>>>> serialising the contents of the local array of structures into a byte > array, >>>>> which is >>>>>>> then saved to file "f" using MPI_File_set_view( f, MPI_COMM_WORLD, >>> offset, >>>>>>> MPI_CHAR, MPI_CHAR, "native", MPI_INFO_NULL ) to skip the initial > integer >>>>>>> "header" and then a call to MPI_File_write_all( f, local_bytearray, >>>>> local_n_bytes, >>>>>>> MPI_CHAR, &status ). Here, "offset" is simply the size of an integer (in >>> bytes) >>>>> + >>>>>>> the summation of the number of bytes each preceeding rank wishes > to >>> write >>>>> to >>>>>>> the file (received via an earlier MPI_Allgather call). This seems to work, > as >>>>> when I >>>>>>> read the file back in on a single MPI rank and deserialise the data into > an >>>>> array of >>>>>>> structures I get the results I expect. >>>>>>> >>>>>>> The second approach is to use MPI's derived data types to create MPI >>>>>>> representations of the structures, and then treat the arrays of > structures >>> as >>>>> MPI >>>>>>> data types. This allows me to avoid copying the local data into an >>>>> intermediate >>>>>>> buffer etc, and seems the more elegant approach. I cannot, however, >>> seem >>>>> to >>>>>>> make this approach work. >>>>>>> >>>>>>> I'm pretty sure the problem lies in my use of the file views, but I'm not >>> sure >>>>>>> where I'm going wrong. The reading of the integer "header" always > works >>>>> fine, >>>>>>> but the proceeding data is garbled. I'm using the "native" data >>> representation >>>>> for >>>>>>> testing, but will likely change that to something more portable when I > get >>>>> this >>>>>>> code working. >>>>>>> >>>>>>> I've included the important excerpts of the test code I'm trying to use >>> below >>>>>>> (with some printf()s and error handling etc removed to make it a little >>> more >>>>>>> concise). I have previously tested that std::vector allocates a > contiguous >>> flat >>>>>>> array of the appropriate data type in memory, so passing a >>> pointer/reference >>>>> to >>>>>>> the first element in such a data structure behaves the same way as > simply >>>>>>> passing a conventional array of the appropriate data type: >>>>>>> >>>>>>> struct atom >>>>>>> { >>>>>>> int global_id; >>>>>>> double xyz[3]; >>>>>>> }; >>>>>>> >>>>>>> void write( char * fpath, std::vector<struct atom> &atoms, int rank, > int >>>>> nranks ) >>>>>>> { >>>>>>> /* >>>>>>> Memory layout information for the structure we wish to > convert >>> into >>>>>>> an >>>>>>> MPI derived data type. >>>>>>> */ >>>>>>> std::vector<int> s_blocklengths; >>>>>>> std::vector<MPI_Aint> s_displacements; >>>>>>> std::vector<MPI_Datatype> s_datatypes; >>>>>>> MPI_Aint addr_start, addr; >>>>>>> MPI_Datatype mpi_atom_type, mpi_atom_type_resized; >>>>>>> int type_size; >>>>>>> >>>>>>> struct atom a; >>>>>>> >>>>>>> MPI_File f; >>>>>>> MPI_Status stat; >>>>>>> MPI_Offset offset; >>>>>>> char *datarep = (char *)"native"; >>>>>>> >>>>>>> std::vector<int> all_N; >>>>>>> int local_N, global_N; >>>>>>> >>>>>>> /* >>>>>>> Set up the structure data type: single integer, and 3 double >>> precision >>>>>>> floats. >>>>>>> We use the temporary "a" structure to determine the layout > of >>> memory >>>>>>> inside >>>>>>> atom structures. >>>>>>> */ >>>>>>> MPI_Get_address( &a, &addr_start ); >>>>>>> >>>>>>> s_blocklengths.push_back( 1 ); >>>>>>> s_datatypes.push_back( MPI_INT ); >>>>>>> MPI_Get_address( &a.global_id, &addr ); >>>>>>> s_displacements.push_back( addr - addr_start ); >>>>>>> >>>>>>> s_blocklengths.push_back( 3 ); >>>>>>> s_datatypes.push_back( MPI_DOUBLE ); >>>>>>> MPI_Get_address( &a.xyz[0], &addr ); >>>>>>> s_displacements.push_back( addr - addr_start ); >>>>>>> >>>>>>> MPI_Type_create_struct( (int)s_blocklengths.size(), > &s_blocklengths[0], >>>>>>> &s_displacements[0], &s_datatypes[0], &mpi_atom_type ); >>>>>>> MPI_Type_commit( &mpi_atom_type ); >>>>>>> >>>>>>> /* >>>>>>> Take into account any compiler padding in creating an array > of >>>>>>> structures. >>>>>>> */ >>>>>>> MPI_Type_create_resized( mpi_atom_type, 0, sizeof(struct atom), >>>>>>> &mpi_atom_type_resized ); >>>>>>> MPI_Type_commit( &mpi_atom_type_resized ); >>>>>>> >>>>>>> MPI_Type_size( mpi_atom_type_resized, &type_size ); >>>>>>> >>>>>>> local_N = (int)atoms.size(); >>>>>>> all_N.resize( nranks ); >>>>>>> >>>>>>> MPI_Allgather( &local_N, 1, MPI_INT, &all_N[0], 1, MPI_INT, >>>>>>> MPI_COMM_WORLD ); >>>>>>> >>>>>>> global_N = 0; >>>>>>> for( size_t i=0; i<all_N.size(); i++ ) global_N += all_N[i]; >>>>>>> >>>>>>> offset = 0; >>>>>>> for( int i=0; i<rank; i++ ) offset += all_N[i]; >>>>>>> >>>>>>> offset *= type_size; // convert from structure counts -> bytes > into file >>> for >>>>>>> true structure size >>>>>>> offset += sizeof( int ); // skip leading integer (global_N) in file. >>>>>>> >>>>>>> MPI_File_open( MPI_COMM_WORLD, fpath, MPI_MODE_CREATE | >>>>>>> MPI_MODE_WRONLY, MPI_INFO_NULL, &f ); >>>>>>> if( rank == 0 ) >>>>>>> { >>>>>>> MPI_File_write( f, &global_N, 1, MPI_INT, &stat ); >>>>>>> } >>>>>>> MPI_File_set_view( f, offset, mpi_atom_type_resized, >>>>>>> mpi_atom_type_resized, datarep, MPI_INFO_NULL ); >>>>>>> >>>>>>> MPI_File_write_all( f, &atoms[0], (int)atoms.size(), >>> mpi_atom_type_resized, >>>>>>> &stat ); >>>>>>> MPI_File_close( &f ); >>>>>>> >>>>>>> MPI_Type_free( &mpi_atom_type ); >>>>>>> MPI_Type_free( &mpi_atom_type_resized ); >>>>>>> >>>>>>> return; >>>>>>> } >>>>>>> >>>>>>> void read( char * fpath, std::vector<struct atom> &atoms ) >>>>>>> { >>>>>>> std::vector<int> s_blocklengths; >>>>>>> std::vector<MPI_Aint> s_displacements; >>>>>>> std::vector<MPI_Datatype> s_datatypes; >>>>>>> MPI_Datatype mpi_atom_type, mpi_atom_type_resized; >>>>>>> >>>>>>> struct atom a; >>>>>>> MPI_Aint addr_start, addr; >>>>>>> >>>>>>> MPI_File f; >>>>>>> MPI_Status stat; >>>>>>> >>>>>>> int global_N; >>>>>>> char *datarep = (char *)"native"; >>>>>>> >>>>>>> int type_size; >>>>>>> >>>>>>> /* >>>>>>> Set up the structure data type >>>>>>> */ >>>>>>> MPI_Get_address( &a, &addr_start ); >>>>>>> >>>>>>> s_blocklengths.push_back( 1 ); >>>>>>> s_datatypes.push_back( MPI_INT ); >>>>>>> MPI_Get_address( &a.global_id, &addr ); >>>>>>> s_displacements.push_back( addr - addr_start ); >>>>>>> >>>>>>> s_blocklengths.push_back( 3 ); >>>>>>> s_datatypes.push_back( MPI_DOUBLE ); >>>>>>> MPI_Get_address( &a.xyz[0], &addr ); >>>>>>> s_displacements.push_back( addr - addr_start ); >>>>>>> >>>>>>> MPI_Type_create_struct( (int)s_blocklengths.size(), > &s_blocklengths[0], >>>>>>> &s_displacements[0], &s_datatypes[0], &mpi_atom_type ); >>>>>>> MPI_Type_commit( &mpi_atom_type ); >>>>>>> >>>>>>> /* >>>>>>> Take into account any compiler padding in creating an array > of >>>>>>> structures. >>>>>>> */ >>>>>>> MPI_Type_create_resized( mpi_atom_type, 0, sizeof(struct atom), >>>>>>> &mpi_atom_type_resized ); >>>>>>> MPI_Type_commit( &mpi_atom_type_resized ); >>>>>>> >>>>>>> MPI_Type_size( mpi_atom_type_resized, &type_size ); >>>>>>> >>>>>>> MPI_File_open( MPI_COMM_SELF, fpath, MPI_MODE_RDONLY, >>>>>>> MPI_INFO_NULL, &f ); >>>>>>> >>>>>>> MPI_File_read( f, &global_N, 1, MPI_INT, &stat ); >>>>>>> >>>>>>> atoms.clear(); >>>>>>> atoms.resize( global_N ); >>>>>>> >>>>>>> errcode = MPI_File_set_view( f, sizeof(int), > mpi_atom_type_resized, >>>>>>> mpi_atom_type_resized, datarep, MPI_INFO_NULL ); >>>>>>> errcode = MPI_File_read( f, &atoms[0], global_N, >>> mpi_atom_type_resized, >>>>>>> &stat ); >>>>>>> errcode = MPI_File_close( &f ); >>>>>>> >>>>>>> MPI_Type_free( &mpi_atom_type ); >>>>>>> MPI_Type_free( &mpi_atom_type_resized ); >>>>>>> >>>>>>> return; >>>>>>> } >>>>>>> >>>>>>> Calling MPI_Type_get_extent() and MPI_Type_get_true_extent() for > both >>>>>>> mpi_atom_type and mpi_atom_type_resized returns (0,32) bytes in all >>> cases. >>>>>>> Calling MPI_Type_size() on both derived data types returns 28 bytes. >>>>>>> >>>>>>> If I call MPI_File_get_type_extent() on both derived data types after >>> opening >>>>> the >>>>>>> file, they both resolve to 32 bytes - so I think the problem is in the >>>>> difference >>>>>>> between the data representation in memory and on disk. If I explicitly > use >>> 32 >>>>>>> bytes in the offset calculation in the write() routine above, it still > doesn't >>>>> work. >>>>>>> >>>>>>> I'm finding it remarkably difficult to do something very simple using > MPI's >>>>>>> derived data types and the parallel IO, and hence I'm guessing that I > have >>>>>>> fundamentally misunderstood one or more aspects of this. If anyone > can >>>>> help >>>>>>> clarify where I'm going wrong, that would be much appreciated! >>>>>>> >>>>>>> Cheers, >>>>>>> >>>>>>> John. >>>>>>> _______________________________________________ >>>>>>> discuss mailing list discuss(a)mpich.org >>>>>>> To manage subscription options or unsubscribe: >>>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>> >>>>>> _______________________________________________ >>>>>> discuss mailing list discuss(a)mpich.org >>>>>> To manage subscription options or unsubscribe: >>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Use of MPI derived data types / MPI file IO
by jgrime＠uchicago.edu 19 Nov '12

19 Nov '12

Hi Wei-keng, I checked this out with the latest mpich2 installation I have on my desktop, and I can indeed call MPI_Type_free() on a derived type which has not been committed with no error code returned. Maybe this could/should be clarified in the MPI standard at some point, although I'm probably one of only 5 people who ever cared! J. ---- Original message ---- >Date: Mon, 19 Nov 2012 09:17:36 -0600 >From: discuss-bounces(a)mpich.org (on behalf of Wei-keng Liao <wkliao(a)ece.northwestern.edu>) >Subject: Re: [mpich-discuss] Use of MPI derived data types / MPI file IO >To: discuss(a)mpich.org > >Hi, John, > >I did not find from the MPI standard specific about this question. However, >if you check the man page of MPI_Type_free, one of the errors is > MPI_ERR_TYPE > Invalid datatype argument. May be an uncommitted MPI_Datatype (see MPI_Type_commit). > >So, it seems to imply one should only free the data types that have been committed. >But, I think the MPICH development team should confirm if this is the case. > > >Wei-keng > > >On Nov 19, 2012, at 8:46 AM, <jgrime(a)uchicago.edu> <jgrime(a)uchicago.edu> wrote: > >> Hi Wei-keng, >> >> It now works! Thanks for the help! >> >> One last question to the list: >> >> I've looked at the MPI 2 standards documents, and I'm still a little confused as to >> the precise semantics of MPI_Type_free(); as "MPI_Datatype" is an opaque type, >> I'm assuming that there is a certain amount of background allocation going on >> inside the MPI runtime when I call something like MPI_Type_create_struct() or >> similar routines. >> >> Am I right in assuming that I should call MPI_Type_free() on *all* derived data >> types I generate, even if they are not subsequently registered using >> MPI_Type_commit()? I would imagine that any other behaviour is likely to lead to >> memory leaks! >> >> Cheers, >> >> J. >> >> ---- Original message ---- >>> Date: Sun, 18 Nov 2012 18:58:37 -0600 >>> From: discuss-bounces(a)mpich.org (on behalf of Wei-keng Liao >> <wkliao(a)ece.northwestern.edu>) >>> Subject: Re: [mpich-discuss] Use of MPI derived data types / MPI file IO >>> To: discuss(a)mpich.org >>> >>> Hi, John, >>> >>> You certainly are on the right track to achieve that. Your code is almost >>> there, only the call to MPI_File_set_view is incorrect. In fact, you don't need it. >>> >>> Try remove the call to MPI_File_set_view and replace the MPI_File_write_all with: >>> MPI_File_write_at_all(f, offset, &atoms[0], (int)atoms.size(), >> mpi_atom_type_resized, &stat); >>> >>> On the reader side, you need to set the offset based on the new struct. Other >> than >>> that, it is the same as the writer case. (no need of MPI_File_set_view either). >>> >>> As for the portability issue, I would suggest you to use high-level I/O libraries, >>> such as PnetCDF. >>> >>> Wei-keng >>> >>> On Nov 18, 2012, at 12:38 PM, <jgrime(a)uchicago.edu> >> <jgrime(a)uchicago.edu> wrote: >>> >>>> Hi Wei-keng, >>>> >>>> That's a good point, thanks! >>>> >>>> However, I actually only want to save certain parts of the "atom" structure to >> file, >>>> and saving the whole array as a raw dump could waste a lot of disk space. >>>> >>>> For example, the "atom" structure I used in the example code in reality >> contains >>>> not only an integer and three contiguous doubles, but also at least another >> two >>>> double[3] entries which I may not want to save to disk. As the full data set >> can >>>> be hundreds of millions (or even billions) of "atom" structures, using a >> derived >>>> data type with only a restricted subset of the data in each "atom" structure >> will >>>> produce considerably smaller file sizes! >>>> >>>> There's also the problem of making the resultant file "portable" - raw >> memory >>>> dumps could make life difficult in trying to use output files on machines with >>>> different processor architectures. Once I get the derived data types working, >> I >>>> can then switch from the "native" representation to something else >> ("external32" >>>> etc), which should allow me to create portable output files, provided I'm >> careful >>>> with using MPIs file offset routines etc if the file is larger than plain old 32 >> bit >>>> offsets can handle. >>>> >>>> Cheers, >>>> >>>> J. >>>> >>>> ---- Original message ---- >>>>> Date: Sun, 18 Nov 2012 12:27:04 -0600 >>>>> From: discuss-bounces(a)mpich.org (on behalf of Wei-keng Liao >>>> <wkliao(a)ece.northwestern.edu>) >>>>> Subject: Re: [mpich-discuss] Use of MPI derived data types / MPI file IO >>>>> To: discuss(a)mpich.org >>>>> >>>>> Hi, John >>>>> >>>>> If your I/O is simply appending one process's data after another and the >> I/O >>>> buffers in memory >>>>> are all contiguous, then you can simply do the following without defining >> MPI >>>>> derived data types or setting the file view. >>>>> >>>>> MPI_File_write_at_all(f, offset, &atoms[0], (int)atoms.size() * sizeof(struct >> atom), >>>> MPI_BYTE, &stat); >>>>> >>>>> Using derived data types is usually when you have noncontiguous buffer in >>>> memory or >>>>> want to access non-contiguous data in files. >>>>> >>>>> >>>>> Wei-keng >>>>> >>>>> On Nov 18, 2012, at 11:52 AM, <jgrime(a)uchicago.edu> >>>> <jgrime(a)uchicago.edu> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> I'm having some problems with using derived data types and MPI parallel >> IO, >>>> and >>>>>> was wondering if anyone could help. I tried to search the archives in case >>>> this >>>>>> was covered earlier, but that just gave me "ht://Dig error" messages. >>>>>> >>>>>> Outline: I have written a C++ program where each MPI rank acts on data >>>> stored >>>>>> in a local array of structures. The arrays are typically of different lengths >> on >>>> each >>>>>> rank. I wish to write and read the contents of these arrays to disk using >> MPI's >>>>>> parallel IO routines. The file format is simply an initial integer which >>>> describes >>>>>> how many "structures" are in the file, followed by the data which >> represents >>>> the >>>>>> "structure information" from all ranks (ie the total data set). >>>>>> >>>>>> So far, I've tried two different approaches: the first consists of each rank >>>>>> serialising the contents of the local array of structures into a byte array, >>>> which is >>>>>> then saved to file "f" using MPI_File_set_view( f, MPI_COMM_WORLD, >> offset, >>>>>> MPI_CHAR, MPI_CHAR, "native", MPI_INFO_NULL ) to skip the initial integer >>>>>> "header" and then a call to MPI_File_write_all( f, local_bytearray, >>>> local_n_bytes, >>>>>> MPI_CHAR, &status ). Here, "offset" is simply the size of an integer (in >> bytes) >>>> + >>>>>> the summation of the number of bytes each preceeding rank wishes to >> write >>>> to >>>>>> the file (received via an earlier MPI_Allgather call). This seems to work, as >>>> when I >>>>>> read the file back in on a single MPI rank and deserialise the data into an >>>> array of >>>>>> structures I get the results I expect. >>>>>> >>>>>> The second approach is to use MPI's derived data types to create MPI >>>>>> representations of the structures, and then treat the arrays of structures >> as >>>> MPI >>>>>> data types. This allows me to avoid copying the local data into an >>>> intermediate >>>>>> buffer etc, and seems the more elegant approach. I cannot, however, >> seem >>>> to >>>>>> make this approach work. >>>>>> >>>>>> I'm pretty sure the problem lies in my use of the file views, but I'm not >> sure >>>>>> where I'm going wrong. The reading of the integer "header" always works >>>> fine, >>>>>> but the proceeding data is garbled. I'm using the "native" data >> representation >>>> for >>>>>> testing, but will likely change that to something more portable when I get >>>> this >>>>>> code working. >>>>>> >>>>>> I've included the important excerpts of the test code I'm trying to use >> below >>>>>> (with some printf()s and error handling etc removed to make it a little >> more >>>>>> concise). I have previously tested that std::vector allocates a contiguous >> flat >>>>>> array of the appropriate data type in memory, so passing a >> pointer/reference >>>> to >>>>>> the first element in such a data structure behaves the same way as simply >>>>>> passing a conventional array of the appropriate data type: >>>>>> >>>>>> struct atom >>>>>> { >>>>>> int global_id; >>>>>> double xyz[3]; >>>>>> }; >>>>>> >>>>>> void write( char * fpath, std::vector<struct atom> &atoms, int rank, int >>>> nranks ) >>>>>> { >>>>>> /* >>>>>> Memory layout information for the structure we wish to convert >> into >>>>>> an >>>>>> MPI derived data type. >>>>>> */ >>>>>> std::vector<int> s_blocklengths; >>>>>> std::vector<MPI_Aint> s_displacements; >>>>>> std::vector<MPI_Datatype> s_datatypes; >>>>>> MPI_Aint addr_start, addr; >>>>>> MPI_Datatype mpi_atom_type, mpi_atom_type_resized; >>>>>> int type_size; >>>>>> >>>>>> struct atom a; >>>>>> >>>>>> MPI_File f; >>>>>> MPI_Status stat; >>>>>> MPI_Offset offset; >>>>>> char *datarep = (char *)"native"; >>>>>> >>>>>> std::vector<int> all_N; >>>>>> int local_N, global_N; >>>>>> >>>>>> /* >>>>>> Set up the structure data type: single integer, and 3 double >> precision >>>>>> floats. >>>>>> We use the temporary "a" structure to determine the layout of >> memory >>>>>> inside >>>>>> atom structures. >>>>>> */ >>>>>> MPI_Get_address( &a, &addr_start ); >>>>>> >>>>>> s_blocklengths.push_back( 1 ); >>>>>> s_datatypes.push_back( MPI_INT ); >>>>>> MPI_Get_address( &a.global_id, &addr ); >>>>>> s_displacements.push_back( addr - addr_start ); >>>>>> >>>>>> s_blocklengths.push_back( 3 ); >>>>>> s_datatypes.push_back( MPI_DOUBLE ); >>>>>> MPI_Get_address( &a.xyz[0], &addr ); >>>>>> s_displacements.push_back( addr - addr_start ); >>>>>> >>>>>> MPI_Type_create_struct( (int)s_blocklengths.size(), &s_blocklengths[0], >>>>>> &s_displacements[0], &s_datatypes[0], &mpi_atom_type ); >>>>>> MPI_Type_commit( &mpi_atom_type ); >>>>>> >>>>>> /* >>>>>> Take into account any compiler padding in creating an array of >>>>>> structures. >>>>>> */ >>>>>> MPI_Type_create_resized( mpi_atom_type, 0, sizeof(struct atom), >>>>>> &mpi_atom_type_resized ); >>>>>> MPI_Type_commit( &mpi_atom_type_resized ); >>>>>> >>>>>> MPI_Type_size( mpi_atom_type_resized, &type_size ); >>>>>> >>>>>> local_N = (int)atoms.size(); >>>>>> all_N.resize( nranks ); >>>>>> >>>>>> MPI_Allgather( &local_N, 1, MPI_INT, &all_N[0], 1, MPI_INT, >>>>>> MPI_COMM_WORLD ); >>>>>> >>>>>> global_N = 0; >>>>>> for( size_t i=0; i<all_N.size(); i++ ) global_N += all_N[i]; >>>>>> >>>>>> offset = 0; >>>>>> for( int i=0; i<rank; i++ ) offset += all_N[i]; >>>>>> >>>>>> offset *= type_size; // convert from structure counts -> bytes into file >> for >>>>>> true structure size >>>>>> offset += sizeof( int ); // skip leading integer (global_N) in file. >>>>>> >>>>>> MPI_File_open( MPI_COMM_WORLD, fpath, MPI_MODE_CREATE | >>>>>> MPI_MODE_WRONLY, MPI_INFO_NULL, &f ); >>>>>> if( rank == 0 ) >>>>>> { >>>>>> MPI_File_write( f, &global_N, 1, MPI_INT, &stat ); >>>>>> } >>>>>> MPI_File_set_view( f, offset, mpi_atom_type_resized, >>>>>> mpi_atom_type_resized, datarep, MPI_INFO_NULL ); >>>>>> >>>>>> MPI_File_write_all( f, &atoms[0], (int)atoms.size(), >> mpi_atom_type_resized, >>>>>> &stat ); >>>>>> MPI_File_close( &f ); >>>>>> >>>>>> MPI_Type_free( &mpi_atom_type ); >>>>>> MPI_Type_free( &mpi_atom_type_resized ); >>>>>> >>>>>> return; >>>>>> } >>>>>> >>>>>> void read( char * fpath, std::vector<struct atom> &atoms ) >>>>>> { >>>>>> std::vector<int> s_blocklengths; >>>>>> std::vector<MPI_Aint> s_displacements; >>>>>> std::vector<MPI_Datatype> s_datatypes; >>>>>> MPI_Datatype mpi_atom_type, mpi_atom_type_resized; >>>>>> >>>>>> struct atom a; >>>>>> MPI_Aint addr_start, addr; >>>>>> >>>>>> MPI_File f; >>>>>> MPI_Status stat; >>>>>> >>>>>> int global_N; >>>>>> char *datarep = (char *)"native"; >>>>>> >>>>>> int type_size; >>>>>> >>>>>> /* >>>>>> Set up the structure data type >>>>>> */ >>>>>> MPI_Get_address( &a, &addr_start ); >>>>>> >>>>>> s_blocklengths.push_back( 1 ); >>>>>> s_datatypes.push_back( MPI_INT ); >>>>>> MPI_Get_address( &a.global_id, &addr ); >>>>>> s_displacements.push_back( addr - addr_start ); >>>>>> >>>>>> s_blocklengths.push_back( 3 ); >>>>>> s_datatypes.push_back( MPI_DOUBLE ); >>>>>> MPI_Get_address( &a.xyz[0], &addr ); >>>>>> s_displacements.push_back( addr - addr_start ); >>>>>> >>>>>> MPI_Type_create_struct( (int)s_blocklengths.size(), &s_blocklengths[0], >>>>>> &s_displacements[0], &s_datatypes[0], &mpi_atom_type ); >>>>>> MPI_Type_commit( &mpi_atom_type ); >>>>>> >>>>>> /* >>>>>> Take into account any compiler padding in creating an array of >>>>>> structures. >>>>>> */ >>>>>> MPI_Type_create_resized( mpi_atom_type, 0, sizeof(struct atom), >>>>>> &mpi_atom_type_resized ); >>>>>> MPI_Type_commit( &mpi_atom_type_resized ); >>>>>> >>>>>> MPI_Type_size( mpi_atom_type_resized, &type_size ); >>>>>> >>>>>> MPI_File_open( MPI_COMM_SELF, fpath, MPI_MODE_RDONLY, >>>>>> MPI_INFO_NULL, &f ); >>>>>> >>>>>> MPI_File_read( f, &global_N, 1, MPI_INT, &stat ); >>>>>> >>>>>> atoms.clear(); >>>>>> atoms.resize( global_N ); >>>>>> >>>>>> errcode = MPI_File_set_view( f, sizeof(int), mpi_atom_type_resized, >>>>>> mpi_atom_type_resized, datarep, MPI_INFO_NULL ); >>>>>> errcode = MPI_File_read( f, &atoms[0], global_N, >> mpi_atom_type_resized, >>>>>> &stat ); >>>>>> errcode = MPI_File_close( &f ); >>>>>> >>>>>> MPI_Type_free( &mpi_atom_type ); >>>>>> MPI_Type_free( &mpi_atom_type_resized ); >>>>>> >>>>>> return; >>>>>> } >>>>>> >>>>>> Calling MPI_Type_get_extent() and MPI_Type_get_true_extent() for both >>>>>> mpi_atom_type and mpi_atom_type_resized returns (0,32) bytes in all >> cases. >>>>>> Calling MPI_Type_size() on both derived data types returns 28 bytes. >>>>>> >>>>>> If I call MPI_File_get_type_extent() on both derived data types after >> opening >>>> the >>>>>> file, they both resolve to 32 bytes - so I think the problem is in the >>>> difference >>>>>> between the data representation in memory and on disk. If I explicitly use >> 32 >>>>>> bytes in the offset calculation in the write() routine above, it still doesn't >>>> work. >>>>>> >>>>>> I'm finding it remarkably difficult to do something very simple using MPI's >>>>>> derived data types and the parallel IO, and hence I'm guessing that I have >>>>>> fundamentally misunderstood one or more aspects of this. If anyone can >>>> help >>>>>> clarify where I'm going wrong, that would be much appreciated! >>>>>> >>>>>> Cheers, >>>>>> >>>>>> John. >>>>>> _______________________________________________ >>>>>> discuss mailing list discuss(a)mpich.org >>>>>> To manage subscription options or unsubscribe: >>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > >_______________________________________________ >discuss mailing list discuss(a)mpich.org >To manage subscription options or unsubscribe: >https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Use of MPI derived data types / MPI file IO
by Dave Goodell 19 Nov '12

19 Nov '12

All datatypes that have been created (by any means) should eventually be freed by the user. Whether or not the type has been committed has no bearing on whether it should be freed. Although, IIRC, a committed type can consume more memory under the hood and therefore failure to free it may result in a larger memory leak than for an uncommitted type :) -Dave On Nov 19, 2012, at 9:17 AM CST, Wei-keng Liao wrote: > Hi, John, > > I did not find from the MPI standard specific about this question. However, > if you check the man page of MPI_Type_free, one of the errors is > MPI_ERR_TYPE > Invalid datatype argument. May be an uncommitted MPI_Datatype (see MPI_Type_commit). > > So, it seems to imply one should only free the data types that have been committed. > But, I think the MPICH development team should confirm if this is the case. > > > Wei-keng > > > On Nov 19, 2012, at 8:46 AM, <jgrime(a)uchicago.edu> <jgrime(a)uchicago.edu> wrote: > >> Hi Wei-keng, >> >> It now works! Thanks for the help! >> >> One last question to the list: >> >> I've looked at the MPI 2 standards documents, and I'm still a little confused as to >> the precise semantics of MPI_Type_free(); as "MPI_Datatype" is an opaque type, >> I'm assuming that there is a certain amount of background allocation going on >> inside the MPI runtime when I call something like MPI_Type_create_struct() or >> similar routines. >> >> Am I right in assuming that I should call MPI_Type_free() on *all* derived data >> types I generate, even if they are not subsequently registered using >> MPI_Type_commit()? I would imagine that any other behaviour is likely to lead to >> memory leaks! >> >> Cheers, >> >> J. >> >> ---- Original message ---- >>> Date: Sun, 18 Nov 2012 18:58:37 -0600 >>> From: discuss-bounces(a)mpich.org (on behalf of Wei-keng Liao >> <wkliao(a)ece.northwestern.edu>) >>> Subject: Re: [mpich-discuss] Use of MPI derived data types / MPI file IO >>> To: discuss(a)mpich.org >>> >>> Hi, John, >>> >>> You certainly are on the right track to achieve that. Your code is almost >>> there, only the call to MPI_File_set_view is incorrect. In fact, you don't need it. >>> >>> Try remove the call to MPI_File_set_view and replace the MPI_File_write_all with: >>> MPI_File_write_at_all(f, offset, &atoms[0], (int)atoms.size(), >> mpi_atom_type_resized, &stat); >>> >>> On the reader side, you need to set the offset based on the new struct. Other >> than >>> that, it is the same as the writer case. (no need of MPI_File_set_view either). >>> >>> As for the portability issue, I would suggest you to use high-level I/O libraries, >>> such as PnetCDF. >>> >>> Wei-keng >>> >>> On Nov 18, 2012, at 12:38 PM, <jgrime(a)uchicago.edu> >> <jgrime(a)uchicago.edu> wrote: >>> >>>> Hi Wei-keng, >>>> >>>> That's a good point, thanks! >>>> >>>> However, I actually only want to save certain parts of the "atom" structure to >> file, >>>> and saving the whole array as a raw dump could waste a lot of disk space. >>>> >>>> For example, the "atom" structure I used in the example code in reality >> contains >>>> not only an integer and three contiguous doubles, but also at least another >> two >>>> double[3] entries which I may not want to save to disk. As the full data set >> can >>>> be hundreds of millions (or even billions) of "atom" structures, using a >> derived >>>> data type with only a restricted subset of the data in each "atom" structure >> will >>>> produce considerably smaller file sizes! >>>> >>>> There's also the problem of making the resultant file "portable" - raw >> memory >>>> dumps could make life difficult in trying to use output files on machines with >>>> different processor architectures. Once I get the derived data types working, >> I >>>> can then switch from the "native" representation to something else >> ("external32" >>>> etc), which should allow me to create portable output files, provided I'm >> careful >>>> with using MPIs file offset routines etc if the file is larger than plain old 32 >> bit >>>> offsets can handle. >>>> >>>> Cheers, >>>> >>>> J. >>>> >>>> ---- Original message ---- >>>>> Date: Sun, 18 Nov 2012 12:27:04 -0600 >>>>> From: discuss-bounces(a)mpich.org (on behalf of Wei-keng Liao >>>> <wkliao(a)ece.northwestern.edu>) >>>>> Subject: Re: [mpich-discuss] Use of MPI derived data types / MPI file IO >>>>> To: discuss(a)mpich.org >>>>> >>>>> Hi, John >>>>> >>>>> If your I/O is simply appending one process's data after another and the >> I/O >>>> buffers in memory >>>>> are all contiguous, then you can simply do the following without defining >> MPI >>>>> derived data types or setting the file view. >>>>> >>>>> MPI_File_write_at_all(f, offset, &atoms[0], (int)atoms.size() * sizeof(struct >> atom), >>>> MPI_BYTE, &stat); >>>>> >>>>> Using derived data types is usually when you have noncontiguous buffer in >>>> memory or >>>>> want to access non-contiguous data in files. >>>>> >>>>> >>>>> Wei-keng >>>>> >>>>> On Nov 18, 2012, at 11:52 AM, <jgrime(a)uchicago.edu> >>>> <jgrime(a)uchicago.edu> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> I'm having some problems with using derived data types and MPI parallel >> IO, >>>> and >>>>>> was wondering if anyone could help. I tried to search the archives in case >>>> this >>>>>> was covered earlier, but that just gave me "ht://Dig error" messages. >>>>>> >>>>>> Outline: I have written a C++ program where each MPI rank acts on data >>>> stored >>>>>> in a local array of structures. The arrays are typically of different lengths >> on >>>> each >>>>>> rank. I wish to write and read the contents of these arrays to disk using >> MPI's >>>>>> parallel IO routines. The file format is simply an initial integer which >>>> describes >>>>>> how many "structures" are in the file, followed by the data which >> represents >>>> the >>>>>> "structure information" from all ranks (ie the total data set). >>>>>> >>>>>> So far, I've tried two different approaches: the first consists of each rank >>>>>> serialising the contents of the local array of structures into a byte array, >>>> which is >>>>>> then saved to file "f" using MPI_File_set_view( f, MPI_COMM_WORLD, >> offset, >>>>>> MPI_CHAR, MPI_CHAR, "native", MPI_INFO_NULL ) to skip the initial integer >>>>>> "header" and then a call to MPI_File_write_all( f, local_bytearray, >>>> local_n_bytes, >>>>>> MPI_CHAR, &status ). Here, "offset" is simply the size of an integer (in >> bytes) >>>> + >>>>>> the summation of the number of bytes each preceeding rank wishes to >> write >>>> to >>>>>> the file (received via an earlier MPI_Allgather call). This seems to work, as >>>> when I >>>>>> read the file back in on a single MPI rank and deserialise the data into an >>>> array of >>>>>> structures I get the results I expect. >>>>>> >>>>>> The second approach is to use MPI's derived data types to create MPI >>>>>> representations of the structures, and then treat the arrays of structures >> as >>>> MPI >>>>>> data types. This allows me to avoid copying the local data into an >>>> intermediate >>>>>> buffer etc, and seems the more elegant approach. I cannot, however, >> seem >>>> to >>>>>> make this approach work. >>>>>> >>>>>> I'm pretty sure the problem lies in my use of the file views, but I'm not >> sure >>>>>> where I'm going wrong. The reading of the integer "header" always works >>>> fine, >>>>>> but the proceeding data is garbled. I'm using the "native" data >> representation >>>> for >>>>>> testing, but will likely change that to something more portable when I get >>>> this >>>>>> code working. >>>>>> >>>>>> I've included the important excerpts of the test code I'm trying to use >> below >>>>>> (with some printf()s and error handling etc removed to make it a little >> more >>>>>> concise). I have previously tested that std::vector allocates a contiguous >> flat >>>>>> array of the appropriate data type in memory, so passing a >> pointer/reference >>>> to >>>>>> the first element in such a data structure behaves the same way as simply >>>>>> passing a conventional array of the appropriate data type: >>>>>> >>>>>> struct atom >>>>>> { >>>>>> int global_id; >>>>>> double xyz[3]; >>>>>> }; >>>>>> >>>>>> void write( char * fpath, std::vector<struct atom> &atoms, int rank, int >>>> nranks ) >>>>>> { >>>>>> /* >>>>>> Memory layout information for the structure we wish to convert >> into >>>>>> an >>>>>> MPI derived data type. >>>>>> */ >>>>>> std::vector<int> s_blocklengths; >>>>>> std::vector<MPI_Aint> s_displacements; >>>>>> std::vector<MPI_Datatype> s_datatypes; >>>>>> MPI_Aint addr_start, addr; >>>>>> MPI_Datatype mpi_atom_type, mpi_atom_type_resized; >>>>>> int type_size; >>>>>> >>>>>> struct atom a; >>>>>> >>>>>> MPI_File f; >>>>>> MPI_Status stat; >>>>>> MPI_Offset offset; >>>>>> char *datarep = (char *)"native"; >>>>>> >>>>>> std::vector<int> all_N; >>>>>> int local_N, global_N; >>>>>> >>>>>> /* >>>>>> Set up the structure data type: single integer, and 3 double >> precision >>>>>> floats. >>>>>> We use the temporary "a" structure to determine the layout of >> memory >>>>>> inside >>>>>> atom structures. >>>>>> */ >>>>>> MPI_Get_address( &a, &addr_start ); >>>>>> >>>>>> s_blocklengths.push_back( 1 ); >>>>>> s_datatypes.push_back( MPI_INT ); >>>>>> MPI_Get_address( &a.global_id, &addr ); >>>>>> s_displacements.push_back( addr - addr_start ); >>>>>> >>>>>> s_blocklengths.push_back( 3 ); >>>>>> s_datatypes.push_back( MPI_DOUBLE ); >>>>>> MPI_Get_address( &a.xyz[0], &addr ); >>>>>> s_displacements.push_back( addr - addr_start ); >>>>>> >>>>>> MPI_Type_create_struct( (int)s_blocklengths.size(), &s_blocklengths[0], >>>>>> &s_displacements[0], &s_datatypes[0], &mpi_atom_type ); >>>>>> MPI_Type_commit( &mpi_atom_type ); >>>>>> >>>>>> /* >>>>>> Take into account any compiler padding in creating an array of >>>>>> structures. >>>>>> */ >>>>>> MPI_Type_create_resized( mpi_atom_type, 0, sizeof(struct atom), >>>>>> &mpi_atom_type_resized ); >>>>>> MPI_Type_commit( &mpi_atom_type_resized ); >>>>>> >>>>>> MPI_Type_size( mpi_atom_type_resized, &type_size ); >>>>>> >>>>>> local_N = (int)atoms.size(); >>>>>> all_N.resize( nranks ); >>>>>> >>>>>> MPI_Allgather( &local_N, 1, MPI_INT, &all_N[0], 1, MPI_INT, >>>>>> MPI_COMM_WORLD ); >>>>>> >>>>>> global_N = 0; >>>>>> for( size_t i=0; i<all_N.size(); i++ ) global_N += all_N[i]; >>>>>> >>>>>> offset = 0; >>>>>> for( int i=0; i<rank; i++ ) offset += all_N[i]; >>>>>> >>>>>> offset *= type_size; // convert from structure counts -> bytes into file >> for >>>>>> true structure size >>>>>> offset += sizeof( int ); // skip leading integer (global_N) in file. >>>>>> >>>>>> MPI_File_open( MPI_COMM_WORLD, fpath, MPI_MODE_CREATE | >>>>>> MPI_MODE_WRONLY, MPI_INFO_NULL, &f ); >>>>>> if( rank == 0 ) >>>>>> { >>>>>> MPI_File_write( f, &global_N, 1, MPI_INT, &stat ); >>>>>> } >>>>>> MPI_File_set_view( f, offset, mpi_atom_type_resized, >>>>>> mpi_atom_type_resized, datarep, MPI_INFO_NULL ); >>>>>> >>>>>> MPI_File_write_all( f, &atoms[0], (int)atoms.size(), >> mpi_atom_type_resized, >>>>>> &stat ); >>>>>> MPI_File_close( &f ); >>>>>> >>>>>> MPI_Type_free( &mpi_atom_type ); >>>>>> MPI_Type_free( &mpi_atom_type_resized ); >>>>>> >>>>>> return; >>>>>> } >>>>>> >>>>>> void read( char * fpath, std::vector<struct atom> &atoms ) >>>>>> { >>>>>> std::vector<int> s_blocklengths; >>>>>> std::vector<MPI_Aint> s_displacements; >>>>>> std::vector<MPI_Datatype> s_datatypes; >>>>>> MPI_Datatype mpi_atom_type, mpi_atom_type_resized; >>>>>> >>>>>> struct atom a; >>>>>> MPI_Aint addr_start, addr; >>>>>> >>>>>> MPI_File f; >>>>>> MPI_Status stat; >>>>>> >>>>>> int global_N; >>>>>> char *datarep = (char *)"native"; >>>>>> >>>>>> int type_size; >>>>>> >>>>>> /* >>>>>> Set up the structure data type >>>>>> */ >>>>>> MPI_Get_address( &a, &addr_start ); >>>>>> >>>>>> s_blocklengths.push_back( 1 ); >>>>>> s_datatypes.push_back( MPI_INT ); >>>>>> MPI_Get_address( &a.global_id, &addr ); >>>>>> s_displacements.push_back( addr - addr_start ); >>>>>> >>>>>> s_blocklengths.push_back( 3 ); >>>>>> s_datatypes.push_back( MPI_DOUBLE ); >>>>>> MPI_Get_address( &a.xyz[0], &addr ); >>>>>> s_displacements.push_back( addr - addr_start ); >>>>>> >>>>>> MPI_Type_create_struct( (int)s_blocklengths.size(), &s_blocklengths[0], >>>>>> &s_displacements[0], &s_datatypes[0], &mpi_atom_type ); >>>>>> MPI_Type_commit( &mpi_atom_type ); >>>>>> >>>>>> /* >>>>>> Take into account any compiler padding in creating an array of >>>>>> structures. >>>>>> */ >>>>>> MPI_Type_create_resized( mpi_atom_type, 0, sizeof(struct atom), >>>>>> &mpi_atom_type_resized ); >>>>>> MPI_Type_commit( &mpi_atom_type_resized ); >>>>>> >>>>>> MPI_Type_size( mpi_atom_type_resized, &type_size ); >>>>>> >>>>>> MPI_File_open( MPI_COMM_SELF, fpath, MPI_MODE_RDONLY, >>>>>> MPI_INFO_NULL, &f ); >>>>>> >>>>>> MPI_File_read( f, &global_N, 1, MPI_INT, &stat ); >>>>>> >>>>>> atoms.clear(); >>>>>> atoms.resize( global_N ); >>>>>> >>>>>> errcode = MPI_File_set_view( f, sizeof(int), mpi_atom_type_resized, >>>>>> mpi_atom_type_resized, datarep, MPI_INFO_NULL ); >>>>>> errcode = MPI_File_read( f, &atoms[0], global_N, >> mpi_atom_type_resized, >>>>>> &stat ); >>>>>> errcode = MPI_File_close( &f ); >>>>>> >>>>>> MPI_Type_free( &mpi_atom_type ); >>>>>> MPI_Type_free( &mpi_atom_type_resized ); >>>>>> >>>>>> return; >>>>>> } >>>>>> >>>>>> Calling MPI_Type_get_extent() and MPI_Type_get_true_extent() for both >>>>>> mpi_atom_type and mpi_atom_type_resized returns (0,32) bytes in all >> cases. >>>>>> Calling MPI_Type_size() on both derived data types returns 28 bytes. >>>>>> >>>>>> If I call MPI_File_get_type_extent() on both derived data types after >> opening >>>> the >>>>>> file, they both resolve to 32 bytes - so I think the problem is in the >>>> difference >>>>>> between the data representation in memory and on disk. If I explicitly use >> 32 >>>>>> bytes in the offset calculation in the write() routine above, it still doesn't >>>> work. >>>>>> >>>>>> I'm finding it remarkably difficult to do something very simple using MPI's >>>>>> derived data types and the parallel IO, and hence I'm guessing that I have >>>>>> fundamentally misunderstood one or more aspects of this. If anyone can >>>> help >>>>>> clarify where I'm going wrong, that would be much appreciated! >>>>>> >>>>>> Cheers, >>>>>> >>>>>> John. >>>>>> _______________________________________________ >>>>>> discuss mailing list discuss(a)mpich.org >>>>>> To manage subscription options or unsubscribe: >>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Use of MPI derived data types / MPI file IO
by Wei-keng Liao 19 Nov '12

19 Nov '12

Hi, John, I did not find from the MPI standard specific about this question. However, if you check the man page of MPI_Type_free, one of the errors is MPI_ERR_TYPE Invalid datatype argument. May be an uncommitted MPI_Datatype (see MPI_Type_commit). So, it seems to imply one should only free the data types that have been committed. But, I think the MPICH development team should confirm if this is the case. Wei-keng On Nov 19, 2012, at 8:46 AM, <jgrime(a)uchicago.edu> <jgrime(a)uchicago.edu> wrote: > Hi Wei-keng, > > It now works! Thanks for the help! > > One last question to the list: > > I've looked at the MPI 2 standards documents, and I'm still a little confused as to > the precise semantics of MPI_Type_free(); as "MPI_Datatype" is an opaque type, > I'm assuming that there is a certain amount of background allocation going on > inside the MPI runtime when I call something like MPI_Type_create_struct() or > similar routines. > > Am I right in assuming that I should call MPI_Type_free() on *all* derived data > types I generate, even if they are not subsequently registered using > MPI_Type_commit()? I would imagine that any other behaviour is likely to lead to > memory leaks! > > Cheers, > > J. > > ---- Original message ---- >> Date: Sun, 18 Nov 2012 18:58:37 -0600 >> From: discuss-bounces(a)mpich.org (on behalf of Wei-keng Liao > <wkliao(a)ece.northwestern.edu>) >> Subject: Re: [mpich-discuss] Use of MPI derived data types / MPI file IO >> To: discuss(a)mpich.org >> >> Hi, John, >> >> You certainly are on the right track to achieve that. Your code is almost >> there, only the call to MPI_File_set_view is incorrect. In fact, you don't need it. >> >> Try remove the call to MPI_File_set_view and replace the MPI_File_write_all with: >> MPI_File_write_at_all(f, offset, &atoms[0], (int)atoms.size(), > mpi_atom_type_resized, &stat); >> >> On the reader side, you need to set the offset based on the new struct. Other > than >> that, it is the same as the writer case. (no need of MPI_File_set_view either). >> >> As for the portability issue, I would suggest you to use high-level I/O libraries, >> such as PnetCDF. >> >> Wei-keng >> >> On Nov 18, 2012, at 12:38 PM, <jgrime(a)uchicago.edu> > <jgrime(a)uchicago.edu> wrote: >> >>> Hi Wei-keng, >>> >>> That's a good point, thanks! >>> >>> However, I actually only want to save certain parts of the "atom" structure to > file, >>> and saving the whole array as a raw dump could waste a lot of disk space. >>> >>> For example, the "atom" structure I used in the example code in reality > contains >>> not only an integer and three contiguous doubles, but also at least another > two >>> double[3] entries which I may not want to save to disk. As the full data set > can >>> be hundreds of millions (or even billions) of "atom" structures, using a > derived >>> data type with only a restricted subset of the data in each "atom" structure > will >>> produce considerably smaller file sizes! >>> >>> There's also the problem of making the resultant file "portable" - raw > memory >>> dumps could make life difficult in trying to use output files on machines with >>> different processor architectures. Once I get the derived data types working, > I >>> can then switch from the "native" representation to something else > ("external32" >>> etc), which should allow me to create portable output files, provided I'm > careful >>> with using MPIs file offset routines etc if the file is larger than plain old 32 > bit >>> offsets can handle. >>> >>> Cheers, >>> >>> J. >>> >>> ---- Original message ---- >>>> Date: Sun, 18 Nov 2012 12:27:04 -0600 >>>> From: discuss-bounces(a)mpich.org (on behalf of Wei-keng Liao >>> <wkliao(a)ece.northwestern.edu>) >>>> Subject: Re: [mpich-discuss] Use of MPI derived data types / MPI file IO >>>> To: discuss(a)mpich.org >>>> >>>> Hi, John >>>> >>>> If your I/O is simply appending one process's data after another and the > I/O >>> buffers in memory >>>> are all contiguous, then you can simply do the following without defining > MPI >>>> derived data types or setting the file view. >>>> >>>> MPI_File_write_at_all(f, offset, &atoms[0], (int)atoms.size() * sizeof(struct > atom), >>> MPI_BYTE, &stat); >>>> >>>> Using derived data types is usually when you have noncontiguous buffer in >>> memory or >>>> want to access non-contiguous data in files. >>>> >>>> >>>> Wei-keng >>>> >>>> On Nov 18, 2012, at 11:52 AM, <jgrime(a)uchicago.edu> >>> <jgrime(a)uchicago.edu> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I'm having some problems with using derived data types and MPI parallel > IO, >>> and >>>>> was wondering if anyone could help. I tried to search the archives in case >>> this >>>>> was covered earlier, but that just gave me "ht://Dig error" messages. >>>>> >>>>> Outline: I have written a C++ program where each MPI rank acts on data >>> stored >>>>> in a local array of structures. The arrays are typically of different lengths > on >>> each >>>>> rank. I wish to write and read the contents of these arrays to disk using > MPI's >>>>> parallel IO routines. The file format is simply an initial integer which >>> describes >>>>> how many "structures" are in the file, followed by the data which > represents >>> the >>>>> "structure information" from all ranks (ie the total data set). >>>>> >>>>> So far, I've tried two different approaches: the first consists of each rank >>>>> serialising the contents of the local array of structures into a byte array, >>> which is >>>>> then saved to file "f" using MPI_File_set_view( f, MPI_COMM_WORLD, > offset, >>>>> MPI_CHAR, MPI_CHAR, "native", MPI_INFO_NULL ) to skip the initial integer >>>>> "header" and then a call to MPI_File_write_all( f, local_bytearray, >>> local_n_bytes, >>>>> MPI_CHAR, &status ). Here, "offset" is simply the size of an integer (in > bytes) >>> + >>>>> the summation of the number of bytes each preceeding rank wishes to > write >>> to >>>>> the file (received via an earlier MPI_Allgather call). This seems to work, as >>> when I >>>>> read the file back in on a single MPI rank and deserialise the data into an >>> array of >>>>> structures I get the results I expect. >>>>> >>>>> The second approach is to use MPI's derived data types to create MPI >>>>> representations of the structures, and then treat the arrays of structures > as >>> MPI >>>>> data types. This allows me to avoid copying the local data into an >>> intermediate >>>>> buffer etc, and seems the more elegant approach. I cannot, however, > seem >>> to >>>>> make this approach work. >>>>> >>>>> I'm pretty sure the problem lies in my use of the file views, but I'm not > sure >>>>> where I'm going wrong. The reading of the integer "header" always works >>> fine, >>>>> but the proceeding data is garbled. I'm using the "native" data > representation >>> for >>>>> testing, but will likely change that to something more portable when I get >>> this >>>>> code working. >>>>> >>>>> I've included the important excerpts of the test code I'm trying to use > below >>>>> (with some printf()s and error handling etc removed to make it a little > more >>>>> concise). I have previously tested that std::vector allocates a contiguous > flat >>>>> array of the appropriate data type in memory, so passing a > pointer/reference >>> to >>>>> the first element in such a data structure behaves the same way as simply >>>>> passing a conventional array of the appropriate data type: >>>>> >>>>> struct atom >>>>> { >>>>> int global_id; >>>>> double xyz[3]; >>>>> }; >>>>> >>>>> void write( char * fpath, std::vector<struct atom> &atoms, int rank, int >>> nranks ) >>>>> { >>>>> /* >>>>> Memory layout information for the structure we wish to convert > into >>>>> an >>>>> MPI derived data type. >>>>> */ >>>>> std::vector<int> s_blocklengths; >>>>> std::vector<MPI_Aint> s_displacements; >>>>> std::vector<MPI_Datatype> s_datatypes; >>>>> MPI_Aint addr_start, addr; >>>>> MPI_Datatype mpi_atom_type, mpi_atom_type_resized; >>>>> int type_size; >>>>> >>>>> struct atom a; >>>>> >>>>> MPI_File f; >>>>> MPI_Status stat; >>>>> MPI_Offset offset; >>>>> char *datarep = (char *)"native"; >>>>> >>>>> std::vector<int> all_N; >>>>> int local_N, global_N; >>>>> >>>>> /* >>>>> Set up the structure data type: single integer, and 3 double > precision >>>>> floats. >>>>> We use the temporary "a" structure to determine the layout of > memory >>>>> inside >>>>> atom structures. >>>>> */ >>>>> MPI_Get_address( &a, &addr_start ); >>>>> >>>>> s_blocklengths.push_back( 1 ); >>>>> s_datatypes.push_back( MPI_INT ); >>>>> MPI_Get_address( &a.global_id, &addr ); >>>>> s_displacements.push_back( addr - addr_start ); >>>>> >>>>> s_blocklengths.push_back( 3 ); >>>>> s_datatypes.push_back( MPI_DOUBLE ); >>>>> MPI_Get_address( &a.xyz[0], &addr ); >>>>> s_displacements.push_back( addr - addr_start ); >>>>> >>>>> MPI_Type_create_struct( (int)s_blocklengths.size(), &s_blocklengths[0], >>>>> &s_displacements[0], &s_datatypes[0], &mpi_atom_type ); >>>>> MPI_Type_commit( &mpi_atom_type ); >>>>> >>>>> /* >>>>> Take into account any compiler padding in creating an array of >>>>> structures. >>>>> */ >>>>> MPI_Type_create_resized( mpi_atom_type, 0, sizeof(struct atom), >>>>> &mpi_atom_type_resized ); >>>>> MPI_Type_commit( &mpi_atom_type_resized ); >>>>> >>>>> MPI_Type_size( mpi_atom_type_resized, &type_size ); >>>>> >>>>> local_N = (int)atoms.size(); >>>>> all_N.resize( nranks ); >>>>> >>>>> MPI_Allgather( &local_N, 1, MPI_INT, &all_N[0], 1, MPI_INT, >>>>> MPI_COMM_WORLD ); >>>>> >>>>> global_N = 0; >>>>> for( size_t i=0; i<all_N.size(); i++ ) global_N += all_N[i]; >>>>> >>>>> offset = 0; >>>>> for( int i=0; i<rank; i++ ) offset += all_N[i]; >>>>> >>>>> offset *= type_size; // convert from structure counts -> bytes into file > for >>>>> true structure size >>>>> offset += sizeof( int ); // skip leading integer (global_N) in file. >>>>> >>>>> MPI_File_open( MPI_COMM_WORLD, fpath, MPI_MODE_CREATE | >>>>> MPI_MODE_WRONLY, MPI_INFO_NULL, &f ); >>>>> if( rank == 0 ) >>>>> { >>>>> MPI_File_write( f, &global_N, 1, MPI_INT, &stat ); >>>>> } >>>>> MPI_File_set_view( f, offset, mpi_atom_type_resized, >>>>> mpi_atom_type_resized, datarep, MPI_INFO_NULL ); >>>>> >>>>> MPI_File_write_all( f, &atoms[0], (int)atoms.size(), > mpi_atom_type_resized, >>>>> &stat ); >>>>> MPI_File_close( &f ); >>>>> >>>>> MPI_Type_free( &mpi_atom_type ); >>>>> MPI_Type_free( &mpi_atom_type_resized ); >>>>> >>>>> return; >>>>> } >>>>> >>>>> void read( char * fpath, std::vector<struct atom> &atoms ) >>>>> { >>>>> std::vector<int> s_blocklengths; >>>>> std::vector<MPI_Aint> s_displacements; >>>>> std::vector<MPI_Datatype> s_datatypes; >>>>> MPI_Datatype mpi_atom_type, mpi_atom_type_resized; >>>>> >>>>> struct atom a; >>>>> MPI_Aint addr_start, addr; >>>>> >>>>> MPI_File f; >>>>> MPI_Status stat; >>>>> >>>>> int global_N; >>>>> char *datarep = (char *)"native"; >>>>> >>>>> int type_size; >>>>> >>>>> /* >>>>> Set up the structure data type >>>>> */ >>>>> MPI_Get_address( &a, &addr_start ); >>>>> >>>>> s_blocklengths.push_back( 1 ); >>>>> s_datatypes.push_back( MPI_INT ); >>>>> MPI_Get_address( &a.global_id, &addr ); >>>>> s_displacements.push_back( addr - addr_start ); >>>>> >>>>> s_blocklengths.push_back( 3 ); >>>>> s_datatypes.push_back( MPI_DOUBLE ); >>>>> MPI_Get_address( &a.xyz[0], &addr ); >>>>> s_displacements.push_back( addr - addr_start ); >>>>> >>>>> MPI_Type_create_struct( (int)s_blocklengths.size(), &s_blocklengths[0], >>>>> &s_displacements[0], &s_datatypes[0], &mpi_atom_type ); >>>>> MPI_Type_commit( &mpi_atom_type ); >>>>> >>>>> /* >>>>> Take into account any compiler padding in creating an array of >>>>> structures. >>>>> */ >>>>> MPI_Type_create_resized( mpi_atom_type, 0, sizeof(struct atom), >>>>> &mpi_atom_type_resized ); >>>>> MPI_Type_commit( &mpi_atom_type_resized ); >>>>> >>>>> MPI_Type_size( mpi_atom_type_resized, &type_size ); >>>>> >>>>> MPI_File_open( MPI_COMM_SELF, fpath, MPI_MODE_RDONLY, >>>>> MPI_INFO_NULL, &f ); >>>>> >>>>> MPI_File_read( f, &global_N, 1, MPI_INT, &stat ); >>>>> >>>>> atoms.clear(); >>>>> atoms.resize( global_N ); >>>>> >>>>> errcode = MPI_File_set_view( f, sizeof(int), mpi_atom_type_resized, >>>>> mpi_atom_type_resized, datarep, MPI_INFO_NULL ); >>>>> errcode = MPI_File_read( f, &atoms[0], global_N, > mpi_atom_type_resized, >>>>> &stat ); >>>>> errcode = MPI_File_close( &f ); >>>>> >>>>> MPI_Type_free( &mpi_atom_type ); >>>>> MPI_Type_free( &mpi_atom_type_resized ); >>>>> >>>>> return; >>>>> } >>>>> >>>>> Calling MPI_Type_get_extent() and MPI_Type_get_true_extent() for both >>>>> mpi_atom_type and mpi_atom_type_resized returns (0,32) bytes in all > cases. >>>>> Calling MPI_Type_size() on both derived data types returns 28 bytes. >>>>> >>>>> If I call MPI_File_get_type_extent() on both derived data types after > opening >>> the >>>>> file, they both resolve to 32 bytes - so I think the problem is in the >>> difference >>>>> between the data representation in memory and on disk. If I explicitly use > 32 >>>>> bytes in the offset calculation in the write() routine above, it still doesn't >>> work. >>>>> >>>>> I'm finding it remarkably difficult to do something very simple using MPI's >>>>> derived data types and the parallel IO, and hence I'm guessing that I have >>>>> fundamentally misunderstood one or more aspects of this. If anyone can >>> help >>>>> clarify where I'm going wrong, that would be much appreciated! >>>>> >>>>> Cheers, >>>>> >>>>> John. >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Use of MPI derived data types / MPI file IO
by Darius Buntinas 19 Nov '12

19 Nov '12

Yep, creating datatypes results in internal allocation, so freeing the datatypes will avoid memory leaks. -d On Nov 19, 2012, at 8:46 AM, <jgrime(a)uchicago.edu> <jgrime(a)uchicago.edu> wrote: > Hi Wei-keng, > > It now works! Thanks for the help! > > One last question to the list: > > I've looked at the MPI 2 standards documents, and I'm still a little confused as to > the precise semantics of MPI_Type_free(); as "MPI_Datatype" is an opaque type, > I'm assuming that there is a certain amount of background allocation going on > inside the MPI runtime when I call something like MPI_Type_create_struct() or > similar routines. > > Am I right in assuming that I should call MPI_Type_free() on *all* derived data > types I generate, even if they are not subsequently registered using > MPI_Type_commit()? I would imagine that any other behaviour is likely to lead to > memory leaks! > > Cheers, > > J. > > ---- Original message ---- >> Date: Sun, 18 Nov 2012 18:58:37 -0600 >> From: discuss-bounces(a)mpich.org (on behalf of Wei-keng Liao > <wkliao(a)ece.northwestern.edu>) >> Subject: Re: [mpich-discuss] Use of MPI derived data types / MPI file IO >> To: discuss(a)mpich.org >> >> Hi, John, >> >> You certainly are on the right track to achieve that. Your code is almost >> there, only the call to MPI_File_set_view is incorrect. In fact, you don't need it. >> >> Try remove the call to MPI_File_set_view and replace the MPI_File_write_all with: >> MPI_File_write_at_all(f, offset, &atoms[0], (int)atoms.size(), > mpi_atom_type_resized, &stat); >> >> On the reader side, you need to set the offset based on the new struct. Other > than >> that, it is the same as the writer case. (no need of MPI_File_set_view either). >> >> As for the portability issue, I would suggest you to use high-level I/O libraries, >> such as PnetCDF. >> >> Wei-keng >> >> On Nov 18, 2012, at 12:38 PM, <jgrime(a)uchicago.edu> > <jgrime(a)uchicago.edu> wrote: >> >>> Hi Wei-keng, >>> >>> That's a good point, thanks! >>> >>> However, I actually only want to save certain parts of the "atom" structure to > file, >>> and saving the whole array as a raw dump could waste a lot of disk space. >>> >>> For example, the "atom" structure I used in the example code in reality > contains >>> not only an integer and three contiguous doubles, but also at least another > two >>> double[3] entries which I may not want to save to disk. As the full data set > can >>> be hundreds of millions (or even billions) of "atom" structures, using a > derived >>> data type with only a restricted subset of the data in each "atom" structure > will >>> produce considerably smaller file sizes! >>> >>> There's also the problem of making the resultant file "portable" - raw > memory >>> dumps could make life difficult in trying to use output files on machines with >>> different processor architectures. Once I get the derived data types working, > I >>> can then switch from the "native" representation to something else > ("external32" >>> etc), which should allow me to create portable output files, provided I'm > careful >>> with using MPIs file offset routines etc if the file is larger than plain old 32 > bit >>> offsets can handle. >>> >>> Cheers, >>> >>> J. >>> >>> ---- Original message ---- >>>> Date: Sun, 18 Nov 2012 12:27:04 -0600 >>>> From: discuss-bounces(a)mpich.org (on behalf of Wei-keng Liao >>> <wkliao(a)ece.northwestern.edu>) >>>> Subject: Re: [mpich-discuss] Use of MPI derived data types / MPI file IO >>>> To: discuss(a)mpich.org >>>> >>>> Hi, John >>>> >>>> If your I/O is simply appending one process's data after another and the > I/O >>> buffers in memory >>>> are all contiguous, then you can simply do the following without defining > MPI >>>> derived data types or setting the file view. >>>> >>>> MPI_File_write_at_all(f, offset, &atoms[0], (int)atoms.size() * sizeof(struct > atom), >>> MPI_BYTE, &stat); >>>> >>>> Using derived data types is usually when you have noncontiguous buffer in >>> memory or >>>> want to access non-contiguous data in files. >>>> >>>> >>>> Wei-keng >>>> >>>> On Nov 18, 2012, at 11:52 AM, <jgrime(a)uchicago.edu> >>> <jgrime(a)uchicago.edu> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I'm having some problems with using derived data types and MPI parallel > IO, >>> and >>>>> was wondering if anyone could help. I tried to search the archives in case >>> this >>>>> was covered earlier, but that just gave me "ht://Dig error" messages. >>>>> >>>>> Outline: I have written a C++ program where each MPI rank acts on data >>> stored >>>>> in a local array of structures. The arrays are typically of different lengths > on >>> each >>>>> rank. I wish to write and read the contents of these arrays to disk using > MPI's >>>>> parallel IO routines. The file format is simply an initial integer which >>> describes >>>>> how many "structures" are in the file, followed by the data which > represents >>> the >>>>> "structure information" from all ranks (ie the total data set). >>>>> >>>>> So far, I've tried two different approaches: the first consists of each rank >>>>> serialising the contents of the local array of structures into a byte array, >>> which is >>>>> then saved to file "f" using MPI_File_set_view( f, MPI_COMM_WORLD, > offset, >>>>> MPI_CHAR, MPI_CHAR, "native", MPI_INFO_NULL ) to skip the initial integer >>>>> "header" and then a call to MPI_File_write_all( f, local_bytearray, >>> local_n_bytes, >>>>> MPI_CHAR, &status ). Here, "offset" is simply the size of an integer (in > bytes) >>> + >>>>> the summation of the number of bytes each preceeding rank wishes to > write >>> to >>>>> the file (received via an earlier MPI_Allgather call). This seems to work, as >>> when I >>>>> read the file back in on a single MPI rank and deserialise the data into an >>> array of >>>>> structures I get the results I expect. >>>>> >>>>> The second approach is to use MPI's derived data types to create MPI >>>>> representations of the structures, and then treat the arrays of structures > as >>> MPI >>>>> data types. This allows me to avoid copying the local data into an >>> intermediate >>>>> buffer etc, and seems the more elegant approach. I cannot, however, > seem >>> to >>>>> make this approach work. >>>>> >>>>> I'm pretty sure the problem lies in my use of the file views, but I'm not > sure >>>>> where I'm going wrong. The reading of the integer "header" always works >>> fine, >>>>> but the proceeding data is garbled. I'm using the "native" data > representation >>> for >>>>> testing, but will likely change that to something more portable when I get >>> this >>>>> code working. >>>>> >>>>> I've included the important excerpts of the test code I'm trying to use > below >>>>> (with some printf()s and error handling etc removed to make it a little > more >>>>> concise). I have previously tested that std::vector allocates a contiguous > flat >>>>> array of the appropriate data type in memory, so passing a > pointer/reference >>> to >>>>> the first element in such a data structure behaves the same way as simply >>>>> passing a conventional array of the appropriate data type: >>>>> >>>>> struct atom >>>>> { >>>>> int global_id; >>>>> double xyz[3]; >>>>> }; >>>>> >>>>> void write( char * fpath, std::vector<struct atom> &atoms, int rank, int >>> nranks ) >>>>> { >>>>> /* >>>>> Memory layout information for the structure we wish to convert > into >>>>> an >>>>> MPI derived data type. >>>>> */ >>>>> std::vector<int> s_blocklengths; >>>>> std::vector<MPI_Aint> s_displacements; >>>>> std::vector<MPI_Datatype> s_datatypes; >>>>> MPI_Aint addr_start, addr; >>>>> MPI_Datatype mpi_atom_type, mpi_atom_type_resized; >>>>> int type_size; >>>>> >>>>> struct atom a; >>>>> >>>>> MPI_File f; >>>>> MPI_Status stat; >>>>> MPI_Offset offset; >>>>> char *datarep = (char *)"native"; >>>>> >>>>> std::vector<int> all_N; >>>>> int local_N, global_N; >>>>> >>>>> /* >>>>> Set up the structure data type: single integer, and 3 double > precision >>>>> floats. >>>>> We use the temporary "a" structure to determine the layout of > memory >>>>> inside >>>>> atom structures. >>>>> */ >>>>> MPI_Get_address( &a, &addr_start ); >>>>> >>>>> s_blocklengths.push_back( 1 ); >>>>> s_datatypes.push_back( MPI_INT ); >>>>> MPI_Get_address( &a.global_id, &addr ); >>>>> s_displacements.push_back( addr - addr_start ); >>>>> >>>>> s_blocklengths.push_back( 3 ); >>>>> s_datatypes.push_back( MPI_DOUBLE ); >>>>> MPI_Get_address( &a.xyz[0], &addr ); >>>>> s_displacements.push_back( addr - addr_start ); >>>>> >>>>> MPI_Type_create_struct( (int)s_blocklengths.size(), &s_blocklengths[0], >>>>> &s_displacements[0], &s_datatypes[0], &mpi_atom_type ); >>>>> MPI_Type_commit( &mpi_atom_type ); >>>>> >>>>> /* >>>>> Take into account any compiler padding in creating an array of >>>>> structures. >>>>> */ >>>>> MPI_Type_create_resized( mpi_atom_type, 0, sizeof(struct atom), >>>>> &mpi_atom_type_resized ); >>>>> MPI_Type_commit( &mpi_atom_type_resized ); >>>>> >>>>> MPI_Type_size( mpi_atom_type_resized, &type_size ); >>>>> >>>>> local_N = (int)atoms.size(); >>>>> all_N.resize( nranks ); >>>>> >>>>> MPI_Allgather( &local_N, 1, MPI_INT, &all_N[0], 1, MPI_INT, >>>>> MPI_COMM_WORLD ); >>>>> >>>>> global_N = 0; >>>>> for( size_t i=0; i<all_N.size(); i++ ) global_N += all_N[i]; >>>>> >>>>> offset = 0; >>>>> for( int i=0; i<rank; i++ ) offset += all_N[i]; >>>>> >>>>> offset *= type_size; // convert from structure counts -> bytes into file > for >>>>> true structure size >>>>> offset += sizeof( int ); // skip leading integer (global_N) in file. >>>>> >>>>> MPI_File_open( MPI_COMM_WORLD, fpath, MPI_MODE_CREATE | >>>>> MPI_MODE_WRONLY, MPI_INFO_NULL, &f ); >>>>> if( rank == 0 ) >>>>> { >>>>> MPI_File_write( f, &global_N, 1, MPI_INT, &stat ); >>>>> } >>>>> MPI_File_set_view( f, offset, mpi_atom_type_resized, >>>>> mpi_atom_type_resized, datarep, MPI_INFO_NULL ); >>>>> >>>>> MPI_File_write_all( f, &atoms[0], (int)atoms.size(), > mpi_atom_type_resized, >>>>> &stat ); >>>>> MPI_File_close( &f ); >>>>> >>>>> MPI_Type_free( &mpi_atom_type ); >>>>> MPI_Type_free( &mpi_atom_type_resized ); >>>>> >>>>> return; >>>>> } >>>>> >>>>> void read( char * fpath, std::vector<struct atom> &atoms ) >>>>> { >>>>> std::vector<int> s_blocklengths; >>>>> std::vector<MPI_Aint> s_displacements; >>>>> std::vector<MPI_Datatype> s_datatypes; >>>>> MPI_Datatype mpi_atom_type, mpi_atom_type_resized; >>>>> >>>>> struct atom a; >>>>> MPI_Aint addr_start, addr; >>>>> >>>>> MPI_File f; >>>>> MPI_Status stat; >>>>> >>>>> int global_N; >>>>> char *datarep = (char *)"native"; >>>>> >>>>> int type_size; >>>>> >>>>> /* >>>>> Set up the structure data type >>>>> */ >>>>> MPI_Get_address( &a, &addr_start ); >>>>> >>>>> s_blocklengths.push_back( 1 ); >>>>> s_datatypes.push_back( MPI_INT ); >>>>> MPI_Get_address( &a.global_id, &addr ); >>>>> s_displacements.push_back( addr - addr_start ); >>>>> >>>>> s_blocklengths.push_back( 3 ); >>>>> s_datatypes.push_back( MPI_DOUBLE ); >>>>> MPI_Get_address( &a.xyz[0], &addr ); >>>>> s_displacements.push_back( addr - addr_start ); >>>>> >>>>> MPI_Type_create_struct( (int)s_blocklengths.size(), &s_blocklengths[0], >>>>> &s_displacements[0], &s_datatypes[0], &mpi_atom_type ); >>>>> MPI_Type_commit( &mpi_atom_type ); >>>>> >>>>> /* >>>>> Take into account any compiler padding in creating an array of >>>>> structures. >>>>> */ >>>>> MPI_Type_create_resized( mpi_atom_type, 0, sizeof(struct atom), >>>>> &mpi_atom_type_resized ); >>>>> MPI_Type_commit( &mpi_atom_type_resized ); >>>>> >>>>> MPI_Type_size( mpi_atom_type_resized, &type_size ); >>>>> >>>>> MPI_File_open( MPI_COMM_SELF, fpath, MPI_MODE_RDONLY, >>>>> MPI_INFO_NULL, &f ); >>>>> >>>>> MPI_File_read( f, &global_N, 1, MPI_INT, &stat ); >>>>> >>>>> atoms.clear(); >>>>> atoms.resize( global_N ); >>>>> >>>>> errcode = MPI_File_set_view( f, sizeof(int), mpi_atom_type_resized, >>>>> mpi_atom_type_resized, datarep, MPI_INFO_NULL ); >>>>> errcode = MPI_File_read( f, &atoms[0], global_N, > mpi_atom_type_resized, >>>>> &stat ); >>>>> errcode = MPI_File_close( &f ); >>>>> >>>>> MPI_Type_free( &mpi_atom_type ); >>>>> MPI_Type_free( &mpi_atom_type_resized ); >>>>> >>>>> return; >>>>> } >>>>> >>>>> Calling MPI_Type_get_extent() and MPI_Type_get_true_extent() for both >>>>> mpi_atom_type and mpi_atom_type_resized returns (0,32) bytes in all > cases. >>>>> Calling MPI_Type_size() on both derived data types returns 28 bytes. >>>>> >>>>> If I call MPI_File_get_type_extent() on both derived data types after > opening >>> the >>>>> file, they both resolve to 32 bytes - so I think the problem is in the >>> difference >>>>> between the data representation in memory and on disk. If I explicitly use > 32 >>>>> bytes in the offset calculation in the write() routine above, it still doesn't >>> work. >>>>> >>>>> I'm finding it remarkably difficult to do something very simple using MPI's >>>>> derived data types and the parallel IO, and hence I'm guessing that I have >>>>> fundamentally misunderstood one or more aspects of this. If anyone can >>> help >>>>> clarify where I'm going wrong, that would be much appreciated! >>>>> >>>>> Cheers, >>>>> >>>>> John. >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] shell in emacs
by Dave Goodell 19 Nov '12

19 Nov '12

On Nov 17, 2012, at 2:06 AM CST, Hui Zhang wrote: > I M-x shell in emacs and compile, run the hell world. > My shell is closed by mpi after running which I do not expect. Just to rule out problems in your "Hello World" program, can you try the same thing with the "examples/cpi.c" program included in the MPICH distribution? There's probably some strange interaction between the hydra mpiexec and emacs' shell. If you were running on Linux, I would recommend running the mpiexec under "strace". On your Mac you can try "dtruss", but it's a poor imitation of both "truss" and "strace", so that may or may not help us here. I don't personally use emacs, but perhaps one of the other MPICH developers who uses emacs has a suggestion. -Dave

1 0

Re: [mpich-discuss] Use of MPI derived data types / MPI file IO
by jgrime＠uchicago.edu 19 Nov '12

19 Nov '12

Hi Wei-keng, It now works! Thanks for the help! One last question to the list: I've looked at the MPI 2 standards documents, and I'm still a little confused as to the precise semantics of MPI_Type_free(); as "MPI_Datatype" is an opaque type, I'm assuming that there is a certain amount of background allocation going on inside the MPI runtime when I call something like MPI_Type_create_struct() or similar routines. Am I right in assuming that I should call MPI_Type_free() on *all* derived data types I generate, even if they are not subsequently registered using MPI_Type_commit()? I would imagine that any other behaviour is likely to lead to memory leaks! Cheers, J. ---- Original message ---- >Date: Sun, 18 Nov 2012 18:58:37 -0600 >From: discuss-bounces(a)mpich.org (on behalf of Wei-keng Liao <wkliao(a)ece.northwestern.edu>) >Subject: Re: [mpich-discuss] Use of MPI derived data types / MPI file IO >To: discuss(a)mpich.org > >Hi, John, > >You certainly are on the right track to achieve that. Your code is almost >there, only the call to MPI_File_set_view is incorrect. In fact, you don't need it. > >Try remove the call to MPI_File_set_view and replace the MPI_File_write_all with: >MPI_File_write_at_all(f, offset, &atoms[0], (int)atoms.size(), mpi_atom_type_resized, &stat); > >On the reader side, you need to set the offset based on the new struct. Other than >that, it is the same as the writer case. (no need of MPI_File_set_view either). > >As for the portability issue, I would suggest you to use high-level I/O libraries, >such as PnetCDF. > >Wei-keng > >On Nov 18, 2012, at 12:38 PM, <jgrime(a)uchicago.edu> <jgrime(a)uchicago.edu> wrote: > >> Hi Wei-keng, >> >> That's a good point, thanks! >> >> However, I actually only want to save certain parts of the "atom" structure to file, >> and saving the whole array as a raw dump could waste a lot of disk space. >> >> For example, the "atom" structure I used in the example code in reality contains >> not only an integer and three contiguous doubles, but also at least another two >> double[3] entries which I may not want to save to disk. As the full data set can >> be hundreds of millions (or even billions) of "atom" structures, using a derived >> data type with only a restricted subset of the data in each "atom" structure will >> produce considerably smaller file sizes! >> >> There's also the problem of making the resultant file "portable" - raw memory >> dumps could make life difficult in trying to use output files on machines with >> different processor architectures. Once I get the derived data types working, I >> can then switch from the "native" representation to something else ("external32" >> etc), which should allow me to create portable output files, provided I'm careful >> with using MPIs file offset routines etc if the file is larger than plain old 32 bit >> offsets can handle. >> >> Cheers, >> >> J. >> >> ---- Original message ---- >>> Date: Sun, 18 Nov 2012 12:27:04 -0600 >>> From: discuss-bounces(a)mpich.org (on behalf of Wei-keng Liao >> <wkliao(a)ece.northwestern.edu>) >>> Subject: Re: [mpich-discuss] Use of MPI derived data types / MPI file IO >>> To: discuss(a)mpich.org >>> >>> Hi, John >>> >>> If your I/O is simply appending one process's data after another and the I/O >> buffers in memory >>> are all contiguous, then you can simply do the following without defining MPI >>> derived data types or setting the file view. >>> >>> MPI_File_write_at_all(f, offset, &atoms[0], (int)atoms.size() * sizeof(struct atom), >> MPI_BYTE, &stat); >>> >>> Using derived data types is usually when you have noncontiguous buffer in >> memory or >>> want to access non-contiguous data in files. >>> >>> >>> Wei-keng >>> >>> On Nov 18, 2012, at 11:52 AM, <jgrime(a)uchicago.edu> >> <jgrime(a)uchicago.edu> wrote: >>> >>>> Hi all, >>>> >>>> I'm having some problems with using derived data types and MPI parallel IO, >> and >>>> was wondering if anyone could help. I tried to search the archives in case >> this >>>> was covered earlier, but that just gave me "ht://Dig error" messages. >>>> >>>> Outline: I have written a C++ program where each MPI rank acts on data >> stored >>>> in a local array of structures. The arrays are typically of different lengths on >> each >>>> rank. I wish to write and read the contents of these arrays to disk using MPI's >>>> parallel IO routines. The file format is simply an initial integer which >> describes >>>> how many "structures" are in the file, followed by the data which represents >> the >>>> "structure information" from all ranks (ie the total data set). >>>> >>>> So far, I've tried two different approaches: the first consists of each rank >>>> serialising the contents of the local array of structures into a byte array, >> which is >>>> then saved to file "f" using MPI_File_set_view( f, MPI_COMM_WORLD, offset, >>>> MPI_CHAR, MPI_CHAR, "native", MPI_INFO_NULL ) to skip the initial integer >>>> "header" and then a call to MPI_File_write_all( f, local_bytearray, >> local_n_bytes, >>>> MPI_CHAR, &status ). Here, "offset" is simply the size of an integer (in bytes) >> + >>>> the summation of the number of bytes each preceeding rank wishes to write >> to >>>> the file (received via an earlier MPI_Allgather call). This seems to work, as >> when I >>>> read the file back in on a single MPI rank and deserialise the data into an >> array of >>>> structures I get the results I expect. >>>> >>>> The second approach is to use MPI's derived data types to create MPI >>>> representations of the structures, and then treat the arrays of structures as >> MPI >>>> data types. This allows me to avoid copying the local data into an >> intermediate >>>> buffer etc, and seems the more elegant approach. I cannot, however, seem >> to >>>> make this approach work. >>>> >>>> I'm pretty sure the problem lies in my use of the file views, but I'm not sure >>>> where I'm going wrong. The reading of the integer "header" always works >> fine, >>>> but the proceeding data is garbled. I'm using the "native" data representation >> for >>>> testing, but will likely change that to something more portable when I get >> this >>>> code working. >>>> >>>> I've included the important excerpts of the test code I'm trying to use below >>>> (with some printf()s and error handling etc removed to make it a little more >>>> concise). I have previously tested that std::vector allocates a contiguous flat >>>> array of the appropriate data type in memory, so passing a pointer/reference >> to >>>> the first element in such a data structure behaves the same way as simply >>>> passing a conventional array of the appropriate data type: >>>> >>>> struct atom >>>> { >>>> int global_id; >>>> double xyz[3]; >>>> }; >>>> >>>> void write( char * fpath, std::vector<struct atom> &atoms, int rank, int >> nranks ) >>>> { >>>> /* >>>> Memory layout information for the structure we wish to convert into >>>> an >>>> MPI derived data type. >>>> */ >>>> std::vector<int> s_blocklengths; >>>> std::vector<MPI_Aint> s_displacements; >>>> std::vector<MPI_Datatype> s_datatypes; >>>> MPI_Aint addr_start, addr; >>>> MPI_Datatype mpi_atom_type, mpi_atom_type_resized; >>>> int type_size; >>>> >>>> struct atom a; >>>> >>>> MPI_File f; >>>> MPI_Status stat; >>>> MPI_Offset offset; >>>> char *datarep = (char *)"native"; >>>> >>>> std::vector<int> all_N; >>>> int local_N, global_N; >>>> >>>> /* >>>> Set up the structure data type: single integer, and 3 double precision >>>> floats. >>>> We use the temporary "a" structure to determine the layout of memory >>>> inside >>>> atom structures. >>>> */ >>>> MPI_Get_address( &a, &addr_start ); >>>> >>>> s_blocklengths.push_back( 1 ); >>>> s_datatypes.push_back( MPI_INT ); >>>> MPI_Get_address( &a.global_id, &addr ); >>>> s_displacements.push_back( addr - addr_start ); >>>> >>>> s_blocklengths.push_back( 3 ); >>>> s_datatypes.push_back( MPI_DOUBLE ); >>>> MPI_Get_address( &a.xyz[0], &addr ); >>>> s_displacements.push_back( addr - addr_start ); >>>> >>>> MPI_Type_create_struct( (int)s_blocklengths.size(), &s_blocklengths[0], >>>> &s_displacements[0], &s_datatypes[0], &mpi_atom_type ); >>>> MPI_Type_commit( &mpi_atom_type ); >>>> >>>> /* >>>> Take into account any compiler padding in creating an array of >>>> structures. >>>> */ >>>> MPI_Type_create_resized( mpi_atom_type, 0, sizeof(struct atom), >>>> &mpi_atom_type_resized ); >>>> MPI_Type_commit( &mpi_atom_type_resized ); >>>> >>>> MPI_Type_size( mpi_atom_type_resized, &type_size ); >>>> >>>> local_N = (int)atoms.size(); >>>> all_N.resize( nranks ); >>>> >>>> MPI_Allgather( &local_N, 1, MPI_INT, &all_N[0], 1, MPI_INT, >>>> MPI_COMM_WORLD ); >>>> >>>> global_N = 0; >>>> for( size_t i=0; i<all_N.size(); i++ ) global_N += all_N[i]; >>>> >>>> offset = 0; >>>> for( int i=0; i<rank; i++ ) offset += all_N[i]; >>>> >>>> offset *= type_size; // convert from structure counts -> bytes into file for >>>> true structure size >>>> offset += sizeof( int ); // skip leading integer (global_N) in file. >>>> >>>> MPI_File_open( MPI_COMM_WORLD, fpath, MPI_MODE_CREATE | >>>> MPI_MODE_WRONLY, MPI_INFO_NULL, &f ); >>>> if( rank == 0 ) >>>> { >>>> MPI_File_write( f, &global_N, 1, MPI_INT, &stat ); >>>> } >>>> MPI_File_set_view( f, offset, mpi_atom_type_resized, >>>> mpi_atom_type_resized, datarep, MPI_INFO_NULL ); >>>> >>>> MPI_File_write_all( f, &atoms[0], (int)atoms.size(), mpi_atom_type_resized, >>>> &stat ); >>>> MPI_File_close( &f ); >>>> >>>> MPI_Type_free( &mpi_atom_type ); >>>> MPI_Type_free( &mpi_atom_type_resized ); >>>> >>>> return; >>>> } >>>> >>>> void read( char * fpath, std::vector<struct atom> &atoms ) >>>> { >>>> std::vector<int> s_blocklengths; >>>> std::vector<MPI_Aint> s_displacements; >>>> std::vector<MPI_Datatype> s_datatypes; >>>> MPI_Datatype mpi_atom_type, mpi_atom_type_resized; >>>> >>>> struct atom a; >>>> MPI_Aint addr_start, addr; >>>> >>>> MPI_File f; >>>> MPI_Status stat; >>>> >>>> int global_N; >>>> char *datarep = (char *)"native"; >>>> >>>> int type_size; >>>> >>>> /* >>>> Set up the structure data type >>>> */ >>>> MPI_Get_address( &a, &addr_start ); >>>> >>>> s_blocklengths.push_back( 1 ); >>>> s_datatypes.push_back( MPI_INT ); >>>> MPI_Get_address( &a.global_id, &addr ); >>>> s_displacements.push_back( addr - addr_start ); >>>> >>>> s_blocklengths.push_back( 3 ); >>>> s_datatypes.push_back( MPI_DOUBLE ); >>>> MPI_Get_address( &a.xyz[0], &addr ); >>>> s_displacements.push_back( addr - addr_start ); >>>> >>>> MPI_Type_create_struct( (int)s_blocklengths.size(), &s_blocklengths[0], >>>> &s_displacements[0], &s_datatypes[0], &mpi_atom_type ); >>>> MPI_Type_commit( &mpi_atom_type ); >>>> >>>> /* >>>> Take into account any compiler padding in creating an array of >>>> structures. >>>> */ >>>> MPI_Type_create_resized( mpi_atom_type, 0, sizeof(struct atom), >>>> &mpi_atom_type_resized ); >>>> MPI_Type_commit( &mpi_atom_type_resized ); >>>> >>>> MPI_Type_size( mpi_atom_type_resized, &type_size ); >>>> >>>> MPI_File_open( MPI_COMM_SELF, fpath, MPI_MODE_RDONLY, >>>> MPI_INFO_NULL, &f ); >>>> >>>> MPI_File_read( f, &global_N, 1, MPI_INT, &stat ); >>>> >>>> atoms.clear(); >>>> atoms.resize( global_N ); >>>> >>>> errcode = MPI_File_set_view( f, sizeof(int), mpi_atom_type_resized, >>>> mpi_atom_type_resized, datarep, MPI_INFO_NULL ); >>>> errcode = MPI_File_read( f, &atoms[0], global_N, mpi_atom_type_resized, >>>> &stat ); >>>> errcode = MPI_File_close( &f ); >>>> >>>> MPI_Type_free( &mpi_atom_type ); >>>> MPI_Type_free( &mpi_atom_type_resized ); >>>> >>>> return; >>>> } >>>> >>>> Calling MPI_Type_get_extent() and MPI_Type_get_true_extent() for both >>>> mpi_atom_type and mpi_atom_type_resized returns (0,32) bytes in all cases. >>>> Calling MPI_Type_size() on both derived data types returns 28 bytes. >>>> >>>> If I call MPI_File_get_type_extent() on both derived data types after opening >> the >>>> file, they both resolve to 32 bytes - so I think the problem is in the >> difference >>>> between the data representation in memory and on disk. If I explicitly use 32 >>>> bytes in the offset calculation in the write() routine above, it still doesn't >> work. >>>> >>>> I'm finding it remarkably difficult to do something very simple using MPI's >>>> derived data types and the parallel IO, and hence I'm guessing that I have >>>> fundamentally misunderstood one or more aspects of this. If anyone can >> help >>>> clarify where I'm going wrong, that would be much appreciated! >>>> >>>> Cheers, >>>> >>>> John. >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > >_______________________________________________ >discuss mailing list discuss(a)mpich.org >To manage subscription options or unsubscribe: >https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Use of MPI derived data types / MPI file IO
by Wei-keng Liao 18 Nov '12

18 Nov '12

Hi, John, You certainly are on the right track to achieve that. Your code is almost there, only the call to MPI_File_set_view is incorrect. In fact, you don't need it. Try remove the call to MPI_File_set_view and replace the MPI_File_write_all with: MPI_File_write_at_all(f, offset, &atoms[0], (int)atoms.size(), mpi_atom_type_resized, &stat); On the reader side, you need to set the offset based on the new struct. Other than that, it is the same as the writer case. (no need of MPI_File_set_view either). As for the portability issue, I would suggest you to use high-level I/O libraries, such as PnetCDF. Wei-keng On Nov 18, 2012, at 12:38 PM, <jgrime(a)uchicago.edu> <jgrime(a)uchicago.edu> wrote: > Hi Wei-keng, > > That's a good point, thanks! > > However, I actually only want to save certain parts of the "atom" structure to file, > and saving the whole array as a raw dump could waste a lot of disk space. > > For example, the "atom" structure I used in the example code in reality contains > not only an integer and three contiguous doubles, but also at least another two > double[3] entries which I may not want to save to disk. As the full data set can > be hundreds of millions (or even billions) of "atom" structures, using a derived > data type with only a restricted subset of the data in each "atom" structure will > produce considerably smaller file sizes! > > There's also the problem of making the resultant file "portable" - raw memory > dumps could make life difficult in trying to use output files on machines with > different processor architectures. Once I get the derived data types working, I > can then switch from the "native" representation to something else ("external32" > etc), which should allow me to create portable output files, provided I'm careful > with using MPIs file offset routines etc if the file is larger than plain old 32 bit > offsets can handle. > > Cheers, > > J. > > ---- Original message ---- >> Date: Sun, 18 Nov 2012 12:27:04 -0600 >> From: discuss-bounces(a)mpich.org (on behalf of Wei-keng Liao > <wkliao(a)ece.northwestern.edu>) >> Subject: Re: [mpich-discuss] Use of MPI derived data types / MPI file IO >> To: discuss(a)mpich.org >> >> Hi, John >> >> If your I/O is simply appending one process's data after another and the I/O > buffers in memory >> are all contiguous, then you can simply do the following without defining MPI >> derived data types or setting the file view. >> >> MPI_File_write_at_all(f, offset, &atoms[0], (int)atoms.size() * sizeof(struct atom), > MPI_BYTE, &stat); >> >> Using derived data types is usually when you have noncontiguous buffer in > memory or >> want to access non-contiguous data in files. >> >> >> Wei-keng >> >> On Nov 18, 2012, at 11:52 AM, <jgrime(a)uchicago.edu> > <jgrime(a)uchicago.edu> wrote: >> >>> Hi all, >>> >>> I'm having some problems with using derived data types and MPI parallel IO, > and >>> was wondering if anyone could help. I tried to search the archives in case > this >>> was covered earlier, but that just gave me "ht://Dig error" messages. >>> >>> Outline: I have written a C++ program where each MPI rank acts on data > stored >>> in a local array of structures. The arrays are typically of different lengths on > each >>> rank. I wish to write and read the contents of these arrays to disk using MPI's >>> parallel IO routines. The file format is simply an initial integer which > describes >>> how many "structures" are in the file, followed by the data which represents > the >>> "structure information" from all ranks (ie the total data set). >>> >>> So far, I've tried two different approaches: the first consists of each rank >>> serialising the contents of the local array of structures into a byte array, > which is >>> then saved to file "f" using MPI_File_set_view( f, MPI_COMM_WORLD, offset, >>> MPI_CHAR, MPI_CHAR, "native", MPI_INFO_NULL ) to skip the initial integer >>> "header" and then a call to MPI_File_write_all( f, local_bytearray, > local_n_bytes, >>> MPI_CHAR, &status ). Here, "offset" is simply the size of an integer (in bytes) > + >>> the summation of the number of bytes each preceeding rank wishes to write > to >>> the file (received via an earlier MPI_Allgather call). This seems to work, as > when I >>> read the file back in on a single MPI rank and deserialise the data into an > array of >>> structures I get the results I expect. >>> >>> The second approach is to use MPI's derived data types to create MPI >>> representations of the structures, and then treat the arrays of structures as > MPI >>> data types. This allows me to avoid copying the local data into an > intermediate >>> buffer etc, and seems the more elegant approach. I cannot, however, seem > to >>> make this approach work. >>> >>> I'm pretty sure the problem lies in my use of the file views, but I'm not sure >>> where I'm going wrong. The reading of the integer "header" always works > fine, >>> but the proceeding data is garbled. I'm using the "native" data representation > for >>> testing, but will likely change that to something more portable when I get > this >>> code working. >>> >>> I've included the important excerpts of the test code I'm trying to use below >>> (with some printf()s and error handling etc removed to make it a little more >>> concise). I have previously tested that std::vector allocates a contiguous flat >>> array of the appropriate data type in memory, so passing a pointer/reference > to >>> the first element in such a data structure behaves the same way as simply >>> passing a conventional array of the appropriate data type: >>> >>> struct atom >>> { >>> int global_id; >>> double xyz[3]; >>> }; >>> >>> void write( char * fpath, std::vector<struct atom> &atoms, int rank, int > nranks ) >>> { >>> /* >>> Memory layout information for the structure we wish to convert into >>> an >>> MPI derived data type. >>> */ >>> std::vector<int> s_blocklengths; >>> std::vector<MPI_Aint> s_displacements; >>> std::vector<MPI_Datatype> s_datatypes; >>> MPI_Aint addr_start, addr; >>> MPI_Datatype mpi_atom_type, mpi_atom_type_resized; >>> int type_size; >>> >>> struct atom a; >>> >>> MPI_File f; >>> MPI_Status stat; >>> MPI_Offset offset; >>> char *datarep = (char *)"native"; >>> >>> std::vector<int> all_N; >>> int local_N, global_N; >>> >>> /* >>> Set up the structure data type: single integer, and 3 double precision >>> floats. >>> We use the temporary "a" structure to determine the layout of memory >>> inside >>> atom structures. >>> */ >>> MPI_Get_address( &a, &addr_start ); >>> >>> s_blocklengths.push_back( 1 ); >>> s_datatypes.push_back( MPI_INT ); >>> MPI_Get_address( &a.global_id, &addr ); >>> s_displacements.push_back( addr - addr_start ); >>> >>> s_blocklengths.push_back( 3 ); >>> s_datatypes.push_back( MPI_DOUBLE ); >>> MPI_Get_address( &a.xyz[0], &addr ); >>> s_displacements.push_back( addr - addr_start ); >>> >>> MPI_Type_create_struct( (int)s_blocklengths.size(), &s_blocklengths[0], >>> &s_displacements[0], &s_datatypes[0], &mpi_atom_type ); >>> MPI_Type_commit( &mpi_atom_type ); >>> >>> /* >>> Take into account any compiler padding in creating an array of >>> structures. >>> */ >>> MPI_Type_create_resized( mpi_atom_type, 0, sizeof(struct atom), >>> &mpi_atom_type_resized ); >>> MPI_Type_commit( &mpi_atom_type_resized ); >>> >>> MPI_Type_size( mpi_atom_type_resized, &type_size ); >>> >>> local_N = (int)atoms.size(); >>> all_N.resize( nranks ); >>> >>> MPI_Allgather( &local_N, 1, MPI_INT, &all_N[0], 1, MPI_INT, >>> MPI_COMM_WORLD ); >>> >>> global_N = 0; >>> for( size_t i=0; i<all_N.size(); i++ ) global_N += all_N[i]; >>> >>> offset = 0; >>> for( int i=0; i<rank; i++ ) offset += all_N[i]; >>> >>> offset *= type_size; // convert from structure counts -> bytes into file for >>> true structure size >>> offset += sizeof( int ); // skip leading integer (global_N) in file. >>> >>> MPI_File_open( MPI_COMM_WORLD, fpath, MPI_MODE_CREATE | >>> MPI_MODE_WRONLY, MPI_INFO_NULL, &f ); >>> if( rank == 0 ) >>> { >>> MPI_File_write( f, &global_N, 1, MPI_INT, &stat ); >>> } >>> MPI_File_set_view( f, offset, mpi_atom_type_resized, >>> mpi_atom_type_resized, datarep, MPI_INFO_NULL ); >>> >>> MPI_File_write_all( f, &atoms[0], (int)atoms.size(), mpi_atom_type_resized, >>> &stat ); >>> MPI_File_close( &f ); >>> >>> MPI_Type_free( &mpi_atom_type ); >>> MPI_Type_free( &mpi_atom_type_resized ); >>> >>> return; >>> } >>> >>> void read( char * fpath, std::vector<struct atom> &atoms ) >>> { >>> std::vector<int> s_blocklengths; >>> std::vector<MPI_Aint> s_displacements; >>> std::vector<MPI_Datatype> s_datatypes; >>> MPI_Datatype mpi_atom_type, mpi_atom_type_resized; >>> >>> struct atom a; >>> MPI_Aint addr_start, addr; >>> >>> MPI_File f; >>> MPI_Status stat; >>> >>> int global_N; >>> char *datarep = (char *)"native"; >>> >>> int type_size; >>> >>> /* >>> Set up the structure data type >>> */ >>> MPI_Get_address( &a, &addr_start ); >>> >>> s_blocklengths.push_back( 1 ); >>> s_datatypes.push_back( MPI_INT ); >>> MPI_Get_address( &a.global_id, &addr ); >>> s_displacements.push_back( addr - addr_start ); >>> >>> s_blocklengths.push_back( 3 ); >>> s_datatypes.push_back( MPI_DOUBLE ); >>> MPI_Get_address( &a.xyz[0], &addr ); >>> s_displacements.push_back( addr - addr_start ); >>> >>> MPI_Type_create_struct( (int)s_blocklengths.size(), &s_blocklengths[0], >>> &s_displacements[0], &s_datatypes[0], &mpi_atom_type ); >>> MPI_Type_commit( &mpi_atom_type ); >>> >>> /* >>> Take into account any compiler padding in creating an array of >>> structures. >>> */ >>> MPI_Type_create_resized( mpi_atom_type, 0, sizeof(struct atom), >>> &mpi_atom_type_resized ); >>> MPI_Type_commit( &mpi_atom_type_resized ); >>> >>> MPI_Type_size( mpi_atom_type_resized, &type_size ); >>> >>> MPI_File_open( MPI_COMM_SELF, fpath, MPI_MODE_RDONLY, >>> MPI_INFO_NULL, &f ); >>> >>> MPI_File_read( f, &global_N, 1, MPI_INT, &stat ); >>> >>> atoms.clear(); >>> atoms.resize( global_N ); >>> >>> errcode = MPI_File_set_view( f, sizeof(int), mpi_atom_type_resized, >>> mpi_atom_type_resized, datarep, MPI_INFO_NULL ); >>> errcode = MPI_File_read( f, &atoms[0], global_N, mpi_atom_type_resized, >>> &stat ); >>> errcode = MPI_File_close( &f ); >>> >>> MPI_Type_free( &mpi_atom_type ); >>> MPI_Type_free( &mpi_atom_type_resized ); >>> >>> return; >>> } >>> >>> Calling MPI_Type_get_extent() and MPI_Type_get_true_extent() for both >>> mpi_atom_type and mpi_atom_type_resized returns (0,32) bytes in all cases. >>> Calling MPI_Type_size() on both derived data types returns 28 bytes. >>> >>> If I call MPI_File_get_type_extent() on both derived data types after opening > the >>> file, they both resolve to 32 bytes - so I think the problem is in the > difference >>> between the data representation in memory and on disk. If I explicitly use 32 >>> bytes in the offset calculation in the write() routine above, it still doesn't > work. >>> >>> I'm finding it remarkably difficult to do something very simple using MPI's >>> derived data types and the parallel IO, and hence I'm guessing that I have >>> fundamentally misunderstood one or more aspects of this. If anyone can > help >>> clarify where I'm going wrong, that would be much appreciated! >>> >>> Cheers, >>> >>> John. >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0