December 2023 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Hang during MPI_Finalize using ch4:ofi:shm in mpich-4.1.2
by Edric Ellis 14 Dec '23

14 Dec '23

I don't think there's any uncompleted communication. The example I show below is one of the very simple examples shipping with MPICH itself (sorry if that wasn't completely clear). Cheers, Edric. ________________________________ From: Joachim Jenke via discuss <discuss(a)mpich.org> Sent: 13 December 2023 17:16 To: discuss(a)mpich.org <discuss(a)mpich.org>; Zhou, Hui <zhouh(a)anl.gov> Cc: Joachim Jenke <jenke(a)itc.rwth-aachen.de> Subject: Re: [mpich-discuss] Hang during MPI_Finalize using ch4:ofi:shm in mpich-4.1.2 If your code hangs in MPI_Finalize for certain communication implementation, this sounds like an uncompleted communication. Are you sure that you have no MPI communication ongoing when calling MPI_Finalize? - Joachim Am 13.12.23 um 17:45 schrieb Edric Ellis via discuss: > Ok, that's good to know, I'll stick with simply "ofi:tcp" for now. > > Thanks, > Edric. > ------------------------------------------------------------------------ > *From:* Zhou, Hui <zhouh(a)anl.gov> > *Sent:* 13 December 2023 15:39 > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* Edric Ellis <eellis(a)mathworks.com> > *Subject:* Re: Hang during MPI_Finalize using ch4:ofi:shm in mpich-4.1.2 > Hi Edric, > > I am not sure which part is hanging, but you don't need to enable > |ofi:shm| (libfabric shm provider). The ch4 device comes with its own > shared memory functionality. > > -- > Hui > ------------------------------------------------------------------------ > *From:* Edric Ellis via discuss <discuss(a)mpich.org> > *Sent:* Wednesday, December 13, 2023 7:05 AM > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* Edric Ellis <eellis(a)mathworks.com> > *Subject:* [mpich-discuss] Hang during MPI_Finalize using ch4:ofi:shm in > mpich-4.1.2 > I'm working on getting a build of mpich-4.1.2 ready to replace our old > build of mpich-3.3.2. With older MPICH releases, we used the "nemesis" > channel via ch3 to provide support for shared-memory configurations as > well as TCP/IP. In ch4, I thought the nearest equivalent would be: > > --with-device=ch4:ofi:tcp,shm > > The "tcp" portion of this seems to work just fine, but "shm" hangs > during (I think) MPI_Finalize, requiring a CTRL-C to kill it. For > example, in the build area, > > $ ./src/pm/hydra/mpiexec.hydra -n 2 ./examples/cpi > Process 0 of 2 is on uk-eellis-l > Process 1 of 2 is on uk-eellis-l > pi is approximately 3.1415926544231318, Error is 0.0000000008333387 > wall clock time = 0.000019 > ^C[mpiexec@uk-eellis-l] Sending Ctrl-C to processes as requested > [mpiexec@uk-eellis-l] Press Ctrl-C again to force abort > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 829015 RUNNING AT uk-eellis-l > = EXIT CODE: 2 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Interrupt (signal 2) > This typically refers to a problem with your application. > Please see the FAQ page for debugging suggestions > > Things work fine if I force FI_PROVIDER=tcp. Am I missing something? > > Here's the configure line I'm using: > > $ ./configure --prefix <prefix> --with-device=ch4:ofi:tcp,shm > --enable-shared --with-libfabric=embedded --enable-fortran --enable-efa=no > > This is running on a Debian 11 system, gcc 10.3.0. > > Cheers, > Edric. > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss<https://lists.mpich.org/mailman/listinfo/discuss> -- Dr. rer. nat. Joachim Jenke IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 jenke(a)itc.rwth-aachen.de www.itc.rwth-aachen.de<http://www.itc.rwth-aachen.de>

1 0

Re: [mpich-discuss] Hang during MPI_Finalize using ch4:ofi:shm in mpich-4.1.2
by Joachim Jenke 13 Dec '23

13 Dec '23

If your code hangs in MPI_Finalize for certain communication implementation, this sounds like an uncompleted communication. Are you sure that you have no MPI communication ongoing when calling MPI_Finalize? - Joachim Am 13.12.23 um 17:45 schrieb Edric Ellis via discuss: > Ok, that's good to know, I'll stick with simply "ofi:tcp" for now. > > Thanks, > Edric. > ------------------------------------------------------------------------ > *From:* Zhou, Hui <zhouh(a)anl.gov> > *Sent:* 13 December 2023 15:39 > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* Edric Ellis <eellis(a)mathworks.com> > *Subject:* Re: Hang during MPI_Finalize using ch4:ofi:shm in mpich-4.1.2 > Hi Edric, > > I am not sure which part is hanging, but you don't need to enable > |ofi:shm| (libfabric shm provider). The ch4 device comes with its own > shared memory functionality. > > -- > Hui > ------------------------------------------------------------------------ > *From:* Edric Ellis via discuss <discuss(a)mpich.org> > *Sent:* Wednesday, December 13, 2023 7:05 AM > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* Edric Ellis <eellis(a)mathworks.com> > *Subject:* [mpich-discuss] Hang during MPI_Finalize using ch4:ofi:shm in > mpich-4.1.2 > I'm working on getting a build of mpich-4.1.2 ready to replace our old > build of mpich-3.3.2. With older MPICH releases, we used the "nemesis" > channel via ch3 to provide support for shared-memory configurations as > well as TCP/IP. In ch4, I thought the nearest equivalent would be: > > --with-device=ch4:ofi:tcp,shm > > The "tcp" portion of this seems to work just fine, but "shm" hangs > during (I think) MPI_Finalize, requiring a CTRL-C to kill it. For > example, in the build area, > > $ ./src/pm/hydra/mpiexec.hydra -n 2 ./examples/cpi > Process 0 of 2 is on uk-eellis-l > Process 1 of 2 is on uk-eellis-l > pi is approximately 3.1415926544231318, Error is 0.0000000008333387 > wall clock time = 0.000019 > ^C[mpiexec@uk-eellis-l] Sending Ctrl-C to processes as requested > [mpiexec@uk-eellis-l] Press Ctrl-C again to force abort > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 829015 RUNNING AT uk-eellis-l > = EXIT CODE: 2 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Interrupt (signal 2) > This typically refers to a problem with your application. > Please see the FAQ page for debugging suggestions > > Things work fine if I force FI_PROVIDER=tcp. Am I missing something? > > Here's the configure line I'm using: > > $ ./configure --prefix <prefix> --with-device=ch4:ofi:tcp,shm > --enable-shared --with-libfabric=embedded --enable-fortran --enable-efa=no > > This is running on a Debian 11 system, gcc 10.3.0. > > Cheers, > Edric. > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Dr. rer. nat. Joachim Jenke IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 jenke(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpich-discuss] Hang during MPI_Finalize using ch4:ofi:shm in mpich-4.1.2
by Edric Ellis 13 Dec '23

13 Dec '23

Ok, that's good to know, I'll stick with simply "ofi:tcp" for now. Thanks, Edric. ________________________________ From: Zhou, Hui <zhouh(a)anl.gov> Sent: 13 December 2023 15:39 To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Edric Ellis <eellis(a)mathworks.com> Subject: Re: Hang during MPI_Finalize using ch4:ofi:shm in mpich-4.1.2 Hi Edric, I am not sure which part is hanging, but you don't need to enable ofi:shm (libfabric shm provider). The ch4 device comes with its own shared memory functionality. -- Hui ________________________________ From: Edric Ellis via discuss <discuss(a)mpich.org> Sent: Wednesday, December 13, 2023 7:05 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Edric Ellis <eellis(a)mathworks.com> Subject: [mpich-discuss] Hang during MPI_Finalize using ch4:ofi:shm in mpich-4.1.2 I'm working on getting a build of mpich-4.1.2 ready to replace our old build of mpich-3.3.2. With older MPICH releases, we used the "nemesis" channel via ch3 to provide support for shared-memory configurations as well as TCP/IP. In ch4, I thought the nearest equivalent would be: --with-device=ch4:ofi:tcp,shm The "tcp" portion of this seems to work just fine, but "shm" hangs during (I think) MPI_Finalize, requiring a CTRL-C to kill it. For example, in the build area, $ ./src/pm/hydra/mpiexec.hydra -n 2 ./examples/cpi Process 0 of 2 is on uk-eellis-l Process 1 of 2 is on uk-eellis-l pi is approximately 3.1415926544231318, Error is 0.0000000008333387 wall clock time = 0.000019 ^C[mpiexec@uk-eellis-l] Sending Ctrl-C to processes as requested [mpiexec@uk-eellis-l] Press Ctrl-C again to force abort =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 829015 RUNNING AT uk-eellis-l = EXIT CODE: 2 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Interrupt (signal 2) This typically refers to a problem with your application. Please see the FAQ page for debugging suggestions Things work fine if I force FI_PROVIDER=tcp. Am I missing something? Here's the configure line I'm using: $ ./configure --prefix <prefix> --with-device=ch4:ofi:tcp,shm --enable-shared --with-libfabric=embedded --enable-fortran --enable-efa=no This is running on a Debian 11 system, gcc 10.3.0. Cheers, Edric.

1 0

Re: [mpich-discuss] Hang during MPI_Finalize using ch4:ofi:shm in mpich-4.1.2
by Zhou, Hui 13 Dec '23

13 Dec '23

Hi Edric, I am not sure which part is hanging, but you don't need to enable ofi:shm (libfabric shm provider). The ch4 device comes with its own shared memory functionality. -- Hui ________________________________ From: Edric Ellis via discuss <discuss(a)mpich.org> Sent: Wednesday, December 13, 2023 7:05 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Edric Ellis <eellis(a)mathworks.com> Subject: [mpich-discuss] Hang during MPI_Finalize using ch4:ofi:shm in mpich-4.1.2 I'm working on getting a build of mpich-4.1.2 ready to replace our old build of mpich-3.3.2. With older MPICH releases, we used the "nemesis" channel via ch3 to provide support for shared-memory configurations as well as TCP/IP. In ch4, I thought the nearest equivalent would be: --with-device=ch4:ofi:tcp,shm The "tcp" portion of this seems to work just fine, but "shm" hangs during (I think) MPI_Finalize, requiring a CTRL-C to kill it. For example, in the build area, $ ./src/pm/hydra/mpiexec.hydra -n 2 ./examples/cpi Process 0 of 2 is on uk-eellis-l Process 1 of 2 is on uk-eellis-l pi is approximately 3.1415926544231318, Error is 0.0000000008333387 wall clock time = 0.000019 ^C[mpiexec@uk-eellis-l] Sending Ctrl-C to processes as requested [mpiexec@uk-eellis-l] Press Ctrl-C again to force abort =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 829015 RUNNING AT uk-eellis-l = EXIT CODE: 2 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Interrupt (signal 2) This typically refers to a problem with your application. Please see the FAQ page for debugging suggestions Things work fine if I force FI_PROVIDER=tcp. Am I missing something? Here's the configure line I'm using: $ ./configure --prefix <prefix> --with-device=ch4:ofi:tcp,shm --enable-shared --with-libfabric=embedded --enable-fortran --enable-efa=no This is running on a Debian 11 system, gcc 10.3.0. Cheers, Edric.

1 0

Hang during MPI_Finalize using ch4:ofi:shm in mpich-4.1.2
by Edric Ellis 13 Dec '23

13 Dec '23

I'm working on getting a build of mpich-4.1.2 ready to replace our old build of mpich-3.3.2. With older MPICH releases, we used the "nemesis" channel via ch3 to provide support for shared-memory configurations as well as TCP/IP. In ch4, I thought the nearest equivalent would be: --with-device=ch4:ofi:tcp,shm The "tcp" portion of this seems to work just fine, but "shm" hangs during (I think) MPI_Finalize, requiring a CTRL-C to kill it. For example, in the build area, $ ./src/pm/hydra/mpiexec.hydra -n 2 ./examples/cpi Process 0 of 2 is on uk-eellis-l Process 1 of 2 is on uk-eellis-l pi is approximately 3.1415926544231318, Error is 0.0000000008333387 wall clock time = 0.000019 ^C[mpiexec@uk-eellis-l] Sending Ctrl-C to processes as requested [mpiexec@uk-eellis-l] Press Ctrl-C again to force abort =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 829015 RUNNING AT uk-eellis-l = EXIT CODE: 2 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Interrupt (signal 2) This typically refers to a problem with your application. Please see the FAQ page for debugging suggestions Things work fine if I force FI_PROVIDER=tcp. Am I missing something? Here's the configure line I'm using: $ ./configure --prefix <prefix> --with-device=ch4:ofi:tcp,shm --enable-shared --with-libfabric=embedded --enable-fortran --enable-efa=no This is running on a Debian 11 system, gcc 10.3.0. Cheers, Edric.

1 0

Re: [mpich-discuss] MPI user-defined datatype size
by Zhou, Hui 04 Dec '23

04 Dec '23

Hi Kurt, MPI_Type_size returns the size of a datatype excluding the gaps (hole). You should use MPI_Type_get_extent to query the extent of the datatype, which should come out as 568. Alternatively, if you never going to break the struct apart during communication, you may try simply treating the struct as an opaque blob, i.e. use MPI_Type_contiguous(sizeof(struct MsgTupleSeq), MPI_BYTE, &newtype). It is likely to perform better by avoiding treating it as a non-contiguous datatype. -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Sent: Sunday, December 3, 2023 2:55 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] MPI user-defined datatype size Hi, I’m running MPICH 4.1.2. Should a MPI user-defined datatype created by MPI_Type_create_struct have the same size as the C++ object on which it is based? Comparing sizeof(cpp_obj) and the output From MPI_Type_size(mpi_data_type_, &size1), the C++ object is four bytes larger than the MPI datatype. I don’t know if this is because I have to do something special to take padding into account when I create the MPI user datatype. Here is my C++ message class: struct MsgTupleSeq { static const int N_INDICES_MAX_ = 4; // culprit? int n_vars_; int n_ndxs_; short start_[N_INDICES_MAX_]; short end_[N_INDICES_MAX_]; short current_[N_INDICES_MAX_]; long long n_tuples_; int unique_id_; int subordinate_id_; int opcode_; char error_msg_[512]; } That is 564 bytes for the non-static members, but such an object is 568 bytes long with the 4 bytes of padding that the compiler inserts. I define the MPI user datatype with the code below, using the MsgTupleSeq object to compute the offsets. MPI_Type_size returns 564 bytes for the user datatype, so it seems like it isn’t seeing the padding. MsgTupleSeq obj; MPI_Datatype mpi_data_type; int struct_len = 10, i = 0; int block_len[struct_len]; MPI_Datatype types[struct_len]; MPI_Aint displacements[struct_len]; // the integer "n_vars_" member block_len[i] = 1; types[i] = MPI_INT; displacements[i] = (size_t) &obj.n_vars_ - (size_t) &obj; // the integer "n_ndxs_" member ++i; block_len[i] = 1; types[i] = MPI_INT; displacements[i] = (size_t) &obj.n_ndxs_ - (size_t) &obj; // the short integer "start_" array member ++i; block_len[i] = N_INDICES_MAX_; types[i] = MPI_SHORT; displacements[i] = (size_t) &obj.start_ - (size_t) &obj; // the short integer "end_" array member ++i; block_len[i] = N_INDICES_MAX_; types[i] = MPI_SHORT; displacements[i] = (size_t) &obj.end_ - (size_t) &obj; // the short integer "current_" array member ++i; block_len[i] = N_INDICES_MAX_; types[i] = MPI_SHORT; displacements[i] = (size_t) &obj.current_ - (size_t) &obj; // the long long integer "n_tuples_" member ++i; block_len[i] = 1; types[i] = MPI_LONG_LONG; displacements[i] = (size_t) &obj.n_tuples_ - (size_t) &obj; // the integer "unique_id_" member ++i; block_len[i] = 1; types[i] = MPI_INT; displacements[i] = (size_t) &obj.unique_id_ - (size_t) &obj; // the integer "subordinate_id_" member ++i; block_len[i] = 1; types[i] = MPI_INT; displacements[i] = (size_t) &obj.subordinate_id_ - (size_t) &obj; // the integer "opcode_" member ++i; block_len[i] = 1; types[i] = MPI_INT; displacements[i] = (size_t) &obj.opcode_ - (size_t) &obj; // the char array "error_msg_" member ++i; block_len[i] = 512; types[i] = MPI_CHAR; displacements[i] = (size_t) &obj.error_msg_[0] - (size_t) &obj; MPI_Type_create_struct(struct_len, block_len, displacements, types, &mpi_data_type); MPI_Type_commit(&mpi_data_type);

1 0

Re: [mpich-discuss] MPI user-defined datatype size
by George Bosilca 04 Dec '23

04 Dec '23

MPI does not know about the padding because it does not really create a struct in the same sense as the compiler. As a result, you need to resize the datatype to account for the padding. Check the examples 5.16 and 5.17 in the MPI standard to see how to use it. For your particular example, replacing the last 2 lines of code with: MPI_Type_create_struct(struct_len, block_len, displacements, types, &tmp_ddt); MPI_Type_create_resized(tmp_ddt, 0, sizeof(), &mpi_data_type); MPI_Type_connit(mpi_data_type); should make the MPI struct match the compiler struct. While you can manipulate the pointers directly and compute the displacements by hand, you should not use a size_t for that. MPI_Aint is a signed type, ssize_t or some form of ptrdiff_t. Instead you should use the MPI suggested way via MPI_Get_address (to obtain the address of an object as an MPI_Aint), and then MPI_Aint_diff to compute relative displacements. George. On Sun, Dec 3, 2023 at 3:55 PM Mccall, Kurt E. (MSFC-EV41) via discuss < discuss(a)mpich.org> wrote: > Hi, > > > > I’m running MPICH 4.1.2. > > > > Should a MPI user-defined datatype created by MPI_Type_create_struct have > the same size as the C++ object on which it is based? Comparing > sizeof(cpp_obj) and the output > > From MPI_Type_size(mpi_data_type_, &size1), the C++ object is four bytes > larger than the MPI datatype. I don’t know if this is because I have to > do something special to take > > padding into account when I create the MPI user datatype. > > > > Here is my C++ message class: > > > > struct MsgTupleSeq > > { > > static const int N_INDICES_MAX_ = 4; // culprit? > > > > int n_vars_; > > int n_ndxs_; > > short start_[N_INDICES_MAX_]; > > short end_[N_INDICES_MAX_]; > > short current_[N_INDICES_MAX_]; > > long long n_tuples_; > > int unique_id_; > > int subordinate_id_; > > int opcode_; > > char error_msg_[512]; > > } > > > > That is 564 bytes for the non-static members, but such an object is 568 > bytes long with the 4 bytes of padding that the compiler inserts. > > > > I define the MPI user datatype with the code below, using the MsgTupleSeq > object to compute the offsets. MPI_Type_size returns 564 bytes for the > user datatype, > > so it seems like it isn’t seeing the padding. > > > > MsgTupleSeq obj; > > MPI_Datatype mpi_data_type; > > int struct_len = 10, i = 0; > > > > int block_len[struct_len]; > > MPI_Datatype types[struct_len]; > > MPI_Aint displacements[struct_len]; > > > > // the integer "n_vars_" member > > block_len[i] = 1; > > types[i] = MPI_INT; > > displacements[i] = (size_t) &obj.n_vars_ - (size_t) &obj; > > > > // the integer "n_ndxs_" member > > ++i; > > block_len[i] = 1; > > types[i] = MPI_INT; > > displacements[i] = (size_t) &obj.n_ndxs_ - (size_t) &obj; > > > > // the short integer "start_" array member > > ++i; > > block_len[i] = N_INDICES_MAX_; > > types[i] = MPI_SHORT; > > displacements[i] = (size_t) &obj.start_ - (size_t) &obj; > > > > // the short integer "end_" array member > > ++i; > > block_len[i] = N_INDICES_MAX_; > > types[i] = MPI_SHORT; > > displacements[i] = (size_t) &obj.end_ - (size_t) &obj; > > > > // the short integer "current_" array member > > ++i; > > block_len[i] = N_INDICES_MAX_; > > types[i] = MPI_SHORT; > > displacements[i] = (size_t) &obj.current_ - (size_t) &obj; > > > > // the long long integer "n_tuples_" member > > ++i; > > block_len[i] = 1; > > types[i] = MPI_LONG_LONG; > > displacements[i] = (size_t) &obj.n_tuples_ - (size_t) &obj; > > > > // the integer "unique_id_" member > > ++i; > > block_len[i] = 1; > > types[i] = MPI_INT; > > displacements[i] = (size_t) &obj.unique_id_ - (size_t) &obj; > > > > // the integer "subordinate_id_" member > > ++i; > > block_len[i] = 1; > > types[i] = MPI_INT; > > displacements[i] = (size_t) &obj.subordinate_id_ - (size_t) &obj; > > > > // the integer "opcode_" member > > ++i; > > block_len[i] = 1; > > types[i] = MPI_INT; > > displacements[i] = (size_t) &obj.opcode_ - (size_t) &obj; > > > > // the char array "error_msg_" member > > ++i; > > block_len[i] = 512; > > types[i] = MPI_CHAR; > > displacements[i] = (size_t) &obj.error_msg_[0] - (size_t) &obj; > > > > MPI_Type_create_struct(struct_len, block_len, displacements, types, > &mpi_data_type); > > MPI_Type_commit(&mpi_data_type); > > > > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

MPI user-defined datatype size
by Mccall, Kurt E. (MSFC-EV41) 03 Dec '23

03 Dec '23

Hi, I'm running MPICH 4.1.2. Should a MPI user-defined datatype created by MPI_Type_create_struct have the same size as the C++ object on which it is based? Comparing sizeof(cpp_obj) and the output >From MPI_Type_size(mpi_data_type_, &size1), the C++ object is four bytes larger than the MPI datatype. I don't know if this is because I have to do something special to take padding into account when I create the MPI user datatype. Here is my C++ message class: struct MsgTupleSeq { static const int N_INDICES_MAX_ = 4; // culprit? int n_vars_; int n_ndxs_; short start_[N_INDICES_MAX_]; short end_[N_INDICES_MAX_]; short current_[N_INDICES_MAX_]; long long n_tuples_; int unique_id_; int subordinate_id_; int opcode_; char error_msg_[512]; } That is 564 bytes for the non-static members, but such an object is 568 bytes long with the 4 bytes of padding that the compiler inserts. I define the MPI user datatype with the code below, using the MsgTupleSeq object to compute the offsets. MPI_Type_size returns 564 bytes for the user datatype, so it seems like it isn't seeing the padding. MsgTupleSeq obj; MPI_Datatype mpi_data_type; int struct_len = 10, i = 0; int block_len[struct_len]; MPI_Datatype types[struct_len]; MPI_Aint displacements[struct_len]; // the integer "n_vars_" member block_len[i] = 1; types[i] = MPI_INT; displacements[i] = (size_t) &obj.n_vars_ - (size_t) &obj; // the integer "n_ndxs_" member ++i; block_len[i] = 1; types[i] = MPI_INT; displacements[i] = (size_t) &obj.n_ndxs_ - (size_t) &obj; // the short integer "start_" array member ++i; block_len[i] = N_INDICES_MAX_; types[i] = MPI_SHORT; displacements[i] = (size_t) &obj.start_ - (size_t) &obj; // the short integer "end_" array member ++i; block_len[i] = N_INDICES_MAX_; types[i] = MPI_SHORT; displacements[i] = (size_t) &obj.end_ - (size_t) &obj; // the short integer "current_" array member ++i; block_len[i] = N_INDICES_MAX_; types[i] = MPI_SHORT; displacements[i] = (size_t) &obj.current_ - (size_t) &obj; // the long long integer "n_tuples_" member ++i; block_len[i] = 1; types[i] = MPI_LONG_LONG; displacements[i] = (size_t) &obj.n_tuples_ - (size_t) &obj; // the integer "unique_id_" member ++i; block_len[i] = 1; types[i] = MPI_INT; displacements[i] = (size_t) &obj.unique_id_ - (size_t) &obj; // the integer "subordinate_id_" member ++i; block_len[i] = 1; types[i] = MPI_INT; displacements[i] = (size_t) &obj.subordinate_id_ - (size_t) &obj; // the integer "opcode_" member ++i; block_len[i] = 1; types[i] = MPI_INT; displacements[i] = (size_t) &obj.opcode_ - (size_t) &obj; // the char array "error_msg_" member ++i; block_len[i] = 512; types[i] = MPI_CHAR; displacements[i] = (size_t) &obj.error_msg_[0] - (size_t) &obj; MPI_Type_create_struct(struct_len, block_len, displacements, types, &mpi_data_type); MPI_Type_commit(&mpi_data_type);

1 0

Slides from the SC23 MPICH BoF now available
by Raffenetti, Ken 01 Dec '23

01 Dec '23

Hi all, Thank again to all the speakers and everyone who attended the MPICH BoF. It was a lively event :). We have posted the slides here for anyone to access: https://github.com/pmodels/mpich/wiki/2023-Supercomputing-BoF Cheers, Ken

1 0