- discuss - mailman3.cels.anl.gov

attribute value of MPI_TAG_UB
by Wei-keng Liao 26 Apr '19

26 Apr '19

According to MPI 3.1, Section 8.1.2, the attribute value of MPI_TAG_UB attached to MPI_COMM_WORLD can be inquired by function MPI_Comm_get_attr(). It has the same value on all processes of MPI_COMM_WORLD. My first question is should this value be the same across different runs? A small test program using MPICH 3.3 shows they are not the same across runs and across processes. But when compiled with the latest master branch, they are the same. So, can I assume the answer to my question is YES and MPICH has fixed this in the master branch? My second question is that I notice MPI_TAG_UB is defined as a constant of 0x64400001 (=1681915905) in mpi.h. That value is not the one returned by MPI_Comm_get_attr(), which is 10345120. Is this intended? I also tested OpenMPI 4.0.0. The inconsistency occurs across different runs, as well as different processes. I can see OpenMPI defines MPI_TAG_UB as an enum type, rather than a constant. Wei-keng

1 0

Re: [mpich-discuss] Possible integer-overflow for MPI_COMM_WORLD in MPI_Iprobe
by Jeff Hammond 26 Apr '19

26 Apr '19

For anyone who cares about this bug because of Intel MPI, I am told it is fixed in Intel MPI 2019 update 3. Jeff On Mon, Jan 21, 2019 at 7:46 PM Jeff Hammond <jeff.science(a)gmail.com> wrote: > I was able to reproduce with my own test ( > https://github.com/jeffhammond/HPCInfo/blob/master/mpi/bugs/iprobe-overflow…) > with Intel MPI 2019, so I will report that bug to the Intel MPI team. It > should be easy enough for them to figure out if this bug is from MPICH or > not. > > 2139000000 iterations, 627.143272 seconds > > 2140000000 iterations, 627.436206 seconds > > 2141000000 iterations, 627.729135 seconds > > 2142000000 iterations, 628.022049 seconds > > 2143000000 iterations, 628.315015 seconds > > 2144000000 iterations, 628.608066 seconds > > 2145000000 iterations, 628.901065 seconds > > 2146000000 iterations, 629.193992 seconds > > 2147000000 iterations, 629.488107 seconds > > Abort(738833413) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Iprobe: > Invalid communicator, error stack: > > PMPI_Iprobe(123): MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, > MPI_COMM_WORLD, flag=0x7ffdf75a396c, status=0x7ffdf75a3970) failed > > PMPI_Iprobe(90).: Invalid communicator > > Jeff, who works for Intel but knows more about MPICH than Intel MPI > > On Mon, Jan 21, 2019 at 11:19 AM Joachim Protze via discuss < > discuss(a)mpich.org> wrote: > >> Hi all, >> >> we detected the behavior with IntelMPI 2019 (which is based on MPICH >> 3.3). Reproducing it with MPICH-3.3 was not yet successful. But I fear, >> that our built of MPICH just not uses the necessary code path / build >> flags. >> >> When calling MPI_Iprobe with the same communicator for ~2^31 times >> (which can take 10-30 minutes), the execution stops with: >> >> Abort(201962501) on node 0 (rank 0 in comm 0): Fatal error in >> PMPI_Iprobe: Invalid communicator, error stack: >> PMPI_Iprobe(123): MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, >> MPI_COMM_WORLD, flag=0x7ffd925056c0, status=0x7ffd92505694) failed >> PMPI_Iprobe(90).: Invalid communicator >> >> From my understanding of the referenced MPICH code lines, I guess, that >> the ref-count for MPI_COMM_WORLD overflows, which triggers this error >> message. >> >> Best >> Joachim >> >> -- >> Dipl.-Inf. Joachim Protze >> >> IT Center >> Group: High Performance Computing >> Division: Computational Science and Engineering >> RWTH Aachen University >> Seffenter Weg 23 >> D 52074 Aachen (Germany) >> Tel: +49 241 80- 24765 >> Fax: +49 241 80-624765 >> protze(a)itc.rwth-aachen.de >> www.itc.rwth-aachen.de >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

PROFILING MPICH
by SANJAY KUMAR 19 Apr '19

19 Apr '19

During profiling MPICH internal calls, I found that there is so many calls made to MPID_nem_mpich_blocking_recv under file named "mpid_nem_inline.h". Can anyone explain why is it so? Thanks Sanjay Kumar

1 0

Re: [mpich-discuss] Debugging MPI Window
by Palmer, Bruce J 19 Apr '19

19 Apr '19

Thanks for all the suggestions. I’ll try these and see how I do. Bruce From: Jeff Hammond via discuss <discuss(a)mpich.org> Sent: Thursday, April 18, 2019 10:26 PM To: discuss(a)mpich.org Cc: Jeff Hammond <jeff.science(a)gmail.com> Subject: Re: [mpich-discuss] Debugging MPI Window MPI_Win_{get,set}_name might be an even simpler way to do that. Jeff On Thu, Apr 18, 2019 at 1:04 AM Joachim Protze via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi Bruce, I think, you could use the keyval and set/get-attr functions for that purpose and it should even be portable :) Without testing the code, I would so something like the following: #define S1(x) #x #define S2(x) S1(x) #define LOCATION __FILE__ " : " S2(__LINE__) int win_debug_string_key; const int max_win_debug_string_len = 30; int My_copy_attr_function( MPI_Win oldwin, int win_keyval, void *extra_state, void *attribute_val_in, void *attribute_val_out, int *flag) { attribute_val_out = strndup(attribute_val_in, *(int*)extra_state); flag = 1; /* or set 0 if a copy is intended to be different */ } int My_delete_attr_function( MPI_Win win, int win_keyval, void *attribute_val, void *extra_state) { free(attribute_val); } /* Define the keyval once */ MPI_Win_create_keyval( My_copy_attr_function, My_delete_attr_function, &win_debug_string_key, &max_win_debug_string_len ); /* When creating a window */ MPI_Win_create( ..., &win ); /* be aware of the extra byte for the \0 byte */ char * win_debug_string = malloc(sizeof(char)*(max_win_debug_string_len+1)); /* Write the current location in there: */ strncpy(win_debug_string, LOCATION); /* Bind the debug-string to the new window */ MPI_Win_set_attr( win, win_debug_string_key, win_debug_string ); Best Joachim On 4/18/19 12:32 AM, Palmer, Bruce J via discuss wrote: > Hi, > > Strictly for debugging purposes, is there a way in MPICH to determine if > a MPI_Win created at point A in a code is the same window being accessed > at point B? It doesn’t need to be portable, I’m just interested in using > it to fix some bugs (that are probably not associated with MPI). > > Bruce Palmer > > Computer Scientist > > Pacific Northwest National Laboratory > > (509) 375-3899 > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org<mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de<mailto:[email protected]> www.itc.rwth-aachen.de<http://www.itc.rwth-aachen.de> _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> http://jeffhammond.github.io/

1 0

Re: [mpich-discuss] Debugging MPI Window
by Jeff Hammond 19 Apr '19

19 Apr '19

MPI_Win_{get,set}_name might be an even simpler way to do that. Jeff On Thu, Apr 18, 2019 at 1:04 AM Joachim Protze via discuss < discuss(a)mpich.org> wrote: > Hi Bruce, > > I think, you could use the keyval and set/get-attr functions for that > purpose and it should even be portable :) > Without testing the code, I would so something like the following: > > > #define S1(x) #x > #define S2(x) S1(x) > #define LOCATION __FILE__ " : " S2(__LINE__) > int win_debug_string_key; > const int max_win_debug_string_len = 30; > > int My_copy_attr_function( > MPI_Win oldwin, int win_keyval, > void *extra_state, void *attribute_val_in, > void *attribute_val_out, int *flag) > { > attribute_val_out = strndup(attribute_val_in, *(int*)extra_state); > flag = 1; /* or set 0 if a copy is intended to be different */ > } > > int My_delete_attr_function( > MPI_Win win, int win_keyval, > void *attribute_val, void *extra_state) > { > free(attribute_val); > } > > > > /* Define the keyval once */ > MPI_Win_create_keyval( My_copy_attr_function, My_delete_attr_function, > &win_debug_string_key, &max_win_debug_string_len ); > > > /* When creating a window */ > MPI_Win_create( ..., &win ); > > /* be aware of the extra byte for the \0 byte */ > char * win_debug_string = > malloc(sizeof(char)*(max_win_debug_string_len+1)); > /* Write the current location in there: */ > strncpy(win_debug_string, LOCATION); > /* Bind the debug-string to the new window */ > MPI_Win_set_attr( win, win_debug_string_key, win_debug_string ); > > > Best > Joachim > > > On 4/18/19 12:32 AM, Palmer, Bruce J via discuss wrote: > > Hi, > > > > Strictly for debugging purposes, is there a way in MPICH to determine if > > a MPI_Win created at point A in a code is the same window being accessed > > at point B? It doesn’t need to be portable, I’m just interested in using > > it to fix some bugs (that are probably not associated with MPI). > > > > Bruce Palmer > > > > Computer Scientist > > > > Pacific Northwest National Laboratory > > > > (509) 375-3899 > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > -- > Dipl.-Inf. Joachim Protze > > IT Center > Group: High Performance Computing > Division: Computational Science and Engineering > RWTH Aachen University > Seffenter Weg 23 > D 52074 Aachen (Germany) > Tel: +49 241 80- 24765 > Fax: +49 241 80-624765 > protze(a)itc.rwth-aachen.de > www.itc.rwth-aachen.de > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-discuss] Debugging MPI Window
by Joachim Protze 18 Apr '19

18 Apr '19

Hi Bruce, I think, you could use the keyval and set/get-attr functions for that purpose and it should even be portable :) Without testing the code, I would so something like the following: #define S1(x) #x #define S2(x) S1(x) #define LOCATION __FILE__ " : " S2(__LINE__) int win_debug_string_key; const int max_win_debug_string_len = 30; int My_copy_attr_function( MPI_Win oldwin, int win_keyval, void *extra_state, void *attribute_val_in, void *attribute_val_out, int *flag) { attribute_val_out = strndup(attribute_val_in, *(int*)extra_state); flag = 1; /* or set 0 if a copy is intended to be different */ } int My_delete_attr_function( MPI_Win win, int win_keyval, void *attribute_val, void *extra_state) { free(attribute_val); } /* Define the keyval once */ MPI_Win_create_keyval( My_copy_attr_function, My_delete_attr_function, &win_debug_string_key, &max_win_debug_string_len ); /* When creating a window */ MPI_Win_create( ..., &win ); /* be aware of the extra byte for the \0 byte */ char * win_debug_string = malloc(sizeof(char)*(max_win_debug_string_len+1)); /* Write the current location in there: */ strncpy(win_debug_string, LOCATION); /* Bind the debug-string to the new window */ MPI_Win_set_attr( win, win_debug_string_key, win_debug_string ); Best Joachim On 4/18/19 12:32 AM, Palmer, Bruce J via discuss wrote: > Hi, > > Strictly for debugging purposes, is there a way in MPICH to determine if > a MPI_Win created at point A in a code is the same window being accessed > at point B? It doesn’t need to be portable, I’m just interested in using > it to fix some bugs (that are probably not associated with MPI). > > Bruce Palmer > > Computer Scientist > > Pacific Northwest National Laboratory > > (509) 375-3899 > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpich-discuss] Debugging MPI Window
by Si, Min 17 Apr '19

17 Apr '19

Hi Bruce, If you want to check if the window object is the same, you can compare the window handle value. Just note that if you do win_create-win_free-win_create, the second window may be assigned with the same handle as the first one. Best regards, Min On 2019/04/17 17:32, Palmer, Bruce J via discuss wrote: Hi, Strictly for debugging purposes, is there a way in MPICH to determine if a MPI_Win created at point A in a code is the same window being accessed at point B? It doesn’t need to be portable, I’m just interested in using it to fix some bugs (that are probably not associated with MPI). Bruce Palmer Computer Scientist Pacific Northwest National Laboratory (509) 375-3899 _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Debugging MPI Window
by Palmer, Bruce J 17 Apr '19

17 Apr '19

Hi, Strictly for debugging purposes, is there a way in MPICH to determine if a MPI_Win created at point A in a code is the same window being accessed at point B? It doesn't need to be portable, I'm just interested in using it to fix some bugs (that are probably not associated with MPI). Bruce Palmer Computer Scientist Pacific Northwest National Laboratory (509) 375-3899

1 0

Re: [mpich-discuss] Reduction with MPI_CHAR
by Jeff Hammond 06 Apr '19

06 Apr '19

I’d create a GitHub issue for this. Checking for incorrect usage is a quality-of-implementation issue but I’m sure the devs want to fix this. It’s easy enough. Jeff On Fri, Apr 5, 2019 at 8:21 AM Zhang, Junchao via discuss <discuss(a)mpich.org> wrote: > MPI 3.1 Chapter 5.9.3 reads "MPI_CHAR, MPI_WCHAR, and MPI_CHARACTER (which > represent printable characters) cannot be used in reduction operations". > But the following code with MPICH does not abort. > > #include <mpi.h> > #include <stdlib.h> > int main(int argc, char* argv[]) > { > char a ='A',b; > MPI_Init(&argc,&argv); > if(MPI_Allreduce(&a,&b,1,MPI_CHAR,MPI_SUM,MPI_COMM_WORLD)) abort(); > MPI_Finalize(); > } > --Junchao Zhang > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-discuss] Profiling MPICH Source Code
by Amer, Abdelhalim 05 Apr '19

05 Apr '19

I'll throw in a couple of options that are good at large scale: Open Source: HPCToolkit (http://hpctoolkit.org/) Commercial: Arm MAP Similar to Jeff's comment about the other tools, you need to build MPICH with debug symbols for these tools to give you useful information. Halim www.mcs.anl.gov/~aamer On 4/3/19 3:00 PM, Jeff Hammond via discuss wrote: > First, you need to compile MPICH with debug symbols. > https://wiki.mpich.org/mpich/index.php/Building_MPICH_For_Development has details. > > Second, use a profiler like Linux perf or Intel Vtune to see what the > hotspots are. > > Jeff > > On Wed, Apr 3, 2019 at 11:54 AM SANJAY KUMAR via discuss > <discuss(a)mpich.org <mailto:[email protected]>> wrote: > > Here at Indian Institute of Technology Kanpur, Computer Science > Department, we are trying to profile mpich source code itself. So as to > analyze the internal function calls made by the mpich functions like > MPI_Bcast etc. I have tried so many profilers like IPM, gprof to > compile the MPICH source code. But It turned out nothing works. I have > also so many flags but nothing worked out. Could you please tell me how > to profile the mpich source code itself. > > > Thanks > > Sanjay Kumar > > Masters of Technology > > Compute Science Department > > Indian Institute of Technology Kanpur > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Jeff Hammond > jeff.science(a)gmail.com <mailto:[email protected]> > http://jeffhammond.github.io/ > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0