April 2019 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] attribute value of MPI_TAG_UB
by Wesley Bland 29 Apr '19

29 Apr '19

There’s been a lot of churn on the TAG_UB code in the last year or two, but I think it finally stabilized last October with this pull request: https://github.com/pmodels/mpich/pull/3111 <https://github.com/pmodels/mpich/pull/3111> Jeff was right. The value of TAG_UB will depend on a lot of things, but if you use the exact same configuration, hardware, and environment variables from run to run, it should be the same value every time. As long as your application is querying the value instead of assuming a static value, you’ll be fine. > On Apr 27, 2019, at 11:20 AM, Jeff Hammond via discuss <discuss(a)mpich.org> wrote: > > > > On Fri, Apr 26, 2019 at 12:18 PM Wei-keng Liao via discuss <discuss(a)mpich.org <mailto:[email protected]>> wrote: > According to MPI 3.1, Section 8.1.2, the attribute value of > MPI_TAG_UB attached to MPI_COMM_WORLD can be inquired by function > MPI_Comm_get_attr(). It has the same value on all processes of > MPI_COMM_WORLD. > > My first question is should this value be the same across different runs? > > I would expect it to be the same when running with the same options. It is possible that it will change for different numbers of processes. It is also possible that it changes based upon the network or which back-end is used (eg libfabric provider). > > > A small test program using MPICH 3.3 shows they are not the same across > runs and across processes. But when compiled with the latest master branch, > they are the same. So, can I assume the answer to my question is YES and > MPICH has fixed this in the master branch? > > My second question is that I notice MPI_TAG_UB is defined as a constant > of 0x64400001 (=1681915905) in mpi.h. That value is not the one returned > by MPI_Comm_get_attr(), which is 10345120. Is this intended? > > I don’t think that’s the value but the key used to query it. You need to call get_attr rather than look at this value. The key is meaningless by itself. > > Jeff > > > I also tested OpenMPI 4.0.0. The inconsistency occurs across different > runs, as well as different processes. I can see OpenMPI defines > MPI_TAG_UB as an enum type, rather than a constant. > > Wei-keng > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss <https://lists.mpich.org/mailman/listinfo/discuss> > -- > Jeff Hammond > jeff.science(a)gmail.com <mailto:[email protected]> > http://jeffhammond.github.io/ <http://jeffhammond.github.io/>_______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI snow cluster in R
by Benson Muite 27 Apr '19

27 Apr '19

Hi, Example linux and OpenMPI for pbdR (https://pbdr.org/) that you might be able to adapt is here: https://github.com/Digitaalhumanitaaria/biohpc May want to check with pbdR Regards, Benson On 4/27/19 8:44 PM, Mark Sale via discuss wrote: > > Anyone have any (Windows)ï¿½experience they can share on setting up a > cluster in Rï¿½(using Rmpi, or other Rï¿½package) and the snow package, > then using parallel? > > > thanks > > > > > Mark Sale M.D. > > Senior Vice President, Pharmacometrics > > Nuventra Inc. > 2525 Meridian Parkway, Suite 200 > Durham, NC 27713 > > Phone (919)-973-0383 > > msale(a)nuventra.com <msale(a)kinetigen.com> > > // > > CONFIDENTIALITY NOTICE The information in this transmittal (including > attachments, if any) may be privileged and confidential and is > intended only for the recipient(s) listed above. Any review, use, > disclosure, distribution or copying of this transmittal, in any form, > is prohibited except by or on behalf of the intended recipient(s). If > you have received this transmittal in error, please notify me > immediately by reply email and destroy all copies of the transmittal. > > > > ------------------------------------------------------------------------ > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPI snow cluster in R
by Mark Sale 27 Apr '19

27 Apr '19

Anyone have any (Windows) experience they can share on setting up a cluster in R (using Rmpi, or other R package) and the snow package, then using parallel? thanks Mark Sale M.D. Senior Vice President, Pharmacometrics Nuventra Inc. 2525 Meridian Parkway, Suite 200 Durham, NC 27713 Phone (919)-973-0383 msale(a)nuventra.com<msale(a)kinetigen.com> CONFIDENTIALITY NOTICE The information in this transmittal (including attachments, if any) may be privileged and confidential and is intended only for the recipient(s) listed above. Any review, use, disclosure, distribution or copying of this transmittal, in any form, is prohibited except by or on behalf of the intended recipient(s). If you have received this transmittal in error, please notify me immediately by reply email and destroy all copies of the transmittal. ________________________________ From: Jeff Hammond via discuss <discuss(a)mpich.org> Sent: Saturday, April 27, 2019 9:20 AM To: discuss(a)mpich.org Cc: Jeff Hammond Subject: Re: [mpich-discuss] attribute value of MPI_TAG_UB WARNING: This email originated from outside of the company. Do not click links or open attachments unless you recognize the sender and are expecting the message. On Fri, Apr 26, 2019 at 12:18 PM Wei-keng Liao via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: According to MPI 3.1, Section 8.1.2, the attribute value of MPI_TAG_UB attached to MPI_COMM_WORLD can be inquired by function MPI_Comm_get_attr(). It has the same value on all processes of MPI_COMM_WORLD. My first question is should this value be the same across different runs? I would expect it to be the same when running with the same options. It is possible that it will change for different numbers of processes. It is also possible that it changes based upon the network or which back-end is used (eg libfabric provider). A small test program using MPICH 3.3 shows they are not the same across runs and across processes. But when compiled with the latest master branch, they are the same. So, can I assume the answer to my question is YES and MPICH has fixed this in the master branch? My second question is that I notice MPI_TAG_UB is defined as a constant of 0x64400001 (=1681915905) in mpi.h. That value is not the one returned by MPI_Comm_get_attr(), which is 10345120. Is this intended? I don’t think that’s the value but the key used to query it. You need to call get_attr rather than look at this value. The key is meaningless by itself. Jeff I also tested OpenMPI 4.0.0. The inconsistency occurs across different runs, as well as different processes. I can see OpenMPI defines MPI_TAG_UB as an enum type, rather than a constant. Wei-keng _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> http://jeffhammond.github.io/

1 0

Re: [mpich-discuss] attribute value of MPI_TAG_UB
by Jeff Hammond 27 Apr '19

27 Apr '19

On Fri, Apr 26, 2019 at 12:18 PM Wei-keng Liao via discuss < discuss(a)mpich.org> wrote: > According to MPI 3.1, Section 8.1.2, the attribute value of > MPI_TAG_UB attached to MPI_COMM_WORLD can be inquired by function > MPI_Comm_get_attr(). It has the same value on all processes of > MPI_COMM_WORLD. > > My first question is should this value be the same across different runs? I would expect it to be the same when running with the same options. It is possible that it will change for different numbers of processes. It is also possible that it changes based upon the network or which back-end is used (eg libfabric provider). > A small test program using MPICH 3.3 shows they are not the same across > runs and across processes. But when compiled with the latest master branch, > they are the same. So, can I assume the answer to my question is YES and > MPICH has fixed this in the master branch? > > My second question is that I notice MPI_TAG_UB is defined as a constant > of 0x64400001 (=1681915905) in mpi.h. That value is not the one returned > by MPI_Comm_get_attr(), which is 10345120. Is this intended? > I don’t think that’s the value but the key used to query it. You need to call get_attr rather than look at this value. The key is meaningless by itself. Jeff > I also tested OpenMPI 4.0.0. The inconsistency occurs across different > runs, as well as different processes. I can see OpenMPI defines > MPI_TAG_UB as an enum type, rather than a constant. > > Wei-keng > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

attribute value of MPI_TAG_UB
by Wei-keng Liao 26 Apr '19

26 Apr '19

According to MPI 3.1, Section 8.1.2, the attribute value of MPI_TAG_UB attached to MPI_COMM_WORLD can be inquired by function MPI_Comm_get_attr(). It has the same value on all processes of MPI_COMM_WORLD. My first question is should this value be the same across different runs? A small test program using MPICH 3.3 shows they are not the same across runs and across processes. But when compiled with the latest master branch, they are the same. So, can I assume the answer to my question is YES and MPICH has fixed this in the master branch? My second question is that I notice MPI_TAG_UB is defined as a constant of 0x64400001 (=1681915905) in mpi.h. That value is not the one returned by MPI_Comm_get_attr(), which is 10345120. Is this intended? I also tested OpenMPI 4.0.0. The inconsistency occurs across different runs, as well as different processes. I can see OpenMPI defines MPI_TAG_UB as an enum type, rather than a constant. Wei-keng

1 0

Re: [mpich-discuss] Possible integer-overflow for MPI_COMM_WORLD in MPI_Iprobe
by Jeff Hammond 26 Apr '19

26 Apr '19

For anyone who cares about this bug because of Intel MPI, I am told it is fixed in Intel MPI 2019 update 3. Jeff On Mon, Jan 21, 2019 at 7:46 PM Jeff Hammond <jeff.science(a)gmail.com> wrote: > I was able to reproduce with my own test ( > https://github.com/jeffhammond/HPCInfo/blob/master/mpi/bugs/iprobe-overflow…) > with Intel MPI 2019, so I will report that bug to the Intel MPI team. It > should be easy enough for them to figure out if this bug is from MPICH or > not. > > 2139000000 iterations, 627.143272 seconds > > 2140000000 iterations, 627.436206 seconds > > 2141000000 iterations, 627.729135 seconds > > 2142000000 iterations, 628.022049 seconds > > 2143000000 iterations, 628.315015 seconds > > 2144000000 iterations, 628.608066 seconds > > 2145000000 iterations, 628.901065 seconds > > 2146000000 iterations, 629.193992 seconds > > 2147000000 iterations, 629.488107 seconds > > Abort(738833413) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Iprobe: > Invalid communicator, error stack: > > PMPI_Iprobe(123): MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, > MPI_COMM_WORLD, flag=0x7ffdf75a396c, status=0x7ffdf75a3970) failed > > PMPI_Iprobe(90).: Invalid communicator > > Jeff, who works for Intel but knows more about MPICH than Intel MPI > > On Mon, Jan 21, 2019 at 11:19 AM Joachim Protze via discuss < > discuss(a)mpich.org> wrote: > >> Hi all, >> >> we detected the behavior with IntelMPI 2019 (which is based on MPICH >> 3.3). Reproducing it with MPICH-3.3 was not yet successful. But I fear, >> that our built of MPICH just not uses the necessary code path / build >> flags. >> >> When calling MPI_Iprobe with the same communicator for ~2^31 times >> (which can take 10-30 minutes), the execution stops with: >> >> Abort(201962501) on node 0 (rank 0 in comm 0): Fatal error in >> PMPI_Iprobe: Invalid communicator, error stack: >> PMPI_Iprobe(123): MPI_Iprobe(src=MPI_ANY_SOURCE, tag=MPI_ANY_TAG, >> MPI_COMM_WORLD, flag=0x7ffd925056c0, status=0x7ffd92505694) failed >> PMPI_Iprobe(90).: Invalid communicator >> >> From my understanding of the referenced MPICH code lines, I guess, that >> the ref-count for MPI_COMM_WORLD overflows, which triggers this error >> message. >> >> Best >> Joachim >> >> -- >> Dipl.-Inf. Joachim Protze >> >> IT Center >> Group: High Performance Computing >> Division: Computational Science and Engineering >> RWTH Aachen University >> Seffenter Weg 23 >> D 52074 Aachen (Germany) >> Tel: +49 241 80- 24765 >> Fax: +49 241 80-624765 >> protze(a)itc.rwth-aachen.de >> www.itc.rwth-aachen.de >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

PROFILING MPICH
by SANJAY KUMAR 19 Apr '19

19 Apr '19

During profiling MPICH internal calls, I found that there is so many calls made to MPID_nem_mpich_blocking_recv under file named "mpid_nem_inline.h". Can anyone explain why is it so? Thanks Sanjay Kumar

1 0

Re: [mpich-discuss] Debugging MPI Window
by Palmer, Bruce J 19 Apr '19

19 Apr '19

Thanks for all the suggestions. I’ll try these and see how I do. Bruce From: Jeff Hammond via discuss <discuss(a)mpich.org> Sent: Thursday, April 18, 2019 10:26 PM To: discuss(a)mpich.org Cc: Jeff Hammond <jeff.science(a)gmail.com> Subject: Re: [mpich-discuss] Debugging MPI Window MPI_Win_{get,set}_name might be an even simpler way to do that. Jeff On Thu, Apr 18, 2019 at 1:04 AM Joachim Protze via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi Bruce, I think, you could use the keyval and set/get-attr functions for that purpose and it should even be portable :) Without testing the code, I would so something like the following: #define S1(x) #x #define S2(x) S1(x) #define LOCATION __FILE__ " : " S2(__LINE__) int win_debug_string_key; const int max_win_debug_string_len = 30; int My_copy_attr_function( MPI_Win oldwin, int win_keyval, void *extra_state, void *attribute_val_in, void *attribute_val_out, int *flag) { attribute_val_out = strndup(attribute_val_in, *(int*)extra_state); flag = 1; /* or set 0 if a copy is intended to be different */ } int My_delete_attr_function( MPI_Win win, int win_keyval, void *attribute_val, void *extra_state) { free(attribute_val); } /* Define the keyval once */ MPI_Win_create_keyval( My_copy_attr_function, My_delete_attr_function, &win_debug_string_key, &max_win_debug_string_len ); /* When creating a window */ MPI_Win_create( ..., &win ); /* be aware of the extra byte for the \0 byte */ char * win_debug_string = malloc(sizeof(char)*(max_win_debug_string_len+1)); /* Write the current location in there: */ strncpy(win_debug_string, LOCATION); /* Bind the debug-string to the new window */ MPI_Win_set_attr( win, win_debug_string_key, win_debug_string ); Best Joachim On 4/18/19 12:32 AM, Palmer, Bruce J via discuss wrote: > Hi, > > Strictly for debugging purposes, is there a way in MPICH to determine if > a MPI_Win created at point A in a code is the same window being accessed > at point B? It doesn’t need to be portable, I’m just interested in using > it to fix some bugs (that are probably not associated with MPI). > > Bruce Palmer > > Computer Scientist > > Pacific Northwest National Laboratory > > (509) 375-3899 > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org<mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de<mailto:[email protected]> www.itc.rwth-aachen.de<http://www.itc.rwth-aachen.de> _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> http://jeffhammond.github.io/

1 0

Re: [mpich-discuss] Debugging MPI Window
by Jeff Hammond 19 Apr '19

19 Apr '19

MPI_Win_{get,set}_name might be an even simpler way to do that. Jeff On Thu, Apr 18, 2019 at 1:04 AM Joachim Protze via discuss < discuss(a)mpich.org> wrote: > Hi Bruce, > > I think, you could use the keyval and set/get-attr functions for that > purpose and it should even be portable :) > Without testing the code, I would so something like the following: > > > #define S1(x) #x > #define S2(x) S1(x) > #define LOCATION __FILE__ " : " S2(__LINE__) > int win_debug_string_key; > const int max_win_debug_string_len = 30; > > int My_copy_attr_function( > MPI_Win oldwin, int win_keyval, > void *extra_state, void *attribute_val_in, > void *attribute_val_out, int *flag) > { > attribute_val_out = strndup(attribute_val_in, *(int*)extra_state); > flag = 1; /* or set 0 if a copy is intended to be different */ > } > > int My_delete_attr_function( > MPI_Win win, int win_keyval, > void *attribute_val, void *extra_state) > { > free(attribute_val); > } > > > > /* Define the keyval once */ > MPI_Win_create_keyval( My_copy_attr_function, My_delete_attr_function, > &win_debug_string_key, &max_win_debug_string_len ); > > > /* When creating a window */ > MPI_Win_create( ..., &win ); > > /* be aware of the extra byte for the \0 byte */ > char * win_debug_string = > malloc(sizeof(char)*(max_win_debug_string_len+1)); > /* Write the current location in there: */ > strncpy(win_debug_string, LOCATION); > /* Bind the debug-string to the new window */ > MPI_Win_set_attr( win, win_debug_string_key, win_debug_string ); > > > Best > Joachim > > > On 4/18/19 12:32 AM, Palmer, Bruce J via discuss wrote: > > Hi, > > > > Strictly for debugging purposes, is there a way in MPICH to determine if > > a MPI_Win created at point A in a code is the same window being accessed > > at point B? It doesn’t need to be portable, I’m just interested in using > > it to fix some bugs (that are probably not associated with MPI). > > > > Bruce Palmer > > > > Computer Scientist > > > > Pacific Northwest National Laboratory > > > > (509) 375-3899 > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > -- > Dipl.-Inf. Joachim Protze > > IT Center > Group: High Performance Computing > Division: Computational Science and Engineering > RWTH Aachen University > Seffenter Weg 23 > D 52074 Aachen (Germany) > Tel: +49 241 80- 24765 > Fax: +49 241 80-624765 > protze(a)itc.rwth-aachen.de > www.itc.rwth-aachen.de > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-discuss] Debugging MPI Window
by Joachim Protze 18 Apr '19

18 Apr '19

Hi Bruce, I think, you could use the keyval and set/get-attr functions for that purpose and it should even be portable :) Without testing the code, I would so something like the following: #define S1(x) #x #define S2(x) S1(x) #define LOCATION __FILE__ " : " S2(__LINE__) int win_debug_string_key; const int max_win_debug_string_len = 30; int My_copy_attr_function( MPI_Win oldwin, int win_keyval, void *extra_state, void *attribute_val_in, void *attribute_val_out, int *flag) { attribute_val_out = strndup(attribute_val_in, *(int*)extra_state); flag = 1; /* or set 0 if a copy is intended to be different */ } int My_delete_attr_function( MPI_Win win, int win_keyval, void *attribute_val, void *extra_state) { free(attribute_val); } /* Define the keyval once */ MPI_Win_create_keyval( My_copy_attr_function, My_delete_attr_function, &win_debug_string_key, &max_win_debug_string_len ); /* When creating a window */ MPI_Win_create( ..., &win ); /* be aware of the extra byte for the \0 byte */ char * win_debug_string = malloc(sizeof(char)*(max_win_debug_string_len+1)); /* Write the current location in there: */ strncpy(win_debug_string, LOCATION); /* Bind the debug-string to the new window */ MPI_Win_set_attr( win, win_debug_string_key, win_debug_string ); Best Joachim On 4/18/19 12:32 AM, Palmer, Bruce J via discuss wrote: > Hi, > > Strictly for debugging purposes, is there a way in MPICH to determine if > a MPI_Win created at point A in a code is the same window being accessed > at point B? It doesn’t need to be portable, I’m just interested in using > it to fix some bugs (that are probably not associated with MPI). > > Bruce Palmer > > Computer Scientist > > Pacific Northwest National Laboratory > > (509) 375-3899 > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0