December 2015 - discuss - mailman3.cels.anl.gov

Redirected links on http://www.mcs.anl.gov/research/projects/perfvis/download/
by Schilling, Herbert W. (GRC-VE00) 03 Dec '15

03 Dec '15

Hi, Some of the links on http://www.mcs.anl.gov/research/projects/perfvis/download/ ( like the links to the User Guide ) get redirected to the MPICH home page. Which might not be too bad but I can't find the MPE Users Guide there even when using the Search. I think I did find it so I am OK, but I thought you should know! ftp://ftp.mcs.anl.gov/pub/mpi/mpeman.pdf Thanks --------- Herb Schilling Scientific Applications and Visualization Team http://ocio.grc.nasa.gov/gvis Office of the Chief Information Officer NASA Glenn Research Center hschilling(a)nasa.gov<mailto:[email protected]>

1 0

mpi_scatter
by Jen B 03 Dec '15

03 Dec '15

Hello, I am trying to understand how the mpi_scatter is implemented according to MPICH2 for long messages. I went through the paper of R. Thakur, et al., titled "Optimization of Collective Communication Operations in MPICH." It is stated there that for long messages mpi_bcast is composed of scatter and allgather (where latter two are based on the Van de Geijn's algorithm). However I wasn't able to find the mpi_scatter details. Could you please explain this? Also, the another thing is that I think for long messages (that are not divisible exactly by the number of processes) this part would use a scatterv + allgatherv. Am I right? Thank you! Best, Jen _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_Gather fails with 2048 processes and 4096 MB total
by Florian.Willich＠dlr.de 03 Dec '15

03 Dec '15

Since this Bug does not concern MPICH I ask to close the reported bug: http://trac.mpich.org/projects/mpich/ticket/2317 Thank you all for helping. Florian ________________________________________ Von: Balaji, Pavan [balaji(a)anl.gov] Gesendet: Mittwoch, 2. Dezember 2015 19:42 An: discuss(a)mpich.org Betreff: Re: [mpich-discuss] MPI_Gather fails with 2048 processes and 4096 MB total I believe Cray has a newer version based on mpich-3.2 that they are releasing soon-ish. If you are a good customer, they might give you an engineering build to try out. :-) -- Pavan > On Dec 2, 2015, at 9:28 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > This was fixed after Cray forked MPICH 3.1.2. You should file a support issue with Cray via the appropriate channel. > > # master > jrhammon-mac01:git jrhammon$ git grep MPIC_Send > src/include/mpiimpl.h:int MPIC_Send(const void *buf, MPI_Aint count, MPI_Datatype datatype, int dest, int tag, > ... > > # 3.1 branch > jrhammon-mac01:git jrhammon$ git checkout 3.1.x > jrhammon-mac01:git jrhammon$ git grep MPIC_Send > src/include/mpiimpl.h:int MPIC_Send(const void *buf, int count, MPI_Datatype datatype, int dest, int tag, > ... > > Best, > > Jeff > > On Wed, Dec 2, 2015 at 6:40 AM, <Florian.Willich(a)dlr.de> wrote: > > > > Thanks for the "bash line", the output is: > > > > $ grep "define MPICH_VERSION" $CRAY_MPICH2_DIR/include/mpi.h > > #define MPICH_VERSION "3.1.2" > > > > Florian > > > > ________________________________ > > Von: Jeff Hammond [jeff.science(a)gmail.com] > > Gesendet: Mittwoch, 2. Dezember 2015 14:58 > > An: MPICH > > > > Betreff: Re: [mpich-discuss] MPI_Gather fails with 2048 processes and 4096 MB total > > > > 1) You should address this with Cray. Their implementation of MPI has closed-source modifications to MPICH that we cannot analyze for count-safety. > > > > 2) You can discover the version of MPICH associated with Cray MPI like this: > > jhammond@cori11:~> grep "define MPICH_VERSION" $CRAY_MPICH2_DIR/include/mpi.h > > #define MPICH_VERSION "3.1.2" > > > > Best, > > > > Jeff > > > > On Wed, Dec 2, 2015 at 12:15 AM, <Florian.Willich(a)dlr.de> wrote: > >> > >> Hi Rob, > >> > >> well maybe I was addressing the wrong organisation... I am currently testing on the Cray swan super computer which provides the module cray-mpich/7.2.6 ("Cray Message Passing Toolkit 7.2.6"). > >> > >> I can not determine whether the cray mpich version is mpich with additional implementations or if it is totally different from the official mpich releases. Additioanlly, I can not figure out on which mpich version this cray-mpich module is based on. I'll continue investigation and keep you updated. > >> > >> Best Regards > >> > >> Florian > >> ________________________________________ > >> Von: Rob Latham [robl(a)mcs.anl.gov] > >> Gesendet: Dienstag, 1. Dezember 2015 16:48 > >> An: discuss(a)mpich.org > >> Betreff: Re: [mpich-discuss] MPI_Gather fails with 2048 processes and 4096 MB total > >> > >> On 11/26/2015 12:38 PM, Archer, Charles J wrote: > >> > FYI, we hit various flavors of this problem when I was still at IBM, I think mostly in weather codes. > >> > Apparently Cray hit this too: > >> > > >> > https://trac.mpich.org/projects/mpich/ticket/1767 > >> > > >> > We pretty much told our customers back then that a fix was forthcoming (with no ETA :) )with the revamp of datatypes to use internal 64-bit counts. > >> > We also provided workarounds. > >> > > >> > In the case of this gather operation, we asked the customer to implement gather as a flat tree using point to point. > >> > Root posts irecvs, then barrier, children send to root. > >> > > >> > IRC, the giant gather we were debugging was at the very end of the application and used to gather some statistics for IO at the root, so it wasn’t critical to perform well. > >> > I also attempted a workaround using some derived datatypes, but I hit another truncation in the datatype code itself :\ > >> > I should see if I can dig up that implementation and make sure it isn’t still broken for large counts. > >> > >> those are all fine approaches to work around the problem. the internals > >> of MPICH, though, need to be 64 bit clean -- there are still 4500 places > >> where clang warns of a 64 bit value being assigned to a 32 bit type. > >> > >> Florian Willich, what version of MPICH is this? The line numbers in > >> the back trace don't match up with what I've got, and > >> I really thought we fixed this class of bug with commits 31d95ed7b18c > >> and 68f8c7aa7 over the summer. > >> > >> ==rob > >> > >> > >> > > >> > > >> > > >> > > >> > > >> > On Nov 26, 2015, at 12:14 PM, Balaji, Pavan <balaji(a)anl.gov<mailto:[email protected]>> wrote: > >> > > >> > > >> > Thanks for reporting. This looks like an integer-overflow issue, which fails when the summation of data elements from all processes is larger than INT_MAX (2 billion). We'll look into it. I've created a ticket for it, and added you as the reporter, so you'll get notified as they are updates. > >> > > >> > http://trac.mpich.org/projects/mpich/ticket/2317 > >> > > >> > Rob: can you create a simple test program for this and add it to the test bucket, so it shows up on the nightlies? > >> > > >> > Thanks, > >> > > >> > -- Pavan > >> > > >> > On Nov 26, 2015, at 10:18 AM, Florian.Willich(a)dlr.de wrote: > >> > > >> > Dear mpich discussion group, > >> > > >> > the following issue appeared when running some benchmarks with MPI Gather: > >> > > >> > Gathering data (calling MPI_Gather(...) ) involing 2048 processes and 2 MB of data (4096 MB total) that each process sends fails with the following output: > >> > ____________________________ > >> > > >> > Rank 1024 [Thu Nov 26 09:43:16 2015] [c1-0c1s12n3] Fatal error in PMPI_Gather: Invalid count, error stack: > >> > PMPI_Gather(959)......: MPI_Gather(sbuf=0x2aaab826c010, scount=524288, MPI_INT, rbuf=(nil), rcount=524288, MPI_INT, root=0, MPI_COMM_WORLD) failed > >> > MPIR_Gather_impl(775).: > >> > MPIR_Gather(735)......: > >> > MPIR_Gather_intra(347): > >> > MPIC_Send(360)........: Negative count, value is -2147483648 > >> > _pmiu_daemon(SIGCHLD): [NID 00307] [c1-0c1s12n3] [Thu Nov 26 09:43:16 2015] PE RANK 1024 exit signal Aborted > >> > [NID 00307] 2015-11-26 09:43:16 Apid 949450: initiated application termination > >> > Application 949450 exit codes: 134 > >> > Application 949450 exit signals: Killed > >> > Application 949450 resources: utime ~1s, stime ~137s, Rss ~2110448, inblocks ~617782, outblocks ~1659320 > >> > ____________________________ > >> > > >> > The following are some tests that I ran to better understand the problem: > >> > > >> > 2047 processes - 2 MB (4094 MB total) -> works! > >> > > >> > 2048 processes - 2047.5 KB (~1.999512 MB) (4095 MB total) -> works! > >> > > >> > 2048 processes - 3 MB (6144 MB total) -> fails: > >> > ____________________________ > >> > > >> > Rank 1024 [Thu Nov 26 09:41:15 2015] [c1-0c1s12n3] Fatal error in PMPI_Gather: Invalid count, error stack: > >> > PMPI_Gather(959)......: MPI_Gather(sbuf=0x2aaab826c010, scount=786432, MPI_INT, rbuf=(nil), rcount=786432, MPI_INT, root=0, MPI_COMM_WORLD) failed > >> > MPIR_Gather_impl(775).: > >> > MPIR_Gather(735)......: > >> > MPIR_Gather_intra(347): > >> > MPIC_Send(360)........: Negative count, value is -1073741824 > >> > _pmiu_daemon(SIGCHLD): [NID 00307] [c1-0c1s12n3] [Thu Nov 26 09:41:15 2015] PE RANK 1024 exit signal Aborted > >> > [NID 00307] 2015-11-26 09:41:15 Apid 949448: initiated application termination > >> > Application 949448 exit codes: 134 > >> > Application 949448 exit signals: Killed > >> > Application 949448 resources: utime ~1s, stime ~139s, Rss ~3159984, inblocks ~617782, outblocks ~1659351 > >> > ____________________________ > >> > > >> > 2047 processes - 3 MB (6141 MB total) -> fails: > >> > ____________________________ > >> > > >> > Rank 1024 [Thu Nov 26 09:40:31 2015] [c1-0c1s12n3] Fatal error in PMPI_Gather: Invalid count, error stack: > >> > PMPI_Gather(959)......: MPI_Gather(sbuf=0x2aaab826c010, scount=786432, MPI_INT, rbuf=(nil), rcount=786432, MPI_INT, root=0, MPI_COMM_WORLD) failed > >> > MPIR_Gather_impl(775).: > >> > MPIR_Gather(735)......: > >> > MPIR_Gather_intra(347): > >> > MPIC_Send(360)........: Negative count, value is -1076887552 > >> > _pmiu_daemon(SIGCHLD): [NID 00307] [c1-0c1s12n3] [Thu Nov 26 09:40:32 2015] PE RANK 1024 exit signal Aborted > >> > [NID 00307] 2015-11-26 09:40:32 Apid 949446: initiated application termination > >> > Application 949446 exit codes: 134 > >> > Application 949446 exit signals: Killed > >> > Application 949446 resources: utime ~1s, stime ~134s, Rss ~3157072, inblocks ~617780, outblocks ~1659351 > >> > ____________________________ > >> > > >> > 8 processes - 625 MB (5000 MB total) -> works! > >> > > >> > I can think of some pitfalls that might cause this issue but I do not have the knowledge of the internally called routines to check them. Is someone familier with the implementation of MPI_Gather(...) and willing to help me? > >> > > >> > Best regards > >> > > >> > Florian > >> > > >> > Deutsches Zentrum für Luft- und Raumfahrt e. V. (DLR) > >> > German Aerospace Center > >> > Institute of Planetary Research | Planetary Physics | Rutherfordstraße 2 | 12489 Berlin > >> > > >> > Florian Willich| Intern - Software Developer (Parallel Applications) > >> > florian.willlich(a)dlr.de > >> > DLR.de > >> > _______________________________________________ > >> > discuss mailing list discuss(a)mpich.org > >> > To manage subscription options or unsubscribe: > >> > https://lists.mpich.org/mailman/listinfo/discuss > >> > > >> > _______________________________________________ > >> > discuss mailing list discuss(a)mpich.org > >> > To manage subscription options or unsubscribe: > >> > https://lists.mpich.org/mailman/listinfo/discuss > >> > > >> > _______________________________________________ > >> > discuss mailing list discuss(a)mpich.org > >> > To manage subscription options or unsubscribe: > >> > https://lists.mpich.org/mailman/listinfo/discuss > >> > > >> > >> -- > >> Rob Latham > >> Mathematics and Computer Science Division > >> Argonne National Lab, IL USA > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > > > > > > > > > > -- > > Jeff Hammond > > jeff.science(a)gmail.com > > http://jeffhammond.github.io/ > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Building MPE for use with OpenMPI
by Schilling, Herbert W. (GRC-VE00) 02 Dec '15

02 Dec '15

Hi Bill, Thank you ! That worked perfectly. Will start to make use of MPE on our code tomorrow. On Dec 2, 2015, at 5:49 PM, William Gropp <wgropp(a)illinois.edu<mailto:[email protected]>> wrote: I’ve fixed this by testing for the bug in using const in the removed routines; in the process of testing with Open MPI, I found a few other bugs that I also fixed. I’ve pushed the fixes to mpe2.git, and created a new tarball, which is installed at the ftp site (with link to mpe2.tar.gz). Bill William Gropp Director, Parallel Computing Institute Thomas M. Siebel Chair in Computer Science Chief Scientist, NCSA University of Illinois Urbana-Champaign On Dec 2, 2015, at 9:17 AM, Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> wrote: But you are right that a configure test could be useful to make MPE work with MPICH releases that have the unnecessary const attributes in MPI_Type_hindexed, MPI_Type_struct, MPI_Address, PMPI_Type_hindexed, PMPI_Type_struct, PMPI_Address. Best, Jeff On Wed, Dec 2, 2015 at 7:16 AM, Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> wrote: Hi Bill, My original idea was wrong. Since those deprecated MPI-2 functions should never have const in the declaration, the right solution is to remove MPE_CONST from them. The macro logic is fine as-is as long as an MPI implementation only adds const to MPI-3 functions. The attached patch implements the aforementioned change. Of course, it must be coupled to my patch to MPICH for the function signatures to align. I don't have write privileges on the MPE repo, not that it is critical. Best, Jeff On Wed, Dec 2, 2015 at 7:08 AM, William Gropp <wgropp(a)illinois.edu<mailto:[email protected]>> wrote: It sounds like this one needs a special case for the current MPICH bug. That isn’t hard and I can do that, Jeff, if you don’t want to. Let me know. It will need a configure test for the suspect cases. Bill William Gropp Director, Parallel Computing Institute Thomas M. Siebel Chair in Computer Science Chief Scientist, NCSA University of Illinois Urbana-Champaign On Dec 1, 2015, at 10:42 PM, Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> wrote: Ok, I didn't understand your issue before. I thought your issue was with MPICH not agreeing with OpenMPI. That needed to be fixed anyways, so it's good that your query prompted it, but I'm sorry it wasn't what you needed. MPE already has MPE_CONST, but it will be "const" whenever MPI_VERSION>=3, which is what late-model MPICH will give you. The solution is probably to the logic below to allow for manual override, rather than OR-ing on MPI_VERSION, which cannot be overridden. #if defined(HAVE_CONST_IN_MPI_DECL) || MPI_VERSION >= 3 #define MPE_CONST const #else #define MPE_CONST #endif I'll try to work up a patch later. As for configure, the documentation is rightly written for the user who downloads a release tarball, not the git repo. So yes, you should generate it with autogen.sh. Jeff On Tue, Dec 1, 2015 at 7:04 PM, Schilling, Herbert W. (GRC-VE00) <hschilling(a)nasa.gov<mailto:[email protected]>> wrote: > > Hi Jeff, > > Thanks. Unfortunately, that is basically the same source code from > > ftp://ftp.mcs.anl.gov/pub/mpi/mpe/mpe2.tar.gz<https://urldefense.proofpoint.com/v2/url?u=ftp-3A__ftp.mcs.anl.gov_pub_mpi_…> > > where I had the issues with the conflicting declarations as Pavan talked about. > > I could remove the deprecated routines as most of the problems are caused by them, but it looks like some non-deprecated routines also have the const issue, e.g. > > visual_mess.c:560: error: conflicting types for 'MPI_Issend' > /u/hschilli/local/include/mpi.h:1543: error: previous declaration of 'MPI_Issend' was here > > > BTW, that repo for MPE doesn't appear to have a configure file. I know I could create it with autogen.sh but the INSTALL file doesn't say that. It just says run configure. Might be confusing for some people, I think! > > Thanks for everyone's help > > > > > > On Dec 1, 2015, at 2:03 PM, Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> > wrote: > > MPE lives here now: http://git.mpich.org/mpe.git/<https://urldefense.proofpoint.com/v2/url?u=http-3A__git.mpich.org_mpe.git_&…> > > Jeff > > On Tue, Dec 1, 2015 at 10:15 AM, Schilling, Herbert W. (GRC-VE00) <hschilling(a)nasa.gov<mailto:[email protected]>> wrote: >> >> Hi Jeff, >> >> Thanks. Pardon my ignorance but I I wanted to just use MPE with an existing OpenMPI installation and it's not clear to me ( even after searching and reading ), how to do that from the full MPICH code base at that repo. I can't really even find MPE in that code at all. What am I missing? >> >> Thanks >> >> On Dec 1, 2015, at 10:03 AM, Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> >> wrote: >> >> Herbert: >> >> Please try https://github.com/jeffhammond/mpich/tree/hammond-mpi2-deprecated-functions…<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_jeffhammond…> and let me know if you see any problems. >> >> Best, >> >> Jeff >> >> On Tue, Dec 1, 2015 at 5:46 AM, Schilling, Herbert W. (GRC-VE00) <hschilling(a)nasa.gov<mailto:[email protected]>> wrote: >>> >>> Hi Jeff and Pavan, >>> >>> Thanks for all the help on this. Jeff, when you have the fix up there, I can give it a shot! >>> >>> >>> On Dec 1, 2015, at 1:15 AM, Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> >>> wrote: >>> >>> Facepalm. Indeed, changing the header but not the implementation was pretty dumb on my part. I'll fix that tomorrow. >>> >>> I have no opinion on the ABI situation, particularly since I know there are other folks that Intel that do. >>> >>> Jeff >>> >>> On Mon, Nov 30, 2015 at 9:00 PM, Balaji, Pavan <balaji(a)anl.gov<mailto:[email protected]>> wrote: >>>> >>>> >>>> Thanks, Jeff. The actual function definitions should be modified as well, which your patch seems to miss. >>>> >>>> But I think the potentially time consuming part is to get buy-in from all the ABI consortium folks for this change. It doesn't exactly break ABI for applications, but will be an annoying change for tools. Ken is looking into this. >>>> >>>> -- Pavan >>>> >>>> > On Nov 30, 2015, at 10:54 PM, Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> wrote: >>>> > >>>> > Pavan: >>>> > >>>> > See hammond-mpi2-deprecated-functions-noconst branch on mpich-review. I found three functions where this change should be made. I do not know if that is the full list, so someone should verify. >>>> > >>>> > Best, >>>> > >>>> > Jeff >>>> > >>>> > On Mon, Nov 30, 2015 at 8:08 PM, Balaji, Pavan <balaji(a)anl.gov<mailto:[email protected]>> wrote: >>>> > >>>> > MPI_Type_struct is not a part of the MPI standard anymore and is replaced by MPI_Type_create_struct. Both MPICH and Open MPI decided to retain it, but followed different directions for it. MPICH decided to follow the same convention as MPI_Type_create_struct with the const additions. Open MPI decided to keep the old prototypes as is. >>>> > >>>> > Unfortunately, MPE got stuck in between the two. The best approach would be to simply remove those functions from MPE and this problem just goes away. Would you be able to do that and contribute a patch? >>>> > >>>> > More generally, I think MPICH was overzealous in trying to constify prototypes for the deleted functions. It should have just left them as they are, like what Open MPI did. We'll figure out what to do about that internally, but that's not going to be a quick change because of ABI and other such stuff. >>>> > >>>> > Thanks, >>>> > >>>> > -- Pavan >>>> > >>>> > > On Nov 30, 2015, at 8:14 PM, Schilling, Herbert W. (GRC-VE00) <hschilling(a)nasa.gov<mailto:[email protected]>> wrote: >>>> > > >>>> > > Hi, >>>> > > >>>> > > I am trying to build MPE for use with OpenMPI. I built from source Version 1.10.1 of OpenMPI. >>>> > > >>>> > > I am using this version of MPE ftp://ftp.mcs.anl.gov/pub/mpi/mpe/mpe2.tar.gz<https://urldefense.proofpoint.com/v2/url?u=ftp-3A__ftp.mcs.anl.gov_pub_mpi_…> >>>> > > >>>> > > Should this work fine? I followed the instructions to build like this: >>>> > > >>>> > > ${MPE_SRC_DIR}/configure MPI_CC=${OPENMPI_INSTALL_DIR}/bin/mpicc \ >>>> > > CC=<C_COMPILER_used_to_build_OpenMPI> \ >>>> > > MPI_F77=${OPENMPI_INSTALL_DIR}/bin/mpif77 \ >>>> > > F77=<FORTRAN_COMPILER_used_to_build_OpenMPI> \ >>>> > > --with-java2=/usr/local/j2sdk-1.4.2 >>>> > > --prefix=<MPE_INSTALL_DIR> >>>> > > make >>>> > > make install >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > I keep running into problems with conflicting declarations >>>> > > >>>> > > >>>> > > log_mpi_core.c:5611: error: conflicting types for 'MPI_Type_struct' >>>> > > /u/hschilli/local/include/mpi.h:1796: error: previous declaration of 'MPI_Type_struct' was here >>>> > > >>>> > > >>>> > > I think the issue is that MPE's declaration uses some const qualifiers while OpenMPI does not. >>>> > > >>>> > > >>>> > > >>>> > > I tried hacking my way through this with the macro >>>> > > >>>> > > HAVE_CONST_IN_MPI_DECL >>>> > > and MPE_CONST but no luck so far. >>>> > > >>>> > > >>>> > > >>>> > > What am I doing wrong? >>>> > > >>>> > > >>>> > > Thanks in advance. >>>> > > >>>> > > --------- >>>> > > Herb Schilling >>>> > > Scientific Applications and Visualization Team >>>> > > http://ocio.grc.nasa.gov/gvis<https://urldefense.proofpoint.com/v2/url?u=http-3A__ocio.grc.nasa.gov_gvis&…> >>>> > > Office of the Chief Information Officer >>>> > > NASA Glenn Research Center >>>> > > hschilling(a)nasa.gov<mailto:[email protected]> >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > _______________________________________________ >>>> > > discuss mailing list discuss(a)mpich.org<mailto:[email protected]> >>>> > > To manage subscription options or unsubscribe: >>>> > > https://lists.mpich.org/mailman/listinfo/discuss<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> >>>> > >>>> > _______________________________________________ >>>> > discuss mailing list discuss(a)mpich.org<mailto:[email protected]> >>>> > To manage subscription options or unsubscribe: >>>> > https://lists.mpich.org/mailman/listinfo/discuss<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> >>>> > >>>> > >>>> > >>>> > -- >>>> > Jeff Hammond >>>> > jeff.science(a)gmail.com<mailto:[email protected]> >>>> > http://jeffhammond.github.io/<https://urldefense.proofpoint.com/v2/url?u=http-3A__jeffhammond.github.io_&…> >>>> > _______________________________________________ >>>> > discuss mailing list discuss(a)mpich.org<mailto:[email protected]> >>>> > To manage subscription options or unsubscribe: >>>> > https://lists.mpich.org/mailman/listinfo/discuss<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org<mailto:[email protected]> >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> >>> >>> >>> >>> >>> -- >>> Jeff Hammond >>> jeff.science(a)gmail.com<mailto:[email protected]> >>> http://jeffhammond.github.io/<https://urldefense.proofpoint.com/v2/url?u=http-3A__jeffhammond.github.io_&…> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org<mailto:[email protected]> >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> >>> >>> >>> --------- >>> Herb Schilling >>> Scientific Applications and Visualization Team >>> http://ocio.grc.nasa.gov/gvis<https://urldefense.proofpoint.com/v2/url?u=http-3A__ocio.grc.nasa.gov_gvis&…> >>> Office of the Chief Information Officer >>> NASA Glenn Research Center >>> hschilling(a)nasa.gov<mailto:[email protected]> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org<mailto:[email protected]> >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> >> >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com<mailto:[email protected]> >> http://jeffhammond.github.io/<https://urldefense.proofpoint.com/v2/url?u=http-3A__jeffhammond.github.io_&…> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org<mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> >> >> >> --------- >> Herb Schilling >> Scientific Applications and Visualization Team >> http://ocio.grc.nasa.gov/gvis<https://urldefense.proofpoint.com/v2/url?u=http-3A__ocio.grc.nasa.gov_gvis&…> >> Office of the Chief Information Officer >> NASA Glenn Research Center >> hschilling(a)nasa.gov<mailto:[email protected]> >> >> >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org<mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> > > > > > -- > Jeff Hammond > jeff.science(a)gmail.com<mailto:[email protected]> > http://jeffhammond.github.io/<https://urldefense.proofpoint.com/v2/url?u=http-3A__jeffhammond.github.io_&…> > _______________________________________________ > discuss mailing list discuss(a)mpich.org<mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> > > > --------- > Herb Schilling > Scientific Applications and Visualization Team > http://ocio.grc.nasa.gov/gvis<https://urldefense.proofpoint.com/v2/url?u=http-3A__ocio.grc.nasa.gov_gvis&…> > Office of the Chief Information Officer > NASA Glenn Research Center > hschilling(a)nasa.gov<mailto:[email protected]> > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org<mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> http://jeffhammond.github.io/<https://urldefense.proofpoint.com/v2/url?u=http-3A__jeffhammond.github.io_&…> _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> http://jeffhammond.github.io/<https://urldefense.proofpoint.com/v2/url?u=http-3A__jeffhammond.github.io_&…> -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> http://jeffhammond.github.io/<https://urldefense.proofpoint.com/v2/url?u=http-3A__jeffhammond.github.io_&…> _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss --------- Herb Schilling Scientific Applications and Visualization Team http://ocio.grc.nasa.gov/gvis Office of the Chief Information Officer NASA Glenn Research Center hschilling(a)nasa.gov<mailto:[email protected]> _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Building MPE for use with OpenMPI
by William Gropp 02 Dec '15

02 Dec '15

I’ve fixed this by testing for the bug in using const in the removed routines; in the process of testing with Open MPI, I found a few other bugs that I also fixed. I’ve pushed the fixes to mpe2.git, and created a new tarball, which is installed at the ftp site (with link to mpe2.tar.gz). Bill William Gropp Director, Parallel Computing Institute Thomas M. Siebel Chair in Computer Science Chief Scientist, NCSA University of Illinois Urbana-Champaign On Dec 2, 2015, at 9:17 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > But you are right that a configure test could be useful to make MPE work with MPICH releases that have the unnecessary const attributes in MPI_Type_hindexed, MPI_Type_struct, MPI_Address, PMPI_Type_hindexed, PMPI_Type_struct, PMPI_Address. > > Best, > > Jeff > > On Wed, Dec 2, 2015 at 7:16 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > Hi Bill, > > My original idea was wrong. Since those deprecated MPI-2 functions should never have const in the declaration, the right solution is to remove MPE_CONST from them. The macro logic is fine as-is as long as an MPI implementation only adds const to MPI-3 functions. > > The attached patch implements the aforementioned change. Of course, it must be coupled to my patch to MPICH for the function signatures to align. > > I don't have write privileges on the MPE repo, not that it is critical. > > Best, > > Jeff > > On Wed, Dec 2, 2015 at 7:08 AM, William Gropp <wgropp(a)illinois.edu> wrote: > It sounds like this one needs a special case for the current MPICH bug. That isn’t hard and I can do that, Jeff, if you don’t want to. Let me know. It will need a configure test for the suspect cases. > > Bill > > William Gropp > Director, Parallel Computing Institute > Thomas M. Siebel Chair in Computer Science > Chief Scientist, NCSA > University of Illinois Urbana-Champaign > > > > > > On Dec 1, 2015, at 10:42 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > >> Ok, I didn't understand your issue before. I thought your issue was with MPICH not agreeing with OpenMPI. That needed to be fixed anyways, so it's good that your query prompted it, but I'm sorry it wasn't what you needed. >> >> MPE already has MPE_CONST, but it will be "const" whenever MPI_VERSION>=3, which is what late-model MPICH will give you. The solution is probably to the logic below to allow for manual override, rather than OR-ing on MPI_VERSION, which cannot be overridden. >> >> #if defined(HAVE_CONST_IN_MPI_DECL) || MPI_VERSION >= 3 >> #define MPE_CONST const >> #else >> #define MPE_CONST >> #endif >> >> I'll try to work up a patch later. >> >> As for configure, the documentation is rightly written for the user who downloads a release tarball, not the git repo. So yes, you should generate it with autogen.sh. >> >> Jeff >> >> On Tue, Dec 1, 2015 at 7:04 PM, Schilling, Herbert W. (GRC-VE00) <hschilling(a)nasa.gov> wrote: >> > >> > Hi Jeff, >> > >> > Thanks. Unfortunately, that is basically the same source code from >> > >> > ftp://ftp.mcs.anl.gov/pub/mpi/mpe/mpe2.tar.gz >> > >> > where I had the issues with the conflicting declarations as Pavan talked about. >> > >> > I could remove the deprecated routines as most of the problems are caused by them, but it looks like some non-deprecated routines also have the const issue, e.g. >> > >> > visual_mess.c:560: error: conflicting types for 'MPI_Issend' >> > /u/hschilli/local/include/mpi.h:1543: error: previous declaration of 'MPI_Issend' was here >> > >> > >> > BTW, that repo for MPE doesn't appear to have a configure file. I know I could create it with autogen.sh but the INSTALL file doesn't say that. It just says run configure. Might be confusing for some people, I think! >> > >> > Thanks for everyone's help >> > >> > >> > >> > >> > >> > On Dec 1, 2015, at 2:03 PM, Jeff Hammond <jeff.science(a)gmail.com> >> > wrote: >> > >> > MPE lives here now: http://git.mpich.org/mpe.git/ >> > >> > Jeff >> > >> > On Tue, Dec 1, 2015 at 10:15 AM, Schilling, Herbert W. (GRC-VE00) <hschilling(a)nasa.gov> wrote: >> >> >> >> Hi Jeff, >> >> >> >> Thanks. Pardon my ignorance but I I wanted to just use MPE with an existing OpenMPI installation and it's not clear to me ( even after searching and reading ), how to do that from the full MPICH code base at that repo. I can't really even find MPE in that code at all. What am I missing? >> >> >> >> Thanks >> >> >> >> On Dec 1, 2015, at 10:03 AM, Jeff Hammond <jeff.science(a)gmail.com> >> >> wrote: >> >> >> >> Herbert: >> >> >> >> Please try https://github.com/jeffhammond/mpich/tree/hammond-mpi2-deprecated-functions… and let me know if you see any problems. >> >> >> >> Best, >> >> >> >> Jeff >> >> >> >> On Tue, Dec 1, 2015 at 5:46 AM, Schilling, Herbert W. (GRC-VE00) <hschilling(a)nasa.gov> wrote: >> >>> >> >>> Hi Jeff and Pavan, >> >>> >> >>> Thanks for all the help on this. Jeff, when you have the fix up there, I can give it a shot! >> >>> >> >>> >> >>> On Dec 1, 2015, at 1:15 AM, Jeff Hammond <jeff.science(a)gmail.com> >> >>> wrote: >> >>> >> >>> Facepalm. Indeed, changing the header but not the implementation was pretty dumb on my part. I'll fix that tomorrow. >> >>> >> >>> I have no opinion on the ABI situation, particularly since I know there are other folks that Intel that do. >> >>> >> >>> Jeff >> >>> >> >>> On Mon, Nov 30, 2015 at 9:00 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: >> >>>> >> >>>> >> >>>> Thanks, Jeff. The actual function definitions should be modified as well, which your patch seems to miss. >> >>>> >> >>>> But I think the potentially time consuming part is to get buy-in from all the ABI consortium folks for this change. It doesn't exactly break ABI for applications, but will be an annoying change for tools. Ken is looking into this. >> >>>> >> >>>> -- Pavan >> >>>> >> >>>> > On Nov 30, 2015, at 10:54 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> >>>> > >> >>>> > Pavan: >> >>>> > >> >>>> > See hammond-mpi2-deprecated-functions-noconst branch on mpich-review. I found three functions where this change should be made. I do not know if that is the full list, so someone should verify. >> >>>> > >> >>>> > Best, >> >>>> > >> >>>> > Jeff >> >>>> > >> >>>> > On Mon, Nov 30, 2015 at 8:08 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: >> >>>> > >> >>>> > MPI_Type_struct is not a part of the MPI standard anymore and is replaced by MPI_Type_create_struct. Both MPICH and Open MPI decided to retain it, but followed different directions for it. MPICH decided to follow the same convention as MPI_Type_create_struct with the const additions. Open MPI decided to keep the old prototypes as is. >> >>>> > >> >>>> > Unfortunately, MPE got stuck in between the two. The best approach would be to simply remove those functions from MPE and this problem just goes away. Would you be able to do that and contribute a patch? >> >>>> > >> >>>> > More generally, I think MPICH was overzealous in trying to constify prototypes for the deleted functions. It should have just left them as they are, like what Open MPI did. We'll figure out what to do about that internally, but that's not going to be a quick change because of ABI and other such stuff. >> >>>> > >> >>>> > Thanks, >> >>>> > >> >>>> > -- Pavan >> >>>> > >> >>>> > > On Nov 30, 2015, at 8:14 PM, Schilling, Herbert W. (GRC-VE00) <hschilling(a)nasa.gov> wrote: >> >>>> > > >> >>>> > > Hi, >> >>>> > > >> >>>> > > I am trying to build MPE for use with OpenMPI. I built from source Version 1.10.1 of OpenMPI. >> >>>> > > >> >>>> > > I am using this version of MPE ftp://ftp.mcs.anl.gov/pub/mpi/mpe/mpe2.tar.gz >> >>>> > > >> >>>> > > Should this work fine? I followed the instructions to build like this: >> >>>> > > >> >>>> > > ${MPE_SRC_DIR}/configure MPI_CC=${OPENMPI_INSTALL_DIR}/bin/mpicc \ >> >>>> > > CC=<C_COMPILER_used_to_build_OpenMPI> \ >> >>>> > > MPI_F77=${OPENMPI_INSTALL_DIR}/bin/mpif77 \ >> >>>> > > F77=<FORTRAN_COMPILER_used_to_build_OpenMPI> \ >> >>>> > > --with-java2=/usr/local/j2sdk-1.4.2 >> >>>> > > --prefix=<MPE_INSTALL_DIR> >> >>>> > > make >> >>>> > > make install >> >>>> > > >> >>>> > > >> >>>> > > >> >>>> > > >> >>>> > > I keep running into problems with conflicting declarations >> >>>> > > >> >>>> > > >> >>>> > > log_mpi_core.c:5611: error: conflicting types for 'MPI_Type_struct' >> >>>> > > /u/hschilli/local/include/mpi.h:1796: error: previous declaration of 'MPI_Type_struct' was here >> >>>> > > >> >>>> > > >> >>>> > > I think the issue is that MPE's declaration uses some const qualifiers while OpenMPI does not. >> >>>> > > >> >>>> > > >> >>>> > > >> >>>> > > I tried hacking my way through this with the macro >> >>>> > > >> >>>> > > HAVE_CONST_IN_MPI_DECL >> >>>> > > and MPE_CONST but no luck so far. >> >>>> > > >> >>>> > > >> >>>> > > >> >>>> > > What am I doing wrong? >> >>>> > > >> >>>> > > >> >>>> > > Thanks in advance. >> >>>> > > >> >>>> > > --------- >> >>>> > > Herb Schilling >> >>>> > > Scientific Applications and Visualization Team >> >>>> > > http://ocio.grc.nasa.gov/gvis >> >>>> > > Office of the Chief Information Officer >> >>>> > > NASA Glenn Research Center >> >>>> > > hschilling(a)nasa.gov >> >>>> > > >> >>>> > > >> >>>> > > >> >>>> > > >> >>>> > > _______________________________________________ >> >>>> > > discuss mailing list discuss(a)mpich.org >> >>>> > > To manage subscription options or unsubscribe: >> >>>> > > https://lists.mpich.org/mailman/listinfo/discuss >> >>>> > >> >>>> > _______________________________________________ >> >>>> > discuss mailing list discuss(a)mpich.org >> >>>> > To manage subscription options or unsubscribe: >> >>>> > https://lists.mpich.org/mailman/listinfo/discuss >> >>>> > >> >>>> > >> >>>> > >> >>>> > -- >> >>>> > Jeff Hammond >> >>>> > jeff.science(a)gmail.com >> >>>> > http://jeffhammond.github.io/ >> >>>> > _______________________________________________ >> >>>> > discuss mailing list discuss(a)mpich.org >> >>>> > To manage subscription options or unsubscribe: >> >>>> > https://lists.mpich.org/mailman/listinfo/discuss >> >>>> >> >>>> _______________________________________________ >> >>>> discuss mailing list discuss(a)mpich.org >> >>>> To manage subscription options or unsubscribe: >> >>>> https://lists.mpich.org/mailman/listinfo/discuss >> >>> >> >>> >> >>> >> >>> >> >>> -- >> >>> Jeff Hammond >> >>> jeff.science(a)gmail.com >> >>> http://jeffhammond.github.io/ >> >>> _______________________________________________ >> >>> discuss mailing list discuss(a)mpich.org >> >>> To manage subscription options or unsubscribe: >> >>> https://lists.mpich.org/mailman/listinfo/discuss >> >>> >> >>> >> >>> --------- >> >>> Herb Schilling >> >>> Scientific Applications and Visualization Team >> >>> http://ocio.grc.nasa.gov/gvis >> >>> Office of the Chief Information Officer >> >>> NASA Glenn Research Center >> >>> hschilling(a)nasa.gov >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> _______________________________________________ >> >>> discuss mailing list discuss(a)mpich.org >> >>> To manage subscription options or unsubscribe: >> >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> >> >> >> >> >> >> -- >> >> Jeff Hammond >> >> jeff.science(a)gmail.com >> >> http://jeffhammond.github.io/ >> >> _______________________________________________ >> >> discuss mailing list discuss(a)mpich.org >> >> To manage subscription options or unsubscribe: >> >> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> >> >> --------- >> >> Herb Schilling >> >> Scientific Applications and Visualization Team >> >> http://ocio.grc.nasa.gov/gvis >> >> Office of the Chief Information Officer >> >> NASA Glenn Research Center >> >> hschilling(a)nasa.gov >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> discuss mailing list discuss(a)mpich.org >> >> To manage subscription options or unsubscribe: >> >> https://lists.mpich.org/mailman/listinfo/discuss >> > >> > >> > >> > >> > -- >> > Jeff Hammond >> > jeff.science(a)gmail.com >> > http://jeffhammond.github.io/ >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> > >> > >> > --------- >> > Herb Schilling >> > Scientific Applications and Visualization Team >> > http://ocio.grc.nasa.gov/gvis >> > Office of the Chief Information Officer >> > NASA Glenn Research Center >> > hschilling(a)nasa.gov >> > >> > >> > >> > >> > >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> http://jeffhammond.github.io/ >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_Gather fails with 2048 processes and 4096 MB total
by Balaji, Pavan 02 Dec '15

02 Dec '15

I believe Cray has a newer version based on mpich-3.2 that they are releasing soon-ish. If you are a good customer, they might give you an engineering build to try out. :-) -- Pavan > On Dec 2, 2015, at 9:28 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > This was fixed after Cray forked MPICH 3.1.2. You should file a support issue with Cray via the appropriate channel. > > # master > jrhammon-mac01:git jrhammon$ git grep MPIC_Send > src/include/mpiimpl.h:int MPIC_Send(const void *buf, MPI_Aint count, MPI_Datatype datatype, int dest, int tag, > ... > > # 3.1 branch > jrhammon-mac01:git jrhammon$ git checkout 3.1.x > jrhammon-mac01:git jrhammon$ git grep MPIC_Send > src/include/mpiimpl.h:int MPIC_Send(const void *buf, int count, MPI_Datatype datatype, int dest, int tag, > ... > > Best, > > Jeff > > On Wed, Dec 2, 2015 at 6:40 AM, <Florian.Willich(a)dlr.de> wrote: > > > > Thanks for the "bash line", the output is: > > > > $ grep "define MPICH_VERSION" $CRAY_MPICH2_DIR/include/mpi.h > > #define MPICH_VERSION "3.1.2" > > > > Florian > > > > ________________________________ > > Von: Jeff Hammond [jeff.science(a)gmail.com] > > Gesendet: Mittwoch, 2. Dezember 2015 14:58 > > An: MPICH > > > > Betreff: Re: [mpich-discuss] MPI_Gather fails with 2048 processes and 4096 MB total > > > > 1) You should address this with Cray. Their implementation of MPI has closed-source modifications to MPICH that we cannot analyze for count-safety. > > > > 2) You can discover the version of MPICH associated with Cray MPI like this: > > jhammond@cori11:~> grep "define MPICH_VERSION" $CRAY_MPICH2_DIR/include/mpi.h > > #define MPICH_VERSION "3.1.2" > > > > Best, > > > > Jeff > > > > On Wed, Dec 2, 2015 at 12:15 AM, <Florian.Willich(a)dlr.de> wrote: > >> > >> Hi Rob, > >> > >> well maybe I was addressing the wrong organisation... I am currently testing on the Cray swan super computer which provides the module cray-mpich/7.2.6 ("Cray Message Passing Toolkit 7.2.6"). > >> > >> I can not determine whether the cray mpich version is mpich with additional implementations or if it is totally different from the official mpich releases. Additioanlly, I can not figure out on which mpich version this cray-mpich module is based on. I'll continue investigation and keep you updated. > >> > >> Best Regards > >> > >> Florian > >> ________________________________________ > >> Von: Rob Latham [robl(a)mcs.anl.gov] > >> Gesendet: Dienstag, 1. Dezember 2015 16:48 > >> An: discuss(a)mpich.org > >> Betreff: Re: [mpich-discuss] MPI_Gather fails with 2048 processes and 4096 MB total > >> > >> On 11/26/2015 12:38 PM, Archer, Charles J wrote: > >> > FYI, we hit various flavors of this problem when I was still at IBM, I think mostly in weather codes. > >> > Apparently Cray hit this too: > >> > > >> > https://trac.mpich.org/projects/mpich/ticket/1767 > >> > > >> > We pretty much told our customers back then that a fix was forthcoming (with no ETA :) )with the revamp of datatypes to use internal 64-bit counts. > >> > We also provided workarounds. > >> > > >> > In the case of this gather operation, we asked the customer to implement gather as a flat tree using point to point. > >> > Root posts irecvs, then barrier, children send to root. > >> > > >> > IRC, the giant gather we were debugging was at the very end of the application and used to gather some statistics for IO at the root, so it wasn’t critical to perform well. > >> > I also attempted a workaround using some derived datatypes, but I hit another truncation in the datatype code itself :\ > >> > I should see if I can dig up that implementation and make sure it isn’t still broken for large counts. > >> > >> those are all fine approaches to work around the problem. the internals > >> of MPICH, though, need to be 64 bit clean -- there are still 4500 places > >> where clang warns of a 64 bit value being assigned to a 32 bit type. > >> > >> Florian Willich, what version of MPICH is this? The line numbers in > >> the back trace don't match up with what I've got, and > >> I really thought we fixed this class of bug with commits 31d95ed7b18c > >> and 68f8c7aa7 over the summer. > >> > >> ==rob > >> > >> > >> > > >> > > >> > > >> > > >> > > >> > On Nov 26, 2015, at 12:14 PM, Balaji, Pavan <balaji(a)anl.gov<mailto:[email protected]>> wrote: > >> > > >> > > >> > Thanks for reporting. This looks like an integer-overflow issue, which fails when the summation of data elements from all processes is larger than INT_MAX (2 billion). We'll look into it. I've created a ticket for it, and added you as the reporter, so you'll get notified as they are updates. > >> > > >> > http://trac.mpich.org/projects/mpich/ticket/2317 > >> > > >> > Rob: can you create a simple test program for this and add it to the test bucket, so it shows up on the nightlies? > >> > > >> > Thanks, > >> > > >> > -- Pavan > >> > > >> > On Nov 26, 2015, at 10:18 AM, Florian.Willich(a)dlr.de wrote: > >> > > >> > Dear mpich discussion group, > >> > > >> > the following issue appeared when running some benchmarks with MPI Gather: > >> > > >> > Gathering data (calling MPI_Gather(...) ) involing 2048 processes and 2 MB of data (4096 MB total) that each process sends fails with the following output: > >> > ____________________________ > >> > > >> > Rank 1024 [Thu Nov 26 09:43:16 2015] [c1-0c1s12n3] Fatal error in PMPI_Gather: Invalid count, error stack: > >> > PMPI_Gather(959)......: MPI_Gather(sbuf=0x2aaab826c010, scount=524288, MPI_INT, rbuf=(nil), rcount=524288, MPI_INT, root=0, MPI_COMM_WORLD) failed > >> > MPIR_Gather_impl(775).: > >> > MPIR_Gather(735)......: > >> > MPIR_Gather_intra(347): > >> > MPIC_Send(360)........: Negative count, value is -2147483648 > >> > _pmiu_daemon(SIGCHLD): [NID 00307] [c1-0c1s12n3] [Thu Nov 26 09:43:16 2015] PE RANK 1024 exit signal Aborted > >> > [NID 00307] 2015-11-26 09:43:16 Apid 949450: initiated application termination > >> > Application 949450 exit codes: 134 > >> > Application 949450 exit signals: Killed > >> > Application 949450 resources: utime ~1s, stime ~137s, Rss ~2110448, inblocks ~617782, outblocks ~1659320 > >> > ____________________________ > >> > > >> > The following are some tests that I ran to better understand the problem: > >> > > >> > 2047 processes - 2 MB (4094 MB total) -> works! > >> > > >> > 2048 processes - 2047.5 KB (~1.999512 MB) (4095 MB total) -> works! > >> > > >> > 2048 processes - 3 MB (6144 MB total) -> fails: > >> > ____________________________ > >> > > >> > Rank 1024 [Thu Nov 26 09:41:15 2015] [c1-0c1s12n3] Fatal error in PMPI_Gather: Invalid count, error stack: > >> > PMPI_Gather(959)......: MPI_Gather(sbuf=0x2aaab826c010, scount=786432, MPI_INT, rbuf=(nil), rcount=786432, MPI_INT, root=0, MPI_COMM_WORLD) failed > >> > MPIR_Gather_impl(775).: > >> > MPIR_Gather(735)......: > >> > MPIR_Gather_intra(347): > >> > MPIC_Send(360)........: Negative count, value is -1073741824 > >> > _pmiu_daemon(SIGCHLD): [NID 00307] [c1-0c1s12n3] [Thu Nov 26 09:41:15 2015] PE RANK 1024 exit signal Aborted > >> > [NID 00307] 2015-11-26 09:41:15 Apid 949448: initiated application termination > >> > Application 949448 exit codes: 134 > >> > Application 949448 exit signals: Killed > >> > Application 949448 resources: utime ~1s, stime ~139s, Rss ~3159984, inblocks ~617782, outblocks ~1659351 > >> > ____________________________ > >> > > >> > 2047 processes - 3 MB (6141 MB total) -> fails: > >> > ____________________________ > >> > > >> > Rank 1024 [Thu Nov 26 09:40:31 2015] [c1-0c1s12n3] Fatal error in PMPI_Gather: Invalid count, error stack: > >> > PMPI_Gather(959)......: MPI_Gather(sbuf=0x2aaab826c010, scount=786432, MPI_INT, rbuf=(nil), rcount=786432, MPI_INT, root=0, MPI_COMM_WORLD) failed > >> > MPIR_Gather_impl(775).: > >> > MPIR_Gather(735)......: > >> > MPIR_Gather_intra(347): > >> > MPIC_Send(360)........: Negative count, value is -1076887552 > >> > _pmiu_daemon(SIGCHLD): [NID 00307] [c1-0c1s12n3] [Thu Nov 26 09:40:32 2015] PE RANK 1024 exit signal Aborted > >> > [NID 00307] 2015-11-26 09:40:32 Apid 949446: initiated application termination > >> > Application 949446 exit codes: 134 > >> > Application 949446 exit signals: Killed > >> > Application 949446 resources: utime ~1s, stime ~134s, Rss ~3157072, inblocks ~617780, outblocks ~1659351 > >> > ____________________________ > >> > > >> > 8 processes - 625 MB (5000 MB total) -> works! > >> > > >> > I can think of some pitfalls that might cause this issue but I do not have the knowledge of the internally called routines to check them. Is someone familier with the implementation of MPI_Gather(...) and willing to help me? > >> > > >> > Best regards > >> > > >> > Florian > >> > > >> > Deutsches Zentrum für Luft- und Raumfahrt e. V. (DLR) > >> > German Aerospace Center > >> > Institute of Planetary Research | Planetary Physics | Rutherfordstraße 2 | 12489 Berlin > >> > > >> > Florian Willich| Intern - Software Developer (Parallel Applications) > >> > florian.willlich(a)dlr.de > >> > DLR.de > >> > _______________________________________________ > >> > discuss mailing list discuss(a)mpich.org > >> > To manage subscription options or unsubscribe: > >> > https://lists.mpich.org/mailman/listinfo/discuss > >> > > >> > _______________________________________________ > >> > discuss mailing list discuss(a)mpich.org > >> > To manage subscription options or unsubscribe: > >> > https://lists.mpich.org/mailman/listinfo/discuss > >> > > >> > _______________________________________________ > >> > discuss mailing list discuss(a)mpich.org > >> > To manage subscription options or unsubscribe: > >> > https://lists.mpich.org/mailman/listinfo/discuss > >> > > >> > >> -- > >> Rob Latham > >> Mathematics and Computer Science Division > >> Argonne National Lab, IL USA > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > > > > > > > > > > -- > > Jeff Hammond > > jeff.science(a)gmail.com > > http://jeffhammond.github.io/ > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Building MPE for use with OpenMPI
by Schilling, Herbert W. (GRC-VE00) 02 Dec '15

02 Dec '15

Hi, That's fine. I appreciate all the help. Thanks for the explanation about configure. Makes sense! On Dec 1, 2015, at 11:42 PM, Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> wrote: Ok, I didn't understand your issue before. I thought your issue was with MPICH not agreeing with OpenMPI. That needed to be fixed anyways, so it's good that your query prompted it, but I'm sorry it wasn't what you needed. MPE already has MPE_CONST, but it will be "const" whenever MPI_VERSION>=3, which is what late-model MPICH will give you. The solution is probably to the logic below to allow for manual override, rather than OR-ing on MPI_VERSION, which cannot be overridden. #if defined(HAVE_CONST_IN_MPI_DECL) || MPI_VERSION >= 3 #define MPE_CONST const #else #define MPE_CONST #endif I'll try to work up a patch later. As for configure, the documentation is rightly written for the user who downloads a release tarball, not the git repo. So yes, you should generate it with autogen.sh. Jeff On Tue, Dec 1, 2015 at 7:04 PM, Schilling, Herbert W. (GRC-VE00) <hschilling(a)nasa.gov<mailto:[email protected]>> wrote: > > Hi Jeff, > > Thanks. Unfortunately, that is basically the same source code from > > ftp://ftp.mcs.anl.gov/pub/mpi/mpe/mpe2.tar.gz > > where I had the issues with the conflicting declarations as Pavan talked about. > > I could remove the deprecated routines as most of the problems are caused by them, but it looks like some non-deprecated routines also have the const issue, e.g. > > visual_mess.c:560: error: conflicting types for 'MPI_Issend' > /u/hschilli/local/include/mpi.h:1543: error: previous declaration of 'MPI_Issend' was here > > > BTW, that repo for MPE doesn't appear to have a configure file. I know I could create it with autogen.sh but the INSTALL file doesn't say that. It just says run configure. Might be confusing for some people, I think! > > Thanks for everyone's help > > > > > > On Dec 1, 2015, at 2:03 PM, Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> > wrote: > > MPE lives here now: http://git.mpich.org/mpe.git/ > > Jeff > > On Tue, Dec 1, 2015 at 10:15 AM, Schilling, Herbert W. (GRC-VE00) <hschilling(a)nasa.gov<mailto:[email protected]>> wrote: >> >> Hi Jeff, >> >> Thanks. Pardon my ignorance but I I wanted to just use MPE with an existing OpenMPI installation and it's not clear to me ( even after searching and reading ), how to do that from the full MPICH code base at that repo. I can't really even find MPE in that code at all. What am I missing? >> >> Thanks >> >> On Dec 1, 2015, at 10:03 AM, Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> >> wrote: >> >> Herbert: >> >> Please try https://github.com/jeffhammond/mpich/tree/hammond-mpi2-deprecated-functions… and let me know if you see any problems. >> >> Best, >> >> Jeff >> >> On Tue, Dec 1, 2015 at 5:46 AM, Schilling, Herbert W. (GRC-VE00) <hschilling(a)nasa.gov<mailto:[email protected]>> wrote: >>> >>> Hi Jeff and Pavan, >>> >>> Thanks for all the help on this. Jeff, when you have the fix up there, I can give it a shot! >>> >>> >>> On Dec 1, 2015, at 1:15 AM, Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> >>> wrote: >>> >>> Facepalm. Indeed, changing the header but not the implementation was pretty dumb on my part. I'll fix that tomorrow. >>> >>> I have no opinion on the ABI situation, particularly since I know there are other folks that Intel that do. >>> >>> Jeff >>> >>> On Mon, Nov 30, 2015 at 9:00 PM, Balaji, Pavan <balaji(a)anl.gov<mailto:[email protected]>> wrote: >>>> >>>> >>>> Thanks, Jeff. The actual function definitions should be modified as well, which your patch seems to miss. >>>> >>>> But I think the potentially time consuming part is to get buy-in from all the ABI consortium folks for this change. It doesn't exactly break ABI for applications, but will be an annoying change for tools. Ken is looking into this. >>>> >>>> -- Pavan >>>> >>>> > On Nov 30, 2015, at 10:54 PM, Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> wrote: >>>> > >>>> > Pavan: >>>> > >>>> > See hammond-mpi2-deprecated-functions-noconst branch on mpich-review. I found three functions where this change should be made. I do not know if that is the full list, so someone should verify. >>>> > >>>> > Best, >>>> > >>>> > Jeff >>>> > >>>> > On Mon, Nov 30, 2015 at 8:08 PM, Balaji, Pavan <balaji(a)anl.gov<mailto:[email protected]>> wrote: >>>> > >>>> > MPI_Type_struct is not a part of the MPI standard anymore and is replaced by MPI_Type_create_struct. Both MPICH and Open MPI decided to retain it, but followed different directions for it. MPICH decided to follow the same convention as MPI_Type_create_struct with the const additions. Open MPI decided to keep the old prototypes as is. >>>> > >>>> > Unfortunately, MPE got stuck in between the two. The best approach would be to simply remove those functions from MPE and this problem just goes away. Would you be able to do that and contribute a patch? >>>> > >>>> > More generally, I think MPICH was overzealous in trying to constify prototypes for the deleted functions. It should have just left them as they are, like what Open MPI did. We'll figure out what to do about that internally, but that's not going to be a quick change because of ABI and other such stuff. >>>> > >>>> > Thanks, >>>> > >>>> > -- Pavan >>>> > >>>> > > On Nov 30, 2015, at 8:14 PM, Schilling, Herbert W. (GRC-VE00) <hschilling(a)nasa.gov<mailto:[email protected]>> wrote: >>>> > > >>>> > > Hi, >>>> > > >>>> > > I am trying to build MPE for use with OpenMPI. I built from source Version 1.10.1 of OpenMPI. >>>> > > >>>> > > I am using this version of MPE ftp://ftp.mcs.anl.gov/pub/mpi/mpe/mpe2.tar.gz >>>> > > >>>> > > Should this work fine? I followed the instructions to build like this: >>>> > > >>>> > > ${MPE_SRC_DIR}/configure MPI_CC=${OPENMPI_INSTALL_DIR}/bin/mpicc \ >>>> > > CC=<C_COMPILER_used_to_build_OpenMPI> \ >>>> > > MPI_F77=${OPENMPI_INSTALL_DIR}/bin/mpif77 \ >>>> > > F77=<FORTRAN_COMPILER_used_to_build_OpenMPI> \ >>>> > > --with-java2=/usr/local/j2sdk-1.4.2 >>>> > > --prefix=<MPE_INSTALL_DIR> >>>> > > make >>>> > > make install >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > I keep running into problems with conflicting declarations >>>> > > >>>> > > >>>> > > log_mpi_core.c:5611: error: conflicting types for 'MPI_Type_struct' >>>> > > /u/hschilli/local/include/mpi.h:1796: error: previous declaration of 'MPI_Type_struct' was here >>>> > > >>>> > > >>>> > > I think the issue is that MPE's declaration uses some const qualifiers while OpenMPI does not. >>>> > > >>>> > > >>>> > > >>>> > > I tried hacking my way through this with the macro >>>> > > >>>> > > HAVE_CONST_IN_MPI_DECL >>>> > > and MPE_CONST but no luck so far. >>>> > > >>>> > > >>>> > > >>>> > > What am I doing wrong? >>>> > > >>>> > > >>>> > > Thanks in advance. >>>> > > >>>> > > --------- >>>> > > Herb Schilling >>>> > > Scientific Applications and Visualization Team >>>> > > http://ocio.grc.nasa.gov/gvis >>>> > > Office of the Chief Information Officer >>>> > > NASA Glenn Research Center >>>> > > hschilling(a)nasa.gov<mailto:[email protected]> >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > _______________________________________________ >>>> > > discuss mailing list discuss(a)mpich.org<mailto:[email protected]> >>>> > > To manage subscription options or unsubscribe: >>>> > > https://lists.mpich.org/mailman/listinfo/discuss >>>> > >>>> > _______________________________________________ >>>> > discuss mailing list discuss(a)mpich.org<mailto:[email protected]> >>>> > To manage subscription options or unsubscribe: >>>> > https://lists.mpich.org/mailman/listinfo/discuss >>>> > >>>> > >>>> > >>>> > -- >>>> > Jeff Hammond >>>> > jeff.science(a)gmail.com<mailto:[email protected]> >>>> > http://jeffhammond.github.io/ >>>> > _______________________________________________ >>>> > discuss mailing list discuss(a)mpich.org<mailto:[email protected]> >>>> > To manage subscription options or unsubscribe: >>>> > https://lists.mpich.org/mailman/listinfo/discuss >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org<mailto:[email protected]> >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> >>> >>> >>> -- >>> Jeff Hammond >>> jeff.science(a)gmail.com<mailto:[email protected]> >>> http://jeffhammond.github.io/ >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org<mailto:[email protected]> >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> >>> --------- >>> Herb Schilling >>> Scientific Applications and Visualization Team >>> http://ocio.grc.nasa.gov/gvis >>> Office of the Chief Information Officer >>> NASA Glenn Research Center >>> hschilling(a)nasa.gov<mailto:[email protected]> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org<mailto:[email protected]> >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com<mailto:[email protected]> >> http://jeffhammond.github.io/ >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org<mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> --------- >> Herb Schilling >> Scientific Applications and Visualization Team >> http://ocio.grc.nasa.gov/gvis >> Office of the Chief Information Officer >> NASA Glenn Research Center >> hschilling(a)nasa.gov<mailto:[email protected]> >> >> >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org<mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > > -- > Jeff Hammond > jeff.science(a)gmail.com<mailto:[email protected]> > http://jeffhammond.github.io/ > _______________________________________________ > discuss mailing list discuss(a)mpich.org<mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > --------- > Herb Schilling > Scientific Applications and Visualization Team > http://ocio.grc.nasa.gov/gvis > Office of the Chief Information Officer > NASA Glenn Research Center > hschilling(a)nasa.gov<mailto:[email protected]> > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org<mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss --------- Herb Schilling Scientific Applications and Visualization Team http://ocio.grc.nasa.gov/gvis Office of the Chief Information Officer NASA Glenn Research Center hschilling(a)nasa.gov<mailto:[email protected]> _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_Gather fails with 2048 processes and 4096 MB total
by Jeff Hammond 02 Dec '15

02 Dec '15

This was fixed after Cray forked MPICH 3.1.2. You should file a support issue with Cray via the appropriate channel. # master jrhammon-mac01:git jrhammon$ git grep MPIC_Send src/include/mpiimpl.h:int MPIC_Send(const void *buf, *MPI_Aint count,* MPI_Datatype datatype, int dest, int tag, ... # 3.1 branch jrhammon-mac01:git jrhammon$ git checkout 3.1.x jrhammon-mac01:git jrhammon$ git grep MPIC_Send src/include/mpiimpl.h:int MPIC_Send(const void *buf, *int count*, MPI_Datatype datatype, int dest, int tag, ... Best, Jeff On Wed, Dec 2, 2015 at 6:40 AM, <Florian.Willich(a)dlr.de> wrote: > > Thanks for the "bash line", the output is: > > $ grep "define MPICH_VERSION" $CRAY_MPICH2_DIR/include/mpi.h > #define MPICH_VERSION "3.1.2" > > Florian > > ________________________________ > Von: Jeff Hammond [jeff.science(a)gmail.com] > Gesendet: Mittwoch, 2. Dezember 2015 14:58 > An: MPICH > > Betreff: Re: [mpich-discuss] MPI_Gather fails with 2048 processes and 4096 MB total > > 1) You should address this with Cray. Their implementation of MPI has closed-source modifications to MPICH that we cannot analyze for count-safety. > > 2) You can discover the version of MPICH associated with Cray MPI like this: > jhammond@cori11:~> grep "define MPICH_VERSION" $CRAY_MPICH2_DIR/include/mpi.h > #define MPICH_VERSION "3.1.2" > > Best, > > Jeff > > On Wed, Dec 2, 2015 at 12:15 AM, <Florian.Willich(a)dlr.de> wrote: >> >> Hi Rob, >> >> well maybe I was addressing the wrong organisation... I am currently testing on the Cray swan super computer which provides the module cray-mpich/7.2.6 ("Cray Message Passing Toolkit 7.2.6"). >> >> I can not determine whether the cray mpich version is mpich with additional implementations or if it is totally different from the official mpich releases. Additioanlly, I can not figure out on which mpich version this cray-mpich module is based on. I'll continue investigation and keep you updated. >> >> Best Regards >> >> Florian >> ________________________________________ >> Von: Rob Latham [robl(a)mcs.anl.gov] >> Gesendet: Dienstag, 1. Dezember 2015 16:48 >> An: discuss(a)mpich.org >> Betreff: Re: [mpich-discuss] MPI_Gather fails with 2048 processes and 4096 MB total >> >> On 11/26/2015 12:38 PM, Archer, Charles J wrote: >> > FYI, we hit various flavors of this problem when I was still at IBM, I think mostly in weather codes. >> > Apparently Cray hit this too: >> > >> > https://trac.mpich.org/projects/mpich/ticket/1767 >> > >> > We pretty much told our customers back then that a fix was forthcoming (with no ETA :) )with the revamp of datatypes to use internal 64-bit counts. >> > We also provided workarounds. >> > >> > In the case of this gather operation, we asked the customer to implement gather as a flat tree using point to point. >> > Root posts irecvs, then barrier, children send to root. >> > >> > IRC, the giant gather we were debugging was at the very end of the application and used to gather some statistics for IO at the root, so it wasn’t critical to perform well. >> > I also attempted a workaround using some derived datatypes, but I hit another truncation in the datatype code itself :\ >> > I should see if I can dig up that implementation and make sure it isn’t still broken for large counts. >> >> those are all fine approaches to work around the problem. the internals >> of MPICH, though, need to be 64 bit clean -- there are still 4500 places >> where clang warns of a 64 bit value being assigned to a 32 bit type. >> >> Florian Willich, what version of MPICH is this? The line numbers in >> the back trace don't match up with what I've got, and >> I really thought we fixed this class of bug with commits 31d95ed7b18c >> and 68f8c7aa7 over the summer. >> >> ==rob >> >> >> > >> > >> > >> > >> > >> > On Nov 26, 2015, at 12:14 PM, Balaji, Pavan <balaji(a)anl.gov<mailto: balaji(a)anl.gov>> wrote: >> > >> > >> > Thanks for reporting. This looks like an integer-overflow issue, which fails when the summation of data elements from all processes is larger than INT_MAX (2 billion). We'll look into it. I've created a ticket for it, and added you as the reporter, so you'll get notified as they are updates. >> > >> > http://trac.mpich.org/projects/mpich/ticket/2317 >> > >> > Rob: can you create a simple test program for this and add it to the test bucket, so it shows up on the nightlies? >> > >> > Thanks, >> > >> > -- Pavan >> > >> > On Nov 26, 2015, at 10:18 AM, Florian.Willich(a)dlr.de wrote: >> > >> > Dear mpich discussion group, >> > >> > the following issue appeared when running some benchmarks with MPI Gather: >> > >> > Gathering data (calling MPI_Gather(...) ) involing 2048 processes and 2 MB of data (4096 MB total) that each process sends fails with the following output: >> > ____________________________ >> > >> > Rank 1024 [Thu Nov 26 09:43:16 2015] [c1-0c1s12n3] Fatal error in PMPI_Gather: Invalid count, error stack: >> > PMPI_Gather(959)......: MPI_Gather(sbuf=0x2aaab826c010, scount=524288, MPI_INT, rbuf=(nil), rcount=524288, MPI_INT, root=0, MPI_COMM_WORLD) failed >> > MPIR_Gather_impl(775).: >> > MPIR_Gather(735)......: >> > MPIR_Gather_intra(347): >> > MPIC_Send(360)........: Negative count, value is -2147483648 >> > _pmiu_daemon(SIGCHLD): [NID 00307] [c1-0c1s12n3] [Thu Nov 26 09:43:16 2015] PE RANK 1024 exit signal Aborted >> > [NID 00307] 2015-11-26 09:43:16 Apid 949450: initiated application termination >> > Application 949450 exit codes: 134 >> > Application 949450 exit signals: Killed >> > Application 949450 resources: utime ~1s, stime ~137s, Rss ~2110448, inblocks ~617782, outblocks ~1659320 >> > ____________________________ >> > >> > The following are some tests that I ran to better understand the problem: >> > >> > 2047 processes - 2 MB (4094 MB total) -> works! >> > >> > 2048 processes - 2047.5 KB (~1.999512 MB) (4095 MB total) -> works! >> > >> > 2048 processes - 3 MB (6144 MB total) -> fails: >> > ____________________________ >> > >> > Rank 1024 [Thu Nov 26 09:41:15 2015] [c1-0c1s12n3] Fatal error in PMPI_Gather: Invalid count, error stack: >> > PMPI_Gather(959)......: MPI_Gather(sbuf=0x2aaab826c010, scount=786432, MPI_INT, rbuf=(nil), rcount=786432, MPI_INT, root=0, MPI_COMM_WORLD) failed >> > MPIR_Gather_impl(775).: >> > MPIR_Gather(735)......: >> > MPIR_Gather_intra(347): >> > MPIC_Send(360)........: Negative count, value is -1073741824 >> > _pmiu_daemon(SIGCHLD): [NID 00307] [c1-0c1s12n3] [Thu Nov 26 09:41:15 2015] PE RANK 1024 exit signal Aborted >> > [NID 00307] 2015-11-26 09:41:15 Apid 949448: initiated application termination >> > Application 949448 exit codes: 134 >> > Application 949448 exit signals: Killed >> > Application 949448 resources: utime ~1s, stime ~139s, Rss ~3159984, inblocks ~617782, outblocks ~1659351 >> > ____________________________ >> > >> > 2047 processes - 3 MB (6141 MB total) -> fails: >> > ____________________________ >> > >> > Rank 1024 [Thu Nov 26 09:40:31 2015] [c1-0c1s12n3] Fatal error in PMPI_Gather: Invalid count, error stack: >> > PMPI_Gather(959)......: MPI_Gather(sbuf=0x2aaab826c010, scount=786432, MPI_INT, rbuf=(nil), rcount=786432, MPI_INT, root=0, MPI_COMM_WORLD) failed >> > MPIR_Gather_impl(775).: >> > MPIR_Gather(735)......: >> > MPIR_Gather_intra(347): >> > MPIC_Send(360)........: Negative count, value is -1076887552 >> > _pmiu_daemon(SIGCHLD): [NID 00307] [c1-0c1s12n3] [Thu Nov 26 09:40:32 2015] PE RANK 1024 exit signal Aborted >> > [NID 00307] 2015-11-26 09:40:32 Apid 949446: initiated application termination >> > Application 949446 exit codes: 134 >> > Application 949446 exit signals: Killed >> > Application 949446 resources: utime ~1s, stime ~134s, Rss ~3157072, inblocks ~617780, outblocks ~1659351 >> > ____________________________ >> > >> > 8 processes - 625 MB (5000 MB total) -> works! >> > >> > I can think of some pitfalls that might cause this issue but I do not have the knowledge of the internally called routines to check them. Is someone familier with the implementation of MPI_Gather(...) and willing to help me? >> > >> > Best regards >> > >> > Florian >> > >> > Deutsches Zentrum für Luft- und Raumfahrt e. V. (DLR) >> > German Aerospace Center >> > Institute of Planetary Research | Planetary Physics | Rutherfordstraße 2 | 12489 Berlin >> > >> > Florian Willich| Intern - Software Developer (Parallel Applications) >> > florian.willlich(a)dlr.de >> > DLR.de >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> > >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> > >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> > >> >> -- >> Rob Latham >> Mathematics and Computer Science Division >> Argonne National Lab, IL USA >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Building MPE for use with OpenMPI
by Jeff Hammond 02 Dec '15

02 Dec '15

But you are right that a configure test could be useful to make MPE work with MPICH releases that have the unnecessary const attributes in MPI_Type_hindexed, MPI_Type_struct, MPI_Address, PMPI_Type_hindexed, PMPI_Type_struct, PMPI_Address. Best, Jeff On Wed, Dec 2, 2015 at 7:16 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > Hi Bill, > > My original idea was wrong. Since those deprecated MPI-2 functions should > never have const in the declaration, the right solution is to remove > MPE_CONST from them. The macro logic is fine as-is as long as an MPI > implementation only adds const to MPI-3 functions. > > The attached patch implements the aforementioned change. Of course, it > must be coupled to my patch to MPICH for the function signatures to align. > > I don't have write privileges on the MPE repo, not that it is critical. > > Best, > > Jeff > > On Wed, Dec 2, 2015 at 7:08 AM, William Gropp <wgropp(a)illinois.edu> wrote: > >> It sounds like this one needs a special case for the current MPICH bug. >> That isn’t hard and I can do that, Jeff, if you don’t want to. Let me >> know. It will need a configure test for the suspect cases. >> >> Bill >> >> William Gropp >> Director, Parallel Computing Institute >> Thomas M. Siebel Chair in Computer Science >> Chief Scientist, NCSA >> University of Illinois Urbana-Champaign >> >> >> >> >> >> On Dec 1, 2015, at 10:42 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> >> Ok, I didn't understand your issue before. I thought your issue was with >> MPICH not agreeing with OpenMPI. That needed to be fixed anyways, so it's >> good that your query prompted it, but I'm sorry it wasn't what you needed. >> >> MPE already has MPE_CONST, but it will be "const" whenever >> MPI_VERSION>=3, which is what late-model MPICH will give you. The solution >> is probably to the logic below to allow for manual override, rather than >> OR-ing on MPI_VERSION, which cannot be overridden. >> >> #if defined(HAVE_CONST_IN_MPI_DECL) || MPI_VERSION >= 3 >> #define MPE_CONST const >> #else >> #define MPE_CONST >> #endif >> >> I'll try to work up a patch later. >> >> As for configure, the documentation is rightly written for the user who >> downloads a release tarball, not the git repo. So yes, you should generate >> it with autogen.sh. >> >> Jeff >> >> On Tue, Dec 1, 2015 at 7:04 PM, Schilling, Herbert W. (GRC-VE00) < >> hschilling(a)nasa.gov> wrote: >> > >> > Hi Jeff, >> > >> > Thanks. Unfortunately, that is basically the same source code from >> > >> > ftp://ftp.mcs.anl.gov/pub/mpi/mpe/mpe2.tar.gz >> <https://urldefense.proofpoint.com/v2/url?u=ftp-3A__ftp.mcs.anl.gov_pub_mpi_…> >> > >> > where I had the issues with the conflicting declarations as Pavan >> talked about. >> > >> > I could remove the deprecated routines as most of the problems are >> caused by them, but it looks like some non-deprecated routines also have >> the const issue, e.g. >> > >> > visual_mess.c:560: error: conflicting types for 'MPI_Issend' >> > /u/hschilli/local/include/mpi.h:1543: error: previous declaration of >> 'MPI_Issend' was here >> > >> > >> > BTW, that repo for MPE doesn't appear to have a configure file. I know >> I could create it with autogen.sh but the INSTALL file doesn't say that. It >> just says run configure. Might be confusing for some people, I think! >> > >> > Thanks for everyone's help >> > >> > >> > >> > >> > >> > On Dec 1, 2015, at 2:03 PM, Jeff Hammond <jeff.science(a)gmail.com> >> > wrote: >> > >> > MPE lives here now: http://git.mpich.org/mpe.git/ >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__git.mpich.org_mpe.git_&…> >> > >> > Jeff >> > >> > On Tue, Dec 1, 2015 at 10:15 AM, Schilling, Herbert W. (GRC-VE00) < >> hschilling(a)nasa.gov> wrote: >> >> >> >> Hi Jeff, >> >> >> >> Thanks. Pardon my ignorance but I I wanted to just use MPE with an >> existing OpenMPI installation and it's not clear to me ( even after >> searching and reading ), how to do that from the full MPICH code base at >> that repo. I can't really even find MPE in that code at all. What am I >> missing? >> >> >> >> Thanks >> >> >> >> On Dec 1, 2015, at 10:03 AM, Jeff Hammond <jeff.science(a)gmail.com> >> >> wrote: >> >> >> >> Herbert: >> >> >> >> Please try >> https://github.com/jeffhammond/mpich/tree/hammond-mpi2-deprecated-functions… >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_jeffhammond…> >> and let me know if you see any problems. >> >> >> >> Best, >> >> >> >> Jeff >> >> >> >> On Tue, Dec 1, 2015 at 5:46 AM, Schilling, Herbert W. (GRC-VE00) < >> hschilling(a)nasa.gov> wrote: >> >>> >> >>> Hi Jeff and Pavan, >> >>> >> >>> Thanks for all the help on this. Jeff, when you have the fix up >> there, I can give it a shot! >> >>> >> >>> >> >>> On Dec 1, 2015, at 1:15 AM, Jeff Hammond <jeff.science(a)gmail.com> >> >>> wrote: >> >>> >> >>> Facepalm. Indeed, changing the header but not the implementation was >> pretty dumb on my part. I'll fix that tomorrow. >> >>> >> >>> I have no opinion on the ABI situation, particularly since I know >> there are other folks that Intel that do. >> >>> >> >>> Jeff >> >>> >> >>> On Mon, Nov 30, 2015 at 9:00 PM, Balaji, Pavan <balaji(a)anl.gov> >> wrote: >> >>>> >> >>>> >> >>>> Thanks, Jeff. The actual function definitions should be modified as >> well, which your patch seems to miss. >> >>>> >> >>>> But I think the potentially time consuming part is to get buy-in >> from all the ABI consortium folks for this change. It doesn't exactly >> break ABI for applications, but will be an annoying change for tools. Ken >> is looking into this. >> >>>> >> >>>> -- Pavan >> >>>> >> >>>> > On Nov 30, 2015, at 10:54 PM, Jeff Hammond <jeff.science(a)gmail.com> >> wrote: >> >>>> > >> >>>> > Pavan: >> >>>> > >> >>>> > See hammond-mpi2-deprecated-functions-noconst branch on >> mpich-review. I found three functions where this change should be made. I >> do not know if that is the full list, so someone should verify. >> >>>> > >> >>>> > Best, >> >>>> > >> >>>> > Jeff >> >>>> > >> >>>> > On Mon, Nov 30, 2015 at 8:08 PM, Balaji, Pavan <balaji(a)anl.gov> >> wrote: >> >>>> > >> >>>> > MPI_Type_struct is not a part of the MPI standard anymore and is >> replaced by MPI_Type_create_struct. Both MPICH and Open MPI decided to >> retain it, but followed different directions for it. MPICH decided to >> follow the same convention as MPI_Type_create_struct with the const >> additions. Open MPI decided to keep the old prototypes as is. >> >>>> > >> >>>> > Unfortunately, MPE got stuck in between the two. The best >> approach would be to simply remove those functions from MPE and this >> problem just goes away. Would you be able to do that and contribute a >> patch? >> >>>> > >> >>>> > More generally, I think MPICH was overzealous in trying to >> constify prototypes for the deleted functions. It should have just left >> them as they are, like what Open MPI did. We'll figure out what to do >> about that internally, but that's not going to be a quick change because of >> ABI and other such stuff. >> >>>> > >> >>>> > Thanks, >> >>>> > >> >>>> > -- Pavan >> >>>> > >> >>>> > > On Nov 30, 2015, at 8:14 PM, Schilling, Herbert W. (GRC-VE00) < >> hschilling(a)nasa.gov> wrote: >> >>>> > > >> >>>> > > Hi, >> >>>> > > >> >>>> > > I am trying to build MPE for use with OpenMPI. I built from >> source Version 1.10.1 of OpenMPI. >> >>>> > > >> >>>> > > I am using this version of MPE >> ftp://ftp.mcs.anl.gov/pub/mpi/mpe/mpe2.tar.gz >> <https://urldefense.proofpoint.com/v2/url?u=ftp-3A__ftp.mcs.anl.gov_pub_mpi_…> >> >>>> > > >> >>>> > > Should this work fine? I followed the instructions to build >> like this: >> >>>> > > >> >>>> > > ${MPE_SRC_DIR}/configure >> MPI_CC=${OPENMPI_INSTALL_DIR}/bin/mpicc \ >> >>>> > > >> CC=<C_COMPILER_used_to_build_OpenMPI> \ >> >>>> > > >> MPI_F77=${OPENMPI_INSTALL_DIR}/bin/mpif77 \ >> >>>> > > >> F77=<FORTRAN_COMPILER_used_to_build_OpenMPI> \ >> >>>> > > --with-java2=/usr/local/j2sdk-1.4.2 >> >>>> > > --prefix=<MPE_INSTALL_DIR> >> >>>> > > make >> >>>> > > make install >> >>>> > > >> >>>> > > >> >>>> > > >> >>>> > > >> >>>> > > I keep running into problems with conflicting declarations >> >>>> > > >> >>>> > > >> >>>> > > log_mpi_core.c:5611: error: conflicting types for >> 'MPI_Type_struct' >> >>>> > > /u/hschilli/local/include/mpi.h:1796: error: previous >> declaration of 'MPI_Type_struct' was here >> >>>> > > >> >>>> > > >> >>>> > > I think the issue is that MPE's declaration uses some const >> qualifiers while OpenMPI does not. >> >>>> > > >> >>>> > > >> >>>> > > >> >>>> > > I tried hacking my way through this with the macro >> >>>> > > >> >>>> > > HAVE_CONST_IN_MPI_DECL >> >>>> > > and MPE_CONST but no luck so far. >> >>>> > > >> >>>> > > >> >>>> > > >> >>>> > > What am I doing wrong? >> >>>> > > >> >>>> > > >> >>>> > > Thanks in advance. >> >>>> > > >> >>>> > > --------- >> >>>> > > Herb Schilling >> >>>> > > Scientific Applications and Visualization Team >> >>>> > > http://ocio.grc.nasa.gov/gvis >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__ocio.grc.nasa.gov_gvis&…> >> >>>> > > Office of the Chief Information Officer >> >>>> > > NASA Glenn Research Center >> >>>> > > hschilling(a)nasa.gov >> >>>> > > >> >>>> > > >> >>>> > > >> >>>> > > >> >>>> > > _______________________________________________ >> >>>> > > discuss mailing list discuss(a)mpich.org >> >>>> > > To manage subscription options or unsubscribe: >> >>>> > > https://lists.mpich.org/mailman/listinfo/discuss >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> >> >>>> > >> >>>> > _______________________________________________ >> >>>> > discuss mailing list discuss(a)mpich.org >> >>>> > To manage subscription options or unsubscribe: >> >>>> > https://lists.mpich.org/mailman/listinfo/discuss >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> >> >>>> > >> >>>> > >> >>>> > >> >>>> > -- >> >>>> > Jeff Hammond >> >>>> > jeff.science(a)gmail.com >> >>>> > http://jeffhammond.github.io/ >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__jeffhammond.github.io_&…> >> >>>> > _______________________________________________ >> >>>> > discuss mailing list discuss(a)mpich.org >> >>>> > To manage subscription options or unsubscribe: >> >>>> > https://lists.mpich.org/mailman/listinfo/discuss >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> >> >>>> >> >>>> _______________________________________________ >> >>>> discuss mailing list discuss(a)mpich.org >> >>>> To manage subscription options or unsubscribe: >> >>>> https://lists.mpich.org/mailman/listinfo/discuss >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> >> >>> >> >>> >> >>> >> >>> >> >>> -- >> >>> Jeff Hammond >> >>> jeff.science(a)gmail.com >> >>> http://jeffhammond.github.io/ >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__jeffhammond.github.io_&…> >> >>> _______________________________________________ >> >>> discuss mailing list discuss(a)mpich.org >> >>> To manage subscription options or unsubscribe: >> >>> https://lists.mpich.org/mailman/listinfo/discuss >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> >> >>> >> >>> >> >>> --------- >> >>> Herb Schilling >> >>> Scientific Applications and Visualization Team >> >>> http://ocio.grc.nasa.gov/gvis >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__ocio.grc.nasa.gov_gvis&…> >> >>> Office of the Chief Information Officer >> >>> NASA Glenn Research Center >> >>> hschilling(a)nasa.gov >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> _______________________________________________ >> >>> discuss mailing list discuss(a)mpich.org >> >>> To manage subscription options or unsubscribe: >> >>> https://lists.mpich.org/mailman/listinfo/discuss >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> >> >> >> >> >> >> >> >> >> >> -- >> >> Jeff Hammond >> >> jeff.science(a)gmail.com >> >> http://jeffhammond.github.io/ >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__jeffhammond.github.io_&…> >> >> _______________________________________________ >> >> discuss mailing list discuss(a)mpich.org >> >> To manage subscription options or unsubscribe: >> >> https://lists.mpich.org/mailman/listinfo/discuss >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> >> >> >> >> >> >> --------- >> >> Herb Schilling >> >> Scientific Applications and Visualization Team >> >> http://ocio.grc.nasa.gov/gvis >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__ocio.grc.nasa.gov_gvis&…> >> >> Office of the Chief Information Officer >> >> NASA Glenn Research Center >> >> hschilling(a)nasa.gov >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> discuss mailing list discuss(a)mpich.org >> >> To manage subscription options or unsubscribe: >> >> https://lists.mpich.org/mailman/listinfo/discuss >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> >> > >> > >> > >> > >> > -- >> > Jeff Hammond >> > jeff.science(a)gmail.com >> > http://jeffhammond.github.io/ >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__jeffhammond.github.io_&…> >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> >> > >> > >> > --------- >> > Herb Schilling >> > Scientific Applications and Visualization Team >> > http://ocio.grc.nasa.gov/gvis >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__ocio.grc.nasa.gov_gvis&…> >> > Office of the Chief Information Officer >> > NASA Glenn Research Center >> > hschilling(a)nasa.gov >> > >> > >> > >> > >> > >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> >> >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> http://jeffhammond.github.io/ >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__jeffhammond.github.io_&…> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Building MPE for use with OpenMPI
by Jeff Hammond 02 Dec '15

02 Dec '15

Hi Bill, My original idea was wrong. Since those deprecated MPI-2 functions should never have const in the declaration, the right solution is to remove MPE_CONST from them. The macro logic is fine as-is as long as an MPI implementation only adds const to MPI-3 functions. The attached patch implements the aforementioned change. Of course, it must be coupled to my patch to MPICH for the function signatures to align. I don't have write privileges on the MPE repo, not that it is critical. Best, Jeff On Wed, Dec 2, 2015 at 7:08 AM, William Gropp <wgropp(a)illinois.edu> wrote: > It sounds like this one needs a special case for the current MPICH bug. > That isn’t hard and I can do that, Jeff, if you don’t want to. Let me > know. It will need a configure test for the suspect cases. > > Bill > > William Gropp > Director, Parallel Computing Institute > Thomas M. Siebel Chair in Computer Science > Chief Scientist, NCSA > University of Illinois Urbana-Champaign > > > > > > On Dec 1, 2015, at 10:42 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > Ok, I didn't understand your issue before. I thought your issue was with > MPICH not agreeing with OpenMPI. That needed to be fixed anyways, so it's > good that your query prompted it, but I'm sorry it wasn't what you needed. > > MPE already has MPE_CONST, but it will be "const" whenever MPI_VERSION>=3, > which is what late-model MPICH will give you. The solution is probably to > the logic below to allow for manual override, rather than OR-ing on > MPI_VERSION, which cannot be overridden. > > #if defined(HAVE_CONST_IN_MPI_DECL) || MPI_VERSION >= 3 > #define MPE_CONST const > #else > #define MPE_CONST > #endif > > I'll try to work up a patch later. > > As for configure, the documentation is rightly written for the user who > downloads a release tarball, not the git repo. So yes, you should generate > it with autogen.sh. > > Jeff > > On Tue, Dec 1, 2015 at 7:04 PM, Schilling, Herbert W. (GRC-VE00) < > hschilling(a)nasa.gov> wrote: > > > > Hi Jeff, > > > > Thanks. Unfortunately, that is basically the same source code from > > > > ftp://ftp.mcs.anl.gov/pub/mpi/mpe/mpe2.tar.gz > <https://urldefense.proofpoint.com/v2/url?u=ftp-3A__ftp.mcs.anl.gov_pub_mpi_…> > > > > where I had the issues with the conflicting declarations as Pavan talked > about. > > > > I could remove the deprecated routines as most of the problems are > caused by them, but it looks like some non-deprecated routines also have > the const issue, e.g. > > > > visual_mess.c:560: error: conflicting types for 'MPI_Issend' > > /u/hschilli/local/include/mpi.h:1543: error: previous declaration of > 'MPI_Issend' was here > > > > > > BTW, that repo for MPE doesn't appear to have a configure file. I know I > could create it with autogen.sh but the INSTALL file doesn't say that. It > just says run configure. Might be confusing for some people, I think! > > > > Thanks for everyone's help > > > > > > > > > > > > On Dec 1, 2015, at 2:03 PM, Jeff Hammond <jeff.science(a)gmail.com> > > wrote: > > > > MPE lives here now: http://git.mpich.org/mpe.git/ > <https://urldefense.proofpoint.com/v2/url?u=http-3A__git.mpich.org_mpe.git_&…> > > > > Jeff > > > > On Tue, Dec 1, 2015 at 10:15 AM, Schilling, Herbert W. (GRC-VE00) < > hschilling(a)nasa.gov> wrote: > >> > >> Hi Jeff, > >> > >> Thanks. Pardon my ignorance but I I wanted to just use MPE with an > existing OpenMPI installation and it's not clear to me ( even after > searching and reading ), how to do that from the full MPICH code base at > that repo. I can't really even find MPE in that code at all. What am I > missing? > >> > >> Thanks > >> > >> On Dec 1, 2015, at 10:03 AM, Jeff Hammond <jeff.science(a)gmail.com> > >> wrote: > >> > >> Herbert: > >> > >> Please try > https://github.com/jeffhammond/mpich/tree/hammond-mpi2-deprecated-functions… > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_jeffhammond…> > and let me know if you see any problems. > >> > >> Best, > >> > >> Jeff > >> > >> On Tue, Dec 1, 2015 at 5:46 AM, Schilling, Herbert W. (GRC-VE00) < > hschilling(a)nasa.gov> wrote: > >>> > >>> Hi Jeff and Pavan, > >>> > >>> Thanks for all the help on this. Jeff, when you have the fix up > there, I can give it a shot! > >>> > >>> > >>> On Dec 1, 2015, at 1:15 AM, Jeff Hammond <jeff.science(a)gmail.com> > >>> wrote: > >>> > >>> Facepalm. Indeed, changing the header but not the implementation was > pretty dumb on my part. I'll fix that tomorrow. > >>> > >>> I have no opinion on the ABI situation, particularly since I know > there are other folks that Intel that do. > >>> > >>> Jeff > >>> > >>> On Mon, Nov 30, 2015 at 9:00 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > >>>> > >>>> > >>>> Thanks, Jeff. The actual function definitions should be modified as > well, which your patch seems to miss. > >>>> > >>>> But I think the potentially time consuming part is to get buy-in from > all the ABI consortium folks for this change. It doesn't exactly break ABI > for applications, but will be an annoying change for tools. Ken is looking > into this. > >>>> > >>>> -- Pavan > >>>> > >>>> > On Nov 30, 2015, at 10:54 PM, Jeff Hammond <jeff.science(a)gmail.com> > wrote: > >>>> > > >>>> > Pavan: > >>>> > > >>>> > See hammond-mpi2-deprecated-functions-noconst branch on > mpich-review. I found three functions where this change should be made. I > do not know if that is the full list, so someone should verify. > >>>> > > >>>> > Best, > >>>> > > >>>> > Jeff > >>>> > > >>>> > On Mon, Nov 30, 2015 at 8:08 PM, Balaji, Pavan <balaji(a)anl.gov> > wrote: > >>>> > > >>>> > MPI_Type_struct is not a part of the MPI standard anymore and is > replaced by MPI_Type_create_struct. Both MPICH and Open MPI decided to > retain it, but followed different directions for it. MPICH decided to > follow the same convention as MPI_Type_create_struct with the const > additions. Open MPI decided to keep the old prototypes as is. > >>>> > > >>>> > Unfortunately, MPE got stuck in between the two. The best approach > would be to simply remove those functions from MPE and this problem just > goes away. Would you be able to do that and contribute a patch? > >>>> > > >>>> > More generally, I think MPICH was overzealous in trying to constify > prototypes for the deleted functions. It should have just left them as > they are, like what Open MPI did. We'll figure out what to do about that > internally, but that's not going to be a quick change because of ABI and > other such stuff. > >>>> > > >>>> > Thanks, > >>>> > > >>>> > -- Pavan > >>>> > > >>>> > > On Nov 30, 2015, at 8:14 PM, Schilling, Herbert W. (GRC-VE00) < > hschilling(a)nasa.gov> wrote: > >>>> > > > >>>> > > Hi, > >>>> > > > >>>> > > I am trying to build MPE for use with OpenMPI. I built from > source Version 1.10.1 of OpenMPI. > >>>> > > > >>>> > > I am using this version of MPE > ftp://ftp.mcs.anl.gov/pub/mpi/mpe/mpe2.tar.gz > <https://urldefense.proofpoint.com/v2/url?u=ftp-3A__ftp.mcs.anl.gov_pub_mpi_…> > >>>> > > > >>>> > > Should this work fine? I followed the instructions to build like > this: > >>>> > > > >>>> > > ${MPE_SRC_DIR}/configure > MPI_CC=${OPENMPI_INSTALL_DIR}/bin/mpicc \ > >>>> > > CC=<C_COMPILER_used_to_build_OpenMPI> > \ > >>>> > > > MPI_F77=${OPENMPI_INSTALL_DIR}/bin/mpif77 \ > >>>> > > > F77=<FORTRAN_COMPILER_used_to_build_OpenMPI> \ > >>>> > > --with-java2=/usr/local/j2sdk-1.4.2 > >>>> > > --prefix=<MPE_INSTALL_DIR> > >>>> > > make > >>>> > > make install > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > I keep running into problems with conflicting declarations > >>>> > > > >>>> > > > >>>> > > log_mpi_core.c:5611: error: conflicting types for > 'MPI_Type_struct' > >>>> > > /u/hschilli/local/include/mpi.h:1796: error: previous > declaration of 'MPI_Type_struct' was here > >>>> > > > >>>> > > > >>>> > > I think the issue is that MPE's declaration uses some const > qualifiers while OpenMPI does not. > >>>> > > > >>>> > > > >>>> > > > >>>> > > I tried hacking my way through this with the macro > >>>> > > > >>>> > > HAVE_CONST_IN_MPI_DECL > >>>> > > and MPE_CONST but no luck so far. > >>>> > > > >>>> > > > >>>> > > > >>>> > > What am I doing wrong? > >>>> > > > >>>> > > > >>>> > > Thanks in advance. > >>>> > > > >>>> > > --------- > >>>> > > Herb Schilling > >>>> > > Scientific Applications and Visualization Team > >>>> > > http://ocio.grc.nasa.gov/gvis > <https://urldefense.proofpoint.com/v2/url?u=http-3A__ocio.grc.nasa.gov_gvis&…> > >>>> > > Office of the Chief Information Officer > >>>> > > NASA Glenn Research Center > >>>> > > hschilling(a)nasa.gov > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > _______________________________________________ > >>>> > > discuss mailing list discuss(a)mpich.org > >>>> > > To manage subscription options or unsubscribe: > >>>> > > https://lists.mpich.org/mailman/listinfo/discuss > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> > >>>> > > >>>> > _______________________________________________ > >>>> > discuss mailing list discuss(a)mpich.org > >>>> > To manage subscription options or unsubscribe: > >>>> > https://lists.mpich.org/mailman/listinfo/discuss > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> > >>>> > > >>>> > > >>>> > > >>>> > -- > >>>> > Jeff Hammond > >>>> > jeff.science(a)gmail.com > >>>> > http://jeffhammond.github.io/ > <https://urldefense.proofpoint.com/v2/url?u=http-3A__jeffhammond.github.io_&…> > >>>> > _______________________________________________ > >>>> > discuss mailing list discuss(a)mpich.org > >>>> > To manage subscription options or unsubscribe: > >>>> > https://lists.mpich.org/mailman/listinfo/discuss > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> > >>>> > >>>> _______________________________________________ > >>>> discuss mailing list discuss(a)mpich.org > >>>> To manage subscription options or unsubscribe: > >>>> https://lists.mpich.org/mailman/listinfo/discuss > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> > >>> > >>> > >>> > >>> > >>> -- > >>> Jeff Hammond > >>> jeff.science(a)gmail.com > >>> http://jeffhammond.github.io/ > <https://urldefense.proofpoint.com/v2/url?u=http-3A__jeffhammond.github.io_&…> > >>> _______________________________________________ > >>> discuss mailing list discuss(a)mpich.org > >>> To manage subscription options or unsubscribe: > >>> https://lists.mpich.org/mailman/listinfo/discuss > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> > >>> > >>> > >>> --------- > >>> Herb Schilling > >>> Scientific Applications and Visualization Team > >>> http://ocio.grc.nasa.gov/gvis > <https://urldefense.proofpoint.com/v2/url?u=http-3A__ocio.grc.nasa.gov_gvis&…> > >>> Office of the Chief Information Officer > >>> NASA Glenn Research Center > >>> hschilling(a)nasa.gov > >>> > >>> > >>> > >>> > >>> > >>> _______________________________________________ > >>> discuss mailing list discuss(a)mpich.org > >>> To manage subscription options or unsubscribe: > >>> https://lists.mpich.org/mailman/listinfo/discuss > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> > >> > >> > >> > >> > >> -- > >> Jeff Hammond > >> jeff.science(a)gmail.com > >> http://jeffhammond.github.io/ > <https://urldefense.proofpoint.com/v2/url?u=http-3A__jeffhammond.github.io_&…> > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> > >> > >> > >> --------- > >> Herb Schilling > >> Scientific Applications and Visualization Team > >> http://ocio.grc.nasa.gov/gvis > <https://urldefense.proofpoint.com/v2/url?u=http-3A__ocio.grc.nasa.gov_gvis&…> > >> Office of the Chief Information Officer > >> NASA Glenn Research Center > >> hschilling(a)nasa.gov > >> > >> > >> > >> > >> > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> > > > > > > > > > > -- > > Jeff Hammond > > jeff.science(a)gmail.com > > http://jeffhammond.github.io/ > <https://urldefense.proofpoint.com/v2/url?u=http-3A__jeffhammond.github.io_&…> > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> > > > > > > --------- > > Herb Schilling > > Scientific Applications and Visualization Team > > http://ocio.grc.nasa.gov/gvis > <https://urldefense.proofpoint.com/v2/url?u=http-3A__ocio.grc.nasa.gov_gvis&…> > > Office of the Chief Information Officer > > NASA Glenn Research Center > > hschilling(a)nasa.gov > > > > > > > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…> > > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > <https://urldefense.proofpoint.com/v2/url?u=http-3A__jeffhammond.github.io_&…> > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0