Why not implement the equivalent of the ref-counting init proposal or
at least the simple things that allows multiple inits but only one
finalize? Is it that you don't see the effort being worthwhile or
because there is a principled reason (PMI semantics should match MPI)?
Given that e.g. PGAS runtimes may want to use PMI for bootstrapping (I
do this with DMAPP code on Cray, for example), it seems to benefit
user experience to support this behavior.
Thanks,
Jeff
On Fri, Apr 24, 2015 at 1:12 AM, Balaji, Pavan <balaji(a)anl.gov> wrote:
>
> Multiple PMI Inits are not allowed. It might work in the current
> implementation within MPICH, but is not safe.
>
> If you are looking for MPICH specific behavior, you can look for the PMI_ID
> and PMI_RANK environment variables instead to get your rank.
>
> — Pavan
>
> On Apr 21, 2015, at 11:17 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote:
>
> So MPICH doesn't fail when I try this on my Mac, but it would be good
> to know if I have a reasonable expectation of doing this in general.
>
> Thanks,
>
> Jeff
>
> jrhammon-mac01:ENDPOINTS jrhammon$ mpiexec -n 4 ./a.out
> PMI says I am 2 of 4
> PMI says I am 0 of 4
> PMI says I am 1 of 4
> PMI says I am 3 of 4
> MPI says I am 0 of 4 (provided=1)
> MPI says I am 2 of 4 (provided=1)
> MPI says I am 3 of 4 (provided=3)
> MPI says I am 1 of 4 (provided=3)
>
> jrhammon-mac01:ENDPOINTS jrhammon$ cat pmi.c
> #include <stdio.h>
> #include <stdlib.h>
>
> /*#include <pmi.h>*/
> int PMI_Init( int *spawned );
> int PMI_Finalize( void );
> int PMI_Get_size( int *size );
> int PMI_Get_rank( int *rank );
>
> #include <mpi.h>
>
> int main(int argc, char * argv[])
> {
> int has_parent;
> PMI_Init(&has_parent);
>
> int rank, size;
> PMI_Get_size(&size);
> PMI_Get_rank(&rank);
>
> printf("PMI says I am %d of %d \n", rank, size);
> fflush(stdout);
>
> int requested = (rank%2==0) ? MPI_THREAD_FUNNELED : MPI_THREAD_MULTIPLE;
> int provided;
> MPI_Init_thread(&argc, &argv, requested, &provided);
>
> int wsize, wrank;
> MPI_Comm_size(MPI_COMM_WORLD, &wsize);
> MPI_Comm_rank(MPI_COMM_WORLD, &wrank);
>
> printf("MPI says I am %d of %d (provided=%d)\n", wrank, wsize, provided);
> fflush(stdout);
>
> MPI_Finalize();
> //PMI_Finalize(); /* required? */
>
> return 0;
> }
>
> On Tue, Apr 21, 2015 at 9:09 PM, Jeff Hammond <jeff.science(a)gmail.com>
> wrote:
>
> Is it valid in _MPICH_ to make calls to PMI before MPI is initialized?
> I am interested in PMI_Init, PMI_Get_size and PMI_Get_rank.
>
> The motivating use case is to initialize e.g. half my ranks with
> MPI_THREAD_FUNNELED and the other half with MPI_THREAD_MULTIPLE but
> not use MPMD launching.
>
> I have two reasons for wanting to do it this way. First, it is
> otherwise unnecessary to use MPMD since I can branch at the top of my
> code and meet the requirements of FUNNELED and MULTIPLE. Second, I
> might be interested in running my code on a supercomputer with an
> MPICH-based MPI implementation that does not support MPMD launching in
> an unrestricted manner.
>
> If MPICH can handle this, then I will worry about whether or not the
> aforementioned supercomputer can. It is my assumption that, if MPICH
> cannot handle this, then the MPICH derivative in question cannot
> either.
>
> Thanks,
>
> Jeff
>
> --
> Jeff Hammond
> jeff.science(a)gmail.com
> http://jeffhammond.github.io/
>
>
>
>
> --
> Jeff Hammond
> jeff.science(a)gmail.com
> http://jeffhammond.github.io/
> _______________________________________________
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/devel
>
>
>
> _______________________________________________
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/devel
--
Jeff Hammond
jeff.science(a)gmail.com
http://jeffhammond.github.io/
Multiple PMI Inits are not allowed. It might work in the current implementation within MPICH, but is not safe.
If you are looking for MPICH specific behavior, you can look for the PMI_ID and PMI_RANK environment variables instead to get your rank.
— Pavan
On Apr 21, 2015, at 11:17 PM, Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> wrote:
So MPICH doesn't fail when I try this on my Mac, but it would be good
to know if I have a reasonable expectation of doing this in general.
Thanks,
Jeff
jrhammon-mac01:ENDPOINTS jrhammon$ mpiexec -n 4 ./a.out
PMI says I am 2 of 4
PMI says I am 0 of 4
PMI says I am 1 of 4
PMI says I am 3 of 4
MPI says I am 0 of 4 (provided=1)
MPI says I am 2 of 4 (provided=1)
MPI says I am 3 of 4 (provided=3)
MPI says I am 1 of 4 (provided=3)
jrhammon-mac01:ENDPOINTS jrhammon$ cat pmi.c
#include <stdio.h>
#include <stdlib.h>
/*#include <pmi.h>*/
int PMI_Init( int *spawned );
int PMI_Finalize( void );
int PMI_Get_size( int *size );
int PMI_Get_rank( int *rank );
#include <mpi.h>
int main(int argc, char * argv[])
{
int has_parent;
PMI_Init(&has_parent);
int rank, size;
PMI_Get_size(&size);
PMI_Get_rank(&rank);
printf("PMI says I am %d of %d \n", rank, size);
fflush(stdout);
int requested = (rank%2==0) ? MPI_THREAD_FUNNELED : MPI_THREAD_MULTIPLE;
int provided;
MPI_Init_thread(&argc, &argv, requested, &provided);
int wsize, wrank;
MPI_Comm_size(MPI_COMM_WORLD, &wsize);
MPI_Comm_rank(MPI_COMM_WORLD, &wrank);
printf("MPI says I am %d of %d (provided=%d)\n", wrank, wsize, provided);
fflush(stdout);
MPI_Finalize();
//PMI_Finalize(); /* required? */
return 0;
}
On Tue, Apr 21, 2015 at 9:09 PM, Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> wrote:
Is it valid in _MPICH_ to make calls to PMI before MPI is initialized?
I am interested in PMI_Init, PMI_Get_size and PMI_Get_rank.
The motivating use case is to initialize e.g. half my ranks with
MPI_THREAD_FUNNELED and the other half with MPI_THREAD_MULTIPLE but
not use MPMD launching.
I have two reasons for wanting to do it this way. First, it is
otherwise unnecessary to use MPMD since I can branch at the top of my
code and meet the requirements of FUNNELED and MULTIPLE. Second, I
might be interested in running my code on a supercomputer with an
MPICH-based MPI implementation that does not support MPMD launching in
an unrestricted manner.
If MPICH can handle this, then I will worry about whether or not the
aforementioned supercomputer can. It is my assumption that, if MPICH
cannot handle this, then the MPICH derivative in question cannot
either.
Thanks,
Jeff
--
Jeff Hammond
jeff.science(a)gmail.com<mailto:[email protected]>
http://jeffhammond.github.io/
--
Jeff Hammond
jeff.science(a)gmail.com<mailto:[email protected]>
http://jeffhammond.github.io/
_______________________________________________
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/devel
Yeah, no urgency here. I can push a review branch once I get the QP
stuff implemented.
Jeff
On Thu, Apr 23, 2015 at 12:46 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote:
> OK. MPICH is now in freeze for 3.2b2 release. I will do it after the
> release.
>
> --Junchao Zhang
>
> On Thu, Apr 23, 2015 at 2:39 PM, Jeff Hammond <jeff.science(a)gmail.com>
> wrote:
>>
>> Thanks. Please also apply the attached patch (0004) to fix a trivial bug.
>>
>> The other patch (0003) will make this test XFAIL. You should not
>> apply it yet, but that's where I'm going...
>>
>> Jeff
>>
>>
>> On Thu, Apr 23, 2015 at 10:44 AM, Junchao Zhang <jczhang(a)mcs.anl.gov>
>> wrote:
>> > I will add your patch. Thanks.
>> >
>> > --Junchao Zhang
>> >
>> > On Thu, Apr 23, 2015 at 12:31 PM, Jeff Hammond <jeff.science(a)gmail.com>
>> > wrote:
>> >>
>> >> Patches for test/mpi/f90/f90types/createf90types.c that test
>> >> MPI_Type_create_f90_complex for 64b floats are attached.
>> >>
>> >> If these patches are accepted, I want to extend them to support the
>> >> case of REAL128 (http://fortranwiki.org/fortran/show/Real+precision)
>> >> which is part of Fortran 2008 and therefore should be supported by
>> >> MPICH. Tests for REAL128 will fail since MPICH does not support it
>> >> yet, but I'm hoping to inspire this to change. And I might end up
>> >> doing it myself if I have to.
>> >>
>> >> Thanks,
>> >>
>> >> Jeff
>> >>
>> >> --
>> >> Jeff Hammond
>> >> jeff.science(a)gmail.com
>> >> http://jeffhammond.github.io/
>> >>
>> >> _______________________________________________
>> >> To manage subscription options or unsubscribe:
>> >> https://lists.mpich.org/mailman/listinfo/devel
>> >
>> >
>> >
>> > _______________________________________________
>> > To manage subscription options or unsubscribe:
>> > https://lists.mpich.org/mailman/listinfo/devel
>>
>>
>>
>> --
>> Jeff Hammond
>> jeff.science(a)gmail.com
>> http://jeffhammond.github.io/
>>
>> _______________________________________________
>> To manage subscription options or unsubscribe:
>> https://lists.mpich.org/mailman/listinfo/devel
>
>
>
> _______________________________________________
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/devel
--
Jeff Hammond
jeff.science(a)gmail.com
http://jeffhammond.github.io/
OK. MPICH is now in freeze for 3.2b2 release. I will do it after the
release.
--Junchao Zhang
On Thu, Apr 23, 2015 at 2:39 PM, Jeff Hammond <jeff.science(a)gmail.com>
wrote:
> Thanks. Please also apply the attached patch (0004) to fix a trivial bug.
>
> The other patch (0003) will make this test XFAIL. You should not
> apply it yet, but that's where I'm going...
>
> Jeff
>
>
> On Thu, Apr 23, 2015 at 10:44 AM, Junchao Zhang <jczhang(a)mcs.anl.gov>
> wrote:
> > I will add your patch. Thanks.
> >
> > --Junchao Zhang
> >
> > On Thu, Apr 23, 2015 at 12:31 PM, Jeff Hammond <jeff.science(a)gmail.com>
> > wrote:
> >>
> >> Patches for test/mpi/f90/f90types/createf90types.c that test
> >> MPI_Type_create_f90_complex for 64b floats are attached.
> >>
> >> If these patches are accepted, I want to extend them to support the
> >> case of REAL128 (http://fortranwiki.org/fortran/show/Real+precision)
> >> which is part of Fortran 2008 and therefore should be supported by
> >> MPICH. Tests for REAL128 will fail since MPICH does not support it
> >> yet, but I'm hoping to inspire this to change. And I might end up
> >> doing it myself if I have to.
> >>
> >> Thanks,
> >>
> >> Jeff
> >>
> >> --
> >> Jeff Hammond
> >> jeff.science(a)gmail.com
> >> http://jeffhammond.github.io/
> >>
> >> _______________________________________________
> >> To manage subscription options or unsubscribe:
> >> https://lists.mpich.org/mailman/listinfo/devel
> >
> >
> >
> > _______________________________________________
> > To manage subscription options or unsubscribe:
> > https://lists.mpich.org/mailman/listinfo/devel
>
>
>
> --
> Jeff Hammond
> jeff.science(a)gmail.com
> http://jeffhammond.github.io/
>
> _______________________________________________
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/devel
>
Thanks. Please also apply the attached patch (0004) to fix a trivial bug.
The other patch (0003) will make this test XFAIL. You should not
apply it yet, but that's where I'm going...
Jeff
On Thu, Apr 23, 2015 at 10:44 AM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote:
> I will add your patch. Thanks.
>
> --Junchao Zhang
>
> On Thu, Apr 23, 2015 at 12:31 PM, Jeff Hammond <jeff.science(a)gmail.com>
> wrote:
>>
>> Patches for test/mpi/f90/f90types/createf90types.c that test
>> MPI_Type_create_f90_complex for 64b floats are attached.
>>
>> If these patches are accepted, I want to extend them to support the
>> case of REAL128 (http://fortranwiki.org/fortran/show/Real+precision)
>> which is part of Fortran 2008 and therefore should be supported by
>> MPICH. Tests for REAL128 will fail since MPICH does not support it
>> yet, but I'm hoping to inspire this to change. And I might end up
>> doing it myself if I have to.
>>
>> Thanks,
>>
>> Jeff
>>
>> --
>> Jeff Hammond
>> jeff.science(a)gmail.com
>> http://jeffhammond.github.io/
>>
>> _______________________________________________
>> To manage subscription options or unsubscribe:
>> https://lists.mpich.org/mailman/listinfo/devel
>
>
>
> _______________________________________________
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/devel
--
Jeff Hammond
jeff.science(a)gmail.com
http://jeffhammond.github.io/
I will add your patch. Thanks.
--Junchao Zhang
On Thu, Apr 23, 2015 at 12:31 PM, Jeff Hammond <jeff.science(a)gmail.com>
wrote:
> Patches for test/mpi/f90/f90types/createf90types.c that test
> MPI_Type_create_f90_complex for 64b floats are attached.
>
> If these patches are accepted, I want to extend them to support the
> case of REAL128 (http://fortranwiki.org/fortran/show/Real+precision)
> which is part of Fortran 2008 and therefore should be supported by
> MPICH. Tests for REAL128 will fail since MPICH does not support it
> yet, but I'm hoping to inspire this to change. And I might end up
> doing it myself if I have to.
>
> Thanks,
>
> Jeff
>
> --
> Jeff Hammond
> jeff.science(a)gmail.com
> http://jeffhammond.github.io/
>
> _______________________________________________
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/devel
>
Patches for test/mpi/f90/f90types/createf90types.c that test
MPI_Type_create_f90_complex for 64b floats are attached.
If these patches are accepted, I want to extend them to support the
case of REAL128 (http://fortranwiki.org/fortran/show/Real+precision)
which is part of Fortran 2008 and therefore should be supported by
MPICH. Tests for REAL128 will fail since MPICH does not support it
yet, but I'm hoping to inspire this to change. And I might end up
doing it myself if I have to.
Thanks,
Jeff
--
Jeff Hammond
jeff.science(a)gmail.com
http://jeffhammond.github.io/
So MPICH doesn't fail when I try this on my Mac, but it would be good
to know if I have a reasonable expectation of doing this in general.
Thanks,
Jeff
jrhammon-mac01:ENDPOINTS jrhammon$ mpiexec -n 4 ./a.out
PMI says I am 2 of 4
PMI says I am 0 of 4
PMI says I am 1 of 4
PMI says I am 3 of 4
MPI says I am 0 of 4 (provided=1)
MPI says I am 2 of 4 (provided=1)
MPI says I am 3 of 4 (provided=3)
MPI says I am 1 of 4 (provided=3)
jrhammon-mac01:ENDPOINTS jrhammon$ cat pmi.c
#include <stdio.h>
#include <stdlib.h>
/*#include <pmi.h>*/
int PMI_Init( int *spawned );
int PMI_Finalize( void );
int PMI_Get_size( int *size );
int PMI_Get_rank( int *rank );
#include <mpi.h>
int main(int argc, char * argv[])
{
int has_parent;
PMI_Init(&has_parent);
int rank, size;
PMI_Get_size(&size);
PMI_Get_rank(&rank);
printf("PMI says I am %d of %d \n", rank, size);
fflush(stdout);
int requested = (rank%2==0) ? MPI_THREAD_FUNNELED : MPI_THREAD_MULTIPLE;
int provided;
MPI_Init_thread(&argc, &argv, requested, &provided);
int wsize, wrank;
MPI_Comm_size(MPI_COMM_WORLD, &wsize);
MPI_Comm_rank(MPI_COMM_WORLD, &wrank);
printf("MPI says I am %d of %d (provided=%d)\n", wrank, wsize, provided);
fflush(stdout);
MPI_Finalize();
//PMI_Finalize(); /* required? */
return 0;
}
On Tue, Apr 21, 2015 at 9:09 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote:
> Is it valid in _MPICH_ to make calls to PMI before MPI is initialized?
> I am interested in PMI_Init, PMI_Get_size and PMI_Get_rank.
>
> The motivating use case is to initialize e.g. half my ranks with
> MPI_THREAD_FUNNELED and the other half with MPI_THREAD_MULTIPLE but
> not use MPMD launching.
>
> I have two reasons for wanting to do it this way. First, it is
> otherwise unnecessary to use MPMD since I can branch at the top of my
> code and meet the requirements of FUNNELED and MULTIPLE. Second, I
> might be interested in running my code on a supercomputer with an
> MPICH-based MPI implementation that does not support MPMD launching in
> an unrestricted manner.
>
> If MPICH can handle this, then I will worry about whether or not the
> aforementioned supercomputer can. It is my assumption that, if MPICH
> cannot handle this, then the MPICH derivative in question cannot
> either.
>
> Thanks,
>
> Jeff
>
> --
> Jeff Hammond
> jeff.science(a)gmail.com
> http://jeffhammond.github.io/
--
Jeff Hammond
jeff.science(a)gmail.com
http://jeffhammond.github.io/
Is it valid in _MPICH_ to make calls to PMI before MPI is initialized?
I am interested in PMI_Init, PMI_Get_size and PMI_Get_rank.
The motivating use case is to initialize e.g. half my ranks with
MPI_THREAD_FUNNELED and the other half with MPI_THREAD_MULTIPLE but
not use MPMD launching.
I have two reasons for wanting to do it this way. First, it is
otherwise unnecessary to use MPMD since I can branch at the top of my
code and meet the requirements of FUNNELED and MULTIPLE. Second, I
might be interested in running my code on a supercomputer with an
MPICH-based MPI implementation that does not support MPMD launching in
an unrestricted manner.
If MPICH can handle this, then I will worry about whether or not the
aforementioned supercomputer can. It is my assumption that, if MPICH
cannot handle this, then the MPICH derivative in question cannot
either.
Thanks,
Jeff
--
Jeff Hammond
jeff.science(a)gmail.com
http://jeffhammond.github.io/
>> Oh, but then - <bam> - we see on slide 33, "Hide communication inside
>> "distributed array". OMG it's a high-level abstraction layer on top
>> of MPI that hides the supposed assembly language of parallel
>> programming and makes the programmer productive.
>>
>> But maybe Baidu is exceptional and most HPC programmers who deal with
>> array are doomed to tedious MPI programming. Let's see if there any
>> distributed array abstraction layers out there that hide MPI. It took
>> me about 42 microseconds to find http://libelemental.org/ and
>> https://www.emsl.pnl.gov/docs/global/, and those are just the ones
>> that I have a personal relationship with. And that is just the tip of
>> the MPI library iceberg.
>>
>> So yeah, most people that criticize MPI do so in a manner that is
>> logically equivalent to "Linux is way better than Fortran".
>
>
> This is but one of the points made, but I'm glad you brought it up. Parallel
> HDF5 and Parallel-NetCDF layer on top of MPI-IO routines, as well. so you
> and I know MPI meets one of its goals: be a good foundation for parallel
> libraries.
>
> Can you speak to Dursi's point about Gasnet and Charm++ 's relationship to
> MPI? It was you, not the GASNET folks, that developed the MPI transport
> layer for ARMCI, after all, but I am not familiar with those groups thought
> processes.
You are the zen master, Rob Latham. I was having so much fun being
mad, and you managed to ignore all my sarcasm and find something
constructive. Your talents are clearly wasted on computing; have you
considered a career in peace negotiation? :-)
MPI as a foundation for parallel libraries and thus good software
engineering in general is fundamental to its success. The IO
libraries you note do far more than MPI can or should while at the
same time subtracting nothing from the usability of MPI. I am not
aware of equivalent capability in e.g. Chapel or UPC, either in the
standard IO library of the language or in third-party code. And if it
exists in UPC, I believe that it is constrained by the lack of
layering/scoping capability equivalent to MPI comm, win or file
objects.
To be clear, ARMCI-MPI exists because of Pavan Balaji and Jim Dinan.
When they started that project, I was working on my own non-MPI port
of ARMCI, which I ultimately abandoned upon realizing that the
microseconds I could save with a non-portable implementation were not
worth the lack of portability relative to ARMCI over MPI-2 RMA, which
at that time was standard and supported on Blue Gene, Cray and
InfiniBand systems. Remodeling and optimizing ARMCI-MPI for MPI-3 was
something that I did due to the obvious practical advantages for
running NWChem and to validate the MPI Forum effort to support Global
Arrays effectively in MPI-3.
GASNet exists in part because of the shortcomings of MPI-2 RMA, which
are described in
http://upc.lbl.gov/publications/bonachea-duell-mpi.pdf. GASNet
remains relevant because of implementation shortcomings in some MPI
libraries, which still exist but are monotonically decreasing in
magnitude, and because - like MPI - there is a software ecosystem
build around it that users cannot immediately abandon. My hope is
that GASNet continues to exist, but that the default conduit is one
based upon the latest and greatest RMA and thread-oriented features of
MPI-3 and eventually MPI-4. But the MPI-3 direct port of UPC should
be faster since, unlike GASNet, MPI-3 exposes a rich set of atomics
that obviate the need for active-messages in many cases.
As for Charm++, Argonne has done some work to show that the unexpected
message performance issue can be mitigated with simple parameter
tuning. My hope is MPI implementations provide the necessary features
to make Charm++ run as fast as possible relative to lower-level
conduits and that the portability of MPI is recognized of greater
value than maintaining Charm++ backends for MPI, TCP/UDP/IP, IBV,
PAMI, uGNI, etc. etc. But at extreme scale, the NAMD people probably
need to shave as many microseconds off the comm time as possible,
which may require abandoning MPI. But such efforts should always be a
means of last resort and, in any case, such events do not support
Dursi's thesis.
Best,
Jeff
--
Jeff Hammond
jeff.science(a)gmail.com
http://jeffhammond.github.io/
