December 2012 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] too many communicators
by Jim Dinan 13 Dec '12

13 Dec '12

Hi Jack, It sounds like your application, or maybe the solver, is leaking communicators somewhere. If you configure with --enable-g=handlealloc (or --enable-g=all), MPICH will issue warnings about leaked MPI objects when your program exits. ~Jim. On 12/12/12 6:47 PM, Dave Goodell wrote: > On Dec 13, 2012, at 5:23 AM GMT+09:00, Jack Lee wrote: > >> What I'd like to determine first is whether fault is on my side (e.g. perhaps I'm not calling the clean-up routines properly). Is there a way to find out how many context id's are in use at a given point? > > If your MPICH was configured with "--enable-g=log" (or a superset thereof, such as "=all"), then you can add the following prototype into your application and invoke this function to get a string showing the current bit-vector state on a given process: > > ----8<---- > static char *MPIR_ContextMaskToStr(void); > ----8<---- > > see src/mpi/comm/commutil.c for more insight into how this works and what it's actually telling you. This wiki node also has some useful (and only slightly stale) information: > > http://wiki.mpich.org/mpich/index.php/Communicators_and_Context_IDs > > I wish I had a better option for you, but without just rolling up your sleeves and hacking on the MPICH code, this is about as good as you're likely to get. > > The next best approach would probably be to trap all the communicator creation/destruction calls that you can think of (see the MPI-2.2 standard, chapter 6) via the PMPI_ profiling interface and log the invocations to look for a mismatch. > > -Dave > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] too many communicators
by Dave Goodell 12 Dec '12

12 Dec '12

On Dec 13, 2012, at 5:23 AM GMT+09:00, Jack Lee wrote: > What I'd like to determine first is whether fault is on my side (e.g. perhaps I'm not calling the clean-up routines properly). Is there a way to find out how many context id's are in use at a given point? If your MPICH was configured with "--enable-g=log" (or a superset thereof, such as "=all"), then you can add the following prototype into your application and invoke this function to get a string showing the current bit-vector state on a given process: ----8<---- static char *MPIR_ContextMaskToStr(void); ----8<---- see src/mpi/comm/commutil.c for more insight into how this works and what it's actually telling you. This wiki node also has some useful (and only slightly stale) information: http://wiki.mpich.org/mpich/index.php/Communicators_and_Context_IDs I wish I had a better option for you, but without just rolling up your sleeves and hacking on the MPICH code, this is about as good as you're likely to get. The next best approach would probably be to trap all the communicator creation/destruction calls that you can think of (see the MPI-2.2 standard, chapter 6) via the PMPI_ profiling interface and log the invocations to look for a mismatch. -Dave

1 0

Re: [mpich-discuss] Problems with MPI and virtual functions in C++
by Hélvio Vairinhos 12 Dec '12

12 Dec '12

Thank you again, Dries! You were extremely helpful. I will take a look at the MPI standard. Cheers, Helvio On 12/12/2012 08:22 PM, Dries Kimpe wrote: > * Hï¿½lvio Vairinhos <helvio.vairinhos(a)gmail.com> [2012-12-12 20:00:28]: > >> Thanks for your suggestion, Dries! It works if I use &p.elem[0] as the >> initial address (not &p->elem), and only if I make elem public.I guess >> that's a price to pay (well, I can always fetch the address with a getter). >> Do you know what happens to the memory layout in the presence of virtual >> functions if I have multiple components? For example: >> double a,b; >> instead of: >> double elem[2]; > When you have virtual functions (in fact, according to the <C++98 > standard, as soon as you have non-trivial constructor/destructor), you can > no longer reason about the memory layout of the struct/class. > > They are no longer PODs (plain old datatype). > >> Is it safe to assume that &p.a would give the initial address of a >> contiguous structure? Or should I play safe and use MPI_Pack and MPI_Unpack, >> for example? > Even for a 'contiguous' structure (such as POD struct) the compiler is > allowed to insert padding for alignment reasons. > The difference with non-POD is that you are allowed to copy those bytes as > well to another instance and expect everything to work. > > (i.e. memcpy (&a, &b, sizeof(a)) is allowed for POD). > > Really, this is a larger topic. If I'm not mistaken, there is some > discussion of this in the MPI standard, even though at this point, your > question is more a C/C++ question than an MPI question. > > To give you at least some practical advice: POD types will generally work > as expected. But I would strongly recommend against making similar > assumptions for non-POD types (i.e. types having public/protect/private > modifiers, non-trivial constructors/destructors, and especially virtual > functions). > > Dries > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

too many communicators
by Jack Lee 12 Dec '12

12 Dec '12

Hi there, I'm using an external matrix solver library which is resulting in error, either in mpi_comm_dup or mpi_comm_split. This happens after rebuilding the matrix and calling the solver many times (hundreds of times or more), during which the solver object is destroyed and recreated. These routines are never called from my code. And as far as I can see in the solver source, there are corresponding calls to mpi_comm_free, but it's entirely possible that my assessment is incorrect. The MPI error messages and strack trace printed upon crash are quite brief, and I'm finding it difficult to tackle this issue. What I'd like to determine first is whether fault is on my side (e.g. perhaps I'm not calling the clean-up routines properly). Is there a way to find out how many context id's are in use at a given point? -- Ubuntu 12.04, mpich2 1.4, gcc/gfortran. The linear solver is SuperLU_dist 2.5. Many thanks, Jack

1 0

Re: [mpich-discuss] Problems with MPI and virtual functions in C++
by Dries Kimpe 12 Dec '12

12 Dec '12

* Hélvio Vairinhos <helvio.vairinhos(a)gmail.com> [2012-12-12 20:00:28]: > Thanks for your suggestion, Dries! It works if I use &p.elem[0] as the > initial address (not &p->elem), and only if I make elem public.I guess > that's a price to pay (well, I can always fetch the address with a getter). > Do you know what happens to the memory layout in the presence of virtual > functions if I have multiple components? For example: > double a,b; > instead of: > double elem[2]; When you have virtual functions (in fact, according to the <C++98 standard, as soon as you have non-trivial constructor/destructor), you can no longer reason about the memory layout of the struct/class. They are no longer PODs (plain old datatype). > Is it safe to assume that &p.a would give the initial address of a > contiguous structure? Or should I play safe and use MPI_Pack and MPI_Unpack, > for example? Even for a 'contiguous' structure (such as POD struct) the compiler is allowed to insert padding for alignment reasons. The difference with non-POD is that you are allowed to copy those bytes as well to another instance and expect everything to work. (i.e. memcpy (&a, &b, sizeof(a)) is allowed for POD). Really, this is a larger topic. If I'm not mistaken, there is some discussion of this in the MPI standard, even though at this point, your question is more a C/C++ question than an MPI question. To give you at least some practical advice: POD types will generally work as expected. But I would strongly recommend against making similar assumptions for non-POD types (i.e. types having public/protect/private modifiers, non-trivial constructors/destructors, and especially virtual functions). Dries

1 0

Re: [mpich-discuss] Problems with MPI and virtual functions in C++
by Hélvio Vairinhos 12 Dec '12

12 Dec '12

Thanks for your suggestion, Dries! It works if I use &p.elem[0] as the initial address (not &p->elem), and only if I make elem public.I guess that's a price to pay (well, I can always fetch the address with a getter). Do you know what happens to the memory layout in the presence of virtual functions if I have multiple components? For example: double a,b; instead of: double elem[2]; Is it safe to assume that &p.a would give the initial address of a contiguous structure? Or should I play safe and use MPI_Pack and MPI_Unpack, for example? Helvio On 12/12/2012 07:37 PM, Dries Kimpe wrote: > Classes/structs that have virtual functions have a different memory layout > than structures/classes without them. > > In your code below, you're assuming that the class Pair has a memory > layout of 2 doubles. > This happens to be the case when there are no virtual functions in the > class. > > However, if you add a virtual function, the memory layout changes. > The class now also contains information about its runtime type > (i.e. Pair). > > You can fix this by using &p->elem[0] as the address of the data you want > to send/receive. > > Dries

1 0

Re: [mpich-discuss] Problems with MPI and virtual functions in C++
by Dries Kimpe 12 Dec '12

12 Dec '12

Classes/structs that have virtual functions have a different memory layout than structures/classes without them. In your code below, you're assuming that the class Pair has a memory layout of 2 doubles. This happens to be the case when there are no virtual functions in the class. However, if you add a virtual function, the memory layout changes. The class now also contains information about its runtime type (i.e. Pair). You can fix this by using &p->elem[0] as the address of the data you want to send/receive. Dries * Hélvio Vairinhos <helvio.vairinhos(a)gmail.com> [2012-12-12 19:23:12]: > Hi, > I came across a strange behaviour of MPI when I try to do a simple > send/receive of an object of a class which contains virtual functions. I > wrote a short code to exemplify what happens: > #include <mpi.h> > #include <iostream> > using namespace std; > class Pair { > double elem[2]; > public: > Pair(); > Pair(const double&, const double&); > //virtual > Pair& operator += (const Pair&); > void print(); > static MPI_Datatype mpi_type; > static void Init(); > static void Finalize(); > }; > Pair::Pair() > { elem[0]=0.; elem[1]=0.; } > Pair::Pair(const double& x, const double& y) > { elem[0]=x; elem[1]=y; } > Pair& Pair::operator += (const Pair& q) > { elem[0] += q.elem[0]; elem[1] += q.elem[1]; return *this; } > void Pair::print() > { std::cout << "(" << elem[0] << "," << elem[1] << ")" << std::endl; } > MPI_Datatype Pair::mpi_type; > void Pair::Init() > { MPI_Type_contiguous(2,MPI_DOUBLE,&mpi_type); > MPI_Type_commit(&mpi_type); } > void Pair::Finalize() > { MPI_Type_free(&mpi_type); } > int main() { > int rank; > MPI_Init(0,0); > MPI_Comm_rank(MPI_COMM_WORLD,&rank); > Pair::Init(); > Pair q; > if (rank == 0) { > q = Pair(1,2); > MPI_Send(&q, 1, Pair::mpi_type, 1, 99, MPI_COMM_WORLD); > } > if (rank == 1) { > q = Pair(3,4); > MPI_Status status; > MPI_Recv(&q, 1, Pair::mpi_type, 0, 99, MPI_COMM_WORLD, &status); > q.print(); > } > Pair::Finalize(); > MPI_Finalize(); > return 0; > } > I compiled it with mpic++, and ran it on 2 processes. Process 0 simply sends > the pair q=(1,2) to process 1, which prints the result on the screen. The > MPI datatype is contiguous with 2 MPI_DOUBLE components. > The program above behaves just fine, and prints the correct result (1,2). > However, if I make the operator += virtual within the Pair class (i.e. if I > uncomment the corresponding line above), then the process 1 prints the > result (1,4). In other words, process 0 only sends the 1st element of the > pair. The 2nd element of the pair is only sent if I create an MPI datatype > with 3 MPI_DOUBLE elements, i.e. if I use > MPI_Type_contiguous(3,MPI_DOUBLE,&mpi_type). It behaves as if the buffer of > MPI_Send starts at the address "&q-1" instead of "&q" when the virtual > keyword is present. > Does anyone know why this happens, or if I am being careless at some point? > I need to virtualize some functions in classes like this one, and I would > like to trust that MPI communication is being done correctly. > Thank you in advance, > Helvio Vairinhos > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Problems with MPI and virtual functions in C++
by Helvio Vairinhos 12 Dec '12

12 Dec '12

Errata: For the code to compile correctly, the line "using namespace std;" must be removed. Sorry about that. Cheers, Helvio Vairinhos.

1 0

Problems with MPI and virtual functions in C++
by Hélvio Vairinhos 12 Dec '12

12 Dec '12

Hi, I came across a strange behaviour of MPI when I try to do a simple send/receive of an object of a class which contains virtual functions. I wrote a short code to exemplify what happens: #include <mpi.h> #include <iostream> using namespace std; class Pair { double elem[2]; public: Pair(); Pair(const double&, const double&); //virtual Pair& operator += (const Pair&); void print(); static MPI_Datatype mpi_type; static void Init(); static void Finalize(); }; Pair::Pair() { elem[0]=0.; elem[1]=0.; } Pair::Pair(const double& x, const double& y) { elem[0]=x; elem[1]=y; } Pair& Pair::operator += (const Pair& q) { elem[0] += q.elem[0]; elem[1] += q.elem[1]; return *this; } void Pair::print() { std::cout << "(" << elem[0] << "," << elem[1] << ")" << std::endl; } MPI_Datatype Pair::mpi_type; void Pair::Init() { MPI_Type_contiguous(2,MPI_DOUBLE,&mpi_type); MPI_Type_commit(&mpi_type); } void Pair::Finalize() { MPI_Type_free(&mpi_type); } int main() { int rank; MPI_Init(0,0); MPI_Comm_rank(MPI_COMM_WORLD,&rank); Pair::Init(); Pair q; if (rank == 0) { q = Pair(1,2); MPI_Send(&q, 1, Pair::mpi_type, 1, 99, MPI_COMM_WORLD); } if (rank == 1) { q = Pair(3,4); MPI_Status status; MPI_Recv(&q, 1, Pair::mpi_type, 0, 99, MPI_COMM_WORLD, &status); q.print(); } Pair::Finalize(); MPI_Finalize(); return 0; } I compiled it with mpic++, and ran it on 2 processes. Process 0 simply sends the pair q=(1,2) to process 1, which prints the result on the screen. The MPI datatype is contiguous with 2 MPI_DOUBLE components. The program above behaves just fine, and prints the correct result (1,2). However, if I make the operator += virtual within the Pair class (i.e. if I uncomment the corresponding line above), then the process 1 prints the result (1,4). In other words, process 0 only sends the 1st element of the pair. The 2nd element of the pair is only sent if I create an MPI datatype with 3 MPI_DOUBLE elements, i.e. if I use MPI_Type_contiguous(3,MPI_DOUBLE,&mpi_type). It behaves as if the buffer of MPI_Send starts at the address "&q-1" instead of "&q" when the virtual keyword is present. Does anyone know why this happens, or if I am being careless at some point? I need to virtualize some functions in classes like this one, and I would like to trust that MPI communication is being done correctly. Thank you in advance, Helvio Vairinhos

1 0

Re: [mpich-discuss] Different architectures
by Jim Dinan 10 Dec '12

10 Dec '12

Hi Peter, Yes -- you will have to install a 32 bit build of MPICH on all machines in your cluster, and you will also have to ensure that your application is built as a 32-bit binary on all machines. The problem is that 32/64 bit nodes won't agree on the sizes of basic datatypes (integer, address, etc), which complicates communication and can have a performance impact. Technically, the MPI standard supports such heterogeneous system configurations, but most MPI implementations (including MPICH) do not. Best, ~Jim. On 12/9/12 2:55 PM, Peter Kroon wrote: > Are you suggestion that I should install a 32b version on all machines > because 32b can't *talk* to 64b? > > > 2012/12/9 Jeff Hammond <jhammond(a)alcf.anl.gov > <mailto:[email protected]>> > > It appears you're trying to use the 64b library (ELFCLASS64...) on the > 32b master. In any case, I don't believe MPICH works in the > heterogeneous case. In theory, you should be able to use a 32b > library across both machines though. > > Jeff > > On Sun, Dec 9, 2012 at 1:40 PM, Peter Kroon <plakroon(a)gmail.com > <mailto:[email protected]>> wrote: > > Hello, > > > > I've installed mpich2 on ubuntu i386(master) via > > https://help.ubuntu.com/community/MpichCluster > > I have another machine amd64(slave). > > > > When i run my script on master I get an error: error while > loading shared > > libraries: linbmpich.so.3: wrong ELF CLASS: ELFCLASS64 > > > > Is mpich compatible with different architectures? If so what's > the easiest > > way for me to get rid of my error? > > > > Also note that I need this to work with x86-64 as well. > > > > Best, > > Peter > > > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov <mailto:[email protected]> / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0