June 2021 - moab-dev - mailman3.cels.anl.gov

Re: [MOAB-dev] Help for migration of cells between MPI subdomains with moab
by Vijay S. Mahadevan 17 Jun '21

17 Jun '21

Hi Olivier, As Iulian mentioned, there is a lot of infrastructure written for our Climate workflows to migrate a mesh from a process set A to a process set B, where A and B can have overlaps or can be completely disjoint. And during the process of migration, we can do repartitioning as well since typically n(A) != n(B). Currently, this functionality is available and heavily tested with iMOAB interface as it is consumed by a Fortran90 based code. However, if you are interested in this for arbitrary meshes, we can work with you to update that interface more generally to support all dimensions. Note that this iMOAB migration implementation currently works for all unstructured meshes on a sphere (so the dimension is hard-coded in places). I suggest you not use the iMesh/ITAPS framework as this is a legacy interface and we are not too keen on supporting it going forward. Thanks, Vijay On Fri, Jun 11, 2021 at 10:52 AM Grindeanu, Iulian R. <iulian(a)mcs.anl.gov> wrote: > > Hello Olivier, > Migration of meshes with ParallelComm methods is not a well tested piece of code, it is rather old; > Your attachment did not go through, can you send it again to me directly? > > I would be happy to assist; In principle, the data needs to be packed, then sent/received using MPI , then unpacked; > We do use the packing and unpacking quite extensively in currently tested code; we prefer now the migration of full meshes from a set of processes to another set, while repartitioning on the fly using Zoltan based methods (more exactly using iMOAB newer interface) > > I am pretty sure ParallelComm methods will work with minor fixes. > > We also support an iMesh / ITAPS based framework, which could assist in migration of elements, but that is just another layer / API ; the heavy duty work is still performed by ParallelComm > > Thanks, > Iulian > > ________________________________________ > From: moab-dev <moab-dev-bounces(a)mcs.anl.gov> on behalf of Olivier Jamond <olivier.jamond(a)cea.fr> > Sent: Thursday, June 10, 2021 7:32 AM > To: moab-dev(a)mcs.anl.gov > Subject: [MOAB-dev] Help for migration of cells between MPI subdomains with moab > > Dear Moab developers, > > I am working for the french CEA > (https://en.wikipedia.org/wiki/French_Alternative_Energies_and_Atomic_Energy…) > on the development of a new software dedicated to HPC simulations of > various mechanical scenarii (structures, fluid, FSI, other couplings, > ...) using mesh-based methods (finite-elements, finite-volumes, > hybrid-discontinuous-galerkin, ...). > > This new code uses moab to manage the topological connections between > geometrical entities in a distributed memory context. > > For several reasons, especially for dynamic load balancing, we will need > to be able to migrate cells between MPI processes during calculations. > But at this time I am quite stuck with this with moab... So I wonder if > maybe you could help me with that... > > I struggled for some time with the moab::ParallelComm class to try to > migrate a cell from a process to another one, but without success. I > attached to this email a very simple toy piece of code which illustrates > that. In this simple program, I construct a basic mesh with 2 quads on > the proc0, and I would like to migrate one of these quads to proc1, and > then migrate it back to proc0. As I wrote in some comments in this code, > I tried using different functions ('high-level' send/recv_entities, > send_recv_entities, 'low-level' pack/unpack_entities), but without > success yet... > > I would really appreciate if you could help me with that and maybe take > a look at the attached simple code. > > Best regards, > Olivier Jamond > >

3 11

Help for migration of cells between MPI subdomains with moab
by Olivier Jamond 10 Jun '21

10 Jun '21

Dear Moab developers, I am working for the french CEA (https://en.wikipedia.org/wiki/French_Alternative_Energies_and_Atomic_Energy…) on the development of a new software dedicated to HPC simulations of various mechanical scenarii (structures, fluid, FSI, other couplings, ...) using mesh-based methods (finite-elements, finite-volumes, hybrid-discontinuous-galerkin, ...). This new code uses moab to manage the topological connections between geometrical entities in a distributed memory context. For several reasons, especially for dynamic load balancing, we will need to be able to migrate cells between MPI processes during calculations. But at this time I am quite stuck with this with moab... So I wonder if maybe you could help me with that... I struggled for some time with the moab::ParallelComm class to try to migrate a cell from a process to another one, but without success. I attached to this email a very simple toy piece of code which illustrates that. In this simple program, I construct a basic mesh with 2 quads on the proc0, and I would like to migrate one of these quads to proc1, and then migrate it back to proc0. As I wrote in some comments in this code, I tried using different functions ('high-level' send/recv_entities, send_recv_entities, 'low-level' pack/unpack_entities), but without success yet... I would really appreciate if you could help me with that and maybe take a look at the attached simple code. Best regards, Olivier Jamond

1 0

Re: [MOAB-dev] [EXTERNAL] Re: [CELS WordPress Sites] MOAB DoF numbering
by Mahadevan, Vijay S. 01 Jun '21

01 Jun '21

Ok so if you are using DMMoab, then: > So node ID 0 maps to DoFs 0,1,2; node ID 1 maps to DoF 3,4,5; etc.? Or is that not the case in general? This is in general correct. This is the blocked numbering. There is also an option to turn this off and get an interlaced numbering where all Dofs of first field are arranged first, then next field etc. Such numbering really comes back to the type of solver and preconditioner that you want to use. Hope that helps. Best, Vijay On Jun. 1, 2021 18:02, "Warner, James E. (LARC-D309)" <james.e.warner(a)nasa.gov> wrote: Hi Vijay, Sorry about that – we are calling MOAB through PETSc with the DMMoab interface (DMMoabLoadFromFile, DMMoabGetDofs, etc.). -Jim From: "Mahadevan, Vijay S." <mahadevan(a)anl.gov> Date: Tuesday, June 1, 2021 at 5:57 PM To: "Warner, James E. (LARC-D309)" <james.e.warner(a)nasa.gov> Subject: [EXTERNAL] Re: [CELS WordPress Sites] MOAB DoF numbering Hi Jim, MOAB has no notion of DoF numbering. Any and all numbering are maintained on a layer above by the user. This is the case unless you are using something like DMMoab from PETSc where I have code to generate numbering automatically depending on field descriptions. So if you can give better context to your question, I can give you the right answer/pointers. Vijay On Jun. 1, 2021 17:41, Jim Warner <james.e.warner(a)nasa.gov> wrote: Hello, Quick question about MOAB degree of freedom (DoF) numbering - is it safe to assume that the DoFs are ordered consistently with the node ID numbers? So node ID 0 maps to DoFs 0,1,2; node ID 1 maps to DoF 3,4,5; etc.? Or is that not the case in general? Let me know, thanks! -Jim

1 0