Domain decomposition using DMPLEX
Dear PETSc users and developers, I am working with dmplex to distribute a 3D unstructured mesh made of tetrahedrons in a cuboidal domain. I had a few queries: 1) Is there any way of ensuring load balancing based on the number of vertices per MPI process. 2) As the global domain is cuboidal, is the resulting domain decomposition also cuboidal on every MPI process? If not, is there a way to ensure this? For example in DMDA, the default domain decomposition for a cuboidal domain is cuboidal. Sincerely, SG
On Mon, Nov 25, 2019 at 6:25 PM Swarnava Ghosh <[email protected]> wrote:
Dear PETSc users and developers,
I am working with dmplex to distribute a 3D unstructured mesh made of tetrahedrons in a cuboidal domain. I had a few queries: 1) Is there any way of ensuring load balancing based on the number of vertices per MPI process.
You can now call DMPlexRebalanceSharedPoints() to try and get better balance of vertices.
2) As the global domain is cuboidal, is the resulting domain decomposition also cuboidal on every MPI process? If not, is there a way to ensure this? For example in DMDA, the default domain decomposition for a cuboidal domain is cuboidal.
It sounds like you do not want something that is actually unstructured. Rather, it seems like you want to take a DMDA type thing and split it into tets. You can get a cuboidal decomposition of a hex mesh easily. Call DMPlexCreateBoxMesh() with one cell for every process, distribute, and then uniformly refine. This will not quite work for tets since the mesh partitioner will tend to violate that constraint. You could: a) Prescribe the distribution yourself using the Shell partitioner type or b) Write a refiner that turns hexes into tets We already have a refiner that turns tets into hexes, but we never wrote the other direction because it was not clear that it was useful. Thanks, Matt
Sincerely, SG
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
Hi Matt, https://arxiv.org/pdf/1907.02604.pdf On Mon, Nov 25, 2019 at 7:54 PM Matthew Knepley <[email protected]> wrote:
On Mon, Nov 25, 2019 at 6:25 PM Swarnava Ghosh <[email protected]> wrote:
Dear PETSc users and developers,
I am working with dmplex to distribute a 3D unstructured mesh made of tetrahedrons in a cuboidal domain. I had a few queries: 1) Is there any way of ensuring load balancing based on the number of vertices per MPI process.
You can now call DMPlexRebalanceSharedPoints() to try and get better balance of vertices.
Thank you for pointing out this function!
2) As the global domain is cuboidal, is the resulting domain decomposition
also cuboidal on every MPI process? If not, is there a way to ensure this? For example in DMDA, the default domain decomposition for a cuboidal domain is cuboidal.
It sounds like you do not want something that is actually unstructured. Rather, it seems like you want to take a DMDA type thing and split it into tets. You can get a cuboidal decomposition of a hex mesh easily. Call DMPlexCreateBoxMesh() with one cell for every process, distribute, and then uniformly refine. This will not quite work for tets since the mesh partitioner will tend to violate that constraint. You could:
No, I have an unstructured mesh that increases in resolution away from the center of the cuboid. See Figure: 5 in the ArXiv paper https://arxiv.org/pdf/1907.02604.pdf for a slice through the midplane of the cuboid. Given this type of mesh, will dmplex do a cuboidal domain decomposition?
Sincerely, SG
a) Prescribe the distribution yourself using the Shell partitioner type
or
b) Write a refiner that turns hexes into tets
We already have a refiner that turns tets into hexes, but we never wrote the other direction because it was not clear that it was useful.
Thanks,
Matt
Sincerely, SG
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
"No, I have an unstructured mesh that increases in resolution away from the center of the cuboid. See Figure: 5 in the ArXiv paper https://arxiv.org/pdf/1907.02604.pdf for a slice through the midplane of the cuboid. Given this type of mesh, will dmplex do a cuboidal domain decomposition?" No definitely not. Why do you need a cuboidal domain decomposition? Barry
On Nov 25, 2019, at 10:45 PM, Swarnava Ghosh <[email protected]> wrote:
Hi Matt,
https://arxiv.org/pdf/1907.02604.pdf
On Mon, Nov 25, 2019 at 7:54 PM Matthew Knepley <[email protected]> wrote: On Mon, Nov 25, 2019 at 6:25 PM Swarnava Ghosh <[email protected]> wrote: Dear PETSc users and developers,
I am working with dmplex to distribute a 3D unstructured mesh made of tetrahedrons in a cuboidal domain. I had a few queries: 1) Is there any way of ensuring load balancing based on the number of vertices per MPI process.
You can now call DMPlexRebalanceSharedPoints() to try and get better balance of vertices.
Thank you for pointing out this function!
2) As the global domain is cuboidal, is the resulting domain decomposition also cuboidal on every MPI process? If not, is there a way to ensure this? For example in DMDA, the default domain decomposition for a cuboidal domain is cuboidal.
It sounds like you do not want something that is actually unstructured. Rather, it seems like you want to take a DMDA type thing and split it into tets. You can get a cuboidal decomposition of a hex mesh easily. Call DMPlexCreateBoxMesh() with one cell for every process, distribute, and then uniformly refine. This will not quite work for tets since the mesh partitioner will tend to violate that constraint. You could:
No, I have an unstructured mesh that increases in resolution away from the center of the cuboid. See Figure: 5 in the ArXiv paper https://arxiv.org/pdf/1907.02604.pdf for a slice through the midplane of the cuboid. Given this type of mesh, will dmplex do a cuboidal domain decomposition?
Sincerely, SG
a) Prescribe the distribution yourself using the Shell partitioner type
or
b) Write a refiner that turns hexes into tets
We already have a refiner that turns tets into hexes, but we never wrote the other direction because it was not clear that it was useful.
Thanks,
Matt
Sincerely, SG
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
Hi Barry, "Why do you need a cuboidal domain decomposition?" I gave it some thought. I don't always need a cuboidal decomposition. But I would need something that essentially minimized the surface area of the faces of each decomposition. Is there a way to get this? Could you please direct me to a reference a reference where I can read about the domain decomposition strategies used in petsc dmplex. Sincerely, Swarnava On Mon, Nov 25, 2019 at 9:02 PM Smith, Barry F. <[email protected]> wrote:
"No, I have an unstructured mesh that increases in resolution away from the center of the cuboid. See Figure: 5 in the ArXiv paper https://arxiv.org/pdf/1907.02604.pdf for a slice through the midplane of the cuboid. Given this type of mesh, will dmplex do a cuboidal domain decomposition?"
No definitely not. Why do you need a cuboidal domain decomposition?
Barry
On Nov 25, 2019, at 10:45 PM, Swarnava Ghosh <[email protected]> wrote:
Hi Matt,
https://arxiv.org/pdf/1907.02604.pdf
On Mon, Nov 25, 2019 at 7:54 PM Matthew Knepley <[email protected]> wrote: On Mon, Nov 25, 2019 at 6:25 PM Swarnava Ghosh <[email protected]> wrote: Dear PETSc users and developers,
I am working with dmplex to distribute a 3D unstructured mesh made of tetrahedrons in a cuboidal domain. I had a few queries: 1) Is there any way of ensuring load balancing based on the number of vertices per MPI process.
You can now call DMPlexRebalanceSharedPoints() to try and get better balance of vertices.
Thank you for pointing out this function!
2) As the global domain is cuboidal, is the resulting domain decomposition also cuboidal on every MPI process? If not, is there a way to ensure this? For example in DMDA, the default domain decomposition for a cuboidal domain is cuboidal.
It sounds like you do not want something that is actually unstructured. Rather, it seems like you want to take a DMDA type thing and split it into tets. You can get a cuboidal decomposition of a hex mesh easily. Call DMPlexCreateBoxMesh() with one cell for every process, distribute, and then uniformly refine. This will not quite work for tets since the mesh partitioner will tend to violate that constraint. You could:
No, I have an unstructured mesh that increases in resolution away from the center of the cuboid. See Figure: 5 in the ArXiv paper https://arxiv.org/pdf/1907.02604.pdf for a slice through the midplane of the cuboid. Given this type of mesh, will dmplex do a cuboidal domain decomposition?
Sincerely, SG
a) Prescribe the distribution yourself using the Shell partitioner type
or
b) Write a refiner that turns hexes into tets
We already have a refiner that turns tets into hexes, but we never wrote the other direction because it was not clear that it was useful.
Thanks,
Matt
Sincerely, SG
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
On Thu, Nov 28, 2019 at 9:45 PM Swarnava Ghosh <[email protected]> wrote:
Hi Barry,
"Why do you need a cuboidal domain decomposition?"
I gave it some thought. I don't always need a cuboidal decomposition. But I would need something that essentially minimized the surface area of the faces of each decomposition. Is there a way to get this? Could you please direct me to a reference a reference where I can read about the domain decomposition strategies used in petsc dmplex.
This is the point of graph partitioning, which minimizes the "cut" which the the number of links between one partition and another. The ParMetis manual has this kind of information, and citations. Thanks, Matt
Sincerely, Swarnava
On Mon, Nov 25, 2019 at 9:02 PM Smith, Barry F. <[email protected]> wrote:
"No, I have an unstructured mesh that increases in resolution away from the center of the cuboid. See Figure: 5 in the ArXiv paper https://arxiv.org/pdf/1907.02604.pdf for a slice through the midplane of the cuboid. Given this type of mesh, will dmplex do a cuboidal domain decomposition?"
No definitely not. Why do you need a cuboidal domain decomposition?
Barry
On Nov 25, 2019, at 10:45 PM, Swarnava Ghosh <[email protected]> wrote:
Hi Matt,
https://arxiv.org/pdf/1907.02604.pdf
On Mon, Nov 25, 2019 at 7:54 PM Matthew Knepley <[email protected]> wrote: On Mon, Nov 25, 2019 at 6:25 PM Swarnava Ghosh <[email protected]> wrote: Dear PETSc users and developers,
I am working with dmplex to distribute a 3D unstructured mesh made of tetrahedrons in a cuboidal domain. I had a few queries: 1) Is there any way of ensuring load balancing based on the number of vertices per MPI process.
You can now call DMPlexRebalanceSharedPoints() to try and get better balance of vertices.
Thank you for pointing out this function!
2) As the global domain is cuboidal, is the resulting domain decomposition also cuboidal on every MPI process? If not, is there a way to ensure this? For example in DMDA, the default domain decomposition for a cuboidal domain is cuboidal.
It sounds like you do not want something that is actually unstructured. Rather, it seems like you want to take a DMDA type thing and split it into tets. You can get a cuboidal decomposition of a hex mesh easily. Call DMPlexCreateBoxMesh() with one cell for every process, distribute, and then uniformly refine. This will not quite work for tets since the mesh partitioner will tend to violate that constraint. You could:
No, I have an unstructured mesh that increases in resolution away from the center of the cuboid. See Figure: 5 in the ArXiv paper https://arxiv.org/pdf/1907.02604.pdf for a slice through the midplane of the cuboid. Given this type of mesh, will dmplex do a cuboidal domain decomposition?
Sincerely, SG
a) Prescribe the distribution yourself using the Shell partitioner type
or
b) Write a refiner that turns hexes into tets
We already have a refiner that turns tets into hexes, but we never wrote the other direction because it was not clear that it was useful.
Thanks,
Matt
Sincerely, SG
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
On 2019-11-25 7:54 p.m., Matthew Knepley wrote:
On Mon, Nov 25, 2019 at 6:25 PM Swarnava Ghosh <[email protected] <mailto:[email protected]>> wrote:
Dear PETSc users and developers,
I am working with dmplex to distribute a 3D unstructured mesh made of tetrahedrons in a cuboidal domain. I had a few queries: 1) Is there any way of ensuring load balancing based on the number of vertices per MPI process.
You can now call DMPlexRebalanceSharedPoints() to try and get better balance of vertices.
Hi Matt, I just want to follow up if this new function can help to solve the "Strange Partition in PETSc 3.11" problem I mentioned before. Would you please let me know when shall I call this function? Right before DMPlexDistribute? call DMPlexCreateFromCellList call DMPlexGetPartitioner call PetscPartitionerSetFromOptions call DMPlexDistribute Thanks, Danyang
2) As the global domain is cuboidal, is the resulting domain decomposition also cuboidal on every MPI process? If not, is there a way to ensure this? For example in DMDA, the default domain decomposition for a cuboidal domain is cuboidal.
It sounds like you do not want something that is actually unstructured. Rather, it seems like you want to take a DMDA type thing and split it into tets. You can get a cuboidal decomposition of a hex mesh easily. Call DMPlexCreateBoxMesh() with one cell for every process, distribute, and then uniformly refine. This will not quite work for tets since the mesh partitioner will tend to violate that constraint. You could:
a) Prescribe the distribution yourself using the Shell partitioner type
or
b) Write a refiner that turns hexes into tets
We already have a refiner that turns tets into hexes, but we never wrote the other direction because it was not clear that it was useful.
Thanks,
Matt
Sincerely, SG
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
On Tue, Nov 26, 2019 at 11:43 AM Danyang Su <[email protected]> wrote:
On 2019-11-25 7:54 p.m., Matthew Knepley wrote:
On Mon, Nov 25, 2019 at 6:25 PM Swarnava Ghosh <[email protected]> wrote:
Dear PETSc users and developers,
I am working with dmplex to distribute a 3D unstructured mesh made of tetrahedrons in a cuboidal domain. I had a few queries: 1) Is there any way of ensuring load balancing based on the number of vertices per MPI process.
You can now call DMPlexRebalanceSharedPoints() to try and get better balance of vertices.
Hi Matt,
I just want to follow up if this new function can help to solve the "Strange Partition in PETSc 3.11" problem I mentioned before. Would you please let me know when shall I call this function? Right before DMPlexDistribute?
This is not the problem. I believe the problem is that you are partitioning hybrid cells, and the way we handle them internally changed, which I think screwed up the dual mesh for partitioning in your example. I have been sick, so I have not gotten to your example yet, but I will. Sorry about that, Matt
call DMPlexCreateFromCellList
call DMPlexGetPartitioner
call PetscPartitionerSetFromOptions
call DMPlexDistribute
Thanks,
Danyang
2) As the global domain is cuboidal, is the resulting domain decomposition also cuboidal on every MPI process? If not, is there a way to ensure this? For example in DMDA, the default domain decomposition for a cuboidal domain is cuboidal.
It sounds like you do not want something that is actually unstructured. Rather, it seems like you want to take a DMDA type thing and split it into tets. You can get a cuboidal decomposition of a hex mesh easily. Call DMPlexCreateBoxMesh() with one cell for every process, distribute, and then uniformly refine. This will not quite work for tets since the mesh partitioner will tend to violate that constraint. You could:
a) Prescribe the distribution yourself using the Shell partitioner type
or
b) Write a refiner that turns hexes into tets
We already have a refiner that turns tets into hexes, but we never wrote the other direction because it was not clear that it was useful.
Thanks,
Matt
Sincerely, SG
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
On 2019-11-26 10:18 a.m., Matthew Knepley wrote:
On Tue, Nov 26, 2019 at 11:43 AM Danyang Su <[email protected] <mailto:[email protected]>> wrote:
On 2019-11-25 7:54 p.m., Matthew Knepley wrote:
On Mon, Nov 25, 2019 at 6:25 PM Swarnava Ghosh <[email protected] <mailto:[email protected]>> wrote:
Dear PETSc users and developers,
I am working with dmplex to distribute a 3D unstructured mesh made of tetrahedrons in a cuboidal domain. I had a few queries: 1) Is there any way of ensuring load balancing based on the number of vertices per MPI process.
You can now call DMPlexRebalanceSharedPoints() to try and get better balance of vertices.
Hi Matt,
I just want to follow up if this new function can help to solve the "Strange Partition in PETSc 3.11" problem I mentioned before. Would you please let me know when shall I call this function? Right before DMPlexDistribute?
This is not the problem. I believe the problem is that you are partitioning hybrid cells, and the way we handle them internally changed, which I think screwed up the dual mesh for partitioning in your example. I have been sick, so I have not gotten to your example yet, but I will.
Hope you are getting well soon. The mesh is not hybrid, only prism cells layer by layer. But the height of the prism varies significantly. Thanks, Danyang
Sorry about that,
Matt
call DMPlexCreateFromCellList
call DMPlexGetPartitioner
call PetscPartitionerSetFromOptions
call DMPlexDistribute
Thanks,
Danyang
2) As the global domain is cuboidal, is the resulting domain decomposition also cuboidal on every MPI process? If not, is there a way to ensure this? For example in DMDA, the default domain decomposition for a cuboidal domain is cuboidal.
It sounds like you do not want something that is actually unstructured. Rather, it seems like you want to take a DMDA type thing and split it into tets. You can get a cuboidal decomposition of a hex mesh easily. Call DMPlexCreateBoxMesh() with one cell for every process, distribute, and then uniformly refine. This will not quite work for tets since the mesh partitioner will tend to violate that constraint. You could:
a) Prescribe the distribution yourself using the Shell partitioner type
or
b) Write a refiner that turns hexes into tets
We already have a refiner that turns tets into hexes, but we never wrote the other direction because it was not clear that it was useful.
Thanks,
Matt
Sincerely, SG
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
On Tue, Nov 26, 2019 at 12:24 PM Danyang Su <[email protected]> wrote:
On 2019-11-26 10:18 a.m., Matthew Knepley wrote:
On Tue, Nov 26, 2019 at 11:43 AM Danyang Su <[email protected]> wrote:
On 2019-11-25 7:54 p.m., Matthew Knepley wrote:
On Mon, Nov 25, 2019 at 6:25 PM Swarnava Ghosh <[email protected]> wrote:
Dear PETSc users and developers,
I am working with dmplex to distribute a 3D unstructured mesh made of tetrahedrons in a cuboidal domain. I had a few queries: 1) Is there any way of ensuring load balancing based on the number of vertices per MPI process.
You can now call DMPlexRebalanceSharedPoints() to try and get better balance of vertices.
Hi Matt,
I just want to follow up if this new function can help to solve the "Strange Partition in PETSc 3.11" problem I mentioned before. Would you please let me know when shall I call this function? Right before DMPlexDistribute?
This is not the problem. I believe the problem is that you are partitioning hybrid cells, and the way we handle them internally changed, which I think screwed up the dual mesh for partitioning in your example. I have been sick, so I have not gotten to your example yet, but I will.
Hope you are getting well soon. The mesh is not hybrid, only prism cells layer by layer.
Prism cells are called "hybrid" right now, which is indeed a bad term and I will change. Thanks, Matt
But the height of the prism varies significantly.
Thanks,
Danyang
Sorry about that,
Matt
call DMPlexCreateFromCellList
call DMPlexGetPartitioner
call PetscPartitionerSetFromOptions
call DMPlexDistribute
Thanks,
Danyang
2) As the global domain is cuboidal, is the resulting domain decomposition also cuboidal on every MPI process? If not, is there a way to ensure this? For example in DMDA, the default domain decomposition for a cuboidal domain is cuboidal.
It sounds like you do not want something that is actually unstructured. Rather, it seems like you want to take a DMDA type thing and split it into tets. You can get a cuboidal decomposition of a hex mesh easily. Call DMPlexCreateBoxMesh() with one cell for every process, distribute, and then uniformly refine. This will not quite work for tets since the mesh partitioner will tend to violate that constraint. You could:
a) Prescribe the distribution yourself using the Shell partitioner type
or
b) Write a refiner that turns hexes into tets
We already have a refiner that turns tets into hexes, but we never wrote the other direction because it was not clear that it was useful.
Thanks,
Matt
Sincerely, SG
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
participants (4)
-
Danyang Su -
Matthew Knepley -
Smith, Barry F. -
Swarnava Ghosh