Load imbalance in DMSwarm & DMDA simulations
Dear all, I am currently running a PIC-like implementation based on DMSwarm and DMDA. In my setup, a particle (DMSwarm) discretization represents, for example, a sphere placed at the center of a box-shaped domain discretized with a regular mesh (DMDA). The implementation uses both MPI and OpenMP to accelerate the computations. The domain decomposition of the sphere follows a PIC-like approach: each particle is assigned to a rank based on the mesh element it belongs to, and thus inherits the rank that “owns” that element. This is essentially the same strategy used in one of the PETSc examples. However, I have observed that, for certain configurations, some ranks end up with very few or even no particles, which leads to load imbalance. Has anyone experienced a similar issue? Thanks, Miguel
Hi Miguel, This is a common problem. To my knowledge you need to deal with it on your side, but PETSc does have tools to help. I would look in the literature to get ideas and acclimate yourself to the problem. The tool that PETSc has is you can repartition your mesh. Create a dual graph (any petsc examples?) put weights on the "vertice" (now elements, use particle count) and repartition. I am not sure of the best way to proceed from here ... you could create a new DMSwarm with this new "cell" DM and add particles from the old to the new, delete the old, call DMMigrate on the new DMSwarm and that will do the moving that we want. Others will probably be able to add to this. Thanks, Mark On Tue, Mar 17, 2026 at 4:49 AM Miguel Molinos <[email protected]> wrote:
Dear all,
I am currently running a PIC-like implementation based on DMSwarm and DMDA. In my setup, a particle (DMSwarm) discretization represents, for example, a sphere placed at the center of a box-shaped domain discretized with a regular mesh (DMDA).
The implementation uses both MPI and OpenMP to accelerate the computations. The domain decomposition of the sphere follows a PIC-like approach: each particle is assigned to a rank based on the mesh element it belongs to, and thus inherits the rank that “owns” that element. This is essentially the same strategy used in one of the PETSc examples.
However, I have observed that, for certain configurations, some ranks end up with very few or even no particles, which leads to load imbalance.
Has anyone experienced a similar issue?
Thanks, Miguel
Thank you Mark. Indeed my question was oriented towards the tools PETSc can offer to address this problem. Currently I’m using DMDA, perhaps DMPlex is more suited? Thanks, Miguel
On 17 Mar 2026, at 12:17, Mark Adams <[email protected]> wrote:
Hi Miguel,
This is a common problem. To my knowledge you need to deal with it on your side, but PETSc does have tools to help. I would look in the literature to get ideas and acclimate yourself to the problem.
The tool that PETSc has is you can repartition your mesh. Create a dual graph (any petsc examples?) put weights on the "vertice" (now elements, use particle count) and repartition. I am not sure of the best way to proceed from here ... you could create a new DMSwarm with this new "cell" DM and add particles from the old to the new, delete the old, call DMMigrate on the new DMSwarm and that will do the moving that we want.
Others will probably be able to add to this.
Thanks, Mark
On Tue, Mar 17, 2026 at 4:49 AM Miguel Molinos <[email protected] <mailto:[email protected]>> wrote:
Dear all,
I am currently running a PIC-like implementation based on DMSwarm and DMDA. In my setup, a particle (DMSwarm) discretization represents, for example, a sphere placed at the center of a box-shaped domain discretized with a regular mesh (DMDA).
The implementation uses both MPI and OpenMP to accelerate the computations. The domain decomposition of the sphere follows a PIC-like approach: each particle is assigned to a rank based on the mesh element it belongs to, and thus inherits the rank that “owns” that element. This is essentially the same strategy used in one of the PETSc examples.
However, I have observed that, for certain configurations, some ranks end up with very few or even no particles, which leads to load imbalance.
Has anyone experienced a similar issue?
Thanks, Miguel
This tool works at the Mat level so it does not matter, but you need to work at the DM level anyway. I don't know of DM tools for this. Let Matt chime in. On Tue, Mar 17, 2026 at 7:54 AM Miguel Molinos <[email protected]> wrote:
Thank you Mark. Indeed my question was oriented towards the tools PETSc can offer to address this problem. Currently I’m using DMDA, perhaps DMPlex is more suited?
Thanks, Miguel
On 17 Mar 2026, at 12:17, Mark Adams <[email protected]> wrote:
Hi Miguel,
This is a common problem. To my knowledge you need to deal with it on your side, but PETSc does have tools to help. I would look in the literature to get ideas and acclimate yourself to the problem.
The tool that PETSc has is you can repartition your mesh. Create a dual graph (any petsc examples?) put weights on the "vertice" (now elements, use particle count) and repartition. I am not sure of the best way to proceed from here ... you could create a new DMSwarm with this new "cell" DM and add particles from the old to the new, delete the old, call DMMigrate on the new DMSwarm and that will do the moving that we want.
Others will probably be able to add to this.
Thanks, Mark
On Tue, Mar 17, 2026 at 4:49 AM Miguel Molinos <[email protected]> wrote:
Dear all,
I am currently running a PIC-like implementation based on DMSwarm and DMDA. In my setup, a particle (DMSwarm) discretization represents, for example, a sphere placed at the center of a box-shaped domain discretized with a regular mesh (DMDA).
The implementation uses both MPI and OpenMP to accelerate the computations. The domain decomposition of the sphere follows a PIC-like approach: each particle is assigned to a rank based on the mesh element it belongs to, and thus inherits the rank that “owns” that element. This is essentially the same strategy used in one of the PETSc examples.
However, I have observed that, for certain configurations, some ranks end up with very few or even no particles, which leads to load imbalance.
Has anyone experienced a similar issue?
Thanks, Miguel
On Tue, Mar 17, 2026 at 8:53 AM Mark Adams <[email protected]> wrote:
This tool works at the Mat level so it does not matter, but you need to work at the DM level anyway.
I don't know of DM tools for this. Let Matt chime in.
Hi Miguel, DMPlex can definitely do point location and load balancing, which is an advantage. A disadvantage is that you have to work harder to get structured variable access, which DMDA has automatically. Depending on your discretization on this grid, this can be restored, but it is some work. What does the algorithm look like? Thanks, Matt
On Tue, Mar 17, 2026 at 7:54 AM Miguel Molinos <[email protected]> wrote:
Thank you Mark. Indeed my question was oriented towards the tools PETSc can offer to address this problem. Currently I’m using DMDA, perhaps DMPlex is more suited?
Thanks, Miguel
On 17 Mar 2026, at 12:17, Mark Adams <[email protected]> wrote:
Hi Miguel,
This is a common problem. To my knowledge you need to deal with it on your side, but PETSc does have tools to help. I would look in the literature to get ideas and acclimate yourself to the problem.
The tool that PETSc has is you can repartition your mesh. Create a dual graph (any petsc examples?) put weights on the "vertice" (now elements, use particle count) and repartition. I am not sure of the best way to proceed from here ... you could create a new DMSwarm with this new "cell" DM and add particles from the old to the new, delete the old, call DMMigrate on the new DMSwarm and that will do the moving that we want.
Others will probably be able to add to this.
Thanks, Mark
On Tue, Mar 17, 2026 at 4:49 AM Miguel Molinos <[email protected]> wrote:
Dear all,
I am currently running a PIC-like implementation based on DMSwarm and DMDA. In my setup, a particle (DMSwarm) discretization represents, for example, a sphere placed at the center of a box-shaped domain discretized with a regular mesh (DMDA).
The implementation uses both MPI and OpenMP to accelerate the computations. The domain decomposition of the sphere follows a PIC-like approach: each particle is assigned to a rank based on the mesh element it belongs to, and thus inherits the rank that “owns” that element. This is essentially the same strategy used in one of the PETSc examples.
However, I have observed that, for certain configurations, some ranks end up with very few or even no particles, which leads to load imbalance.
Has anyone experienced a similar issue?
Thanks, Miguel
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >
Hi Matt and Mark, Thank you for the feedback.
A disadvantage is that you have to work harder to get structured variable access, which DMDA has automatically. Depending on your discretization on this grid, this can be restored, but it is some work.
I’ve been trying to avoid DMPlex because I find harder to create ghost particles (define padding regions) using an unstructured mesh. Regarding variable access, this is not a limitation since I use the background mesh for domain decomposition purposes. The particles carry al the information.
What does the algorithm look like?
My code reproduces interaction between atoms (some sort of MD). The purposes of the mesh are: - Create a domain decomposition - Particle migration - Definition of boundary conditions (it can works like a supercell) Thanks, Miguel
On 17 Mar 2026, at 15:11, Matthew Knepley <[email protected]> wrote:
On Tue, Mar 17, 2026 at 8:53 AM Mark Adams <[email protected] <mailto:[email protected]>> wrote:
This tool works at the Mat level so it does not matter, but you need to work at the DM level anyway.
I don't know of DM tools for this. Let Matt chime in.
Hi Miguel,
DMPlex can definitely do point location and load balancing, which is an advantage.
A disadvantage is that you have to work harder to get structured variable access, which DMDA has automatically. Depending on your discretization on this grid, this can be restored, but it is some work.
What does the algorithm look like?
Thanks,
Matt
On Tue, Mar 17, 2026 at 7:54 AM Miguel Molinos <[email protected] <mailto:[email protected]>> wrote:
Thank you Mark. Indeed my question was oriented towards the tools PETSc can offer to address this problem. Currently I’m using DMDA, perhaps DMPlex is more suited?
Thanks, Miguel
On 17 Mar 2026, at 12:17, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
Hi Miguel,
This is a common problem. To my knowledge you need to deal with it on your side, but PETSc does have tools to help. I would look in the literature to get ideas and acclimate yourself to the problem.
The tool that PETSc has is you can repartition your mesh. Create a dual graph (any petsc examples?) put weights on the "vertice" (now elements, use particle count) and repartition. I am not sure of the best way to proceed from here ... you could create a new DMSwarm with this new "cell" DM and add particles from the old to the new, delete the old, call DMMigrate on the new DMSwarm and that will do the moving that we want.
Others will probably be able to add to this.
Thanks, Mark
On Tue, Mar 17, 2026 at 4:49 AM Miguel Molinos <[email protected] <mailto:[email protected]>> wrote:
Dear all,
I am currently running a PIC-like implementation based on DMSwarm and DMDA. In my setup, a particle (DMSwarm) discretization represents, for example, a sphere placed at the center of a box-shaped domain discretized with a regular mesh (DMDA).
The implementation uses both MPI and OpenMP to accelerate the computations. The domain decomposition of the sphere follows a PIC-like approach: each particle is assigned to a rank based on the mesh element it belongs to, and thus inherits the rank that “owns” that element. This is essentially the same strategy used in one of the PETSc examples.
However, I have observed that, for certain configurations, some ranks end up with very few or even no particles, which leads to load imbalance.
Has anyone experienced a similar issue?
Thanks, Miguel
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.com/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!D9dNQ... >
On Tue, Mar 17, 2026 at 11:22 AM Miguel Molinos <[email protected]> wrote:
Hi Matt and Mark,
Thank you for the feedback.
A disadvantage is that you have to work harder to get structured variable access, which DMDA has automatically. Depending on your discretization on this grid, this can be restored, but it is some work.
I’ve been trying to avoid DMPlex because I find harder to create ghost particles (define padding regions) using an unstructured mesh.
Oh, you can make DMPlex be geometrically structured very easily. So this should not be a problem.
Regarding variable access, this is not a limitation since I use the background mesh for domain decomposition purposes. The particles carry al the information.
What does the algorithm look like?
My code reproduces interaction between atoms (some sort of MD). The purposes of the mesh are: - Create a domain decomposition - Particle migration - Definition of boundary conditions (it can works like a supercell)
Oh, cool. DMPlex should be very easy then. You just create a BoxMesh (I ould probably do it from the command line), and then have it load balanced exactly as Mark said. I can help you if anything does not make sense. Thanks, Matt
Thanks, Miguel
On 17 Mar 2026, at 15:11, Matthew Knepley <[email protected]> wrote:
On Tue, Mar 17, 2026 at 8:53 AM Mark Adams <[email protected]> wrote:
This tool works at the Mat level so it does not matter, but you need to work at the DM level anyway.
I don't know of DM tools for this. Let Matt chime in.
Hi Miguel,
DMPlex can definitely do point location and load balancing, which is an advantage.
A disadvantage is that you have to work harder to get structured variable access, which DMDA has automatically. Depending on your discretization on this grid, this can be restored, but it is some work.
What does the algorithm look like?
Thanks,
Matt
On Tue, Mar 17, 2026 at 7:54 AM Miguel Molinos <[email protected]> wrote:
Thank you Mark. Indeed my question was oriented towards the tools PETSc can offer to address this problem. Currently I’m using DMDA, perhaps DMPlex is more suited?
Thanks, Miguel
On 17 Mar 2026, at 12:17, Mark Adams <[email protected]> wrote:
Hi Miguel,
This is a common problem. To my knowledge you need to deal with it on your side, but PETSc does have tools to help. I would look in the literature to get ideas and acclimate yourself to the problem.
The tool that PETSc has is you can repartition your mesh. Create a dual graph (any petsc examples?) put weights on the "vertice" (now elements, use particle count) and repartition. I am not sure of the best way to proceed from here ... you could create a new DMSwarm with this new "cell" DM and add particles from the old to the new, delete the old, call DMMigrate on the new DMSwarm and that will do the moving that we want.
Others will probably be able to add to this.
Thanks, Mark
On Tue, Mar 17, 2026 at 4:49 AM Miguel Molinos <[email protected]> wrote:
Dear all,
I am currently running a PIC-like implementation based on DMSwarm and DMDA. In my setup, a particle (DMSwarm) discretization represents, for example, a sphere placed at the center of a box-shaped domain discretized with a regular mesh (DMDA).
The implementation uses both MPI and OpenMP to accelerate the computations. The domain decomposition of the sphere follows a PIC-like approach: each particle is assigned to a rank based on the mesh element it belongs to, and thus inherits the rank that “owns” that element. This is essentially the same strategy used in one of the PETSc examples.
However, I have observed that, for certain configurations, some ranks end up with very few or even no particles, which leads to load imbalance.
Has anyone experienced a similar issue?
Thanks, Miguel
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.com/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!D9dNQ... >
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >
Awesome! I’ll give it a try this weekend :-). Thanks, Miguel
On 17 Mar 2026, at 16:47, Matthew Knepley <[email protected]> wrote:
On Tue, Mar 17, 2026 at 11:22 AM Miguel Molinos <[email protected] <mailto:[email protected]>> wrote:
Hi Matt and Mark,
Thank you for the feedback.
A disadvantage is that you have to work harder to get structured variable access, which DMDA has automatically. Depending on your discretization on this grid, this can be restored, but it is some work.
I’ve been trying to avoid DMPlex because I find harder to create ghost particles (define padding regions) using an unstructured mesh.
Oh, you can make DMPlex be geometrically structured very easily. So this should not be a problem.
Regarding variable access, this is not a limitation since I use the background mesh for domain decomposition purposes. The particles carry al the information.
What does the algorithm look like?
My code reproduces interaction between atoms (some sort of MD). The purposes of the mesh are: - Create a domain decomposition - Particle migration - Definition of boundary conditions (it can works like a supercell)
Oh, cool. DMPlex should be very easy then. You just create a BoxMesh (I ould probably do it from the command line), and then have it load balanced exactly as Mark said. I can help you if anything does not make sense.
Thanks,
Matt
Thanks, Miguel
On 17 Mar 2026, at 15:11, Matthew Knepley <[email protected] <mailto:[email protected]>> wrote:
On Tue, Mar 17, 2026 at 8:53 AM Mark Adams <[email protected] <mailto:[email protected]>> wrote:
This tool works at the Mat level so it does not matter, but you need to work at the DM level anyway.
I don't know of DM tools for this. Let Matt chime in.
Hi Miguel,
DMPlex can definitely do point location and load balancing, which is an advantage.
A disadvantage is that you have to work harder to get structured variable access, which DMDA has automatically. Depending on your discretization on this grid, this can be restored, but it is some work.
What does the algorithm look like?
Thanks,
Matt
On Tue, Mar 17, 2026 at 7:54 AM Miguel Molinos <[email protected] <mailto:[email protected]>> wrote:
Thank you Mark. Indeed my question was oriented towards the tools PETSc can offer to address this problem. Currently I’m using DMDA, perhaps DMPlex is more suited?
Thanks, Miguel
On 17 Mar 2026, at 12:17, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
Hi Miguel,
This is a common problem. To my knowledge you need to deal with it on your side, but PETSc does have tools to help. I would look in the literature to get ideas and acclimate yourself to the problem.
The tool that PETSc has is you can repartition your mesh. Create a dual graph (any petsc examples?) put weights on the "vertice" (now elements, use particle count) and repartition. I am not sure of the best way to proceed from here ... you could create a new DMSwarm with this new "cell" DM and add particles from the old to the new, delete the old, call DMMigrate on the new DMSwarm and that will do the moving that we want.
Others will probably be able to add to this.
Thanks, Mark
On Tue, Mar 17, 2026 at 4:49 AM Miguel Molinos <[email protected] <mailto:[email protected]>> wrote:
Dear all,
I am currently running a PIC-like implementation based on DMSwarm and DMDA. In my setup, a particle (DMSwarm) discretization represents, for example, a sphere placed at the center of a box-shaped domain discretized with a regular mesh (DMDA).
The implementation uses both MPI and OpenMP to accelerate the computations. The domain decomposition of the sphere follows a PIC-like approach: each particle is assigned to a rank based on the mesh element it belongs to, and thus inherits the rank that “owns” that element. This is essentially the same strategy used in one of the PETSc examples.
However, I have observed that, for certain configurations, some ranks end up with very few or even no particles, which leads to load imbalance.
Has anyone experienced a similar issue?
Thanks, Miguel
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.com/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!D9dNQ... >
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.com/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!D9dNQ... >
Dear all, I am experiencing an issue where particles created with DMSwarm are being increased across MPI ranks when initializing a DMSwarm over a periodic DMPlex. I follow this procedure on every rank: Create a sequential vector with the global coordinates of all particles (18,000 atoms). Call DMLocatePoints(FE_Mesh, coordinates, DM_POINTLOCATION_NONE, &cellSF) to find which particles belong to the local DMPlex partition. Then call PetscSFGetGraph. Iterate through the results and add any particle with a valid cell index (index != DMLOCATEPOINT_POINT_NOT_FOUND) to the local DMSwarm. The sum of n_atoms_local across all ranks gives 18,740 instead of the expected 18,000. It seems that particles located exactly on periodic boundaries or MPI interfaces are being identified as "found" by multiple ranks simultaneously. Does DMLocatePoints return points found in ghost/halo cells by default when the DMPlex is periodic or distributed with overlap? Thanks, Miguel
On Sun, Mar 29, 2026 at 6:25 PM Miguel Molinos <[email protected]> wrote:
Dear all,
I am experiencing an issue where particles created with DMSwarm are being increased across MPI ranks when initializing a DMSwarm over a periodic DMPlex. I follow this procedure on every rank:
1.
Create a sequential vector with the global coordinates of all particles (18,000 atoms). 2.
Call DMLocatePoints(FE_Mesh, coordinates, DM_POINTLOCATION_NONE, &cellSF) to find which particles belong to the local DMPlex partition. Then call PetscSFGetGraph. 3.
Iterate through the results and add any particle with a valid cell index (index != DMLOCATEPOINT_POINT_NOT_FOUND) to the local DMSwarm.
The sum of n_atoms_local across all ranks gives 18,740 instead of the expected 18,000. It seems that particles located exactly on periodic boundaries or MPI interfaces are being identified as "found" by multiple ranks simultaneously. Does DMLocatePoints return points found in ghost/halo cells by default when the DMPlex is periodic or distributed with overlap?
I do not think this should happen at periodic boundaries, but definitely if you have cell overlap it will happen. It iterates over all local cells to check particle location. You could discard those points by checking if the cell is in the point SF in Step 3. Thanks, Matt
Thanks,
Miguel
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >
Thank you Matt Indeed the right way was to use the rank property of the SF. Thanks, Miguel PetscCall( DMLocatePoints(FE_Mesh, coordinates, DM_POINTLOCATION_NONE, &cellSF)); PetscCall(PetscSFGetGraph(cellSF, &nroots, NULL, NULL, &LA_sfcell)); PetscCall(VecGetArray(coordinates, &coordinates_ptr)); // Note: mean_q_ptr should be filled based on n_atoms_local increment for (PetscInt site_i = 0; site_i < n_atoms; site_i++) { // Check if the point is found in the local process if (LA_sfcell[site_i].rank == rank_MPI) { idx_ptr[n_atoms_local] = site_i; for (int alpha = 0; alpha < dim; alpha++) { mean_q_ptr[n_atoms_local * dim + alpha] = Simulation_file.mean_q[site_i * dim + alpha]; } n_atoms_local++; } }
On Mar 30, 2026, at 2:50 AM, Matthew Knepley <[email protected]> wrote:
On Sun, Mar 29, 2026 at 6:25 PM Miguel Molinos <[email protected] <mailto:[email protected]>> wrote:
Dear all,
I am experiencing an issue where particles created with DMSwarm are being increased across MPI ranks when initializing a DMSwarm over a periodic DMPlex. I follow this procedure on every rank:
Create a sequential vector with the global coordinates of all particles (18,000 atoms).
Call DMLocatePoints(FE_Mesh, coordinates, DM_POINTLOCATION_NONE, &cellSF) to find which particles belong to the local DMPlex partition. Then call PetscSFGetGraph.
Iterate through the results and add any particle with a valid cell index (index != DMLOCATEPOINT_POINT_NOT_FOUND) to the local DMSwarm.
The sum of n_atoms_local across all ranks gives 18,740 instead of the expected 18,000. It seems that particles located exactly on periodic boundaries or MPI interfaces are being identified as "found" by multiple ranks simultaneously. Does DMLocatePoints return points found in ghost/halo cells by default when the DMPlex is periodic or distributed with overlap?
I do not think this should happen at periodic boundaries, but definitely if you have cell overlap it will happen. It iterates over all local cells to check particle location. You could discard those points by checking if the cell is in the point SF in Step 3.
Thanks,
Matt
Thanks,
Miguel
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.com/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!D9dNQ... >
On Mon, Mar 30, 2026 at 2:21 PM Miguel Molinos <[email protected]> wrote:
Thank you Matt
Indeed the right way was to use the rank property of the SF.
Great! Sometime you will have to show me some of the results. Thanks, Matt
Thanks, Miguel
PetscCall( DMLocatePoints(FE_Mesh, coordinates, DM_POINTLOCATION_NONE, &cellSF)); PetscCall(PetscSFGetGraph(cellSF, &nroots, NULL, NULL, &LA_sfcell));
PetscCall(VecGetArray(coordinates, &coordinates_ptr));
// Note: mean_q_ptr should be filled based on n_atoms_local increment for (PetscInt site_i = 0; site_i < n_atoms; site_i++) { // Check if the point is found in the local process if (LA_sfcell[site_i].rank == rank_MPI) { idx_ptr[n_atoms_local] = site_i; for (int alpha = 0; alpha < dim; alpha++) { mean_q_ptr[n_atoms_local * dim + alpha] = Simulation_file.mean_q[site_i * dim + alpha]; } n_atoms_local++; } }
On Mar 30, 2026, at 2:50 AM, Matthew Knepley <[email protected]> wrote:
On Sun, Mar 29, 2026 at 6:25 PM Miguel Molinos <[email protected]> wrote:
Dear all,
I am experiencing an issue where particles created with DMSwarm are being increased across MPI ranks when initializing a DMSwarm over a periodic DMPlex. I follow this procedure on every rank:
1.
Create a sequential vector with the global coordinates of all particles (18,000 atoms). 2.
Call DMLocatePoints(FE_Mesh, coordinates, DM_POINTLOCATION_NONE, &cellSF) to find which particles belong to the local DMPlex partition. Then call PetscSFGetGraph. 3.
Iterate through the results and add any particle with a valid cell index (index != DMLOCATEPOINT_POINT_NOT_FOUND) to the local DMSwarm.
The sum of n_atoms_local across all ranks gives 18,740 instead of the expected 18,000. It seems that particles located exactly on periodic boundaries or MPI interfaces are being identified as "found" by multiple ranks simultaneously. Does DMLocatePoints return points found in ghost/halo cells by default when the DMPlex is periodic or distributed with overlap?
I do not think this should happen at periodic boundaries, but definitely if you have cell overlap it will happen. It iterates over all local cells to check particle location. You could discard those points by checking if the cell is in the point SF in Step 3.
Thanks,
Matt
Thanks,
Miguel
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.com/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!D9dNQ... >
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >
participants (3)
-
Mark Adams -
Matthew Knepley -
Miguel Molinos