Thank you Matt for your time, What you describe seems to me the ideal approach. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST This means, locally, I need to allocate Nlocal + ghost particles (duplicated) for my model? If that so, how to do the communication between the ghost particles living in the rank i and their “real” counterpart in the rank j. Algo, as an alternative, what about: 1) Use an IS tag which contains, for each rank, a list of the global index of the neighbors particles outside of the rank. 2) Use VecCreateGhost to create a new vector which contains extra local space for the ghost components of the vector. 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the transference of data between a vector obtained with DMSwarmCreateGlobalVectorFromField 4) Do necessary computations using the vectors created with VecCreateGhost. Thanks, Miguel On Aug 2, 2024, at 8:58 AM, Matthew Knepley <[email protected]> wrote: On Thu, Aug 1, 2024 at 4:40 PM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Dear all, I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Dear all, I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor. This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something? I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions. This is feasible, but it would be good to develop a set of best practices, since we have been mainly focused on the case of non-redundant particles. Here is how I think I would do what you want. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST 2) At some interval identify particles that should be sent to other processes as ghosts. I would call these "overlap particles". The determination seems application specific, so I would leave this determination to the user right now. We do two things to these particles a) Mark chosen particles as OVERLAP b) Change rank to process we are sending to 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag 4) Mark OVERLAP particles as GHOST when they arrive There is one problem in the above algorithm. It does not allow sending particles to multiple ranks. We would have to do this in phases right now, or make a small adjustment to the interface allowing replication of particles when a set of ranks is specified. THanks, Matt Thank you, Miguel -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >

Thanks again Matt, that makes a lot more sense !! Just to check that we are on the same page. You are saying: 1. create a field define a field called "owner rank" for each particle. 2. Identify the phantom particles and modify the internal variable defined by the DMSwarmField_rank variable. 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the new local vector including the ghost particles. 4. Then, once the calculations are done, rename the DMSwarmField_rank variable using the "owner rank" variable and call DMSwarmMigrate(*,PETSC_FALSE) once again. Thank you, Miguel On Aug 2, 2024, at 5:33 PM, Matthew Knepley <[email protected]> wrote: On Fri, Aug 2, 2024 at 11:15 AM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thank you Matt for your time, What you describe seems to me the ideal approach. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST This means, locally, I need to allocate Nlocal + ghost particles (duplicated) for my model? I would do it another way. I would allocate the particles with no overlap and set them up. Then I would identify the halo particles, mark them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work since we cannot tell the difference between particles we received and particles we sent. Okay, instead of the `ghost` field we need an `owner rank` field. So then we 1) Setup the non-overlapping particles 2) Identify the halo particles 3) Change the `rank`, but not the `owner rank` 4) Call DMSwarmMigrate() Now we can identify ghost particles by the `owner rank` If that so, how to do the communication between the ghost particles living in the rank i and their “real” counterpart in the rank j. Algo, as an alternative, what about: 1) Use an IS tag which contains, for each rank, a list of the global index of the neighbors particles outside of the rank. 2) Use VecCreateGhost to create a new vector which contains extra local space for the ghost components of the vector. 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the transference of data between a vector obtained with DMSwarmCreateGlobalVectorFromField 4) Do necessary computations using the vectors created with VecCreateGhost. This is essentially what Migrate() does. I was trying to reuse the code. Thanks, Matt Thanks, Miguel On Aug 2, 2024, at 8:58 AM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Thu, Aug 1, 2024 at 4:40 PM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: This Message Is From an External Sender This message came from outside your organization. Dear all, I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor. This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something? I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions. This is feasible, but it would be good to develop a set of best practices, since we have been mainly focused on the case of non-redundant particles. Here is how I think I would do what you want. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST 2) At some interval identify particles that should be sent to other processes as ghosts. I would call these "overlap particles". The determination seems application specific, so I would leave this determination to the user right now. We do two things to these particles a) Mark chosen particles as OVERLAP b) Change rank to process we are sending to 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag 4) Mark OVERLAP particles as GHOST when they arrive There is one problem in the above algorithm. It does not allow sending particles to multiple ranks. We would have to do this in phases right now, or make a small adjustment to the interface allowing replication of particles when a set of ranks is specified. THanks, Matt Thank you, Miguel -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >

On Fri, Aug 2, 2024 at 7:15 PM MIGUEL MOLINOS PEREZ <[email protected]> wrote:

Thanks again Matt, that makes a lot more sense !!

Just to check that we are on the same page. You are saying:

1. create a field define a field called "owner rank" for each particle.

2. Identify the phantom particles and modify the internal variable defined by the DMSwarmField_rank variable.

3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the new local vector including the ghost particles.

4. Then, once the calculations are done, rename the DMSwarmField_rank variable using the "owner rank" variable and call DMSwarmMigrate(*,PETSC_FALSE) once again.

I don't think we need this last step. We can just remove those ghost particles for the next step I think. Thanks, Matt

Thank you, Miguel

On Aug 2, 2024, at 5:33 PM, Matthew Knepley <[email protected]> wrote:

On Fri, Aug 2, 2024 at 11:15 AM MIGUEL MOLINOS PEREZ <[email protected]> wrote:

...

Thank you Matt for your time,

What you describe seems to me the ideal approach.

1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST

This means, locally, I need to allocate Nlocal + ghost particles (duplicated) for my model?

I would do it another way. I would allocate the particles with no overlap and set them up. Then I would identify the halo particles, mark them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work since we cannot tell the difference between particles we received and particles we sent. Okay, instead of the `ghost` field we need an `owner rank` field. So then we

1) Setup the non-overlapping particles

2) Identify the halo particles

3) Change the `rank`, but not the `owner rank`

4) Call DMSwarmMigrate()

Now we can identify ghost particles by the `owner rank`

...

If that so, how to do the communication between the ghost particles living in the rank i and their “real” counterpart in the rank j.

Algo, as an alternative, what about: 1) Use an IS tag which contains, for each rank, a list of the global index of the neighbors particles outside of the rank. 2) Use VecCreateGhost to create a new vector which contains extra local space for the ghost components of the vector. 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the transference of data between a vector obtained with DMSwarmCreateGlobalVectorFromField 4) Do necessary computations using the vectors created with VecCreateGhost.

This is essentially what Migrate() does. I was trying to reuse the code.

Thanks,

Matt

...

Thanks, Miguel

On Aug 2, 2024, at 8:58 AM, Matthew Knepley <[email protected]> wrote:

On Thu, Aug 1, 2024 at 4:40 PM MIGUEL MOLINOS PEREZ <[email protected]> wrote:

...

This Message Is From an External Sender This message came from outside your organization.

Dear all,

I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor.

This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something?

I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions.

This is feasible, but it would be good to develop a set of best practices, since we have been mainly focused on the case of non-redundant particles. Here is how I think I would do what you want.

1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST

2) At some interval identify particles that should be sent to other processes as ghosts. I would call these "overlap particles". The determination seems application specific, so I would leave this determination to the user right now. We do two things to these particles

a) Mark chosen particles as OVERLAP

b) Change rank to process we are sending to

3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag

4) Mark OVERLAP particles as GHOST when they arrive

There is one problem in the above algorithm. It does not allow sending particles to multiple ranks. We would have to do this in phases right now, or make a small adjustment to the interface allowing replication of particles when a set of ranks is specified.

THanks,

Matt

...

Thank you, Miguel

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >

Hi Matt et al, I’ve been working on the scheme that you proposed to create ghost particles (atoms in my case), and it works! With a couple of caveats: -1º In general the overlap particles will be migrate from their own rank to more than one neighbor rank, this is specially relevant for those located close to the corners. Therefore, you'll need to call DMSwarmMigrate several times (27 times for 3D cells), during the migration process. -2º You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed algorithm will not work at all! Hope this helps to other folks! I have a follow-up question about periodic bcc on this context, should I open a new thread of keep posting here? Thanks, Miguel On Aug 7, 2024, at 4:22 AM, MIGUEL MOLINOS PEREZ <[email protected]> wrote: Thanks Matt, I think I'll start by making a small program as a proof of concept. Then, if it works I'll implement it in my code and I'll be happy to share it too :-) Miguel On Aug 4, 2024, at 3:30 AM, Matthew Knepley <[email protected]> wrote: On Fri, Aug 2, 2024 at 7:15 PM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thanks again Matt, that makes a lot more sense !! Just to check that we are on the same page. You are saying: 1. create a field define a field called "owner rank" for each particle. 2. Identify the phantom particles and modify the internal variable defined by the DMSwarmField_rank variable. 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the new local vector including the ghost particles. 4. Then, once the calculations are done, rename the DMSwarmField_rank variable using the "owner rank" variable and call DMSwarmMigrate(*,PETSC_FALSE) once again. I don't think we need this last step. We can just remove those ghost particles for the next step I think. Thanks, Matt Thank you, Miguel On Aug 2, 2024, at 5:33 PM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Fri, Aug 2, 2024 at 11:15 AM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thank you Matt for your time, What you describe seems to me the ideal approach. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST This means, locally, I need to allocate Nlocal + ghost particles (duplicated) for my model? I would do it another way. I would allocate the particles with no overlap and set them up. Then I would identify the halo particles, mark them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work since we cannot tell the difference between particles we received and particles we sent. Okay, instead of the `ghost` field we need an `owner rank` field. So then we 1) Setup the non-overlapping particles 2) Identify the halo particles 3) Change the `rank`, but not the `owner rank` 4) Call DMSwarmMigrate() Now we can identify ghost particles by the `owner rank` If that so, how to do the communication between the ghost particles living in the rank i and their “real” counterpart in the rank j. Algo, as an alternative, what about: 1) Use an IS tag which contains, for each rank, a list of the global index of the neighbors particles outside of the rank. 2) Use VecCreateGhost to create a new vector which contains extra local space for the ghost components of the vector. 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the transference of data between a vector obtained with DMSwarmCreateGlobalVectorFromField 4) Do necessary computations using the vectors created with VecCreateGhost. This is essentially what Migrate() does. I was trying to reuse the code. Thanks, Matt Thanks, Miguel On Aug 2, 2024, at 8:58 AM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Thu, Aug 1, 2024 at 4:40 PM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: This Message Is From an External Sender This message came from outside your organization. Dear all, I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor. This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something? I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions. This is feasible, but it would be good to develop a set of best practices, since we have been mainly focused on the case of non-redundant particles. Here is how I think I would do what you want. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST 2) At some interval identify particles that should be sent to other processes as ghosts. I would call these "overlap particles". The determination seems application specific, so I would leave this determination to the user right now. We do two things to these particles a) Mark chosen particles as OVERLAP b) Change rank to process we are sending to 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag 4) Mark OVERLAP particles as GHOST when they arrive There is one problem in the above algorithm. It does not allow sending particles to multiple ranks. We would have to do this in phases right now, or make a small adjustment to the interface allowing replication of particles when a set of ranks is specified. THanks, Matt Thank you, Miguel -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >

Thank you Matt. Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic implementation to see if there is some workaround. The idea of adding new particles is interesting. However, in that case, we need to initialize the new (ghost) particles using the fields of the “real” particle, right? This can be done using something like: VecGhostUpdateBegin<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhos... >(Vec<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;... > globalout,InsertMode<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertM... > ADD_VALUES<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VAL... >, ScatterMode<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Scatter... > SCATTER_REVERSE<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER... >); VecGhostUpdateEnd<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhos... >(Vec<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;... > globalout,InsertMode<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertM... > ADD_VALUES<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VAL... >, ScatterMode<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Scatter... > SCATTER_REVERSE<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER... >); for the particle fields (?). Thanks, Miguel On Sep 26, 2024, at 3:53 PM, Matthew Knepley <[email protected]> wrote: On Thu, Sep 26, 2024 at 6:31 AM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Hi Matt et al, I’ve been working on the scheme that you proposed to create ghost particles (atoms in my case), and it works! With a couple of caveats: -1º In general the overlap particles will be migrate from their own rank to more than one neighbor rank, this is specially relevant for those located close to the corners. Therefore, you'll need to call DMSwarmMigrate several times (27 times for 3D cells), during the migration process. That is terrible. Let's just fix DMSwarmMigrate to have a mode that sends the particle to all overlapping neighbors at once. It can't be that hard. -2º You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed algorithm will not work at all! Oh, I should have thought of that. Sorry. I can help code up that extension. Can you take a quick look at the BASIC code? Right now, we just use the rank attached to the particle to send it. We could have an arrays of ranks, but that seems crazy, and would blow up particle storage. How about just adding new particles with the other ranks right before migration? Thanks, Matt Hope this helps to other folks! I have a follow-up question about periodic bcc on this context, should I open a new thread of keep posting here? Thanks, Miguel On Aug 7, 2024, at 4:22 AM, MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thanks Matt, I think I'll start by making a small program as a proof of concept. Then, if it works I'll implement it in my code and I'll be happy to share it too :-) Miguel On Aug 4, 2024, at 3:30 AM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Fri, Aug 2, 2024 at 7:15 PM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thanks again Matt, that makes a lot more sense !! Just to check that we are on the same page. You are saying: 1. create a field define a field called "owner rank" for each particle. 2. Identify the phantom particles and modify the internal variable defined by the DMSwarmField_rank variable. 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the new local vector including the ghost particles. 4. Then, once the calculations are done, rename the DMSwarmField_rank variable using the "owner rank" variable and call DMSwarmMigrate(*,PETSC_FALSE) once again. I don't think we need this last step. We can just remove those ghost particles for the next step I think. Thanks, Matt Thank you, Miguel On Aug 2, 2024, at 5:33 PM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Fri, Aug 2, 2024 at 11:15 AM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thank you Matt for your time, What you describe seems to me the ideal approach. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST This means, locally, I need to allocate Nlocal + ghost particles (duplicated) for my model? I would do it another way. I would allocate the particles with no overlap and set them up. Then I would identify the halo particles, mark them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work since we cannot tell the difference between particles we received and particles we sent. Okay, instead of the `ghost` field we need an `owner rank` field. So then we 1) Setup the non-overlapping particles 2) Identify the halo particles 3) Change the `rank`, but not the `owner rank` 4) Call DMSwarmMigrate() Now we can identify ghost particles by the `owner rank` If that so, how to do the communication between the ghost particles living in the rank i and their “real” counterpart in the rank j. Algo, as an alternative, what about: 1) Use an IS tag which contains, for each rank, a list of the global index of the neighbors particles outside of the rank. 2) Use VecCreateGhost to create a new vector which contains extra local space for the ghost components of the vector. 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the transference of data between a vector obtained with DMSwarmCreateGlobalVectorFromField 4) Do necessary computations using the vectors created with VecCreateGhost. This is essentially what Migrate() does. I was trying to reuse the code. Thanks, Matt Thanks, Miguel On Aug 2, 2024, at 8:58 AM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Thu, Aug 1, 2024 at 4:40 PM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: This Message Is From an External Sender This message came from outside your organization. Dear all, I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor. This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something? I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions. This is feasible, but it would be good to develop a set of best practices, since we have been mainly focused on the case of non-redundant particles. Here is how I think I would do what you want. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST 2) At some interval identify particles that should be sent to other processes as ghosts. I would call these "overlap particles". The determination seems application specific, so I would leave this determination to the user right now. We do two things to these particles a) Mark chosen particles as OVERLAP b) Change rank to process we are sending to 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag 4) Mark OVERLAP particles as GHOST when they arrive There is one problem in the above algorithm. It does not allow sending particles to multiple ranks. We would have to do this in phases right now, or make a small adjustment to the interface allowing replication of particles when a set of ranks is specified. THanks, Matt Thank you, Miguel -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >

I see, you mean: Create the ghost particles at the local cell with the same properties as particle 1 (duplicate the original particle) but different value DMSwarmField_rank. Then, call DMSwarmMigrate(*,PETSC_FALSE) so we do the migration and delete the local copies of the particle 1. Right? Thanks, Miguel On Sep 26, 2024, at 11:09 PM, Matthew Knepley <[email protected]> wrote: On Thu, Sep 26, 2024 at 11:20 AM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thank you Matt. Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic implementation to see if there is some workaround. The idea of adding new particles is interesting. However, in that case, we need to initialize the new (ghost) particles using the fields of the “real” particle, right? This can be done using something like: VecGhostUpdateBegin<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhos... >(Vec<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;... > globalout,InsertMode<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertM... > ADD_VALUES<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VAL... >, ScatterMode<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Scatter... > SCATTER_REVERSE<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER... >); VecGhostUpdateEnd<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhos... >(Vec<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;... > globalout,InsertMode<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertM... > ADD_VALUES<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VAL... >, ScatterMode<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Scatter... > SCATTER_REVERSE<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER... >); for the particle fields (?). I think we can just copy from the local particle. For example, suppose I decide that particle 1 should go to rank 5, 12, and 27. Then I first set p1.rank = 5, then I add two new particles with the same values as particle 1, but with rank = 12 and 27. Then when I call migrate, it will move these three particles to the correct processes, and delete the original particles and the copies from the local set. Thanks, Matt Thanks, Miguel On Sep 26, 2024, at 3:53 PM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Thu, Sep 26, 2024 at 6:31 AM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Hi Matt et al, I’ve been working on the scheme that you proposed to create ghost particles (atoms in my case), and it works! With a couple of caveats: -1º In general the overlap particles will be migrate from their own rank to more than one neighbor rank, this is specially relevant for those located close to the corners. Therefore, you'll need to call DMSwarmMigrate several times (27 times for 3D cells), during the migration process. That is terrible. Let's just fix DMSwarmMigrate to have a mode that sends the particle to all overlapping neighbors at once. It can't be that hard. -2º You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed algorithm will not work at all! Oh, I should have thought of that. Sorry. I can help code up that extension. Can you take a quick look at the BASIC code? Right now, we just use the rank attached to the particle to send it. We could have an arrays of ranks, but that seems crazy, and would blow up particle storage. How about just adding new particles with the other ranks right before migration? Thanks, Matt Hope this helps to other folks! I have a follow-up question about periodic bcc on this context, should I open a new thread of keep posting here? Thanks, Miguel On Aug 7, 2024, at 4:22 AM, MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thanks Matt, I think I'll start by making a small program as a proof of concept. Then, if it works I'll implement it in my code and I'll be happy to share it too :-) Miguel On Aug 4, 2024, at 3:30 AM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Fri, Aug 2, 2024 at 7:15 PM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thanks again Matt, that makes a lot more sense !! Just to check that we are on the same page. You are saying: 1. create a field define a field called "owner rank" for each particle. 2. Identify the phantom particles and modify the internal variable defined by the DMSwarmField_rank variable. 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the new local vector including the ghost particles. 4. Then, once the calculations are done, rename the DMSwarmField_rank variable using the "owner rank" variable and call DMSwarmMigrate(*,PETSC_FALSE) once again. I don't think we need this last step. We can just remove those ghost particles for the next step I think. Thanks, Matt Thank you, Miguel On Aug 2, 2024, at 5:33 PM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Fri, Aug 2, 2024 at 11:15 AM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thank you Matt for your time, What you describe seems to me the ideal approach. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST This means, locally, I need to allocate Nlocal + ghost particles (duplicated) for my model? I would do it another way. I would allocate the particles with no overlap and set them up. Then I would identify the halo particles, mark them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work since we cannot tell the difference between particles we received and particles we sent. Okay, instead of the `ghost` field we need an `owner rank` field. So then we 1) Setup the non-overlapping particles 2) Identify the halo particles 3) Change the `rank`, but not the `owner rank` 4) Call DMSwarmMigrate() Now we can identify ghost particles by the `owner rank` If that so, how to do the communication between the ghost particles living in the rank i and their “real” counterpart in the rank j. Algo, as an alternative, what about: 1) Use an IS tag which contains, for each rank, a list of the global index of the neighbors particles outside of the rank. 2) Use VecCreateGhost to create a new vector which contains extra local space for the ghost components of the vector. 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the transference of data between a vector obtained with DMSwarmCreateGlobalVectorFromField 4) Do necessary computations using the vectors created with VecCreateGhost. This is essentially what Migrate() does. I was trying to reuse the code. Thanks, Matt Thanks, Miguel On Aug 2, 2024, at 8:58 AM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Thu, Aug 1, 2024 at 4:40 PM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: This Message Is From an External Sender This message came from outside your organization. Dear all, I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor. This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something? I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions. This is feasible, but it would be good to develop a set of best practices, since we have been mainly focused on the case of non-redundant particles. Here is how I think I would do what you want. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST 2) At some interval identify particles that should be sent to other processes as ghosts. I would call these "overlap particles". The determination seems application specific, so I would leave this determination to the user right now. We do two things to these particles a) Mark chosen particles as OVERLAP b) Change rank to process we are sending to 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag 4) Mark OVERLAP particles as GHOST when they arrive There is one problem in the above algorithm. It does not allow sending particles to multiple ranks. We would have to do this in phases right now, or make a small adjustment to the interface allowing replication of particles when a set of ranks is specified. THanks, Matt Thank you, Miguel -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >

Thank you Matt, let me give it try. Miguel On Sep 27, 2024, at 3:44 AM, Matthew Knepley <[email protected]> wrote: On Thu, Sep 26, 2024 at 7:18 PM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: I see, you mean: Create the ghost particles at the local cell with the same properties as particle 1 (duplicate the original particle) but different value DMSwarmField_rank. Then, call DMSwarmMigrate(*,PETSC_FALSE) so we do the migration and delete the local copies of the particle 1. Right? Yep. I think it will work, from what I know about BASIC. Thanks, Matt Thanks, Miguel On Sep 26, 2024, at 11:09 PM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Thu, Sep 26, 2024 at 11:20 AM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thank you Matt. Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic implementation to see if there is some workaround. The idea of adding new particles is interesting. However, in that case, we need to initialize the new (ghost) particles using the fields of the “real” particle, right? This can be done using something like: VecGhostUpdateBegin<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhos... >(Vec<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;... > globalout,InsertMode<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertM... > ADD_VALUES<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VAL... >, ScatterMode<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Scatter... > SCATTER_REVERSE<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER... >); VecGhostUpdateEnd<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhos... >(Vec<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;... > globalout,InsertMode<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertM... > ADD_VALUES<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VAL... >, ScatterMode<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Scatter... > SCATTER_REVERSE<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER... >); for the particle fields (?). I think we can just copy from the local particle. For example, suppose I decide that particle 1 should go to rank 5, 12, and 27. Then I first set p1.rank = 5, then I add two new particles with the same values as particle 1, but with rank = 12 and 27. Then when I call migrate, it will move these three particles to the correct processes, and delete the original particles and the copies from the local set. Thanks, Matt Thanks, Miguel On Sep 26, 2024, at 3:53 PM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Thu, Sep 26, 2024 at 6:31 AM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Hi Matt et al, I’ve been working on the scheme that you proposed to create ghost particles (atoms in my case), and it works! With a couple of caveats: -1º In general the overlap particles will be migrate from their own rank to more than one neighbor rank, this is specially relevant for those located close to the corners. Therefore, you'll need to call DMSwarmMigrate several times (27 times for 3D cells), during the migration process. That is terrible. Let's just fix DMSwarmMigrate to have a mode that sends the particle to all overlapping neighbors at once. It can't be that hard. -2º You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed algorithm will not work at all! Oh, I should have thought of that. Sorry. I can help code up that extension. Can you take a quick look at the BASIC code? Right now, we just use the rank attached to the particle to send it. We could have an arrays of ranks, but that seems crazy, and would blow up particle storage. How about just adding new particles with the other ranks right before migration? Thanks, Matt Hope this helps to other folks! I have a follow-up question about periodic bcc on this context, should I open a new thread of keep posting here? Thanks, Miguel On Aug 7, 2024, at 4:22 AM, MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thanks Matt, I think I'll start by making a small program as a proof of concept. Then, if it works I'll implement it in my code and I'll be happy to share it too :-) Miguel On Aug 4, 2024, at 3:30 AM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Fri, Aug 2, 2024 at 7:15 PM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thanks again Matt, that makes a lot more sense !! Just to check that we are on the same page. You are saying: 1. create a field define a field called "owner rank" for each particle. 2. Identify the phantom particles and modify the internal variable defined by the DMSwarmField_rank variable. 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the new local vector including the ghost particles. 4. Then, once the calculations are done, rename the DMSwarmField_rank variable using the "owner rank" variable and call DMSwarmMigrate(*,PETSC_FALSE) once again. I don't think we need this last step. We can just remove those ghost particles for the next step I think. Thanks, Matt Thank you, Miguel On Aug 2, 2024, at 5:33 PM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Fri, Aug 2, 2024 at 11:15 AM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thank you Matt for your time, What you describe seems to me the ideal approach. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST This means, locally, I need to allocate Nlocal + ghost particles (duplicated) for my model? I would do it another way. I would allocate the particles with no overlap and set them up. Then I would identify the halo particles, mark them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work since we cannot tell the difference between particles we received and particles we sent. Okay, instead of the `ghost` field we need an `owner rank` field. So then we 1) Setup the non-overlapping particles 2) Identify the halo particles 3) Change the `rank`, but not the `owner rank` 4) Call DMSwarmMigrate() Now we can identify ghost particles by the `owner rank` If that so, how to do the communication between the ghost particles living in the rank i and their “real” counterpart in the rank j. Algo, as an alternative, what about: 1) Use an IS tag which contains, for each rank, a list of the global index of the neighbors particles outside of the rank. 2) Use VecCreateGhost to create a new vector which contains extra local space for the ghost components of the vector. 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the transference of data between a vector obtained with DMSwarmCreateGlobalVectorFromField 4) Do necessary computations using the vectors created with VecCreateGhost. This is essentially what Migrate() does. I was trying to reuse the code. Thanks, Matt Thanks, Miguel On Aug 2, 2024, at 8:58 AM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Thu, Aug 1, 2024 at 4:40 PM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: This Message Is From an External Sender This message came from outside your organization. Dear all, I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor. This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something? I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions. This is feasible, but it would be good to develop a set of best practices, since we have been mainly focused on the case of non-redundant particles. Here is how I think I would do what you want. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST 2) At some interval identify particles that should be sent to other processes as ghosts. I would call these "overlap particles". The determination seems application specific, so I would leave this determination to the user right now. We do two things to these particles a) Mark chosen particles as OVERLAP b) Change rank to process we are sending to 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag 4) Mark OVERLAP particles as GHOST when they arrive There is one problem in the above algorithm. It does not allow sending particles to multiple ranks. We would have to do this in phases right now, or make a small adjustment to the interface allowing replication of particles when a set of ranks is specified. THanks, Matt Thank you, Miguel -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >

Hi Dave: On Sep 27, 2024, at 4:43 PM, Dave May <[email protected]> wrote: Hi all, Sorry to be very late coming to this party. Welcome to the party :) I had always intended for swarm to support ghost particles. The way I wanted to do this was via a new migrate type (DMSWARM_MIGRATE_GHOST) (or a better name - suggestions welcome) and a new migrate function. The new migrate function would call a user defined function, with a signature DMSwarmMigrateGhostSelect(DM swarm, void *user_context, PetscInt *nghost, PetscMPIInt *ranks[] PetscInt *pointids[]) DMSwarmMigrateGhostSelect() returns an array of all points indices which are to be ghosted, and the ranks where these ghost points should be sent. I coded myself something similar (see create_ghost_atoms in the attached file) . I use the background DMDA mesh to create the paddles (3D case) for the ghost particles, and DMDAGetNeighbors to get a list of the neighbor ranks. Then in the ghost migrate function one would use ranks[] to define the scatter pattern and then pack swarm points found in pointids[] into buffers and then send em. You mean do something like this lines in DMSwarmMigrate_Push_Basic: for (p = 0; p < npoints; p++) { nrank = rankval[p]; if (nrank != rank) { /* copy point into buffer */ PetscCall(DMSwarmDataBucketFillPackedArray(swarm->db, p, point_buffer)); /* insert point buffer into DMSwarmDataExchanger */ PetscCall(DMSwarmDataExPackData(de, nrank, 1, point_buffer)); } } ??? Removal of points (although unlikely required) could be achieved by sorting and removing dups from pointids[] and deleting indices from largest to smallest. Duplicate indices will likely appear in pointids[] in most ghost scenarios (eg eval an SPH kernel) and thus need to be handled if one is to remove sent points Lastly, I would have a mechanism to "reset" the DMSwarm post ghost scatter to remove the scattered ghost points. As ghosted points would appear at the back of the array a reset would just mean setting the length of the valid points back to the value it had prior to the ghosting operation. Yep, I implemented something similar (I think) in destroy_ghost_atoms inside Ghosts.cpp. I think it’s fairly straightforward to add support for this. Let me know if want to tackle it, or if you want me to code it up. I would be happy to discuss further about it! Let me know how. An other interesting question to address (which I am facing right now) would be the communication between ghosts and real particles during a SNES solver. For instance, something like this: VecScatterBegin(VecScatter scatter,Vec globalin,Vec localin,InsertMode INSERT_VALUES, ScatterMode SCATTER_FORWARD); VecScatterEnd(VecScatter scatter,Vec globalin,Vec localin,InsertMode INSERT_VALUES,ScatterMode SCATTER_FORWARD); /* For example, do local calculations from localin to localout */ VecScatterBegin(VecScatter scatter,Vec localout,Vec globalout,InsertMode ADD_VALUES,ScatterMode SCATTER_REVERSE); VecScatterEnd(VecScatter scatter,Vec localout,Vec globalout,InsertMode ADD_VALUES,ScatterMode SCATTER_REVERSE); Thanks, Miguel Cheers, Dave On Fri, 27 Sept 2024 at 01:51, MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thank you Matt, let me give it try. Miguel On Sep 27, 2024, at 3:44 AM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Thu, Sep 26, 2024 at 7:18 PM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: I see, you mean: Create the ghost particles at the local cell with the same properties as particle 1 (duplicate the original particle) but different value DMSwarmField_rank. Then, call DMSwarmMigrate(*,PETSC_FALSE) so we do the migration and delete the local copies of the particle 1. Right? Yep. I think it will work, from what I know about BASIC. Thanks, Matt Thanks, Miguel On Sep 26, 2024, at 11:09 PM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Thu, Sep 26, 2024 at 11:20 AM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thank you Matt. Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic implementation to see if there is some workaround. The idea of adding new particles is interesting. However, in that case, we need to initialize the new (ghost) particles using the fields of the “real” particle, right? This can be done using something like: VecGhostUpdateBegin<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhostUpdateBegin/__;!!G_uCfscf7eWS!dHNd2Wiqx529QMIoMUJ2N9KCSssD66koKRtx2-UcBvPo__o_PDSsijzmBTWcrfapO5ug6AnIaoy6lJpfWJBnPg$>(Vec<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;!!G_uCfscf7eWS!dHNd2Wiqx529QMIoMUJ2N9KCSssD66koKRtx2-UcBvPo__o_PDSsijzmBTWcrfapO5ug6AnIaoy6lJrHgeioUw$> globalout,InsertMode<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertMode/__;!!G_uCfscf7eWS!dHNd2Wiqx529QMIoMUJ2N9KCSssD66koKRtx2-UcBvPo__o_PDSsijzmBTWcrfapO5ug6AnIaoy6lJrYr3gqrg$> ADD_VALUES<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VALUES/__;!!G_uCfscf7eWS!dHNd2Wiqx529QMIoMUJ2N9KCSssD66koKRtx2-UcBvPo__o_PDSsijzmBTWcrfapO5ug6AnIaoy6lJoF-UyWOQ$>, ScatterMode<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/ScatterMode/__;!!G_uCfscf7eWS!dHNd2Wiqx529QMIoMUJ2N9KCSssD66koKRtx2-UcBvPo__o_PDSsijzmBTWcrfapO5ug6AnIaoy6lJoFdAXMjQ$> SCATTER_REVERSE<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER_REVERSE/__;!!G_uCfscf7eWS!dHNd2Wiqx529QMIoMUJ2N9KCSssD66koKRtx2-UcBvPo__o_PDSsijzmBTWcrfapO5ug6AnIaoy6lJpjVY8L8w$>); VecGhostUpdateEnd<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhostUpdateEnd/__;!!G_uCfscf7eWS!dHNd2Wiqx529QMIoMUJ2N9KCSssD66koKRtx2-UcBvPo__o_PDSsijzmBTWcrfapO5ug6AnIaoy6lJoXZicuuQ$>(Vec<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;!!G_uCfscf7eWS!dHNd2Wiqx529QMIoMUJ2N9KCSssD66koKRtx2-UcBvPo__o_PDSsijzmBTWcrfapO5ug6AnIaoy6lJrHgeioUw$> globalout,InsertMode<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertMode/__;!!G_uCfscf7eWS!dHNd2Wiqx529QMIoMUJ2N9KCSssD66koKRtx2-UcBvPo__o_PDSsijzmBTWcrfapO5ug6AnIaoy6lJrYr3gqrg$> ADD_VALUES<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VALUES/__;!!G_uCfscf7eWS!dHNd2Wiqx529QMIoMUJ2N9KCSssD66koKRtx2-UcBvPo__o_PDSsijzmBTWcrfapO5ug6AnIaoy6lJoF-UyWOQ$>, ScatterMode<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/ScatterMode/__;!!G_uCfscf7eWS!dHNd2Wiqx529QMIoMUJ2N9KCSssD66koKRtx2-UcBvPo__o_PDSsijzmBTWcrfapO5ug6AnIaoy6lJoFdAXMjQ$> SCATTER_REVERSE<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER_REVERSE/__;!!G_uCfscf7eWS!dHNd2Wiqx529QMIoMUJ2N9KCSssD66koKRtx2-UcBvPo__o_PDSsijzmBTWcrfapO5ug6AnIaoy6lJpjVY8L8w$>); for the particle fields (?). I think we can just copy from the local particle. For example, suppose I decide that particle 1 should go to rank 5, 12, and 27. Then I first set p1.rank = 5, then I add two new particles with the same values as particle 1, but with rank = 12 and 27. Then when I call migrate, it will move these three particles to the correct processes, and delete the original particles and the copies from the local set. Thanks, Matt Thanks, Miguel On Sep 26, 2024, at 3:53 PM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Thu, Sep 26, 2024 at 6:31 AM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Hi Matt et al, I’ve been working on the scheme that you proposed to create ghost particles (atoms in my case), and it works! With a couple of caveats: -1º In general the overlap particles will be migrate from their own rank to more than one neighbor rank, this is specially relevant for those located close to the corners. Therefore, you'll need to call DMSwarmMigrate several times (27 times for 3D cells), during the migration process. That is terrible. Let's just fix DMSwarmMigrate to have a mode that sends the particle to all overlapping neighbors at once. It can't be that hard. -2º You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed algorithm will not work at all! Oh, I should have thought of that. Sorry. I can help code up that extension. Can you take a quick look at the BASIC code? Right now, we just use the rank attached to the particle to send it. We could have an arrays of ranks, but that seems crazy, and would blow up particle storage. How about just adding new particles with the other ranks right before migration? Thanks, Matt Hope this helps to other folks! I have a follow-up question about periodic bcc on this context, should I open a new thread of keep posting here? Thanks, Miguel On Aug 7, 2024, at 4:22 AM, MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thanks Matt, I think I'll start by making a small program as a proof of concept. Then, if it works I'll implement it in my code and I'll be happy to share it too :-) Miguel On Aug 4, 2024, at 3:30 AM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Fri, Aug 2, 2024 at 7:15 PM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thanks again Matt, that makes a lot more sense !! Just to check that we are on the same page. You are saying: 1. create a field define a field called "owner rank" for each particle. 2. Identify the phantom particles and modify the internal variable defined by the DMSwarmField_rank variable. 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the new local vector including the ghost particles. 4. Then, once the calculations are done, rename the DMSwarmField_rank variable using the "owner rank" variable and call DMSwarmMigrate(*,PETSC_FALSE) once again. I don't think we need this last step. We can just remove those ghost particles for the next step I think. Thanks, Matt Thank you, Miguel On Aug 2, 2024, at 5:33 PM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Fri, Aug 2, 2024 at 11:15 AM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thank you Matt for your time, What you describe seems to me the ideal approach. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST This means, locally, I need to allocate Nlocal + ghost particles (duplicated) for my model? I would do it another way. I would allocate the particles with no overlap and set them up. Then I would identify the halo particles, mark them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work since we cannot tell the difference between particles we received and particles we sent. Okay, instead of the `ghost` field we need an `owner rank` field. So then we 1) Setup the non-overlapping particles 2) Identify the halo particles 3) Change the `rank`, but not the `owner rank` 4) Call DMSwarmMigrate() Now we can identify ghost particles by the `owner rank` If that so, how to do the communication between the ghost particles living in the rank i and their “real” counterpart in the rank j. Algo, as an alternative, what about: 1) Use an IS tag which contains, for each rank, a list of the global index of the neighbors particles outside of the rank. 2) Use VecCreateGhost to create a new vector which contains extra local space for the ghost components of the vector. 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the transference of data between a vector obtained with DMSwarmCreateGlobalVectorFromField 4) Do necessary computations using the vectors created with VecCreateGhost. This is essentially what Migrate() does. I was trying to reuse the code. Thanks, Matt Thanks, Miguel On Aug 2, 2024, at 8:58 AM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Thu, Aug 1, 2024 at 4:40 PM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: This Message Is From an External Sender This message came from outside your organization. Dear all, I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor. This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something? I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions. This is feasible, but it would be good to develop a set of best practices, since we have been mainly focused on the case of non-redundant particles. Here is how I think I would do what you want. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST 2) At some interval identify particles that should be sent to other processes as ghosts. I would call these "overlap particles". The determination seems application specific, so I would leave this determination to the user right now. We do two things to these particles a) Mark chosen particles as OVERLAP b) Change rank to process we are sending to 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag 4) Mark OVERLAP particles as GHOST when they arrive There is one problem in the above algorithm. It does not allow sending particles to multiple ranks. We would have to do this in phases right now, or make a small adjustment to the interface allowing replication of particles when a set of ranks is specified. THanks, Matt Thank you, Miguel -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dHNd2Wiqx529QMIoMUJ2N9KCSssD66koKRtx2-UcBvPo__o_PDSsijzmBTWcrfapO5ug6AnIaoy6lJqV0G09Wg$> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dHNd2Wiqx529QMIoMUJ2N9KCSssD66koKRtx2-UcBvPo__o_PDSsijzmBTWcrfapO5ug6AnIaoy6lJqV0G09Wg$> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dHNd2Wiqx529QMIoMUJ2N9KCSssD66koKRtx2-UcBvPo__o_PDSsijzmBTWcrfapO5ug6AnIaoy6lJqV0G09Wg$> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dHNd2Wiqx529QMIoMUJ2N9KCSssD66koKRtx2-UcBvPo__o_PDSsijzmBTWcrfapO5ug6AnIaoy6lJqV0G09Wg$> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dHNd2Wiqx529QMIoMUJ2N9KCSssD66koKRtx2-UcBvPo__o_PDSsijzmBTWcrfapO5ug6AnIaoy6lJqV0G09Wg$> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dHNd2Wiqx529QMIoMUJ2N9KCSssD66koKRtx2-UcBvPo__o_PDSsijzmBTWcrfapO5ug6AnIaoy6lJqV0G09Wg$>

Hi Miguel, On Tue 1. Oct 2024 at 07:56, MIGUEL MOLINOS PEREZ <[email protected]> wrote:

Thank you Matt, it works!

The implementation is straightforward: - 1º Define the paddle regions using DMGetLocalBoundingBox with the background DMDA mesh as an auxiliary mesh for the domain-partitioning. - 2º Create an integer to count the local number of particles to be used as ghost particle for other processors (N_ghost). One particle can be counted more than one time. At the same time, fill two arrays: - one with the index of the "main particle” (local particle), - and the other with the target rank of the "main particle”. - 3º Create the new particles using DMSwarmAddNPoints - 4º Fill the new particles with the information of the “main particle” but set the internal variable DMSwarmField_rank with the target rank. - 5º Call DMSwarmMigrate(*,PETSC_TRUE). Therefore, we send the ghost particles to the corresponding processors and we delete them from the original processor. - 6º Do stuff… - 7º Delete ghost particles. This is very easy, we just have to call DMSwarmRemovePoint N_ghost times.

I think this can be easily implemented as closed routine for the DMSwarm class.

The remaining question is: how to do the communication between the “original" particle and the ghost particles? For instance, if we update some particle variable (locally) inside of a SNES context, this same variable should be updated in the ghost particles at the other processors.

I think what you are asking about is an operation similar to VecGhostUpdate{Begin,End}(). In the case of a DMSwarm I’m not sure how to define the InsertMode = ADD_VALUES? Some swarm fields do not make sense to be added. INSERT_VALUES is fine. One solution might be to have something like this DMSwarmCollectViewUpdateGhostOwners(DM dm, InsertMode mode, PetscInt nfields, const char *fieldNames[]); where one can specify the insert mode and the fields on which the insert mode will apply. Would something like that work? Cheers, Dave

PS: Hope this helps someone in the future :-)

On Sep 27, 2024, at 10:50 AM, MIGUEL MOLINOS PEREZ <[email protected]> wrote:

Thank you Matt, let me give it try.

Miguel

On Sep 27, 2024, at 3:44 AM, Matthew Knepley <[email protected]> wrote:

On Thu, Sep 26, 2024 at 7:18 PM MIGUEL MOLINOS PEREZ <[email protected]> wrote:

...

I see, you mean:

Create the ghost particles at the local cell with the same properties as particle 1 (duplicate the original particle) but different value DMSwarmField_rank. Then, call DMSwarmMigrate(*,PETSC_FALSE) so we do the migration and delete the local copies of the particle 1. Right?

Yep. I think it will work, from what I know about BASIC.

Thanks,

Matt

...

Thanks, Miguel

On Sep 26, 2024, at 11:09 PM, Matthew Knepley <[email protected]> wrote:

On Thu, Sep 26, 2024 at 11:20 AM MIGUEL MOLINOS PEREZ <[email protected]> wrote:

...

Thank you Matt.

Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic implementation to see if there is some workaround.

The idea of adding new particles is interesting. However, in that case, we need to initialize the new (ghost) particles using the fields of the “real” particle, right? This can be done using something like:

VecGhostUpdateBegin <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhos... >(Vec <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;... > globalout,InsertMode <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertM... > ADD_VALUES <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VAL... >, ScatterMode <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Scatter... > SCATTER_REVERSE <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER... >);VecGhostUpdateEnd <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhos... >(Vec <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;... > globalout,InsertMode <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertM... > ADD_VALUES <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VAL... >, ScatterMode <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Scatter... > SCATTER_REVERSE <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER... >);

for the particle fields (?).

I think we can just copy from the local particle. For example, suppose I decide that particle 1 should go to rank 5, 12, and 27. Then I first set p1.rank = 5, then I add two new particles with the same values as particle 1, but with rank = 12 and 27. Then when I call migrate, it will move these three particles to the correct processes, and delete the original particles and the copies from the local set.

Thanks,

Matt

...

Thanks, Miguel

On Sep 26, 2024, at 3:53 PM, Matthew Knepley <[email protected]> wrote:

On Thu, Sep 26, 2024 at 6:31 AM MIGUEL MOLINOS PEREZ <[email protected]> wrote:

...

Hi Matt et al,

I’ve been working on the scheme that you proposed to create ghost particles (atoms in my case), and it works! With a couple of caveats: -1º In general the overlap particles will be migrate from their own rank to more than one neighbor rank, this is specially relevant for those located close to the corners. Therefore, you'll need to call DMSwarmMigrate several times (27 times for 3D cells), during the migration process.

That is terrible. Let's just fix DMSwarmMigrate to have a mode that sends the particle to all overlapping neighbors at once. It can't be that hard.

...

-2º You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed algorithm will not work at all!

Oh, I should have thought of that. Sorry.

I can help code up that extension. Can you take a quick look at the BASIC code? Right now, we just use the rank attached to the particle to send it. We could have an arrays of ranks, but that seems crazy, and would blow up particle storage. How about just adding new particles with the other ranks right before migration?

Thanks,

Matt

...

Hope this helps to other folks!

I have a follow-up question about periodic bcc on this context, should I open a new thread of keep posting here?

Thanks, Miguel

On Aug 7, 2024, at 4:22 AM, MIGUEL MOLINOS PEREZ <[email protected]> wrote:

Thanks Matt, I think I'll start by making a small program as a proof of concept. Then, if it works I'll implement it in my code and I'll be happy to share it too :-)

Miguel

On Aug 4, 2024, at 3:30 AM, Matthew Knepley <[email protected]> wrote:

On Fri, Aug 2, 2024 at 7:15 PM MIGUEL MOLINOS PEREZ <[email protected]> wrote:

...

Thanks again Matt, that makes a lot more sense !!

Just to check that we are on the same page. You are saying:

1. create a field define a field called "owner rank" for each particle.

2. Identify the phantom particles and modify the internal variable defined by the DMSwarmField_rank variable.

3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the new local vector including the ghost particles.

4. Then, once the calculations are done, rename the DMSwarmField_rank variable using the "owner rank" variable and call DMSwarmMigrate(*,PETSC_FALSE) once again.

I don't think we need this last step. We can just remove those ghost particles for the next step I think.

Thanks,

Matt

...

Thank you, Miguel

On Aug 2, 2024, at 5:33 PM, Matthew Knepley <[email protected]> wrote:

On Fri, Aug 2, 2024 at 11:15 AM MIGUEL MOLINOS PEREZ <[email protected]> wrote:

...

Thank you Matt for your time,

What you describe seems to me the ideal approach.

1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST

This means, locally, I need to allocate Nlocal + ghost particles (duplicated) for my model?

I would do it another way. I would allocate the particles with no overlap and set them up. Then I would identify the halo particles, mark them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work since we cannot tell the difference between particles we received and particles we sent. Okay, instead of the `ghost` field we need an `owner rank` field. So then we

1) Setup the non-overlapping particles

2) Identify the halo particles

3) Change the `rank`, but not the `owner rank`

4) Call DMSwarmMigrate()

Now we can identify ghost particles by the `owner rank`

...

If that so, how to do the communication between the ghost particles living in the rank i and their “real” counterpart in the rank j.

Algo, as an alternative, what about: 1) Use an IS tag which contains, for each rank, a list of the global index of the neighbors particles outside of the rank. 2) Use VecCreateGhost to create a new vector which contains extra local space for the ghost components of the vector. 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the transference of data between a vector obtained with DMSwarmCreateGlobalVectorFromField 4) Do necessary computations using the vectors created with VecCreateGhost.

This is essentially what Migrate() does. I was trying to reuse the code.

Thanks,

Matt

...

Thanks, Miguel

On Aug 2, 2024, at 8:58 AM, Matthew Knepley <[email protected]> wrote:

On Thu, Aug 1, 2024 at 4:40 PM MIGUEL MOLINOS PEREZ <[email protected]> wrote:

...

This Message Is From an External Sender This message came from outside your organization.

Dear all,

I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor.

This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something?

I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions.

This is feasible, but it would be good to develop a set of best practices, since we have been mainly focused on the case of non-redundant particles. Here is how I think I would do what you want.

1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST

2) At some interval identify particles that should be sent to other processes as ghosts. I would call these "overlap particles". The determination seems application specific, so I would leave this determination to the user right now. We do two things to these particles

a) Mark chosen particles as OVERLAP

b) Change rank to process we are sending to

3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag

4) Mark OVERLAP particles as GHOST when they arrive

There is one problem in the above algorithm. It does not allow sending particles to multiple ranks. We would have to do this in phases right now, or make a small adjustment to the interface allowing replication of particles when a set of ranks is specified.

THanks,

Matt

...

Thank you, Miguel

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >

Sorry, I forgot to add: You think VecScatterCreate is the way to go? Thanks, Miguel On Oct 1, 2024, at 5:56 PM, MIGUEL MOLINOS PEREZ <[email protected]> wrote: Hi Dave, Would something like that work? Yes, this should work! Any idea on where to look so I can try to implement it myself? Best, Miguel On Oct 1, 2024, at 5:22 PM, Dave May <[email protected]> wrote: Hi Miguel, On Tue 1. Oct 2024 at 07:56, MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thank you Matt, it works! The implementation is straightforward: - 1º Define the paddle regions using DMGetLocalBoundingBox with the background DMDA mesh as an auxiliary mesh for the domain-partitioning. - 2º Create an integer to count the local number of particles to be used as ghost particle for other processors (N_ghost). One particle can be counted more than one time. At the same time, fill two arrays: - one with the index of the "main particle” (local particle), - and the other with the target rank of the "main particle”. - 3º Create the new particles using DMSwarmAddNPoints - 4º Fill the new particles with the information of the “main particle” but set the internal variable DMSwarmField_rank with the target rank. - 5º Call DMSwarmMigrate(*,PETSC_TRUE). Therefore, we send the ghost particles to the corresponding processors and we delete them from the original processor. - 6º Do stuff… - 7º Delete ghost particles. This is very easy, we just have to call DMSwarmRemovePoint N_ghost times. I think this can be easily implemented as closed routine for the DMSwarm class. The remaining question is: how to do the communication between the “original" particle and the ghost particles? For instance, if we update some particle variable (locally) inside of a SNES context, this same variable should be updated in the ghost particles at the other processors. I think what you are asking about is an operation similar to VecGhostUpdate{Begin,End}(). In the case of a DMSwarm I’m not sure how to define the InsertMode = ADD_VALUES? Some swarm fields do not make sense to be added. INSERT_VALUES is fine. One solution might be to have something like this DMSwarmCollectViewUpdateGhostOwners(DM dm, InsertMode mode, PetscInt nfields, const char *fieldNames[]); where one can specify the insert mode and the fields on which the insert mode will apply. Would something like that work? Cheers, Dave PS: Hope this helps someone in the future :-) On Sep 27, 2024, at 10:50 AM, MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thank you Matt, let me give it try. Miguel On Sep 27, 2024, at 3:44 AM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Thu, Sep 26, 2024 at 7:18 PM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: I see, you mean: Create the ghost particles at the local cell with the same properties as particle 1 (duplicate the original particle) but different value DMSwarmField_rank. Then, call DMSwarmMigrate(*,PETSC_FALSE) so we do the migration and delete the local copies of the particle 1. Right? Yep. I think it will work, from what I know about BASIC. Thanks, Matt Thanks, Miguel On Sep 26, 2024, at 11:09 PM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Thu, Sep 26, 2024 at 11:20 AM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thank you Matt. Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic implementation to see if there is some workaround. The idea of adding new particles is interesting. However, in that case, we need to initialize the new (ghost) particles using the fields of the “real” particle, right? This can be done using something like: VecGhostUpdateBegin<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhos... >(Vec<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;... > globalout,InsertMode<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertM... > ADD_VALUES<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VAL... >, ScatterMode<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Scatter... > SCATTER_REVERSE<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER... >); VecGhostUpdateEnd<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhos... >(Vec<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;... > globalout,InsertMode<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertM... > ADD_VALUES<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VAL... >, ScatterMode<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Scatter... > SCATTER_REVERSE<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER... >); for the particle fields (?). I think we can just copy from the local particle. For example, suppose I decide that particle 1 should go to rank 5, 12, and 27. Then I first set p1.rank = 5, then I add two new particles with the same values as particle 1, but with rank = 12 and 27. Then when I call migrate, it will move these three particles to the correct processes, and delete the original particles and the copies from the local set. Thanks, Matt Thanks, Miguel On Sep 26, 2024, at 3:53 PM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Thu, Sep 26, 2024 at 6:31 AM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Hi Matt et al, I’ve been working on the scheme that you proposed to create ghost particles (atoms in my case), and it works! With a couple of caveats: -1º In general the overlap particles will be migrate from their own rank to more than one neighbor rank, this is specially relevant for those located close to the corners. Therefore, you'll need to call DMSwarmMigrate several times (27 times for 3D cells), during the migration process. That is terrible. Let's just fix DMSwarmMigrate to have a mode that sends the particle to all overlapping neighbors at once. It can't be that hard. -2º You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed algorithm will not work at all! Oh, I should have thought of that. Sorry. I can help code up that extension. Can you take a quick look at the BASIC code? Right now, we just use the rank attached to the particle to send it. We could have an arrays of ranks, but that seems crazy, and would blow up particle storage. How about just adding new particles with the other ranks right before migration? Thanks, Matt Hope this helps to other folks! I have a follow-up question about periodic bcc on this context, should I open a new thread of keep posting here? Thanks, Miguel On Aug 7, 2024, at 4:22 AM, MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thanks Matt, I think I'll start by making a small program as a proof of concept. Then, if it works I'll implement it in my code and I'll be happy to share it too :-) Miguel On Aug 4, 2024, at 3:30 AM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Fri, Aug 2, 2024 at 7:15 PM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thanks again Matt, that makes a lot more sense !! Just to check that we are on the same page. You are saying: 1. create a field define a field called "owner rank" for each particle. 2. Identify the phantom particles and modify the internal variable defined by the DMSwarmField_rank variable. 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the new local vector including the ghost particles. 4. Then, once the calculations are done, rename the DMSwarmField_rank variable using the "owner rank" variable and call DMSwarmMigrate(*,PETSC_FALSE) once again. I don't think we need this last step. We can just remove those ghost particles for the next step I think. Thanks, Matt Thank you, Miguel On Aug 2, 2024, at 5:33 PM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Fri, Aug 2, 2024 at 11:15 AM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thank you Matt for your time, What you describe seems to me the ideal approach. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST This means, locally, I need to allocate Nlocal + ghost particles (duplicated) for my model? I would do it another way. I would allocate the particles with no overlap and set them up. Then I would identify the halo particles, mark them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work since we cannot tell the difference between particles we received and particles we sent. Okay, instead of the `ghost` field we need an `owner rank` field. So then we 1) Setup the non-overlapping particles 2) Identify the halo particles 3) Change the `rank`, but not the `owner rank` 4) Call DMSwarmMigrate() Now we can identify ghost particles by the `owner rank` If that so, how to do the communication between the ghost particles living in the rank i and their “real” counterpart in the rank j. Algo, as an alternative, what about: 1) Use an IS tag which contains, for each rank, a list of the global index of the neighbors particles outside of the rank. 2) Use VecCreateGhost to create a new vector which contains extra local space for the ghost components of the vector. 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the transference of data between a vector obtained with DMSwarmCreateGlobalVectorFromField 4) Do necessary computations using the vectors created with VecCreateGhost. This is essentially what Migrate() does. I was trying to reuse the code. Thanks, Matt Thanks, Miguel On Aug 2, 2024, at 8:58 AM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Thu, Aug 1, 2024 at 4:40 PM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: This Message Is From an External Sender This message came from outside your organization. Dear all, I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor. This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something? I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions. This is feasible, but it would be good to develop a set of best practices, since we have been mainly focused on the case of non-redundant particles. Here is how I think I would do what you want. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST 2) At some interval identify particles that should be sent to other processes as ghosts. I would call these "overlap particles". The determination seems application specific, so I would leave this determination to the user right now. We do two things to these particles a) Mark chosen particles as OVERLAP b) Change rank to process we are sending to 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag 4) Mark OVERLAP particles as GHOST when they arrive There is one problem in the above algorithm. It does not allow sending particles to multiple ranks. We would have to do this in phases right now, or make a small adjustment to the interface allowing replication of particles when a set of ranks is specified. THanks, Matt Thank you, Miguel -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >

On Tue, 1 Oct 2024 at 08:56, MIGUEL MOLINOS PEREZ <[email protected]> wrote:

Hi Dave,

Would something like that work?

Yes, this should work! Any idea on where to look so I can try to implement it myself?

I am adding support for this right now.

Best, Miguel

On Oct 1, 2024, at 5:22 PM, Dave May <[email protected]> wrote:

Hi Miguel,

On Tue 1. Oct 2024 at 07:56, MIGUEL MOLINOS PEREZ <[email protected]> wrote:

...

Thank you Matt, it works!

The implementation is straightforward: - 1º Define the paddle regions using DMGetLocalBoundingBox with the background DMDA mesh as an auxiliary mesh for the domain-partitioning. - 2º Create an integer to count the local number of particles to be used as ghost particle for other processors (N_ghost). One particle can be counted more than one time. At the same time, fill two arrays: - one with the index of the "main particle” (local particle), - and the other with the target rank of the "main particle”. - 3º Create the new particles using DMSwarmAddNPoints - 4º Fill the new particles with the information of the “main particle” but set the internal variable DMSwarmField_rank with the target rank. - 5º Call DMSwarmMigrate(*,PETSC_TRUE). Therefore, we send the ghost particles to the corresponding processors and we delete them from the original processor. - 6º Do stuff… - 7º Delete ghost particles. This is very easy, we just have to call DMSwarmRemovePoint N_ghost times.

I think this can be easily implemented as closed routine for the DMSwarm class.

The remaining question is: how to do the communication between the “original" particle and the ghost particles? For instance, if we update some particle variable (locally) inside of a SNES context, this same variable should be updated in the ghost particles at the other processors.

I think what you are asking about is an operation similar to VecGhostUpdate{Begin,End}(). In the case of a DMSwarm I’m not sure how to define the InsertMode = ADD_VALUES? Some swarm fields do not make sense to be added. INSERT_VALUES is fine.

One solution might be to have something like this

DMSwarmCollectViewUpdateGhostOwners(DM dm, InsertMode mode, PetscInt nfields, const char *fieldNames[]);

where one can specify the insert mode and the fields on which the insert mode will apply.

Would something like that work?

Cheers, Dave

...

PS: Hope this helps someone in the future :-)

On Sep 27, 2024, at 10:50 AM, MIGUEL MOLINOS PEREZ <[email protected]> wrote:

Thank you Matt, let me give it try.

Miguel

On Sep 27, 2024, at 3:44 AM, Matthew Knepley <[email protected]> wrote:

On Thu, Sep 26, 2024 at 7:18 PM MIGUEL MOLINOS PEREZ <[email protected]> wrote:

...

I see, you mean:

Create the ghost particles at the local cell with the same properties as particle 1 (duplicate the original particle) but different value DMSwarmField_rank. Then, call DMSwarmMigrate(*,PETSC_FALSE) so we do the migration and delete the local copies of the particle 1. Right?

Yep. I think it will work, from what I know about BASIC.

Thanks,

Matt

...

Thanks, Miguel

On Sep 26, 2024, at 11:09 PM, Matthew Knepley <[email protected]> wrote:

On Thu, Sep 26, 2024 at 11:20 AM MIGUEL MOLINOS PEREZ <[email protected]> wrote:

...

Thank you Matt.

Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic implementation to see if there is some workaround.

The idea of adding new particles is interesting. However, in that case, we need to initialize the new (ghost) particles using the fields of the “real” particle, right? This can be done using something like:

VecGhostUpdateBegin <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhos... >(Vec <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;... > globalout,InsertMode <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertM... > ADD_VALUES <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VAL... >, ScatterMode <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Scatter... > SCATTER_REVERSE <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER... >);VecGhostUpdateEnd <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhos... >(Vec <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;... > globalout,InsertMode <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertM... > ADD_VALUES <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VAL... >, ScatterMode <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Scatter... > SCATTER_REVERSE <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER... >);

for the particle fields (?).

I think we can just copy from the local particle. For example, suppose I decide that particle 1 should go to rank 5, 12, and 27. Then I first set p1.rank = 5, then I add two new particles with the same values as particle 1, but with rank = 12 and 27. Then when I call migrate, it will move these three particles to the correct processes, and delete the original particles and the copies from the local set.

Thanks,

Matt

...

Thanks, Miguel

On Sep 26, 2024, at 3:53 PM, Matthew Knepley <[email protected]> wrote:

On Thu, Sep 26, 2024 at 6:31 AM MIGUEL MOLINOS PEREZ <[email protected]> wrote:

...

Hi Matt et al,

I’ve been working on the scheme that you proposed to create ghost particles (atoms in my case), and it works! With a couple of caveats: -1º In general the overlap particles will be migrate from their own rank to more than one neighbor rank, this is specially relevant for those located close to the corners. Therefore, you'll need to call DMSwarmMigrate several times (27 times for 3D cells), during the migration process.

That is terrible. Let's just fix DMSwarmMigrate to have a mode that sends the particle to all overlapping neighbors at once. It can't be that hard.

...

-2º You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed algorithm will not work at all!

Oh, I should have thought of that. Sorry.

I can help code up that extension. Can you take a quick look at the BASIC code? Right now, we just use the rank attached to the particle to send it. We could have an arrays of ranks, but that seems crazy, and would blow up particle storage. How about just adding new particles with the other ranks right before migration?

Thanks,

Matt

...

Hope this helps to other folks!

I have a follow-up question about periodic bcc on this context, should I open a new thread of keep posting here?

Thanks, Miguel

On Aug 7, 2024, at 4:22 AM, MIGUEL MOLINOS PEREZ <[email protected]> wrote:

Thanks Matt, I think I'll start by making a small program as a proof of concept. Then, if it works I'll implement it in my code and I'll be happy to share it too :-)

Miguel

On Aug 4, 2024, at 3:30 AM, Matthew Knepley <[email protected]> wrote:

On Fri, Aug 2, 2024 at 7:15 PM MIGUEL MOLINOS PEREZ <[email protected]> wrote:

...

Thanks again Matt, that makes a lot more sense !!

Just to check that we are on the same page. You are saying:

1. create a field define a field called "owner rank" for each particle.

2. Identify the phantom particles and modify the internal variable defined by the DMSwarmField_rank variable.

3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the new local vector including the ghost particles.

4. Then, once the calculations are done, rename the DMSwarmField_rank variable using the "owner rank" variable and call DMSwarmMigrate(*,PETSC_FALSE) once again.

I don't think we need this last step. We can just remove those ghost particles for the next step I think.

Thanks,

Matt

...

Thank you, Miguel

On Aug 2, 2024, at 5:33 PM, Matthew Knepley <[email protected]> wrote:

On Fri, Aug 2, 2024 at 11:15 AM MIGUEL MOLINOS PEREZ <[email protected]> wrote:

...

Thank you Matt for your time,

What you describe seems to me the ideal approach.

1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST

This means, locally, I need to allocate Nlocal + ghost particles (duplicated) for my model?

I would do it another way. I would allocate the particles with no overlap and set them up. Then I would identify the halo particles, mark them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work since we cannot tell the difference between particles we received and particles we sent. Okay, instead of the `ghost` field we need an `owner rank` field. So then we

1) Setup the non-overlapping particles

2) Identify the halo particles

3) Change the `rank`, but not the `owner rank`

4) Call DMSwarmMigrate()

Now we can identify ghost particles by the `owner rank`

...

If that so, how to do the communication between the ghost particles living in the rank i and their “real” counterpart in the rank j.

Algo, as an alternative, what about: 1) Use an IS tag which contains, for each rank, a list of the global index of the neighbors particles outside of the rank. 2) Use VecCreateGhost to create a new vector which contains extra local space for the ghost components of the vector. 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the transference of data between a vector obtained with DMSwarmCreateGlobalVectorFromField 4) Do necessary computations using the vectors created with VecCreateGhost.

This is essentially what Migrate() does. I was trying to reuse the code.

Thanks,

Matt

...

Thanks, Miguel

On Aug 2, 2024, at 8:58 AM, Matthew Knepley <[email protected]> wrote:

On Thu, Aug 1, 2024 at 4:40 PM MIGUEL MOLINOS PEREZ <[email protected]> wrote:

...

This Message Is From an External Sender This message came from outside your organization.

Dear all,

I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor.

This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something?

I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions.

This is feasible, but it would be good to develop a set of best practices, since we have been mainly focused on the case of non-redundant particles. Here is how I think I would do what you want.

1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST

2) At some interval identify particles that should be sent to other processes as ghosts. I would call these "overlap particles". The determination seems application specific, so I would leave this determination to the user right now. We do two things to these particles

a) Mark chosen particles as OVERLAP

b) Change rank to process we are sending to

3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag

4) Mark OVERLAP particles as GHOST when they arrive

There is one problem in the above algorithm. It does not allow sending particles to multiple ranks. We would have to do this in phases right now, or make a small adjustment to the interface allowing replication of particles when a set of ranks is specified.

THanks,

Matt

...

Thank you, Miguel

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >

Dear all: Just to wrap this thread up. The easiest way to update any variable of the ghost particles is using the following lines of code: PetscCall(VecCreateGhostWithArray(PETSC_COMM_WORLD, n_dof_local, PETSC_DETERMINE, n_dof_ghost, idx_dof_ghost, X_ptr, &X)); PetscCall(VecGhostUpdateBegin(X, INSERT_VALUES, SCATTER_FORWARD)); PetscCall(VecGhostUpdateEnd(X, INSERT_VALUES, SCATTER_FORWARD)); Where X_ptr is the local pointer coming from: DMSwarmGetField Best, Miguel On 1 Oct 2024, at 19:42, MIGUEL MOLINOS PEREZ <[email protected]> wrote: Wow, thank you Dave that’s awesome, let me know if there's anything I can help you with! Miguel On Oct 1, 2024, at 7:20 PM, Dave May <[email protected]> wrote: On Tue, 1 Oct 2024 at 08:56, MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Hi Dave, Would something like that work? Yes, this should work! Any idea on where to look so I can try to implement it myself? I am adding support for this right now. Best, Miguel On Oct 1, 2024, at 5:22 PM, Dave May <[email protected]<mailto:[email protected]>> wrote: Hi Miguel, On Tue 1. Oct 2024 at 07:56, MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thank you Matt, it works! The implementation is straightforward: - 1º Define the paddle regions using DMGetLocalBoundingBox with the background DMDA mesh as an auxiliary mesh for the domain-partitioning. - 2º Create an integer to count the local number of particles to be used as ghost particle for other processors (N_ghost). One particle can be counted more than one time. At the same time, fill two arrays: - one with the index of the "main particle” (local particle), - and the other with the target rank of the "main particle”. - 3º Create the new particles using DMSwarmAddNPoints - 4º Fill the new particles with the information of the “main particle” but set the internal variable DMSwarmField_rank with the target rank. - 5º Call DMSwarmMigrate(*,PETSC_TRUE). Therefore, we send the ghost particles to the corresponding processors and we delete them from the original processor. - 6º Do stuff… - 7º Delete ghost particles. This is very easy, we just have to call DMSwarmRemovePoint N_ghost times. I think this can be easily implemented as closed routine for the DMSwarm class. The remaining question is: how to do the communication between the “original" particle and the ghost particles? For instance, if we update some particle variable (locally) inside of a SNES context, this same variable should be updated in the ghost particles at the other processors. I think what you are asking about is an operation similar to VecGhostUpdate{Begin,End}(). In the case of a DMSwarm I’m not sure how to define the InsertMode = ADD_VALUES? Some swarm fields do not make sense to be added. INSERT_VALUES is fine. One solution might be to have something like this DMSwarmCollectViewUpdateGhostOwners(DM dm, InsertMode mode, PetscInt nfields, const char *fieldNames[]); where one can specify the insert mode and the fields on which the insert mode will apply. Would something like that work? Cheers, Dave PS: Hope this helps someone in the future :-) On Sep 27, 2024, at 10:50 AM, MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thank you Matt, let me give it try. Miguel On Sep 27, 2024, at 3:44 AM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Thu, Sep 26, 2024 at 7:18 PM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: I see, you mean: Create the ghost particles at the local cell with the same properties as particle 1 (duplicate the original particle) but different value DMSwarmField_rank. Then, call DMSwarmMigrate(*,PETSC_FALSE) so we do the migration and delete the local copies of the particle 1. Right? Yep. I think it will work, from what I know about BASIC. Thanks, Matt Thanks, Miguel On Sep 26, 2024, at 11:09 PM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Thu, Sep 26, 2024 at 11:20 AM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thank you Matt. Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic implementation to see if there is some workaround. The idea of adding new particles is interesting. However, in that case, we need to initialize the new (ghost) particles using the fields of the “real” particle, right? This can be done using something like: VecGhostUpdateBegin<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhos... >(Vec<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;... > globalout,InsertMode<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertM... > ADD_VALUES<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VAL... >, ScatterMode<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Scatter... > SCATTER_REVERSE<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER... >); VecGhostUpdateEnd<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhos... >(Vec<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;... > globalout,InsertMode<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertM... > ADD_VALUES<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VAL... >, ScatterMode<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Scatter... > SCATTER_REVERSE<https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER... >); for the particle fields (?). I think we can just copy from the local particle. For example, suppose I decide that particle 1 should go to rank 5, 12, and 27. Then I first set p1.rank = 5, then I add two new particles with the same values as particle 1, but with rank = 12 and 27. Then when I call migrate, it will move these three particles to the correct processes, and delete the original particles and the copies from the local set. Thanks, Matt Thanks, Miguel On Sep 26, 2024, at 3:53 PM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Thu, Sep 26, 2024 at 6:31 AM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Hi Matt et al, I’ve been working on the scheme that you proposed to create ghost particles (atoms in my case), and it works! With a couple of caveats: -1º In general the overlap particles will be migrate from their own rank to more than one neighbor rank, this is specially relevant for those located close to the corners. Therefore, you'll need to call DMSwarmMigrate several times (27 times for 3D cells), during the migration process. That is terrible. Let's just fix DMSwarmMigrate to have a mode that sends the particle to all overlapping neighbors at once. It can't be that hard. -2º You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed algorithm will not work at all! Oh, I should have thought of that. Sorry. I can help code up that extension. Can you take a quick look at the BASIC code? Right now, we just use the rank attached to the particle to send it. We could have an arrays of ranks, but that seems crazy, and would blow up particle storage. How about just adding new particles with the other ranks right before migration? Thanks, Matt Hope this helps to other folks! I have a follow-up question about periodic bcc on this context, should I open a new thread of keep posting here? Thanks, Miguel On Aug 7, 2024, at 4:22 AM, MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thanks Matt, I think I'll start by making a small program as a proof of concept. Then, if it works I'll implement it in my code and I'll be happy to share it too :-) Miguel On Aug 4, 2024, at 3:30 AM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Fri, Aug 2, 2024 at 7:15 PM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thanks again Matt, that makes a lot more sense !! Just to check that we are on the same page. You are saying: 1. create a field define a field called "owner rank" for each particle. 2. Identify the phantom particles and modify the internal variable defined by the DMSwarmField_rank variable. 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the new local vector including the ghost particles. 4. Then, once the calculations are done, rename the DMSwarmField_rank variable using the "owner rank" variable and call DMSwarmMigrate(*,PETSC_FALSE) once again. I don't think we need this last step. We can just remove those ghost particles for the next step I think. Thanks, Matt Thank you, Miguel On Aug 2, 2024, at 5:33 PM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Fri, Aug 2, 2024 at 11:15 AM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: Thank you Matt for your time, What you describe seems to me the ideal approach. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST This means, locally, I need to allocate Nlocal + ghost particles (duplicated) for my model? I would do it another way. I would allocate the particles with no overlap and set them up. Then I would identify the halo particles, mark them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work since we cannot tell the difference between particles we received and particles we sent. Okay, instead of the `ghost` field we need an `owner rank` field. So then we 1) Setup the non-overlapping particles 2) Identify the halo particles 3) Change the `rank`, but not the `owner rank` 4) Call DMSwarmMigrate() Now we can identify ghost particles by the `owner rank` If that so, how to do the communication between the ghost particles living in the rank i and their “real” counterpart in the rank j. Algo, as an alternative, what about: 1) Use an IS tag which contains, for each rank, a list of the global index of the neighbors particles outside of the rank. 2) Use VecCreateGhost to create a new vector which contains extra local space for the ghost components of the vector. 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the transference of data between a vector obtained with DMSwarmCreateGlobalVectorFromField 4) Do necessary computations using the vectors created with VecCreateGhost. This is essentially what Migrate() does. I was trying to reuse the code. Thanks, Matt Thanks, Miguel On Aug 2, 2024, at 8:58 AM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: On Thu, Aug 1, 2024 at 4:40 PM MIGUEL MOLINOS PEREZ <[email protected]<mailto:[email protected]>> wrote: This Message Is From an External Sender This message came from outside your organization. Dear all, I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor. This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something? I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions. This is feasible, but it would be good to develop a set of best practices, since we have been mainly focused on the case of non-redundant particles. Here is how I think I would do what you want. 1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST 2) At some interval identify particles that should be sent to other processes as ghosts. I would call these "overlap particles". The determination seems application specific, so I would leave this determination to the user right now. We do two things to these particles a) Mark chosen particles as OVERLAP b) Change rank to process we are sending to 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag 4) Mark OVERLAP particles as GHOST when they arrive There is one problem in the above algorithm. It does not allow sending particles to multiple ranks. We would have to do this in phases right now, or make a small adjustment to the interface allowing replication of particles when a set of ranks is specified. THanks, Matt Thank you, Miguel -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >