On Wed, Jun 30, 2021 at 7:13 AM Peder Jørgensgaard Olesen <[email protected]> wrote:

Dear Matt

Thank you for your reply. I'm guessing that the line

if (rank) N = 0;

should be changed to something like

if (rank != target_rank) N = 0;

such that an empty SEQVEC and scatter are created on each rank other than target_rank.

Yes Thanks, Matt

Med venlig hilsen / Best regards

Peder ------------------------------ *Fra:* Matthew Knepley <[email protected]> *Sendt:* 30. juni 2021 13:58:49 *Til:* Peder Jørgensgaard Olesen *Cc:* [email protected] *Emne:* Re: [petsc-users] Scatter parallel Vec to sequential Vec on non-zeroth process

On Wed, Jun 30, 2021 at 6:49 AM Peder Jørgensgaard Olesen via petsc-users < [email protected]> wrote:

...

Hello

I have a parallel vector, and I'd like to copy its data to a sequential vector located on a given process that is not necessarily the 0th process, so using VecScatterCreateToZero wouldn't do here.

How might one approach this?

The function is very short: https://www.mcs.anl.gov/petsc/petsc-current/src/vec/is/sf/interface/vscat.c.... Just go in and replace '0' by the rank you want.

Thanks,

Matt

...

Med venlig hilsen / Best Regards

Peder Jørgensgaard Olesen

PhD Student, Turbulence Research Lab

Dept. of Mechanical Engineering

Technical University of Denmark

Koppels Allé

Bygning 403, Rum 105

DK-2800 Kgs. Lyngby

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Dear Junchao Zhang I'm distributing a set of independent tasks over different processes, so I'm afraid sending everything to the zeroth process would rather thoroughly defeat the purpose of what I'm doing. Med venlig hilsen / Best regards Peder ________________________________ Fra: Junchao Zhang <[email protected]> Sendt: 30. juni 2021 16:00:25 Til: Peder Jørgensgaard Olesen Cc: Matthew Knepley; [email protected] Emne: Re: [petsc-users] Scatter parallel Vec to sequential Vec on non-zeroth process That should work. But VecScatterCreateToZero() is more efficient. So if possible, better refactor your code in such a way. --Junchao Zhang On Wed, Jun 30, 2021 at 7:13 AM Peder Jørgensgaard Olesen via petsc-users <[email protected]<mailto:[email protected]>> wrote: Dear Matt Thank you for your reply. I'm guessing that the line if (rank) N = 0; should be changed to something like if (rank != target_rank) N = 0; such that an empty SEQVEC and scatter are created on each rank other than target_rank. Med venlig hilsen / Best regards Peder ________________________________ Fra: Matthew Knepley <[email protected]<mailto:[email protected]>> Sendt: 30. juni 2021 13:58:49 Til: Peder Jørgensgaard Olesen Cc: [email protected]<mailto:[email protected]> Emne: Re: [petsc-users] Scatter parallel Vec to sequential Vec on non-zeroth process On Wed, Jun 30, 2021 at 6:49 AM Peder Jørgensgaard Olesen via petsc-users <[email protected]<mailto:[email protected]>> wrote: Hello I have a parallel vector, and I'd like to copy its data to a sequential vector located on a given process that is not necessarily the 0th process, so using VecScatterCreateToZero wouldn't do here. How might one approach this? The function is very short: https://www.mcs.anl.gov/petsc/petsc-current/src/vec/is/sf/interface/vscat.c.... Just go in and replace '0' by the rank you want. Thanks, Matt Med venlig hilsen / Best Regards Peder Jørgensgaard Olesen PhD Student, Turbulence Research Lab Dept. of Mechanical Engineering Technical University of Denmark Koppels Allé Bygning 403, Rum 105 DK-2800 Kgs. Lyngby -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>

Peder Jørgensgaard Olesen via petsc-users <[email protected]> writes:

I'm distributing a set of independent tasks over different processes, so I'm afraid sending everything to the zeroth process would rather thoroughly defeat the purpose of what I'm doing.

It sounds like you're going to run this a bunch of times. Does it have to be sequential (gather to one rank at a time) or can it be an alltoall?

Peder Jørgensgaard Olesen <[email protected]> writes:

Each process is assigned an indexed subset of the tasks (the tasks are of constant size), and, for each task index, the relevant data is scattered as a SEQVEC to the process (this is done for all processes in each step, using an adaption of the code in Matt's link). This way each process only receives just the data it needs to complete the task. While I'm currently working with very moderate size data sets I'll eventually need to handle something rather more massive, so I want to economize memory where possible and give each process only the data it needs.

From the sounds of it, this pattern ultimately boils down to MPI_Gather being called P times where P is the size of the communicator. This will work okay when P is small, but it's much less efficient than calling MPI_Alltoall (or MPI_Alltoallv), which you can do by creating one PetscSF that ships the needed data to each task and PETSCSF_PATTERN_ALLTOALL. You can see an example.

https://gitlab.com/petsc/petsc/-/blob/main/src/vec/is/sf/tests/ex3.c#L93-151

Matt's method seems to work well, though instead of editing the actual function I put the relevant parts directly into my code. I made the small example attached here. I might look into Star Forests at some point, though it's not really touched upon in the manual (I will probably take a look at your paper, https://arxiv.org/abs/2102.13018). Med venlig hilsen / Best regards Peder ________________________________ Fra: Junchao Zhang <[email protected]> Sendt: 1. juli 2021 16:38:29 Til: Jed Brown Cc: Peder Jørgensgaard Olesen; [email protected] Emne: Re: Sv: [petsc-users] Scatter parallel Vec to sequential Vec on non-zeroth process Peder, PETSCSF_PATTERN_ALLTOALL only supports MPI_Alltoall (not Alltoallv), and is only used by petsc internally at few places. I suggest you can go with Matt's approach. After it solves your problem, you can distill an example to demo the communication pattern. Then we can see how to efficiently support that in petsc. Thanks. --Junchao Zhang On Thu, Jul 1, 2021 at 7:42 AM Jed Brown <[email protected]<mailto:[email protected]>> wrote: Peder Jørgensgaard Olesen <[email protected]<mailto:[email protected]>> writes:

Each process is assigned an indexed subset of the tasks (the tasks are of constant size), and, for each task index, the relevant data is scattered as a SEQVEC to the process (this is done for all processes in each step, using an adaption of the code in Matt's link). This way each process only receives just the data it needs to complete the task. While I'm currently working with very moderate size data sets I'll eventually need to handle something rather more massive, so I want to economize memory where possible and give each process only the data it needs.

From the sounds of it, this pattern ultimately boils down to MPI_Gather being called P times where P is the size of the communicator. This will work okay when P is small, but it's much less efficient than calling MPI_Alltoall (or MPI_Alltoallv), which you can do by creating one PetscSF that ships the needed data to each task and PETSCSF_PATTERN_ALLTOALL. You can see an example.

https://gitlab.com/petsc/petsc/-/blob/main/src/vec/is/sf/tests/ex3.c#L93-151

Yeah, scattering a parallel vector to a sequential on one rank was exactly what I wanted to do (apologies if I didn't phrase that clearly). A code like the one I shared does just what I needed, replacing size-1 with the desired target rank in the if-statement. Isn't what you describe what VecScatterCreateToAll is for? Med venlig hilsen / Best regards Peder ________________________________ Fra: Junchao Zhang <[email protected]> Sendt: 3. juli 2021 04:42:48 Til: Peder Jørgensgaard Olesen Cc: Jed Brown; [email protected] Emne: Re: Sv: [petsc-users] Scatter parallel Vec to sequential Vec on non-zeroth process Peder, Your example scatters a parallel vector to a sequential vector on one rank. It is a pattern like MPI_Gatherv. I want to see how you scatter parallel vectors to sequential vectors on every rank. --Junchao Zhang On Fri, Jul 2, 2021 at 4:07 AM Peder Jørgensgaard Olesen <[email protected]<mailto:[email protected]>> wrote: Matt's method seems to work well, though instead of editing the actual function I put the relevant parts directly into my code. I made the small example attached here. I might look into Star Forests at some point, though it's not really touched upon in the manual (I will probably take a look at your paper, https://arxiv.org/abs/2102.13018). Med venlig hilsen / Best regards Peder ________________________________ Fra: Junchao Zhang <[email protected]<mailto:[email protected]>> Sendt: 1. juli 2021 16:38:29 Til: Jed Brown Cc: Peder Jørgensgaard Olesen; [email protected]<mailto:[email protected]> Emne: Re: Sv: [petsc-users] Scatter parallel Vec to sequential Vec on non-zeroth process Peder, PETSCSF_PATTERN_ALLTOALL only supports MPI_Alltoall (not Alltoallv), and is only used by petsc internally at few places. I suggest you can go with Matt's approach. After it solves your problem, you can distill an example to demo the communication pattern. Then we can see how to efficiently support that in petsc. Thanks. --Junchao Zhang On Thu, Jul 1, 2021 at 7:42 AM Jed Brown <[email protected]<mailto:[email protected]>> wrote: Peder Jørgensgaard Olesen <[email protected]<mailto:[email protected]>> writes:

Each process is assigned an indexed subset of the tasks (the tasks are of constant size), and, for each task index, the relevant data is scattered as a SEQVEC to the process (this is done for all processes in each step, using an adaption of the code in Matt's link). This way each process only receives just the data it needs to complete the task. While I'm currently working with very moderate size data sets I'll eventually need to handle something rather more massive, so I want to economize memory where possible and give each process only the data it needs.

From the sounds of it, this pattern ultimately boils down to MPI_Gather being called P times where P is the size of the communicator. This will work okay when P is small, but it's much less efficient than calling MPI_Alltoall (or MPI_Alltoallv), which you can do by creating one PetscSF that ships the needed data to each task and PETSCSF_PATTERN_ALLTOALL. You can see an example.

https://gitlab.com/petsc/petsc/-/blob/main/src/vec/is/sf/tests/ex3.c#L93-151

I agree, using the loop you describe would definitely not be a clever way of doing it, nor is it at all what I was going for. The code with Matt's method indeed does what I needed. I'd be happy if it could be further optimized. Med venlig hilsen / Best regards Peder ________________________________ Fra: Junchao Zhang <[email protected]> Sendt: 4. juli 2021 05:36:29 Til: Peder Jørgensgaard Olesen Cc: Jed Brown; [email protected] Emne: Re: Sv: [petsc-users] Scatter parallel Vec to sequential Vec on non-zeroth process VecScatterCreateToAll() scatters the MPI vector to a sequential vector on every rank (as if each rank has a duplicate of the same sequential vector). If the sample code you provided is what you want, it is fine and we just need to implement a minor optimization in petsc to make it efficient. But if you want to put the scatter in a loop as follows, then it is a very bad code. for (p=0; p<size; i++) { loc_tgt_size = 0; if (rank == p){ loc_tgt_size = n; } ierr = VecCreateSeq(PETSC_COMM_SELF, loc_tgt_size, &tgt_vec); CHKERRQ(ierr); ierr = VecZeroEntries(tgt_vec); CHKERRQ(ierr); // Scatter source vector to target vector on one process ierr = ISCreateStride(PETSC_COMM_SELF, loc_tgt_size, 0, 1, &is); CHKERRQ(ierr); ierr = VecScatterCreate(src_vec, is, tgt_vec, is, &sctx); CHKERRQ(ierr); ierr = VecScatterBegin(sctx, src_vec, tgt_vec, INSERT_VALUES, SCATTER_FORWARD); CHKERRQ(ierr); ierr = VecScatterEnd(sctx, src_vec, tgt_vec, INSERT_VALUES, SCATTER_FORWARD); CHKERRQ(ierr); ... } --Junchao Zhang On Sat, Jul 3, 2021 at 1:27 PM Peder Jørgensgaard Olesen <[email protected]<mailto:[email protected]>> wrote: Yeah, scattering a parallel vector to a sequential on one rank was exactly what I wanted to do (apologies if I didn't phrase that clearly). A code like the one I shared does just what I needed, replacing size-1 with the desired target rank in the if-statement. Isn't what you describe what VecScatterCreateToAll is for? Med venlig hilsen / Best regards Peder ________________________________ Fra: Junchao Zhang <[email protected]<mailto:[email protected]>> Sendt: 3. juli 2021 04:42:48 Til: Peder Jørgensgaard Olesen Cc: Jed Brown; [email protected]<mailto:[email protected]> Emne: Re: Sv: [petsc-users] Scatter parallel Vec to sequential Vec on non-zeroth process Peder, Your example scatters a parallel vector to a sequential vector on one rank. It is a pattern like MPI_Gatherv. I want to see how you scatter parallel vectors to sequential vectors on every rank. --Junchao Zhang On Fri, Jul 2, 2021 at 4:07 AM Peder Jørgensgaard Olesen <[email protected]<mailto:[email protected]>> wrote: Matt's method seems to work well, though instead of editing the actual function I put the relevant parts directly into my code. I made the small example attached here. I might look into Star Forests at some point, though it's not really touched upon in the manual (I will probably take a look at your paper, https://arxiv.org/abs/2102.13018). Med venlig hilsen / Best regards Peder ________________________________ Fra: Junchao Zhang <[email protected]<mailto:[email protected]>> Sendt: 1. juli 2021 16:38:29 Til: Jed Brown Cc: Peder Jørgensgaard Olesen; [email protected]<mailto:[email protected]> Emne: Re: Sv: [petsc-users] Scatter parallel Vec to sequential Vec on non-zeroth process Peder, PETSCSF_PATTERN_ALLTOALL only supports MPI_Alltoall (not Alltoallv), and is only used by petsc internally at few places. I suggest you can go with Matt's approach. After it solves your problem, you can distill an example to demo the communication pattern. Then we can see how to efficiently support that in petsc. Thanks. --Junchao Zhang On Thu, Jul 1, 2021 at 7:42 AM Jed Brown <[email protected]<mailto:[email protected]>> wrote: Peder Jørgensgaard Olesen <[email protected]<mailto:[email protected]>> writes:

Each process is assigned an indexed subset of the tasks (the tasks are of constant size), and, for each task index, the relevant data is scattered as a SEQVEC to the process (this is done for all processes in each step, using an adaption of the code in Matt's link). This way each process only receives just the data it needs to complete the task. While I'm currently working with very moderate size data sets I'll eventually need to handle something rather more massive, so I want to economize memory where possible and give each process only the data it needs.

From the sounds of it, this pattern ultimately boils down to MPI_Gather being called P times where P is the size of the communicator. This will work okay when P is small, but it's much less efficient than calling MPI_Alltoall (or MPI_Alltoallv), which you can do by creating one PetscSF that ships the needed data to each task and PETSCSF_PATTERN_ALLTOALL. You can see an example.

https://gitlab.com/petsc/petsc/-/blob/main/src/vec/is/sf/tests/ex3.c#L93-151