- petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms
by Barry Smith 06 Nov '25

06 Nov '25

> On Nov 5, 2025, at 6:01 PM, Angus, Justin Ray <angus1(a)llnl.gov> wrote: > > Hi Barry, > > Thanks for the explanation, though I’m not sure I understand exactly what it’s doing. I at least understand now that the overlap doesn’t simply include degrees of freedom from neighboring cells within the range of the specified overlap for each side and direction (which is how chat GPT told me it worked). It kind of does. When you use PCASM and set a requested overlap it "determines" the neighboring cells by the nonzeros in the matrix, not by the geometry. > > For this problem ASM with overlap 16 + LU for the sub pc gave optimal performance. A lower value for the overlap results in too many GMRES iterations. I’m actually surprised that such a large overlap was needed, because the code I’m using is nearly identical to another and in that code the GMRES iterations stay low with an overlap of 4. Perhaps there is something different in the setup that is not clear. It could be, but good preconditioners are very dependent on the exact PDE model you are using as well as the discretization you are using so a slight change can dramatically change what is a good preconditioner for that problem. > > The PETSc preconditioners seem very powerful, bu it is difficult and time-consuming to find the optimal setup for a given problem given my nascent understanding of how the preconditoners work. This is a universal problem, I am hoping someday LLM's will be able to provide good advice for each problem where a preconditioner is needed :-). > Is there someone from PETSc (perhaps yourself) with advanced knowledge of the preconditions that I could schedule a zoom/webex meeting with to discuss? Sorry, but even the little bit you indicated about your problem, electric field on a Yee grid makes it clear to me I have no expert advice on preconditioner selection I can offer to you. I suggest looking at the HPDDM preconditioners that you can use from PETSc https://urldefense.us/v3/__https://petsc.org/release/manualpages/PC/PCHPDDM… . These preconditioners are like the Mona Lisa while basic ASM is like a kindergartener's drawing; they can be quite robust for difficult problems. Barry > > Thanks! > > -Justin > > > From: Barry Smith <bsmith(a)petsc.dev> > Date: Wednesday, November 5, 2025 at 2:48 PM > To: Angus, Justin Ray <angus1(a)llnl.gov> > Cc: MFAdams(a)LBL.GOV <MFAdams(a)LBL.GOV>, Matthew Knepley <knepley(a)gmail.com>, petsc-dev(a)mcs.anl.gov <petsc-dev(a)mcs.anl.gov> > Subject: Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms > > > An overlap of 16 is huge and would rarely if ever, be done in practice. It is not surprising that the subproblems become as large as they do with such a large overlap. > > How the overlap is used. > > while (overlap--) { > add to the subproblem all degrees of freedom that are coupled by nonzeros in the matrix to all the current degrees of freedom in the subproblem. > } > > So it is grabbing all the neighbors for 16 rounds of grabbing. > > > > On Nov 5, 2025, at 3:05 PM, Angus, Justin Ray via petsc-dev <petsc-dev(a)mcs.anl.gov> wrote: > > I think the issue is my overlap is too large. Perhaps I don’t fully understand how the overlap parameter is used. Let me explain my setup below. > > My vector of unknowns is the electric field on a Yee grid in a 2D geometry. I’m using 4x4 grid cells per rank. This gives 4*5 = 20 degrees of freedom for each of the two in-plane components of E, and 5*5 = 25 for the out-of-plane component. The total is 65 degrees of freedom per rank. My global problem size is 224x16 on 224 ranks for one case, and 224x32 on 448 ranks for another. Using ASM overlap 4, I get the following for PC sub blocks on a rank: > PC Object: (sub_) 1 MPI process > type: lu > out-of-place factorization > Reusing fill from past factorization > Reusing reordering from past factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 4.02544 > Factored matrix follows: > Mat Object: (sub_) 1 MPI process > type: seqaij > rows=402, cols=402 > package used to perform factorization: petsc > total: nonzeros=16295, allocated nonzeros=16295 > not using I-node routines > > The above is for a 224x16 size domain in 224 total ranks, but I get the same thing for a 224x32 size domain on 448 ranks, which is what I am expected to get. > > However, if I set the overlap to 16 (which is larger than by box size on a given rank), I get the following > 224x16 gid on 112 ranks: > PC Object: (sub_) 1 MPI process > type: lu > out-of-place factorization > Reusing fill from past factorization > Reusing reordering from past factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 6.52557 > Factored matrix follows: > Mat Object: (sub_) 1 MPI process > type: seqaij > rows=1316, cols=1316 > package used to perform factorization: petsc > total: nonzeros=95195, allocated nonzeros=95195 > not using I-node routines > > 224x16 gid on 112 ranks: > PC Object: (sub_) 1 MPI process > type: lu > out-of-place factorization > Reusing fill from past factorization > Reusing reordering from past factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 8.59182 > Factored matrix follows: > Mat Object: (sub_) 1 MPI process > type: seqaij > rows=2632, cols=2632 > package used to perform factorization: petsc > total: nonzeros=250675, allocated nonzeros=250675 > not using I-node routines > > In this case, with an overlap much larger than the box size, the rows/cols per rank go up by a factor of 2 when doubling the problem size at fixed work per rank. > > Why is this? > How exactly is the overlap parameter used? > > Thank you. > > -Justin > > From: Angus, Justin Ray <angus1(a)llnl.gov> > Date: Wednesday, November 5, 2025 at 8:17 AM > To: MFAdams(a)LBL.GOV <MFAdams(a)LBL.GOV>, Matthew Knepley <knepley(a)gmail.com> > Cc: petsc-dev(a)mcs.anl.gov <petsc-dev(a)mcs.anl.gov> > Subject: Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms > > Thanks for the reply. > > The work per block should be the same for the weak scaling. I know LU is not scalable with respect to the block size. > > Perhaps our setup is not doing what we think it is doing. I’ll look into it further. > > -Justin > > From: Mark Adams <mfadams(a)lbl.gov> > Date: Wednesday, November 5, 2025 at 6:14 AM > To: Matthew Knepley <knepley(a)gmail.com> > Cc: Angus, Justin Ray <angus1(a)llnl.gov>, petsc-dev(a)mcs.anl.gov <petsc-dev(a)mcs.anl.gov> > Subject: Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms > > And we do not have sparse LU on GPUs so that is done on the CPU. > > And I don't know why it would not weak scale well. > Your results are consistent with just using one process with one domain, (re Matt) while you double the problem size. > > On Tue, Nov 4, 2025 at 2:27 PM Matthew Knepley <knepley(a)gmail.com <mailto:[email protected]>> wrote: > On Tue, Nov 4, 2025 at 1:25 PM Angus, Justin Ray via petsc-dev <petsc-dev(a)mcs.anl.gov <mailto:[email protected]>> wrote: > Hi Junchao, > > We have recently been using ASM + LU for 2D problems on both CPU and GPU. However, I found that this method has very bad weak scaling. I find that the cost of PCApply increases by about a factor of 4 each time I increase the problem size in 1 dimension by a factor of 2 while keeping the load per core/gpu the same. The total number of GMRES iterations does not increase, just the cost of PCApply (and PCSetup). Is this scaling behavior expected? Any ideas of how to optimize the preconditioner? > > The cost of PCApply for ASM is dominated by the cost of process-local block solves. You are using LU for the block solve. (Sparse) LU has cost roughly O(N^2) for the apply (depending on the structure of the matrix). So, if you double the size of a local block, your runtime should increase by about 4x. Thus LU is not a scalable method. > > Thanks, > > Matt > > Thank you. > > -Justin > > From: Junchao Zhang <junchao.zhang(a)gmail.com <mailto:[email protected]>> > Date: Monday, April 14, 2025 at 7:35 PM > To: Angus, Justin Ray <angus1(a)llnl.gov <mailto:[email protected]>> > Cc: petsc-dev(a)mcs.anl.gov <mailto:[email protected]> <petsc-dev(a)mcs.anl.gov <mailto:[email protected]>>, Ghosh, Debojyoti <ghosh5(a)llnl.gov <mailto:[email protected]>> > Subject: Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms > > Petsc supports ILU0/ICC0 numeric factorization (without reordering) and then triangular solve on GPUs. It is done by calling vendor libraries (ex. cusparse). > We have options -pc_factor_mat_factor_on_host <bool> -pc_factor_mat_solve_on_host <bool> to force doing the factorization and MatSolve on the host for device matrix types. > > You can try to see if it works for your case. > > --Junchao Zhang > > > On Mon, Apr 14, 2025 at 4:39 PM Angus, Justin Ray via petsc-dev <petsc-dev(a)mcs.anl.gov <mailto:[email protected]>> wrote: > Hello, > > > A project I work on uses GMRES via PETSc. In particular, we have had good successes using the Additive Schwarz Method + ILU preconditioner setup using a CPU-based code. I found online where it is stated that “Parts of most preconditioners run directly on the GPU” (https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!… <https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!…>). Is ASM + ILU also available for GPU platforms? > > > -Justin > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uC… <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf…> >

1 0

Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms
by Angus, Justin Ray 05 Nov '25

05 Nov '25

Hi Barry, Thanks for the explanation, though I’m not sure I understand exactly what it’s doing. I at least understand now that the overlap doesn’t simply include degrees of freedom from neighboring cells within the range of the specified overlap for each side and direction (which is how chat GPT told me it worked). For this problem ASM with overlap 16 + LU for the sub pc gave optimal performance. A lower value for the overlap results in too many GMRES iterations. I’m actually surprised that such a large overlap was needed, because the code I’m using is nearly identical to another and in that code the GMRES iterations stay low with an overlap of 4. Perhaps there is something different in the setup that is not clear. The PETSc preconditioners seem very powerful, bu it is difficult and time-consuming to find the optimal setup for a given problem given my nascent understanding of how the preconditoners work. Is there someone from PETSc (perhaps yourself) with advanced knowledge of the preconditions that I could schedule a zoom/webex meeting with to discuss? Thanks! -Justin From: Barry Smith <bsmith(a)petsc.dev> Date: Wednesday, November 5, 2025 at 2:48 PM To: Angus, Justin Ray <angus1(a)llnl.gov> Cc: MFAdams(a)LBL.GOV <MFAdams(a)LBL.GOV>, Matthew Knepley <knepley(a)gmail.com>, petsc-dev(a)mcs.anl.gov <petsc-dev(a)mcs.anl.gov> Subject: Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms An overlap of 16 is huge and would rarely if ever, be done in practice. It is not surprising that the subproblems become as large as they do with such a large overlap. How the overlap is used. while (overlap--) { add to the subproblem all degrees of freedom that are coupled by nonzeros in the matrix to all the current degrees of freedom in the subproblem. } So it is grabbing all the neighbors for 16 rounds of grabbing. On Nov 5, 2025, at 3:05 PM, Angus, Justin Ray via petsc-dev <petsc-dev(a)mcs.anl.gov> wrote: I think the issue is my overlap is too large. Perhaps I don’t fully understand how the overlap parameter is used. Let me explain my setup below. My vector of unknowns is the electric field on a Yee grid in a 2D geometry. I’m using 4x4 grid cells per rank. This gives 4*5 = 20 degrees of freedom for each of the two in-plane components of E, and 5*5 = 25 for the out-of-plane component. The total is 65 degrees of freedom per rank. My global problem size is 224x16 on 224 ranks for one case, and 224x32 on 448 ranks for another. Using ASM overlap 4, I get the following for PC sub blocks on a rank: PC Object: (sub_) 1 MPI process type: lu out-of-place factorization Reusing fill from past factorization Reusing reordering from past factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd factor fill ratio given 5., needed 4.02544 Factored matrix follows: Mat Object: (sub_) 1 MPI process type: seqaij rows=402, cols=402 package used to perform factorization: petsc total: nonzeros=16295, allocated nonzeros=16295 not using I-node routines The above is for a 224x16 size domain in 224 total ranks, but I get the same thing for a 224x32 size domain on 448 ranks, which is what I am expected to get. However, if I set the overlap to 16 (which is larger than by box size on a given rank), I get the following 224x16 gid on 112 ranks: PC Object: (sub_) 1 MPI process type: lu out-of-place factorization Reusing fill from past factorization Reusing reordering from past factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd factor fill ratio given 5., needed 6.52557 Factored matrix follows: Mat Object: (sub_) 1 MPI process type: seqaij rows=1316, cols=1316 package used to perform factorization: petsc total: nonzeros=95195, allocated nonzeros=95195 not using I-node routines 224x16 gid on 112 ranks: PC Object: (sub_) 1 MPI process type: lu out-of-place factorization Reusing fill from past factorization Reusing reordering from past factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd factor fill ratio given 5., needed 8.59182 Factored matrix follows: Mat Object: (sub_) 1 MPI process type: seqaij rows=2632, cols=2632 package used to perform factorization: petsc total: nonzeros=250675, allocated nonzeros=250675 not using I-node routines In this case, with an overlap much larger than the box size, the rows/cols per rank go up by a factor of 2 when doubling the problem size at fixed work per rank. Why is this? How exactly is the overlap parameter used? Thank you. -Justin From: Angus, Justin Ray <angus1(a)llnl.gov> Date: Wednesday, November 5, 2025 at 8:17 AM To: MFAdams(a)LBL.GOV <MFAdams(a)LBL.GOV>, Matthew Knepley <knepley(a)gmail.com> Cc: petsc-dev(a)mcs.anl.gov <petsc-dev(a)mcs.anl.gov> Subject: Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms Thanks for the reply. The work per block should be the same for the weak scaling. I know LU is not scalable with respect to the block size. Perhaps our setup is not doing what we think it is doing. I’ll look into it further. -Justin From: Mark Adams <mfadams(a)lbl.gov> Date: Wednesday, November 5, 2025 at 6:14 AM To: Matthew Knepley <knepley(a)gmail.com> Cc: Angus, Justin Ray <angus1(a)llnl.gov>, petsc-dev(a)mcs.anl.gov <petsc-dev(a)mcs.anl.gov> Subject: Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms And we do not have sparse LU on GPUs so that is done on the CPU. And I don't know why it would not weak scale well. Your results are consistent with just using one process with one domain, (re Matt) while you double the problem size. On Tue, Nov 4, 2025 at 2:27 PM Matthew Knepley <knepley(a)gmail.com<mailto:[email protected]>> wrote: On Tue, Nov 4, 2025 at 1:25 PM Angus, Justin Ray via petsc-dev <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>> wrote: Hi Junchao, We have recently been using ASM + LU for 2D problems on both CPU and GPU. However, I found that this method has very bad weak scaling. I find that the cost of PCApply increases by about a factor of 4 each time I increase the problem size in 1 dimension by a factor of 2 while keeping the load per core/gpu the same. The total number of GMRES iterations does not increase, just the cost of PCApply (and PCSetup). Is this scaling behavior expected? Any ideas of how to optimize the preconditioner? The cost of PCApply for ASM is dominated by the cost of process-local block solves. You are using LU for the block solve. (Sparse) LU has cost roughly O(N^2) for the apply (depending on the structure of the matrix). So, if you double the size of a local block, your runtime should increase by about 4x. Thus LU is not a scalable method. Thanks, Matt Thank you. -Justin From: Junchao Zhang <junchao.zhang(a)gmail.com<mailto:[email protected]>> Date: Monday, April 14, 2025 at 7:35 PM To: Angus, Justin Ray <angus1(a)llnl.gov<mailto:[email protected]>> Cc: petsc-dev(a)mcs.anl.gov<mailto:[email protected]> <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>>, Ghosh, Debojyoti <ghosh5(a)llnl.gov<mailto:[email protected]>> Subject: Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms Petsc supports ILU0/ICC0 numeric factorization (without reordering) and then triangular solve on GPUs. It is done by calling vendor libraries (ex. cusparse). We have options -pc_factor_mat_factor_on_host <bool> -pc_factor_mat_solve_on_host <bool> to force doing the factorization and MatSolve on the host for device matrix types. You can try to see if it works for your case. --Junchao Zhang On Mon, Apr 14, 2025 at 4:39 PM Angus, Justin Ray via petsc-dev <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>> wrote: Hello, A project I work on uses GMRES via PETSc. In particular, we have had good successes using the Additive Schwarz Method + ILU preconditioner setup using a CPU-based code. I found online where it is stated that “Parts of most preconditioners run directly on the GPU” (https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!… <https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!…>). Is ASM + ILU also available for GPU platforms? -Justin -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uC… <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf…>

1 0

Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms
by Barry Smith 05 Nov '25

05 Nov '25

An overlap of 16 is huge and would rarely if ever, be done in practice. It is not surprising that the subproblems become as large as they do with such a large overlap. How the overlap is used. while (overlap--) { add to the subproblem all degrees of freedom that are coupled by nonzeros in the matrix to all the current degrees of freedom in the subproblem. } So it is grabbing all the neighbors for 16 rounds of grabbing. > On Nov 5, 2025, at 3:05 PM, Angus, Justin Ray via petsc-dev <petsc-dev(a)mcs.anl.gov> wrote: > > I think the issue is my overlap is too large. Perhaps I don’t fully understand how the overlap parameter is used. Let me explain my setup below. > > My vector of unknowns is the electric field on a Yee grid in a 2D geometry. I’m using 4x4 grid cells per rank. This gives 4*5 = 20 degrees of freedom for each of the two in-plane components of E, and 5*5 = 25 for the out-of-plane component. The total is 65 degrees of freedom per rank. My global problem size is 224x16 on 224 ranks for one case, and 224x32 on 448 ranks for another. Using ASM overlap 4, I get the following for PC sub blocks on a rank: > PC Object: (sub_) 1 MPI process > type: lu > out-of-place factorization > Reusing fill from past factorization > Reusing reordering from past factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 4.02544 > Factored matrix follows: > Mat Object: (sub_) 1 MPI process > type: seqaij > rows=402, cols=402 > package used to perform factorization: petsc > total: nonzeros=16295, allocated nonzeros=16295 > not using I-node routines > > The above is for a 224x16 size domain in 224 total ranks, but I get the same thing for a 224x32 size domain on 448 ranks, which is what I am expected to get. > > However, if I set the overlap to 16 (which is larger than by box size on a given rank), I get the following > 224x16 gid on 112 ranks: > PC Object: (sub_) 1 MPI process > type: lu > out-of-place factorization > Reusing fill from past factorization > Reusing reordering from past factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 6.52557 > Factored matrix follows: > Mat Object: (sub_) 1 MPI process > type: seqaij > rows=1316, cols=1316 > package used to perform factorization: petsc > total: nonzeros=95195, allocated nonzeros=95195 > not using I-node routines > > 224x16 gid on 112 ranks: > PC Object: (sub_) 1 MPI process > type: lu > out-of-place factorization > Reusing fill from past factorization > Reusing reordering from past factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 8.59182 > Factored matrix follows: > Mat Object: (sub_) 1 MPI process > type: seqaij > rows=2632, cols=2632 > package used to perform factorization: petsc > total: nonzeros=250675, allocated nonzeros=250675 > not using I-node routines > > In this case, with an overlap much larger than the box size, the rows/cols per rank go up by a factor of 2 when doubling the problem size at fixed work per rank. > > Why is this? > How exactly is the overlap parameter used? > > Thank you. > > -Justin > > From: Angus, Justin Ray <angus1(a)llnl.gov> > Date: Wednesday, November 5, 2025 at 8:17 AM > To: MFAdams(a)LBL.GOV <MFAdams(a)LBL.GOV>, Matthew Knepley <knepley(a)gmail.com> > Cc: petsc-dev(a)mcs.anl.gov <petsc-dev(a)mcs.anl.gov> > Subject: Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms > > Thanks for the reply. > > The work per block should be the same for the weak scaling. I know LU is not scalable with respect to the block size. > > Perhaps our setup is not doing what we think it is doing. I’ll look into it further. > > -Justin > > From: Mark Adams <mfadams(a)lbl.gov> > Date: Wednesday, November 5, 2025 at 6:14 AM > To: Matthew Knepley <knepley(a)gmail.com> > Cc: Angus, Justin Ray <angus1(a)llnl.gov>, petsc-dev(a)mcs.anl.gov <petsc-dev(a)mcs.anl.gov> > Subject: Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms > > And we do not have sparse LU on GPUs so that is done on the CPU. > > And I don't know why it would not weak scale well. > Your results are consistent with just using one process with one domain, (re Matt) while you double the problem size. > > On Tue, Nov 4, 2025 at 2:27 PM Matthew Knepley <knepley(a)gmail.com <mailto:[email protected]>> wrote: > On Tue, Nov 4, 2025 at 1:25 PM Angus, Justin Ray via petsc-dev <petsc-dev(a)mcs.anl.gov <mailto:[email protected]>> wrote: > Hi Junchao, > > We have recently been using ASM + LU for 2D problems on both CPU and GPU. However, I found that this method has very bad weak scaling. I find that the cost of PCApply increases by about a factor of 4 each time I increase the problem size in 1 dimension by a factor of 2 while keeping the load per core/gpu the same. The total number of GMRES iterations does not increase, just the cost of PCApply (and PCSetup). Is this scaling behavior expected? Any ideas of how to optimize the preconditioner? > > The cost of PCApply for ASM is dominated by the cost of process-local block solves. You are using LU for the block solve. (Sparse) LU has cost roughly O(N^2) for the apply (depending on the structure of the matrix). So, if you double the size of a local block, your runtime should increase by about 4x. Thus LU is not a scalable method. > > Thanks, > > Matt > > Thank you. > > -Justin > > From: Junchao Zhang <junchao.zhang(a)gmail.com <mailto:[email protected]>> > Date: Monday, April 14, 2025 at 7:35 PM > To: Angus, Justin Ray <angus1(a)llnl.gov <mailto:[email protected]>> > Cc: petsc-dev(a)mcs.anl.gov <mailto:[email protected]> <petsc-dev(a)mcs.anl.gov <mailto:[email protected]>>, Ghosh, Debojyoti <ghosh5(a)llnl.gov <mailto:[email protected]>> > Subject: Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms > > Petsc supports ILU0/ICC0 numeric factorization (without reordering) and then triangular solve on GPUs. It is done by calling vendor libraries (ex. cusparse). > We have options -pc_factor_mat_factor_on_host <bool> -pc_factor_mat_solve_on_host <bool> to force doing the factorization and MatSolve on the host for device matrix types. > > You can try to see if it works for your case. > > --Junchao Zhang > > > On Mon, Apr 14, 2025 at 4:39 PM Angus, Justin Ray via petsc-dev <petsc-dev(a)mcs.anl.gov <mailto:[email protected]>> wrote: > Hello, > > > A project I work on uses GMRES via PETSc. In particular, we have had good successes using the Additive Schwarz Method + ILU preconditioner setup using a CPU-based code. I found online where it is stated that “Parts of most preconditioners run directly on the GPU” (https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!… <https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!…>). Is ASM + ILU also available for GPU platforms? > > > -Justin > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uC… <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf…>

1 0

Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms
by Angus, Justin Ray 05 Nov '25

05 Nov '25

I think the issue is my overlap is too large. Perhaps I don’t fully understand how the overlap parameter is used. Let me explain my setup below. My vector of unknowns is the electric field on a Yee grid in a 2D geometry. I’m using 4x4 grid cells per rank. This gives 4*5 = 20 degrees of freedom for each of the two in-plane components of E, and 5*5 = 25 for the out-of-plane component. The total is 65 degrees of freedom per rank. My global problem size is 224x16 on 224 ranks for one case, and 224x32 on 448 ranks for another. Using ASM overlap 4, I get the following for PC sub blocks on a rank: PC Object: (sub_) 1 MPI process type: lu out-of-place factorization Reusing fill from past factorization Reusing reordering from past factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd factor fill ratio given 5., needed 4.02544 Factored matrix follows: Mat Object: (sub_) 1 MPI process type: seqaij rows=402, cols=402 package used to perform factorization: petsc total: nonzeros=16295, allocated nonzeros=16295 not using I-node routines The above is for a 224x16 size domain in 224 total ranks, but I get the same thing for a 224x32 size domain on 448 ranks, which is what I am expected to get. However, if I set the overlap to 16 (which is larger than by box size on a given rank), I get the following 224x16 gid on 112 ranks: PC Object: (sub_) 1 MPI process type: lu out-of-place factorization Reusing fill from past factorization Reusing reordering from past factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd factor fill ratio given 5., needed 6.52557 Factored matrix follows: Mat Object: (sub_) 1 MPI process type: seqaij rows=1316, cols=1316 package used to perform factorization: petsc total: nonzeros=95195, allocated nonzeros=95195 not using I-node routines 224x16 gid on 112 ranks: PC Object: (sub_) 1 MPI process type: lu out-of-place factorization Reusing fill from past factorization Reusing reordering from past factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd factor fill ratio given 5., needed 8.59182 Factored matrix follows: Mat Object: (sub_) 1 MPI process type: seqaij rows=2632, cols=2632 package used to perform factorization: petsc total: nonzeros=250675, allocated nonzeros=250675 not using I-node routines In this case, with an overlap much larger than the box size, the rows/cols per rank go up by a factor of 2 when doubling the problem size at fixed work per rank. Why is this? How exactly is the overlap parameter used? Thank you. -Justin From: Angus, Justin Ray <angus1(a)llnl.gov> Date: Wednesday, November 5, 2025 at 8:17 AM To: MFAdams(a)LBL.GOV <MFAdams(a)LBL.GOV>, Matthew Knepley <knepley(a)gmail.com> Cc: petsc-dev(a)mcs.anl.gov <petsc-dev(a)mcs.anl.gov> Subject: Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms Thanks for the reply. The work per block should be the same for the weak scaling. I know LU is not scalable with respect to the block size. Perhaps our setup is not doing what we think it is doing. I’ll look into it further. -Justin From: Mark Adams <mfadams(a)lbl.gov> Date: Wednesday, November 5, 2025 at 6:14 AM To: Matthew Knepley <knepley(a)gmail.com> Cc: Angus, Justin Ray <angus1(a)llnl.gov>, petsc-dev(a)mcs.anl.gov <petsc-dev(a)mcs.anl.gov> Subject: Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms And we do not have sparse LU on GPUs so that is done on the CPU. And I don't know why it would not weak scale well. Your results are consistent with just using one process with one domain, (re Matt) while you double the problem size. On Tue, Nov 4, 2025 at 2:27 PM Matthew Knepley <knepley(a)gmail.com<mailto:[email protected]>> wrote: On Tue, Nov 4, 2025 at 1:25 PM Angus, Justin Ray via petsc-dev <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>> wrote: Hi Junchao, We have recently been using ASM + LU for 2D problems on both CPU and GPU. However, I found that this method has very bad weak scaling. I find that the cost of PCApply increases by about a factor of 4 each time I increase the problem size in 1 dimension by a factor of 2 while keeping the load per core/gpu the same. The total number of GMRES iterations does not increase, just the cost of PCApply (and PCSetup). Is this scaling behavior expected? Any ideas of how to optimize the preconditioner? The cost of PCApply for ASM is dominated by the cost of process-local block solves. You are using LU for the block solve. (Sparse) LU has cost roughly O(N^2) for the apply (depending on the structure of the matrix). So, if you double the size of a local block, your runtime should increase by about 4x. Thus LU is not a scalable method. Thanks, Matt Thank you. -Justin From: Junchao Zhang <junchao.zhang(a)gmail.com<mailto:[email protected]>> Date: Monday, April 14, 2025 at 7:35 PM To: Angus, Justin Ray <angus1(a)llnl.gov<mailto:[email protected]>> Cc: petsc-dev(a)mcs.anl.gov<mailto:[email protected]> <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>>, Ghosh, Debojyoti <ghosh5(a)llnl.gov<mailto:[email protected]>> Subject: Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms Petsc supports ILU0/ICC0 numeric factorization (without reordering) and then triangular solve on GPUs. It is done by calling vendor libraries (ex. cusparse). We have options -pc_factor_mat_factor_on_host <bool> -pc_factor_mat_solve_on_host <bool> to force doing the factorization and MatSolve on the host for device matrix types. You can try to see if it works for your case. --Junchao Zhang On Mon, Apr 14, 2025 at 4:39 PM Angus, Justin Ray via petsc-dev <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>> wrote: Hello, A project I work on uses GMRES via PETSc. In particular, we have had good successes using the Additive Schwarz Method + ILU preconditioner setup using a CPU-based code. I found online where it is stated that “Parts of most preconditioners run directly on the GPU” (https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!… <https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!…>). Is ASM + ILU also available for GPU platforms? -Justin -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uC… <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf…>

1 0

Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms
by Angus, Justin Ray 05 Nov '25

05 Nov '25

Thanks for the reply. The work per block should be the same for the weak scaling. I know LU is not scalable with respect to the block size. Perhaps our setup is not doing what we think it is doing. I’ll look into it further. -Justin From: Mark Adams <mfadams(a)lbl.gov> Date: Wednesday, November 5, 2025 at 6:14 AM To: Matthew Knepley <knepley(a)gmail.com> Cc: Angus, Justin Ray <angus1(a)llnl.gov>, petsc-dev(a)mcs.anl.gov <petsc-dev(a)mcs.anl.gov> Subject: Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms And we do not have sparse LU on GPUs so that is done on the CPU. And I don't know why it would not weak scale well. Your results are consistent with just using one process with one domain, (re Matt) while you double the problem size. On Tue, Nov 4, 2025 at 2:27 PM Matthew Knepley <knepley(a)gmail.com<mailto:[email protected]>> wrote: On Tue, Nov 4, 2025 at 1:25 PM Angus, Justin Ray via petsc-dev <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>> wrote: Hi Junchao, We have recently been using ASM + LU for 2D problems on both CPU and GPU. However, I found that this method has very bad weak scaling. I find that the cost of PCApply increases by about a factor of 4 each time I increase the problem size in 1 dimension by a factor of 2 while keeping the load per core/gpu the same. The total number of GMRES iterations does not increase, just the cost of PCApply (and PCSetup). Is this scaling behavior expected? Any ideas of how to optimize the preconditioner? The cost of PCApply for ASM is dominated by the cost of process-local block solves. You are using LU for the block solve. (Sparse) LU has cost roughly O(N^2) for the apply (depending on the structure of the matrix). So, if you double the size of a local block, your runtime should increase by about 4x. Thus LU is not a scalable method. Thanks, Matt Thank you. -Justin From: Junchao Zhang <junchao.zhang(a)gmail.com<mailto:[email protected]>> Date: Monday, April 14, 2025 at 7:35 PM To: Angus, Justin Ray <angus1(a)llnl.gov<mailto:[email protected]>> Cc: petsc-dev(a)mcs.anl.gov<mailto:[email protected]> <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>>, Ghosh, Debojyoti <ghosh5(a)llnl.gov<mailto:[email protected]>> Subject: Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms Petsc supports ILU0/ICC0 numeric factorization (without reordering) and then triangular solve on GPUs. It is done by calling vendor libraries (ex. cusparse). We have options -pc_factor_mat_factor_on_host <bool> -pc_factor_mat_solve_on_host <bool> to force doing the factorization and MatSolve on the host for device matrix types. You can try to see if it works for your case. --Junchao Zhang On Mon, Apr 14, 2025 at 4:39 PM Angus, Justin Ray via petsc-dev <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>> wrote: Hello, A project I work on uses GMRES via PETSc. In particular, we have had good successes using the Additive Schwarz Method + ILU preconditioner setup using a CPU-based code. I found online where it is stated that “Parts of most preconditioners run directly on the GPU” (https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!… <https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!…>). Is ASM + ILU also available for GPU platforms? -Justin -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uC… <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf…>

1 0

Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms
by Mark Adams 05 Nov '25

05 Nov '25

And we do not have sparse LU on GPUs so that is done on the CPU. And I don't know why it would not weak scale well. Your results are consistent with just using one process with one domain, (re Matt) while you double the problem size. On Tue, Nov 4, 2025 at 2:27 PM Matthew Knepley <knepley(a)gmail.com> wrote: > On Tue, Nov 4, 2025 at 1:25 PM Angus, Justin Ray via petsc-dev < > petsc-dev(a)mcs.anl.gov> wrote: > >> Hi Junchao, >> >> We have recently been using ASM + LU for 2D problems on both CPU and GPU. >> However, I found that this method has very bad weak scaling. I find that >> the cost of PCApply increases by about a factor of 4 each time I increase >> the problem size in 1 dimension by a factor of 2 while keeping the load per >> core/gpu the same. The total number of GMRES iterations does not increase, >> just the cost of PCApply (and PCSetup). Is this scaling behavior expected? >> Any ideas of how to optimize the preconditioner? >> > > The cost of PCApply for ASM is dominated by the cost of process-local > block solves. You are using LU for the block solve. (Sparse) LU has cost > roughly O(N^2) for the apply (depending on the structure of the matrix). > So, if you double the size of a local block, your runtime should increase > by about 4x. Thus LU is not a scalable method. > > Thanks, > > Matt > > >> Thank you. >> >> -Justin >> >> *From: *Junchao Zhang <junchao.zhang(a)gmail.com> >> *Date: *Monday, April 14, 2025 at 7:35 PM >> *To: *Angus, Justin Ray <angus1(a)llnl.gov> >> *Cc: *petsc-dev(a)mcs.anl.gov <petsc-dev(a)mcs.anl.gov>, Ghosh, Debojyoti < >> ghosh5(a)llnl.gov> >> *Subject: *Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms >> >> Petsc supports ILU0/ICC0 numeric factorization (without reordering) and >> then triangular solve on GPUs. It is done by calling vendor libraries (ex. >> cusparse). >> We have options -pc_factor_mat_factor_on_host <bool> >> -pc_factor_mat_solve_on_host <bool> to force doing the factorization and >> MatSolve on the host for device matrix types. >> >> You can try to see if it works for your case. >> >> --Junchao Zhang >> >> >> On Mon, Apr 14, 2025 at 4:39 PM Angus, Justin Ray via petsc-dev < >> petsc-dev(a)mcs.anl.gov> wrote: >> >> Hello, >> >> >> >> A project I work on uses GMRES via PETSc. In particular, we have had good >> successes using the Additive Schwarz Method + ILU preconditioner setup >> using a CPU-based code. I found online where it is stated that “Parts of >> most preconditioners run directly on the GPU” ( >> https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!… >> <https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!…>). >> Is ASM + ILU also available for GPU platforms? >> >> >> >> -Justin >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uC… > <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf…> >

1 0

Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms
by Matthew Knepley 04 Nov '25

04 Nov '25

On Tue, Nov 4, 2025 at 1:25 PM Angus, Justin Ray via petsc-dev < petsc-dev(a)mcs.anl.gov> wrote: > Hi Junchao, > > We have recently been using ASM + LU for 2D problems on both CPU and GPU. > However, I found that this method has very bad weak scaling. I find that > the cost of PCApply increases by about a factor of 4 each time I increase > the problem size in 1 dimension by a factor of 2 while keeping the load per > core/gpu the same. The total number of GMRES iterations does not increase, > just the cost of PCApply (and PCSetup). Is this scaling behavior expected? > Any ideas of how to optimize the preconditioner? > The cost of PCApply for ASM is dominated by the cost of process-local block solves. You are using LU for the block solve. (Sparse) LU has cost roughly O(N^2) for the apply (depending on the structure of the matrix). So, if you double the size of a local block, your runtime should increase by about 4x. Thus LU is not a scalable method. Thanks, Matt > Thank you. > > -Justin > > *From: *Junchao Zhang <junchao.zhang(a)gmail.com> > *Date: *Monday, April 14, 2025 at 7:35 PM > *To: *Angus, Justin Ray <angus1(a)llnl.gov> > *Cc: *petsc-dev(a)mcs.anl.gov <petsc-dev(a)mcs.anl.gov>, Ghosh, Debojyoti < > ghosh5(a)llnl.gov> > *Subject: *Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms > > Petsc supports ILU0/ICC0 numeric factorization (without reordering) and > then triangular solve on GPUs. It is done by calling vendor libraries (ex. > cusparse). > We have options -pc_factor_mat_factor_on_host <bool> > -pc_factor_mat_solve_on_host <bool> to force doing the factorization and > MatSolve on the host for device matrix types. > > You can try to see if it works for your case. > > --Junchao Zhang > > > On Mon, Apr 14, 2025 at 4:39 PM Angus, Justin Ray via petsc-dev < > petsc-dev(a)mcs.anl.gov> wrote: > > Hello, > > > > A project I work on uses GMRES via PETSc. In particular, we have had good > successes using the Additive Schwarz Method + ILU preconditioner setup > using a CPU-based code. I found online where it is stated that “Parts of > most preconditioners run directly on the GPU” ( > https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!… > <https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!…>). > Is ASM + ILU also available for GPU platforms? > > > > -Justin > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uC… <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf… >

1 0

Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms
by Angus, Justin Ray 04 Nov '25

04 Nov '25

Hi Junchao, We have recently been using ASM + LU for 2D problems on both CPU and GPU. However, I found that this method has very bad weak scaling. I find that the cost of PCApply increases by about a factor of 4 each time I increase the problem size in 1 dimension by a factor of 2 while keeping the load per core/gpu the same. The total number of GMRES iterations does not increase, just the cost of PCApply (and PCSetup). Is this scaling behavior expected? Any ideas of how to optimize the preconditioner? Thank you. -Justin From: Junchao Zhang <junchao.zhang(a)gmail.com> Date: Monday, April 14, 2025 at 7:35 PM To: Angus, Justin Ray <angus1(a)llnl.gov> Cc: petsc-dev(a)mcs.anl.gov <petsc-dev(a)mcs.anl.gov>, Ghosh, Debojyoti <ghosh5(a)llnl.gov> Subject: Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms Petsc supports ILU0/ICC0 numeric factorization (without reordering) and then triangular solve on GPUs. It is done by calling vendor libraries (ex. cusparse). We have options -pc_factor_mat_factor_on_host <bool> -pc_factor_mat_solve_on_host <bool> to force doing the factorization and MatSolve on the host for device matrix types. You can try to see if it works for your case. --Junchao Zhang On Mon, Apr 14, 2025 at 4:39 PM Angus, Justin Ray via petsc-dev <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>> wrote: Hello, A project I work on uses GMRES via PETSc. In particular, we have had good successes using the Additive Schwarz Method + ILU preconditioner setup using a CPU-based code. I found online where it is stated that “Parts of most preconditioners run directly on the GPU” (https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!… <https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!…>). Is ASM + ILU also available for GPU platforms? -Justin

1 0

PETSc 3.24 release is out
by Barry Smith 03 Oct '25

03 Oct '25

We are pleased to announce the release of PETSc version 3.24.0 at https://urldefense.us/v3/__https://petsc.org/release/download/__;!!G_uCfscf… This release introduces `PetscRegressor`, https://urldefense.us/v3/__https://petsc.org/release/manual/regressor/__;!!… which provides some basic infrastructure and a general API for supervised machine learning tasks at a higher level of abstraction than a purely algebraic “solvers” view. A list of the major changes and updates can be found at https://urldefense.us/v3/__https://petsc.org/release/changes/324/__;!!G_uCf… The final update to petsc-3.23 i.e petsc-3.23.7 is also available We recommend upgrading to PETSc 3.24.0 soon. As always, please report problems to petsc-maint(a)mcs.anl.gov <mailto:[email protected]> and ask questions at petsc-users(a)mcs.anl.gov <mailto:[email protected]> This release includes contributions from Albert Cowie Alex Lindsay aszaboa Barry Smith Brad Aagaard Connor Ward Daniel Otto de Mentock danofinn Dave Martin David Salac David Wells Davi Yan Hansol Suh James Wright Jared Frazier Jasper Hatton Jed Brown Jeremy L Thompson Jonas Heinzmann josephpu Jose Roman Junchao Zhang Koki Sagiyama Lisandro Dalcin Mark Adams Martin Diehl Matthew Knepley Min RK Mr. Hong Zhang Nuno Nobre Pablo Brubeck Patrick Farrell Paul T. Kühner Pierre Jolivet Pierre Marchand Raphael Zanella Rezgar Shakeri Richard Tran Mills Rylanor Salzman Alexis Satish Balay Stefano Zampini Tapashree Pradhan Toby Isaac Victor A. P. Magri Wisang Sugiarta Zach Atkins and bug reports/proposed improvements received from Aldo Bonfiglioli Alexis Salzman Ali Ahmad Christiaan Klaij Eric Chamberland Félix Kwok Hammond, Glenn E Jose Roman Juan Franco Lawrence Mitchell Lichtner, Peter Lisandro Dalcin PIGNET Nicolas Sebastien Gilles Steven Dargaville Victor Eijkhout Volker Jacht Zongze Yang As always, thanks for your support, Barry

1 0

Fwd: [PETSc] NumFOCUS Code of Conduct – onboarding
by Barry Smith 30 Sep '25

30 Sep '25

Anyone who can fit this into their schedule is welcome to come. Jed should attend if possible. Jed, can you let me know when you are available in the next week or so, and I'll see if I can find an opening? Their scheduling tool, unfortunately, doesn't allow multiple of us to each indicate availability. Barry > Begin forwarded message: > > From: Kamila Stepniowska <kamila(a)numfocus.org> > Subject: [PETSc] NumFOCUS Code of Conduct – onboarding > Date: September 30, 2025 at 7:13:42 PM EDT > To: Barry Smith <bsmith(a)petsc.dev> > Cc: NumFOCUS Code of Conduct Enforcement Team <conduct(a)numfocus.org>, Arliss Collins <arliss(a)numfocus.org>, Kari Hall <kari(a)numfocus.org> > Message-Id: <CAE61Cdx1cvZ-kLLq5AmjHxgwMpqg6f6+RA2f7bMD0ukJXAOP_Q(a)mail.gmail.com> > > Hi Barry, > > Thank you for submitting the NF CoC Project Alignment Form. We are thrilled that PETSc will incorporate the new NumFOCUS CoC! > > The next step is to schedule the onboarding meeting by using the CoC Onboarding Appointment Scheduler <https://urldefense.us/v3/__https://calendar.app.google/vrd3NzK8EWjLe5YF8__;… >. Please let us know your availability. Feel free to share the link with anyone else from your project who should join the meeting, in particular core developers, moderators, and event organizers. > > CC Arliss and Kari (Advocacy Team at NumFOCUS), and for visibility, NumFOCUS Code of Conduct Working Group. > > Best regards, > -- > Kamila Stępniowska > Advocacy Program Lead > numfocus.org <https://urldefense.us/v3/__https://numfocus.org/__;!!G_uCfscf7eWS!akQS2msOb… > > > I work flexibly and may send emails outside standard hours. I'm not expecting a response from you outside of your own working hours. > > My time zone: Pacific Time > >

1 0