Hi, Sawsan, First in test_main.F90, you need to call VecGetArrayF90(temp_solution, H_vector, ierr) and VecRestoreArrayF90 (temp_solution, H_vector, ierr) as Barry mentioned. Secondly, in the loop of test_main.F90, it calls GW_solver(). Within it, it calls PetscInitialize()/PetscFinalize(). But without MPI being initialized, PetscInitialize()/PetscFinalize()* can only be called once. * do timestep =2 , NTSP call GW_boundary_conditions(timestep-1) !print *,HNEW(1,1,1) call GW_elevation() ! print *, GWTOP(2,2,2) call GW_conductance() ! print *, CC(2,2,2) call GW_recharge() ! print *, B_Rech(5,4) call GW_pumping(timestep-1) ! print *, B_pump(2,2,2) call GW_SW(timestep-1) print *,B_RIVER (2,2,2) call GW_solver(timestep-1,N) call GW_deallocate_loop() end do A solution is to delete PetscInitialize()/PetscFinalize() in GW_solver_try.F90 and move it to test_main.F90, outside the do loop. diff --git a/test_main.F90 b/test_main.F90 index b5997c55..107bd3ee 100644 --- a/test_main.F90 +++ b/test_main.F90 @@ -1,5 +1,6 @@ program test_GW +#include <petsc/finclude/petsc.h> use petsc use GW_constants use GW_param_by_user @@ -8,6 +9,9 @@ program test_GW implicit none integer :: N integer :: timestep + PetscErrorCode ierr + + call PetscInitialize(ierr) call GW_domain(N) !print *, "N=",N !print *, DELTAT @@ -37,4 +41,5 @@ program test_GW end do print *, HNEW(NCOL,3,2) call GW_deallocate () + call PetscFinalize(ierr) end program test_GW With that, the MPI error will be fixed. The code could run to gw_deallocate () before abort. There are other memory errors. You can install/use valgrind to fix them. Run it with valgrind ./GW.exe and look through the output Thanks. --Junchao Zhang On Thu, Jan 11, 2024 at 10:49 PM Shatanawi, Sawsan Muhammad < [email protected]> wrote:

Hello,

Thank you all for your help.

I have changed VecGetArray to VecGetArrayF90, and the location of destory call. but I want to make sure that VecGet ArrayF90 is to make a new array( vector) that I can use in the rest of my Fortran code?

when I run it and debugged it, I got

5.2000000E-03 50.00000 10.00000 0.0000000E+00 PETSC: Attaching gdb to /weka/data/lab/richey/sawsan/GW_CODE/code2024/SS_GWM/./GW.exe of pid 33065 on display :0.0 on machine sn16 Unable to start debugger in xterm: No such file or directory 0.0000000E+00 Attempting to use an MPI routine after finalizing MPICH srun: error: sn16: task 0: Exited with exit code 1 [sawsan.shatanawi@login-p2n02 SS_GWM]$ gdb ./GW/exe GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later < http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... ./GW/exe: No such file or directory. (gdb) run Starting program: No executable file specified. Use the "file" or "exec-file" command. (gdb) bt No stack. (gdb)

If the highlighted line is the error, I don't know why when I write gdb , it does not show me the location of error The code : sshatanawi/SS_GWM (github.com) <https://github.com/sshatanawi/SS_GWM>

I really appreciate your helps

Sawsan ------------------------------ *From:* Barry Smith <[email protected]> *Sent:* Wednesday, January 10, 2024 5:35 PM *To:* Junchao Zhang <[email protected]> *Cc:* Shatanawi, Sawsan Muhammad <[email protected]>; Mark Adams < [email protected]>; [email protected] <[email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]*

On Jan 10, 2024, at 6:49 PM, Junchao Zhang <[email protected]> wrote:

Hi, Sawsan, I could build your code and I also could gdb it.

$ gdb ./GW.exe ... $ Thread 1 "GW.exe" received signal SIGSEGV, Segmentation fault. 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 257 *ierr = VecGetArray(*x, &lx); (gdb) bt #0 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 #1 0x000000000040b6e3 in gw_solver (t_s=1.40129846e-45, n=300) at GW_solver_try.F90:169 #2 0x000000000040c6a8 in test_gw () at test_main.F90:35

ierr=0x0 caused the segfault. See https://petsc.org/release/manualpages/Vec/VecGetArray/#vecgetarray <https://urldefense.com/v3/__https://petsc.org/release/manualpages/Vec/VecGetArray/*vecgetarray__;Iw!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ9Km12jA$>, you should use VecGetArrayF90 instead.

BTW, Barry, the code https://github.com/sshatanawi/SS_GWM/blob/main/GW_solver_try.F90#L169 <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM/blob/main/GW_solver_try.F90*L169__;Iw!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZh2eAi4o$> has "call VecGetArray(temp_solution, H_vector, ierr)". I don't find petsc Fortran examples doing VecGetArray. Do we still support it?

This is not the correct calling sequence for VecGetArray() from Fortran.

Regardless, definitely should not be writing any new code that uses VecGetArray() from Fortran. Should use VecGetArrayF90().

--Junchao Zhang

On Wed, Jan 10, 2024 at 2:38 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected]> wrote:

Hello all,

I hope you are doing well.

Generally, I use gdb <the name of my exe.file> to debug the code. I got the attached error message.

I have tried to add the flag -start_in_debugger in the make file, but it didn't work, so it seems I was doing it in the wrong way

This is the link for the whole code: sshatanawi/SS_GWM (github.com) <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ1Veab3M$>

<https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ1Veab3M$> GitHub - sshatanawi/SS_GWM <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ1Veab3M$> Contribute to sshatanawi/SS_GWM development by creating an account on GitHub. github.com <https://urldefense.com/v3/__http://github.com/__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ8rcrPiA$> **

You can read the description of the code in " Model Desprciption.pdf" the compiling file is makefile_f90 where you can find the linked code files

I really appreciate your help

Bests, Sawsan ------------------------------ *From:* Mark Adams <[email protected]> *Sent:* Friday, January 5, 2024 4:53 AM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* Matthew Knepley <[email protected]>; [email protected] < [email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]* This is a segv. As Matt said, you need to use a debugger for this or add print statements to narrow down the place where this happens.

You will need to learn how to use debuggers to do your project so you might as well start now.

If you have a machine with a GUI debugger that is easier but command line debuggers are good to learn anyway.

I tend to run debuggers directly (eg, lldb ./a.out -- program-args ...) and use a GUI debugger (eg, Totalview or DDT) if available.

Mark

On Wed, Dec 20, 2023 at 10:02 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected]> wrote:

Hello Matthew,

Thank you for your help. I am sorry that I keep coming back with my error messages, but I reached a point that I don't know how to fix them, and I don't understand them easily. The list of errors is getting shorter, now I am getting the attached error messages

Thank you again,

Sawsan ------------------------------ *From:* Matthew Knepley <[email protected]> *Sent:* Wednesday, December 20, 2023 6:54 PM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* Barry Smith <[email protected]>; [email protected] < [email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]* On Wed, Dec 20, 2023 at 9:49 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected]> wrote:

Hello Barry,

Thank you a lot for your help, Now I am getting the attached error message.

Do not destroy the PC from KSPGetPC()

THanks,

Matt

Bests, Sawsan ------------------------------ *From:* Barry Smith <[email protected]> *Sent:* Wednesday, December 20, 2023 6:32 PM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* Mark Adams <[email protected]>; [email protected] < [email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]*

Instead of

call PCCreate(PETSC_COMM_WORLD, pc, ierr) call PCSetType(pc, PCILU,ierr) ! Choose a preconditioner type (ILU) call KSPSetPC(ksp, pc,ierr) ! Associate the preconditioner with the KSP solver

do

call KSPGetPC(ksp,pc,ierr) call PCSetType(pc, PCILU,ierr)

Do not call KSPSetUp(). It will be taken care of automatically during the solve

On Dec 20, 2023, at 8:52 PM, Shatanawi, Sawsan Muhammad via petsc-users < [email protected]> wrote:

Hello, I don't think that I set preallocation values when I created the matrix, would you please have look at my code. It is just the petsc related part from my code. I was able to fix some of the error messages. Now I have a new set of error messages related to the KSP solver (attached)

I appreciate your help

Sawsan ------------------------------ *From:* Mark Adams <[email protected]> *Sent:* Wednesday, December 20, 2023 6:44 AM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* [email protected] <[email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]* Did you set preallocation values when you created the matrix? Don't do that.

On Wed, Dec 20, 2023 at 9:36 AM Shatanawi, Sawsan Muhammad < [email protected]> wrote:

Hello,

I am trying to create a sparse matrix( which is as I believe a zero matrix) then adding some nonzero elements to it over a loop, then assembling it

Get Outlook for iOS <https://urldefense.com/v3/__https://aka.ms/o0ukef__;!!JmPEgBY0HMszNaDT!uUJ_jeYf45gcXDGR_PeMjhU7hbd_fKcXJPn0pM9eb-YQihKNYuXMYM9x-hglsbXsCFIwNBWgHXdetHODupsOloE$> ------------------------------ *From:* Mark Adams <[email protected]> *Sent:* Wednesday, December 20, 2023 2:48 AM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* [email protected] <[email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]* I am guessing that you are creating a matrix, adding to it, finalizing it ("assembly"), and then adding to it again, which is fine, but you are adding new non-zeros to the sparsity pattern. If this is what you want then you can tell the matrix to let you do that. Otherwise you have a bug.

Mark

On Tue, Dec 19, 2023 at 9:50 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected]> wrote:

Hello everyone,

I hope this email finds you well.

My Name is Sawsan Shatanawi, and I am currently working on developing a Fortran code for simulating groundwater flow in a 3D system. The code involves solving a nonlinear system, and I have created the matrix to be solved using the PCG solver and Picard iteration. However, when I tried to assign it as a PETSc matrix I started getting a lot of error messages.

I am kindly asking if someone can help me, I would be happy to share my code with him/her.

Please find the attached file contains a list of errors I have gotten

Thank you in advance for your time and assistance.

Best regards,

Sawsan

<Matrix_RHS.F90><out.txt><solver.F90>

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <https://urldefense.com/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!JmPEgBY0HMszNaDT!uskvAyF0pMMWDbMIexr9g4qN46V7Rea17GQdNIVG2vH_HMaX7mXgie4ZYgusmPpss_DS7H1_8vn8arGQNSkC$>

Hello everyone, I hope this email finds you well. My Name is Sawsan Shatanawi, and I was developing a Fortran code for simulating groundwater flow in a 3D system with nonlinear behavior. I solved the nonlinear system using the PCG solver and Picard iteration, but I did not get good results although I checked my matrix and RHS and everything, I decided to change my solver to Newton Rapson method. I checked PETSc documents but I have a few questions: 1) My groundwater system is time-dependent, so should I use TS only instead of SNES? 2) My system has its deltaT, would using deltaT as dt affect my solver, or is it better to use TS-PETSc dt? Also, would using PETSc dt affect the simulation of the groundwater system 3) I want my Jacobian matrix to be calculated by PETSc automatically 4) Do I need to define and calculate the residual vector? My A-Matrix contains coefficients and external sources and my RHS vector includes the boundary conditions Please find the attached file contains a draft of my code Thank you in advance for your time and help. Best regards, Sawsan ________________________________ From: Shatanawi, Sawsan Muhammad <[email protected]> Sent: Tuesday, January 16, 2024 10:43 AM To: Junchao Zhang <[email protected]> Cc: Barry Smith <[email protected]>; Matthew Knepley <[email protected]>; Mark Adams <[email protected]>; [email protected] <[email protected]> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code Hello all, Thank you for your valuable help. I will do your recommendations and hope it will run without any issues. Bests, Sawsan ________________________________ From: Junchao Zhang <[email protected]> Sent: Friday, January 12, 2024 8:46 AM To: Shatanawi, Sawsan Muhammad <[email protected]> Cc: Barry Smith <[email protected]>; Matthew Knepley <[email protected]>; Mark Adams <[email protected]>; [email protected] <[email protected]> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] Hi, Sawsan, First in test_main.F90, you need to call VecGetArrayF90(temp_solution, H_vector, ierr) and VecRestoreArrayF90 (temp_solution, H_vector, ierr) as Barry mentioned. Secondly, in the loop of test_main.F90, it calls GW_solver(). Within it, it calls PetscInitialize()/PetscFinalize(). But without MPI being initialized, PetscInitialize()/PetscFinalize() can only be called once. do timestep =2 , NTSP call GW_boundary_conditions(timestep-1) !print *,HNEW(1,1,1) call GW_elevation() ! print *, GWTOP(2,2,2) call GW_conductance() ! print *, CC(2,2,2) call GW_recharge() ! print *, B_Rech(5,4) call GW_pumping(timestep-1) ! print *, B_pump(2,2,2) call GW_SW(timestep-1) print *,B_RIVER (2,2,2) call GW_solver(timestep-1,N) call GW_deallocate_loop() end do A solution is to delete PetscInitialize()/PetscFinalize() in GW_solver_try.F90 and move it to test_main.F90, outside the do loop. diff --git a/test_main.F90 b/test_main.F90 index b5997c55..107bd3ee 100644 --- a/test_main.F90 +++ b/test_main.F90 @@ -1,5 +1,6 @@ program test_GW +#include <petsc/finclude/petsc.h> use petsc use GW_constants use GW_param_by_user @@ -8,6 +9,9 @@ program test_GW implicit none integer :: N integer :: timestep + PetscErrorCode ierr + + call PetscInitialize(ierr) call GW_domain(N) !print *, "N=",N !print *, DELTAT @@ -37,4 +41,5 @@ program test_GW end do print *, HNEW(NCOL,3,2) call GW_deallocate () + call PetscFinalize(ierr) end program test_GW With that, the MPI error will be fixed. The code could run to gw_deallocate () before abort. There are other memory errors. You can install/use valgrind to fix them. Run it with valgrind ./GW.exe and look through the output Thanks. --Junchao Zhang On Thu, Jan 11, 2024 at 10:49 PM Shatanawi, Sawsan Muhammad <[email protected]<mailto:[email protected]>> wrote: Hello, Thank you all for your help. I have changed VecGetArray to VecGetArrayF90, and the location of destory call. but I want to make sure that VecGet ArrayF90 is to make a new array( vector) that I can use in the rest of my Fortran code? when I run it and debugged it, I got 5.2000000E-03 50.00000 10.00000 0.0000000E+00 PETSC: Attaching gdb to /weka/data/lab/richey/sawsan/GW_CODE/code2024/SS_GWM/./GW.exe of pid 33065 on display :0.0 on machine sn16 Unable to start debugger in xterm: No such file or directory 0.0000000E+00 Attempting to use an MPI routine after finalizing MPICH srun: error: sn16: task 0: Exited with exit code 1 [sawsan.shatanawi@login-p2n02 SS_GWM]$ gdb ./GW/exe GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <https://urldefense.us/v3/__http://gnu.org/licenses/gpl.html__;!!G_uCfscf7eWS... <https://urldefense.com/v3/__http://gnu.org/licenses/gpl.html__;!!JmPEgBY0HMszNaDT!o4qS1zaFLg2L8PlawWJVyYsJpnwYHuL6SIIZsmzTO98RCzbP26HwTi_0-ipCS_D2SBG0X4gtEnM13-nbzFEFKvtraKaM$>> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <https://urldefense.us/v3/__http://www.gnu.org/software/gdb/bugs/__;!!G_uCfsc... <https://urldefense.com/v3/__http://www.gnu.org/software/gdb/bugs/__;!!JmPEgBY0HMszNaDT!o4qS1zaFLg2L8PlawWJVyYsJpnwYHuL6SIIZsmzTO98RCzbP26HwTi_0-ipCS_D2SBG0X4gtEnM13-nbzFEFKjO8jNfs$>>... ./GW/exe: No such file or directory. (gdb) run Starting program: No executable file specified. Use the "file" or "exec-file" command. (gdb) bt No stack. (gdb) If the highlighted line is the error, I don't know why when I write gdb , it does not show me the location of error The code : sshatanawi/SS_GWM (github.com)<https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!o4qS1zaFLg2L8PlawWJVyYsJpnwYHuL6SIIZsmzTO98RCzbP26HwTi_0-ipCS_D2SBG0X4gtEnM13-nbzFEFKp8i33ur$> I really appreciate your helps Sawsan ________________________________ From: Barry Smith <[email protected]<mailto:[email protected]>> Sent: Wednesday, January 10, 2024 5:35 PM To: Junchao Zhang <[email protected]<mailto:[email protected]>> Cc: Shatanawi, Sawsan Muhammad <[email protected]<mailto:[email protected]>>; Mark Adams <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] On Jan 10, 2024, at 6:49 PM, Junchao Zhang <[email protected]<mailto:[email protected]>> wrote: Hi, Sawsan, I could build your code and I also could gdb it. $ gdb ./GW.exe ... $ Thread 1 "GW.exe" received signal SIGSEGV, Segmentation fault. 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 257 *ierr = VecGetArray(*x, &lx); (gdb) bt #0 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 #1 0x000000000040b6e3 in gw_solver (t_s=1.40129846e-45, n=300) at GW_solver_try.F90:169 #2 0x000000000040c6a8 in test_gw () at test_main.F90:35 ierr=0x0 caused the segfault. See https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGetA... <https://urldefense.com/v3/__https://petsc.org/release/manualpages/Vec/VecGetArray/*vecgetarray__;Iw!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ9Km12jA$>, you should use VecGetArrayF90 instead. BTW, Barry, the code https://urldefense.us/v3/__https://github.com/sshatanawi/SS_GWM/blob/main/GW... <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM/blob/main/GW_solver_try.F90*L169__;Iw!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZh2eAi4o$> has "call VecGetArray(temp_solution, H_vector, ierr)". I don't find petsc Fortran examples doing VecGetArray. Do we still support it? This is not the correct calling sequence for VecGetArray() from Fortran. Regardless, definitely should not be writing any new code that uses VecGetArray() from Fortran. Should use VecGetArrayF90(). --Junchao Zhang On Wed, Jan 10, 2024 at 2:38 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected]<mailto:[email protected]>> wrote: Hello all, I hope you are doing well. Generally, I use gdb <the name of my exe.file> to debug the code. I got the attached error message. I have tried to add the flag -start_in_debugger in the make file, but it didn't work, so it seems I was doing it in the wrong way This is the link for the whole code: sshatanawi/SS_GWM (github.com)<https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ1Veab3M$> [https://urldefense.us/v3/__https://opengraph.githubassets.com/9eb6cd14baf12f... ]<https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ1Veab3M$> GitHub - sshatanawi/SS_GWM<https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ1Veab3M$> Contribute to sshatanawi/SS_GWM development by creating an account on GitHub. github.com<https://urldefense.com/v3/__http://github.com/__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ8rcrPiA$>  You can read the description of the code in " Model Desprciption.pdf" the compiling file is makefile_f90 where you can find the linked code files I really appreciate your help Bests, Sawsan ________________________________ From: Mark Adams <[email protected]<mailto:[email protected]>> Sent: Friday, January 5, 2024 4:53 AM To: Shatanawi, Sawsan Muhammad <[email protected]<mailto:[email protected]>> Cc: Matthew Knepley <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] This is a segv. As Matt said, you need to use a debugger for this or add print statements to narrow down the place where this happens. You will need to learn how to use debuggers to do your project so you might as well start now. If you have a machine with a GUI debugger that is easier but command line debuggers are good to learn anyway. I tend to run debuggers directly (eg, lldb ./a.out -- program-args ...) and use a GUI debugger (eg, Totalview or DDT) if available. Mark On Wed, Dec 20, 2023 at 10:02 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected]<mailto:[email protected]>> wrote: Hello Matthew, Thank you for your help. I am sorry that I keep coming back with my error messages, but I reached a point that I don't know how to fix them, and I don't understand them easily. The list of errors is getting shorter, now I am getting the attached error messages Thank you again, Sawsan ________________________________ From: Matthew Knepley <[email protected]<mailto:[email protected]>> Sent: Wednesday, December 20, 2023 6:54 PM To: Shatanawi, Sawsan Muhammad <[email protected]<mailto:[email protected]>> Cc: Barry Smith <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] On Wed, Dec 20, 2023 at 9:49 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected]<mailto:[email protected]>> wrote: Hello Barry, Thank you a lot for your help, Now I am getting the attached error message. Do not destroy the PC from KSPGetPC() THanks, Matt Bests, Sawsan ________________________________ From: Barry Smith <[email protected]<mailto:[email protected]>> Sent: Wednesday, December 20, 2023 6:32 PM To: Shatanawi, Sawsan Muhammad <[email protected]<mailto:[email protected]>> Cc: Mark Adams <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] Instead of call PCCreate(PETSC_COMM_WORLD, pc, ierr) call PCSetType(pc, PCILU,ierr) ! Choose a preconditioner type (ILU) call KSPSetPC(ksp, pc,ierr) ! Associate the preconditioner with the KSP solver do call KSPGetPC(ksp,pc,ierr) call PCSetType(pc, PCILU,ierr) Do not call KSPSetUp(). It will be taken care of automatically during the solve On Dec 20, 2023, at 8:52 PM, Shatanawi, Sawsan Muhammad via petsc-users <[email protected]<mailto:[email protected]>> wrote: Hello, I don't think that I set preallocation values when I created the matrix, would you please have look at my code. It is just the petsc related part from my code. I was able to fix some of the error messages. Now I have a new set of error messages related to the KSP solver (attached) I appreciate your help Sawsan ________________________________ From: Mark Adams <[email protected]<mailto:[email protected]>> Sent: Wednesday, December 20, 2023 6:44 AM To: Shatanawi, Sawsan Muhammad <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] Did you set preallocation values when you created the matrix? Don't do that. On Wed, Dec 20, 2023 at 9:36 AM Shatanawi, Sawsan Muhammad <[email protected]<mailto:[email protected]>> wrote: Hello, I am trying to create a sparse matrix( which is as I believe a zero matrix) then adding some nonzero elements to it over a loop, then assembling it Get Outlook for iOS<https://urldefense.com/v3/__https://aka.ms/o0ukef__;!!JmPEgBY0HMszNaDT!uUJ_jeYf45gcXDGR_PeMjhU7hbd_fKcXJPn0pM9eb-YQihKNYuXMYM9x-hglsbXsCFIwNBWgHXdetHODupsOloE$> ________________________________ From: Mark Adams <[email protected]<mailto:[email protected]>> Sent: Wednesday, December 20, 2023 2:48 AM To: Shatanawi, Sawsan Muhammad <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code [EXTERNAL EMAIL] I am guessing that you are creating a matrix, adding to it, finalizing it ("assembly"), and then adding to it again, which is fine, but you are adding new non-zeros to the sparsity pattern. If this is what you want then you can tell the matrix to let you do that. Otherwise you have a bug. Mark On Tue, Dec 19, 2023 at 9:50 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected]<mailto:[email protected]>> wrote: Hello everyone, I hope this email finds you well. My Name is Sawsan Shatanawi, and I am currently working on developing a Fortran code for simulating groundwater flow in a 3D system. The code involves solving a nonlinear system, and I have created the matrix to be solved using the PCG solver and Picard iteration. However, when I tried to assign it as a PETSc matrix I started getting a lot of error messages. I am kindly asking if someone can help me, I would be happy to share my code with him/her. Please find the attached file contains a list of errors I have gotten Thank you in advance for your time and assistance. Best regards, Sawsan <Matrix_RHS.F90><out.txt><solver.F90> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.com/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!JmPEgBY0HMszNaDT!uskvAyF0pMMWDbMIexr9g4qN46V7Rea17GQdNIVG2vH_HMaX7mXgie4ZYgusmPpss_DS7H1_8vn8arGQNSkC$>

On Tue, May 7, 2024 at 2:23 PM Shatanawi, Sawsan Muhammad via petsc-users < [email protected]> wrote:

Hello everyone, I hope this email finds you well. My Name is Sawsan Shatanawi, and I was developing a Fortran code for simulating groundwater flow in a 3D system with nonlinear behavior. I solved the nonlinear system using the PCG solver and ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization.

ZjQcmQRYFpfptBannerEnd

Hello everyone,

I hope this email finds you well.

My Name is Sawsan Shatanawi, and I was developing a Fortran code for simulating groundwater flow in a 3D system with nonlinear behavior. I solved the nonlinear system using the PCG solver and Picard iteration, but I did not get good results although I checked my matrix and RHS and everything, I decided to change my solver to Newton Rapson method. I checked PETSc documents but I have a few questions: 1) My groundwater system is time-dependent, so should I use TS only instead of SNES?

You could use TS, but it is not necessary. You could use SNES and your own timestepping. THe advantage of TS is that you can try many different timesteppers without recoding (just like you can try many different linear and nonlinear solvers).

2) My system has its deltaT, would using deltaT as dt affect my solver, or is it better to use TS-PETSc dt? Also, would using PETSc dt affect the simulation of the groundwater system

It sounds like your dt comes from your timestepper. If you use TS, you would use the dt from that.

3) I want my Jacobian matrix to be calculated by PETSc automatically

PETSc can calculate a full Jacobian for smaller problems, or a finite-difference Jacobian for any problem (but this impacts the solver). It should be straightfoward to code up the analytic Jacobian. Is there a reason it would be a problem?

4) Do I need to define and calculate the residual vector?

Yes.

My A-Matrix contains coefficients and external sources and my RHS vector includes the boundary conditions

It is strange that your matrix would contain source terms. Do they depend on the flow fields? Barry is right, you should consider PFlotran, and at least know why it would not work for your problem if you don't use it. Thanks, Matt

Please find the attached file contains a draft of my code

Thank you in advance for your time and help.

Best regards,

Sawsan

------------------------------ *From:* Shatanawi, Sawsan Muhammad <[email protected]> *Sent:* Tuesday, January 16, 2024 10:43 AM *To:* Junchao Zhang <[email protected]> *Cc:* Barry Smith <[email protected]>; Matthew Knepley <[email protected]>; Mark Adams <[email protected]>; [email protected] < [email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

Hello all,

Thank you for your valuable help. I will do your recommendations and hope it will run without any issues.

Bests, Sawsan

------------------------------ *From:* Junchao Zhang <[email protected]> *Sent:* Friday, January 12, 2024 8:46 AM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* Barry Smith <[email protected]>; Matthew Knepley <[email protected]>; Mark Adams <[email protected]>; [email protected] < [email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]* Hi, Sawsan, First in test_main.F90, you need to call VecGetArrayF90(temp_solution, H_vector, ierr) and VecRestoreArrayF90 (temp_solution, H_vector, ierr) as Barry mentioned. Secondly, in the loop of test_main.F90, it calls GW_solver(). Within it, it calls PetscInitialize()/PetscFinalize(). But without MPI being initialized, PetscInitialize()/PetscFinalize()* can only be called once. * do timestep =2 , NTSP call GW_boundary_conditions(timestep-1) !print *,HNEW(1,1,1) call GW_elevation() ! print *, GWTOP(2,2,2) call GW_conductance() ! print *, CC(2,2,2) call GW_recharge() ! print *, B_Rech(5,4) call GW_pumping(timestep-1) ! print *, B_pump(2,2,2) call GW_SW(timestep-1) print *,B_RIVER (2,2,2) call GW_solver(timestep-1,N) call GW_deallocate_loop() end do

A solution is to delete PetscInitialize()/PetscFinalize() in GW_solver_try.F90 and move it to test_main.F90, outside the do loop.

diff --git a/test_main.F90 b/test_main.F90 index b5997c55..107bd3ee 100644 --- a/test_main.F90 +++ b/test_main.F90 @@ -1,5 +1,6 @@ program test_GW

+#include <petsc/finclude/petsc.h> use petsc use GW_constants use GW_param_by_user @@ -8,6 +9,9 @@ program test_GW implicit none integer :: N integer :: timestep + PetscErrorCode ierr + + call PetscInitialize(ierr) call GW_domain(N) !print *, "N=",N !print *, DELTAT @@ -37,4 +41,5 @@ program test_GW end do print *, HNEW(NCOL,3,2) call GW_deallocate () + call PetscFinalize(ierr) end program test_GW

With that, the MPI error will be fixed. The code could run to gw_deallocate () before abort. There are other memory errors. You can install/use valgrind to fix them. Run it with valgrind ./GW.exe and look through the output

Thanks. --Junchao Zhang

On Thu, Jan 11, 2024 at 10:49 PM Shatanawi, Sawsan Muhammad < [email protected]> wrote:

Hello,

Thank you all for your help.

I have changed VecGetArray to VecGetArrayF90, and the location of destory call. but I want to make sure that VecGet ArrayF90 is to make a new array( vector) that I can use in the rest of my Fortran code?

when I run it and debugged it, I got

5.2000000E-03 50.00000 10.00000 0.0000000E+00 PETSC: Attaching gdb to /weka/data/lab/richey/sawsan/GW_CODE/code2024/SS_GWM/./GW.exe of pid 33065 on display :0.0 on machine sn16 Unable to start debugger in xterm: No such file or directory 0.0000000E+00 Attempting to use an MPI routine after finalizing MPICH srun: error: sn16: task 0: Exited with exit code 1 [sawsan.shatanawi@login-p2n02 SS_GWM]$ gdb ./GW/exe GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later < https://urldefense.us/v3/__http://gnu.org/licenses/gpl.html__;!!G_uCfscf7eWS... <https://urldefense.com/v3/__http://gnu.org/licenses/gpl.html__;!!JmPEgBY0HMszNaDT!o4qS1zaFLg2L8PlawWJVyYsJpnwYHuL6SIIZsmzTO98RCzbP26HwTi_0-ipCS_D2SBG0X4gtEnM13-nbzFEFKvtraKaM$>

...

This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <https://urldefense.us/v3/__http://www.gnu.org/software/gdb/bugs/__;!!G_uCfsc... <https://urldefense.com/v3/__http://www.gnu.org/software/gdb/bugs/__;!!JmPEgBY0HMszNaDT!o4qS1zaFLg2L8PlawWJVyYsJpnwYHuL6SIIZsmzTO98RCzbP26HwTi_0-ipCS_D2SBG0X4gtEnM13-nbzFEFKjO8jNfs$>

...

... ./GW/exe: No such file or directory. (gdb) run Starting program: No executable file specified. Use the "file" or "exec-file" command. (gdb) bt No stack. (gdb)

If the highlighted line is the error, I don't know why when I write gdb , it does not show me the location of error The code : sshatanawi/SS_GWM (github.com) <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!o4qS1zaFLg2L8PlawWJVyYsJpnwYHuL6SIIZsmzTO98RCzbP26HwTi_0-ipCS_D2SBG0X4gtEnM13-nbzFEFKp8i33ur$>

I really appreciate your helps

Sawsan ------------------------------ *From:* Barry Smith <[email protected]> *Sent:* Wednesday, January 10, 2024 5:35 PM *To:* Junchao Zhang <[email protected]> *Cc:* Shatanawi, Sawsan Muhammad <[email protected]>; Mark Adams < [email protected]>; [email protected] <[email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]*

On Jan 10, 2024, at 6:49 PM, Junchao Zhang <[email protected]> wrote:

Hi, Sawsan, I could build your code and I also could gdb it.

$ gdb ./GW.exe ... $ Thread 1 "GW.exe" received signal SIGSEGV, Segmentation fault. 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 257 *ierr = VecGetArray(*x, &lx); (gdb) bt #0 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 #1 0x000000000040b6e3 in gw_solver (t_s=1.40129846e-45, n=300) at GW_solver_try.F90:169 #2 0x000000000040c6a8 in test_gw () at test_main.F90:35

ierr=0x0 caused the segfault. See https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGetA... <https://urldefense.com/v3/__https://petsc.org/release/manualpages/Vec/VecGetArray/*vecgetarray__;Iw!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ9Km12jA$>, you should use VecGetArrayF90 instead.

BTW, Barry, the code https://urldefense.us/v3/__https://github.com/sshatanawi/SS_GWM/blob/main/GW... <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM/blob/main/GW_solver_try.F90*L169__;Iw!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZh2eAi4o$> has "call VecGetArray(temp_solution, H_vector, ierr)". I don't find petsc Fortran examples doing VecGetArray. Do we still support it?

This is not the correct calling sequence for VecGetArray() from Fortran.

Regardless, definitely should not be writing any new code that uses VecGetArray() from Fortran. Should use VecGetArrayF90().

--Junchao Zhang

On Wed, Jan 10, 2024 at 2:38 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected]> wrote:

Hello all,

I hope you are doing well.

Generally, I use gdb <the name of my exe.file> to debug the code. I got the attached error message.

I have tried to add the flag -start_in_debugger in the make file, but it didn't work, so it seems I was doing it in the wrong way

This is the link for the whole code: sshatanawi/SS_GWM (github.com) <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ1Veab3M$>

<https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ1Veab3M$> GitHub - sshatanawi/SS_GWM <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ1Veab3M$> Contribute to sshatanawi/SS_GWM development by creating an account on GitHub. github.com <https://urldefense.com/v3/__http://github.com/__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ8rcrPiA$> **

You can read the description of the code in " Model Desprciption.pdf" the compiling file is makefile_f90 where you can find the linked code files

I really appreciate your help

Bests, Sawsan ------------------------------ *From:* Mark Adams <[email protected]> *Sent:* Friday, January 5, 2024 4:53 AM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* Matthew Knepley <[email protected]>; [email protected] < [email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]* This is a segv. As Matt said, you need to use a debugger for this or add print statements to narrow down the place where this happens.

You will need to learn how to use debuggers to do your project so you might as well start now.

If you have a machine with a GUI debugger that is easier but command line debuggers are good to learn anyway.

I tend to run debuggers directly (eg, lldb ./a.out -- program-args ...) and use a GUI debugger (eg, Totalview or DDT) if available.

Mark

On Wed, Dec 20, 2023 at 10:02 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected]> wrote:

Hello Matthew,

Thank you for your help. I am sorry that I keep coming back with my error messages, but I reached a point that I don't know how to fix them, and I don't understand them easily. The list of errors is getting shorter, now I am getting the attached error messages

Thank you again,

Sawsan ------------------------------ *From:* Matthew Knepley <[email protected]> *Sent:* Wednesday, December 20, 2023 6:54 PM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* Barry Smith <[email protected]>; [email protected] < [email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]* On Wed, Dec 20, 2023 at 9:49 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected]> wrote:

Hello Barry,

Thank you a lot for your help, Now I am getting the attached error message.

Do not destroy the PC from KSPGetPC()

THanks,

Matt

Bests, Sawsan ------------------------------ *From:* Barry Smith <[email protected]> *Sent:* Wednesday, December 20, 2023 6:32 PM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* Mark Adams <[email protected]>; [email protected] < [email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]*

Instead of

call PCCreate(PETSC_COMM_WORLD, pc, ierr) call PCSetType(pc, PCILU,ierr) ! Choose a preconditioner type (ILU) call KSPSetPC(ksp, pc,ierr) ! Associate the preconditioner with the KSP solver

do

call KSPGetPC(ksp,pc,ierr) call PCSetType(pc, PCILU,ierr)

Do not call KSPSetUp(). It will be taken care of automatically during the solve

On Dec 20, 2023, at 8:52 PM, Shatanawi, Sawsan Muhammad via petsc-users < [email protected]> wrote:

Hello, I don't think that I set preallocation values when I created the matrix, would you please have look at my code. It is just the petsc related part from my code. I was able to fix some of the error messages. Now I have a new set of error messages related to the KSP solver (attached)

I appreciate your help

Sawsan ------------------------------ *From:* Mark Adams <[email protected]> *Sent:* Wednesday, December 20, 2023 6:44 AM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* [email protected] <[email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]* Did you set preallocation values when you created the matrix? Don't do that.

On Wed, Dec 20, 2023 at 9:36 AM Shatanawi, Sawsan Muhammad < [email protected]> wrote:

Hello,

I am trying to create a sparse matrix( which is as I believe a zero matrix) then adding some nonzero elements to it over a loop, then assembling it

Get Outlook for iOS <https://urldefense.com/v3/__https://aka.ms/o0ukef__;!!JmPEgBY0HMszNaDT!uUJ_jeYf45gcXDGR_PeMjhU7hbd_fKcXJPn0pM9eb-YQihKNYuXMYM9x-hglsbXsCFIwNBWgHXdetHODupsOloE$> ------------------------------ *From:* Mark Adams <[email protected]> *Sent:* Wednesday, December 20, 2023 2:48 AM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* [email protected] <[email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]* I am guessing that you are creating a matrix, adding to it, finalizing it ("assembly"), and then adding to it again, which is fine, but you are adding new non-zeros to the sparsity pattern. If this is what you want then you can tell the matrix to let you do that. Otherwise you have a bug.

Mark

On Tue, Dec 19, 2023 at 9:50 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected]> wrote:

Hello everyone,

I hope this email finds you well.

My Name is Sawsan Shatanawi, and I am currently working on developing a Fortran code for simulating groundwater flow in a 3D system. The code involves solving a nonlinear system, and I have created the matrix to be solved using the PCG solver and Picard iteration. However, when I tried to assign it as a PETSc matrix I started getting a lot of error messages.

I am kindly asking if someone can help me, I would be happy to share my code with him/her.

Please find the attached file contains a list of errors I have gotten

Thank you in advance for your time and assistance.

Best regards,

Sawsan

<Matrix_RHS.F90><out.txt><solver.F90>

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.com/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!JmPEgBY0HMszNaDT!uskvAyF0pMMWDbMIexr9g4qN46V7Rea17GQdNIVG2vH_HMaX7mXgie4ZYgusmPpss_DS7H1_8vn8arGQNSkC$>

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >

On May 9, 2024, at 2:52 PM, Shatanawi, Sawsan Muhammad via petsc-users <[email protected]> wrote:

This Message Is From an External Sender This message came from outside your organization. Hello everyone,

Thank you for your responses and feedback,

I checked PFLOTRAN and found that it is a model to simulate groundwater flow, contaminant transport, and other subsurface processes. my goal is not to simulate the groundwater flow, my goal is to develop a code from scratch to simulate the groundwater flow with specific conditions, and then integrate this code with land surface models. Later, the simulation of this code will be on a large scale.

I want PETSc to calculate the Jacobian because the system is large and has complex nonlinear behavior, and I don’t risk calculating the derivative by myself. My A-Matrix has parts of source terms that depend on the flow fields, and independent parts will be in the RHS vector.

With coloring SNESComputeJacobianDefaultColor() PETSc can compute Jacobian's pretty efficiently. You do not to provide the residual function and you need to provide the nonzero pattern of the sparse Jacobian; that is what residual components f_i are coupled to what input variables in the array x_i. This information comes from your PDE and discretization and appears implicitly in your residual function. Barry

I hope I have answered your questions, and I apologize that I wasn’t clear from the beginning, I was trying to keep my descriptions brief.

Bests, Sawsan From: Matthew Knepley <[email protected] <mailto:[email protected]>> Sent: Tuesday, May 7, 2024 5:17 PM To: Shatanawi, Sawsan Muhammad <[email protected] <mailto:[email protected]>> Cc: [email protected] <mailto:[email protected]> <[email protected] <mailto:[email protected]>> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

[EXTERNAL EMAIL] On Tue, May 7, 2024 at 2:23 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected] <mailto:[email protected]>> wrote: This Message Is From an External Sender This message came from outside your organization.

Hello everyone,

I hope this email finds you well.

My Name is Sawsan Shatanawi, and I was developing a Fortran code for simulating groundwater flow in a 3D system with nonlinear behavior. I solved the nonlinear system using the PCG solver and Picard iteration, but I did not get good results although I checked my matrix and RHS and everything, I decided to change my solver to Newton Rapson method. I checked PETSc documents but I have a few questions: 1) My groundwater system is time-dependent, so should I use TS only instead of SNES?

You could use TS, but it is not necessary. You could use SNES and your own timestepping. THe advantage of TS is that you can try many different timesteppers without recoding (just like you can try many different linear and nonlinear solvers).

2) My system has its deltaT, would using deltaT as dt affect my solver, or is it better to use TS-PETSc dt? Also, would using PETSc dt affect the simulation of the groundwater system

It sounds like your dt comes from your timestepper. If you use TS, you would use the dt from that.

3) I want my Jacobian matrix to be calculated by PETSc automatically

PETSc can calculate a full Jacobian for smaller problems, or a finite-difference Jacobian for any problem (but this impacts the solver). It should be straightfoward to code up the analytic Jacobian. Is there a reason it would be a problem?

4) Do I need to define and calculate the residual vector?

Yes.

My A-Matrix contains coefficients and external sources and my RHS vector includes the boundary conditions

It is strange that your matrix would contain source terms. Do they depend on the flow fields?

Barry is right, you should consider PFlotran, and at least know why it would not work for your problem if you don't use it.

Thanks,

Matt

Please find the attached file contains a draft of my code

Thank you in advance for your time and help.

Best regards,

Sawsan

From: Shatanawi, Sawsan Muhammad <[email protected] <mailto:[email protected]>> Sent: Tuesday, January 16, 2024 10:43 AM To: Junchao Zhang <[email protected] <mailto:[email protected]>> Cc: Barry Smith <[email protected] <mailto:[email protected]>>; Matthew Knepley <[email protected] <mailto:[email protected]>>; Mark Adams <[email protected] <mailto:[email protected]>>; [email protected] <mailto:[email protected]> <[email protected] <mailto:[email protected]>> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

Hello all,

Thank you for your valuable help. I will do your recommendations and hope it will run without any issues.

Bests, Sawsan

From: Junchao Zhang <[email protected] <mailto:[email protected]>> Sent: Friday, January 12, 2024 8:46 AM To: Shatanawi, Sawsan Muhammad <[email protected] <mailto:[email protected]>> Cc: Barry Smith <[email protected] <mailto:[email protected]>>; Matthew Knepley <[email protected] <mailto:[email protected]>>; Mark Adams <[email protected] <mailto:[email protected]>>; [email protected] <mailto:[email protected]> <[email protected] <mailto:[email protected]>> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

[EXTERNAL EMAIL] Hi, Sawsan, First in test_main.F90, you need to call VecGetArrayF90(temp_solution, H_vector, ierr) and VecRestoreArrayF90 (temp_solution, H_vector, ierr) as Barry mentioned. Secondly, in the loop of test_main.F90, it calls GW_solver(). Within it, it calls PetscInitialize()/PetscFinalize(). But without MPI being initialized, PetscInitialize()/PetscFinalize() can only be called once. do timestep =2 , NTSP call GW_boundary_conditions(timestep-1) !print *,HNEW(1,1,1) call GW_elevation() ! print *, GWTOP(2,2,2) call GW_conductance() ! print *, CC(2,2,2) call GW_recharge() ! print *, B_Rech(5,4) call GW_pumping(timestep-1) ! print *, B_pump(2,2,2) call GW_SW(timestep-1) print *,B_RIVER (2,2,2) call GW_solver(timestep-1,N) call GW_deallocate_loop() end do

A solution is to delete PetscInitialize()/PetscFinalize() in GW_solver_try.F90 and move it to test_main.F90, outside the do loop.

diff --git a/test_main.F90 b/test_main.F90 index b5997c55..107bd3ee 100644 --- a/test_main.F90 +++ b/test_main.F90 @@ -1,5 +1,6 @@ program test_GW

+#include <petsc/finclude/petsc.h> use petsc use GW_constants use GW_param_by_user @@ -8,6 +9,9 @@ program test_GW implicit none integer :: N integer :: timestep + PetscErrorCode ierr + + call PetscInitialize(ierr) call GW_domain(N) !print *, "N=",N !print *, DELTAT @@ -37,4 +41,5 @@ program test_GW end do print *, HNEW(NCOL,3,2) call GW_deallocate () + call PetscFinalize(ierr) end program test_GW

With that, the MPI error will be fixed. The code could run to gw_deallocate () before abort. There are other memory errors. You can install/use valgrind to fix them. Run it with valgrind ./GW.exe and look through the output

Thanks. --Junchao Zhang

On Thu, Jan 11, 2024 at 10:49 PM Shatanawi, Sawsan Muhammad <[email protected] <mailto:[email protected]>> wrote: Hello,

Thank you all for your help.

I have changed VecGetArray to VecGetArrayF90, and the location of destory call. but I want to make sure that VecGet ArrayF90 is to make a new array( vector) that I can use in the rest of my Fortran code?

when I run it and debugged it, I got

5.2000000E-03 50.00000 10.00000 0.0000000E+00 PETSC: Attaching gdb to /weka/data/lab/richey/sawsan/GW_CODE/code2024/SS_GWM/./GW.exe of pid 33065 on display :0.0 on machine sn16 Unable to start debugger in xterm: No such file or directory 0.0000000E+00 Attempting to use an MPI routine after finalizing MPICH srun: error: sn16: task 0: Exited with exit code 1 [sawsan.shatanawi@login-p2n02 SS_GWM]$ gdb ./GW/exe GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <https://urldefense.us/v3/__http://gnu.org/licenses/gpl.html__;!!G_uCfscf7eWS... <https://urldefense.com/v3/__http://gnu.org/licenses/gpl.html__;!!JmPEgBY0HMszNaDT!o4qS1zaFLg2L8PlawWJVyYsJpnwYHuL6SIIZsmzTO98RCzbP26HwTi_0-ipCS_D2SBG0X4gtEnM13-nbzFEFKvtraKaM$>> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <https://urldefense.us/v3/__http://www.gnu.org/software/gdb/bugs/__;!!G_uCfsc... <https://urldefense.com/v3/__http://www.gnu.org/software/gdb/bugs/__;!!JmPEgBY0HMszNaDT!o4qS1zaFLg2L8PlawWJVyYsJpnwYHuL6SIIZsmzTO98RCzbP26HwTi_0-ipCS_D2SBG0X4gtEnM13-nbzFEFKjO8jNfs$>>... ./GW/exe: No such file or directory. (gdb) run Starting program: No executable file specified. Use the "file" or "exec-file" command. (gdb) bt No stack. (gdb)

If the highlighted line is the error, I don't know why when I write gdb , it does not show me the location of error The code : sshatanawi/SS_GWM (github.com) <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!o4qS1zaFLg2L8PlawWJVyYsJpnwYHuL6SIIZsmzTO98RCzbP26HwTi_0-ipCS_D2SBG0X4gtEnM13-nbzFEFKp8i33ur$>

I really appreciate your helps

Sawsan From: Barry Smith <[email protected] <mailto:[email protected]>> Sent: Wednesday, January 10, 2024 5:35 PM To: Junchao Zhang <[email protected] <mailto:[email protected]>> Cc: Shatanawi, Sawsan Muhammad <[email protected] <mailto:[email protected]>>; Mark Adams <[email protected] <mailto:[email protected]>>; [email protected] <mailto:[email protected]> <[email protected] <mailto:[email protected]>> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

[EXTERNAL EMAIL]

...

On Jan 10, 2024, at 6:49 PM, Junchao Zhang <[email protected] <mailto:[email protected]>> wrote:

Hi, Sawsan, I could build your code and I also could gdb it.

$ gdb ./GW.exe ... $ Thread 1 "GW.exe" received signal SIGSEGV, Segmentation fault. 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 257 *ierr = VecGetArray(*x, &lx); (gdb) bt #0 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 #1 0x000000000040b6e3 in gw_solver (t_s=1.40129846e-45, n=300) at GW_solver_try.F90:169 #2 0x000000000040c6a8 in test_gw () at test_main.F90:35

ierr=0x0 caused the segfault. See https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGetA... <https://urldefense.com/v3/__https://petsc.org/release/manualpages/Vec/VecGetArray/*vecgetarray__;Iw!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ9Km12jA$>, you should use VecGetArrayF90 instead.

BTW, Barry, the code https://urldefense.us/v3/__https://github.com/sshatanawi/SS_GWM/blob/main/GW... <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM/blob/main/GW_solver_try.F90*L169__;Iw!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZh2eAi4o$> has "call VecGetArray(temp_solution, H_vector, ierr)". I don't find petsc Fortran examples doing VecGetArray. Do we still support it?

This is not the correct calling sequence for VecGetArray() from Fortran.

Regardless, definitely should not be writing any new code that uses VecGetArray() from Fortran. Should use VecGetArrayF90().

...

--Junchao Zhang

On Wed, Jan 10, 2024 at 2:38 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected] <mailto:[email protected]>> wrote: Hello all,

I hope you are doing well.

Generally, I use gdb <the name of my exe.file> to debug the code. I got the attached error message.

I have tried to add the flag -start_in_debugger in the make file, but it didn't work, so it seems I was doing it in the wrong way

This is the link for the whole code: sshatanawi/SS_GWM (github.com) <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ1Veab3M$> <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ1Veab3M$> GitHub - sshatanawi/SS_GWM <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ1Veab3M$> Contribute to sshatanawi/SS_GWM development by creating an account on GitHub. github.com <https://urldefense.com/v3/__http://github.com/__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ8rcrPiA$> 

You can read the description of the code in " Model Desprciption.pdf" the compiling file is makefile_f90 where you can find the linked code files

I really appreciate your help

Bests, Sawsan From: Mark Adams <[email protected] <mailto:[email protected]>> Sent: Friday, January 5, 2024 4:53 AM To: Shatanawi, Sawsan Muhammad <[email protected] <mailto:[email protected]>> Cc: Matthew Knepley <[email protected] <mailto:[email protected]>>; [email protected] <mailto:[email protected]> <[email protected] <mailto:[email protected]>> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

[EXTERNAL EMAIL] This is a segv. As Matt said, you need to use a debugger for this or add print statements to narrow down the place where this happens.

You will need to learn how to use debuggers to do your project so you might as well start now.

If you have a machine with a GUI debugger that is easier but command line debuggers are good to learn anyway.

I tend to run debuggers directly (eg, lldb ./a.out -- program-args ...) and use a GUI debugger (eg, Totalview or DDT) if available.

Mark

On Wed, Dec 20, 2023 at 10:02 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected] <mailto:[email protected]>> wrote: Hello Matthew,

Thank you for your help. I am sorry that I keep coming back with my error messages, but I reached a point that I don't know how to fix them, and I don't understand them easily. The list of errors is getting shorter, now I am getting the attached error messages

Thank you again,

Sawsan From: Matthew Knepley <[email protected] <mailto:[email protected]>> Sent: Wednesday, December 20, 2023 6:54 PM To: Shatanawi, Sawsan Muhammad <[email protected] <mailto:[email protected]>> Cc: Barry Smith <[email protected] <mailto:[email protected]>>; [email protected] <mailto:[email protected]> <[email protected] <mailto:[email protected]>> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

[EXTERNAL EMAIL] On Wed, Dec 20, 2023 at 9:49 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected] <mailto:[email protected]>> wrote: Hello Barry,

Thank you a lot for your help, Now I am getting the attached error message.

Do not destroy the PC from KSPGetPC()

THanks,

Matt

Bests, Sawsan From: Barry Smith <[email protected] <mailto:[email protected]>> Sent: Wednesday, December 20, 2023 6:32 PM To: Shatanawi, Sawsan Muhammad <[email protected] <mailto:[email protected]>> Cc: Mark Adams <[email protected] <mailto:[email protected]>>; [email protected] <mailto:[email protected]> <[email protected] <mailto:[email protected]>> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

[EXTERNAL EMAIL]

Instead of

call PCCreate(PETSC_COMM_WORLD, pc, ierr) call PCSetType(pc, PCILU,ierr) ! Choose a preconditioner type (ILU) call KSPSetPC(ksp, pc,ierr) ! Associate the preconditioner with the KSP solver

do

call KSPGetPC(ksp,pc,ierr) call PCSetType(pc, PCILU,ierr)

Do not call KSPSetUp(). It will be taken care of automatically during the solve

...

On Dec 20, 2023, at 8:52 PM, Shatanawi, Sawsan Muhammad via petsc-users <[email protected] <mailto:[email protected]>> wrote:

Hello, I don't think that I set preallocation values when I created the matrix, would you please have look at my code. It is just the petsc related part from my code. I was able to fix some of the error messages. Now I have a new set of error messages related to the KSP solver (attached)

I appreciate your help

Sawsan From: Mark Adams <[email protected] <mailto:[email protected]>> Sent: Wednesday, December 20, 2023 6:44 AM To: Shatanawi, Sawsan Muhammad <[email protected] <mailto:[email protected]>> Cc: [email protected] <mailto:[email protected]> <[email protected] <mailto:[email protected]>> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

[EXTERNAL EMAIL] Did you set preallocation values when you created the matrix? Don't do that.

On Wed, Dec 20, 2023 at 9:36 AM Shatanawi, Sawsan Muhammad <[email protected] <mailto:[email protected]>> wrote: Hello,

I am trying to create a sparse matrix( which is as I believe a zero matrix) then adding some nonzero elements to it over a loop, then assembling it

Get Outlook for iOS <https://urldefense.com/v3/__https://aka.ms/o0ukef__;!!JmPEgBY0HMszNaDT!uUJ_jeYf45gcXDGR_PeMjhU7hbd_fKcXJPn0pM9eb-YQihKNYuXMYM9x-hglsbXsCFIwNBWgHXdetHODupsOloE$> From: Mark Adams <[email protected] <mailto:[email protected]>> Sent: Wednesday, December 20, 2023 2:48 AM To: Shatanawi, Sawsan Muhammad <[email protected] <mailto:[email protected]>> Cc: [email protected] <mailto:[email protected]> <[email protected] <mailto:[email protected]>> Subject: Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

[EXTERNAL EMAIL] I am guessing that you are creating a matrix, adding to it, finalizing it ("assembly"), and then adding to it again, which is fine, but you are adding new non-zeros to the sparsity pattern. If this is what you want then you can tell the matrix to let you do that. Otherwise you have a bug.

Mark

On Tue, Dec 19, 2023 at 9:50 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected] <mailto:[email protected]>> wrote: Hello everyone,

I hope this email finds you well.

My Name is Sawsan Shatanawi, and I am currently working on developing a Fortran code for simulating groundwater flow in a 3D system. The code involves solving a nonlinear system, and I have created the matrix to be solved using the PCG solver and Picard iteration. However, when I tried to assign it as a PETSc matrix I started getting a lot of error messages.

I am kindly asking if someone can help me, I would be happy to share my code with him/her.

Please find the attached file contains a list of errors I have gotten

Thank you in advance for your time and assistance. Best regards,

Sawsan

<Matrix_RHS.F90><out.txt><solver.F90>

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.com/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!JmPEgBY0HMszNaDT!uskvAyF0pMMWDbMIexr9g4qN46V7Rea17GQdNIVG2vH_HMaX7mXgie4ZYgusmPpss_DS7H1_8vn8arGQNSkC$>

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.com/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!JmPEgBY0HMszNaDT!uQ_CqaGbVrhAwdolHoSQABYOvUa_lcGbqSBo0R5vr6xf8cZbIdMji6gOsfHzJJg2Rl4pB9RLvcxAdNkAkI3m$>

On Fri, May 10, 2024 at 6:30 PM Shatanawi, Sawsan Muhammad via petsc-users < [email protected]> wrote:

Good afternoon, I have tried SNESComputeJacobianDefaultColor(), but the arguments needed are confusing me. Would you please have a look at my code and the error messages I am getting? I didn't understand what the nonzero values of the sparse ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization.

ZjQcmQRYFpfptBannerEnd Good afternoon,

I have tried SNESComputeJacobianDefaultColor(), but the arguments needed are confusing me.

Would you please have a look at my code and the error messages I am getting? I didn't understand what the nonzero values of the sparse Jacobian would be.

1) You are not checking any of the return values from PETSc calls. Look at the PETSc Fortran examples. You should wrap all PETSc calls in PetscCall() or PetscCallA(). 2) You are not intended to directly call SNESComputeJacobianDefaultColor(). PETSc will do this automatically if you do not set of Jacobian function. 3) As Barry points out, coloring will not work unless we understand the nonzero structure of your Jacobian. This can happen by either: a) Using a DM: This is the easiest. Find the type that matches your grid, or b) Preallocating your Jacobian: Here you give the nonzero structure of your Jacobian, but not the values. Currently you do not do this. Instead you just give the size, not the nonzero structure (columns for each row). Thanks, Matt

Thank you for your patience and help Bests, Sawsan

------------------------------ *From:* Barry Smith <[email protected]> *Sent:* Thursday, May 9, 2024 12:05 PM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* [email protected] <[email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]*

On May 9, 2024, at 2:52 PM, Shatanawi, Sawsan Muhammad via petsc-users < [email protected]> wrote:

This Message Is From an External Sender This message came from outside your organization. Hello everyone,

Thank you for your responses and feedback,

I checked PFLOTRAN and found that it is a model to simulate groundwater flow, contaminant transport, and other subsurface processes. my goal is not to simulate the groundwater flow, my goal is to develop a code from scratch to simulate the groundwater flow with specific conditions, and then integrate this code with land surface models. Later, the simulation of this code will be on a large scale.

I want PETSc to calculate the Jacobian because the system is large and has complex nonlinear behavior, and I don’t risk calculating the derivative by myself. My A-Matrix has parts of source terms that depend on the flow fields, and independent parts will be in the RHS vector.

With coloring SNESComputeJacobianDefaultColor() PETSc can compute Jacobian's pretty efficiently. You do not to provide the residual function and you need to provide the nonzero pattern of the sparse Jacobian; that is what residual components f_i are coupled to what input variables in the array x_i. This information comes from your PDE and discretization and appears implicitly in your residual function.

Barry

I hope I have answered your questions, and I apologize that I wasn’t clear from the beginning, I was trying to keep my descriptions brief.

Bests, Sawsan ------------------------------ *From:* Matthew Knepley <[email protected]> *Sent:* Tuesday, May 7, 2024 5:17 PM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* [email protected] <[email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]* On Tue, May 7, 2024 at 2:23 PM Shatanawi, Sawsan Muhammad via petsc-users < [email protected]> wrote:

This Message Is From an External Sender This message came from outside your organization.

Hello everyone,

I hope this email finds you well.

My Name is Sawsan Shatanawi, and I was developing a Fortran code for simulating groundwater flow in a 3D system with nonlinear behavior. I solved the nonlinear system using the PCG solver and Picard iteration, but I did not get good results although I checked my matrix and RHS and everything, I decided to change my solver to Newton Rapson method. I checked PETSc documents but I have a few questions: 1) My groundwater system is time-dependent, so should I use TS only instead of SNES?

You could use TS, but it is not necessary. You could use SNES and your own timestepping. THe advantage of TS is that you can try many different timesteppers without recoding (just like you can try many different linear and nonlinear solvers).

2) My system has its deltaT, would using deltaT as dt affect my solver, or is it better to use TS-PETSc dt? Also, would using PETSc dt affect the simulation of the groundwater system

It sounds like your dt comes from your timestepper. If you use TS, you would use the dt from that.

3) I want my Jacobian matrix to be calculated by PETSc automatically

PETSc can calculate a full Jacobian for smaller problems, or a finite-difference Jacobian for any problem (but this impacts the solver). It should be straightfoward to code up the analytic Jacobian. Is there a reason it would be a problem?

4) Do I need to define and calculate the residual vector?

Yes.

My A-Matrix contains coefficients and external sources and my RHS vector includes the boundary conditions

It is strange that your matrix would contain source terms. Do they depend on the flow fields?

Barry is right, you should consider PFlotran, and at least know why it would not work for your problem if you don't use it.

Thanks,

Matt

Please find the attached file contains a draft of my code

Thank you in advance for your time and help.

Best regards,

Sawsan

------------------------------ *From:* Shatanawi, Sawsan Muhammad <[email protected]> *Sent:* Tuesday, January 16, 2024 10:43 AM *To:* Junchao Zhang <[email protected]> *Cc:* Barry Smith <[email protected]>; Matthew Knepley <[email protected]>; Mark Adams <[email protected]>; [email protected] < [email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

Hello all,

Thank you for your valuable help. I will do your recommendations and hope it will run without any issues.

Bests, Sawsan

------------------------------ *From:* Junchao Zhang <[email protected]> *Sent:* Friday, January 12, 2024 8:46 AM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* Barry Smith <[email protected]>; Matthew Knepley <[email protected]>; Mark Adams <[email protected]>; [email protected] < [email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]* Hi, Sawsan, First in test_main.F90, you need to call VecGetArrayF90(temp_solution, H_vector, ierr) and VecRestoreArrayF90 (temp_solution, H_vector, ierr) as Barry mentioned. Secondly, in the loop of test_main.F90, it calls GW_solver(). Within it, it calls PetscInitialize()/PetscFinalize(). But without MPI being initialized, PetscInitialize()/PetscFinalize()* can only be called once.* do timestep =2 , NTSP call GW_boundary_conditions(timestep-1) !print *,HNEW(1,1,1) call GW_elevation() ! print *, GWTOP(2,2,2) call GW_conductance() ! print *, CC(2,2,2) call GW_recharge() ! print *, B_Rech(5,4) call GW_pumping(timestep-1) ! print *, B_pump(2,2,2) call GW_SW(timestep-1) print *,B_RIVER (2,2,2) call GW_solver(timestep-1,N) call GW_deallocate_loop() end do

A solution is to delete PetscInitialize()/PetscFinalize() in GW_solver_try.F90 and move it to test_main.F90, outside the do loop.

diff --git a/test_main.F90 b/test_main.F90 index b5997c55..107bd3ee 100644 --- a/test_main.F90 +++ b/test_main.F90 @@ -1,5 +1,6 @@ program test_GW

+#include <petsc/finclude/petsc.h> use petsc use GW_constants use GW_param_by_user @@ -8,6 +9,9 @@ program test_GW implicit none integer :: N integer :: timestep + PetscErrorCode ierr + + call PetscInitialize(ierr) call GW_domain(N) !print *, "N=",N !print *, DELTAT @@ -37,4 +41,5 @@ program test_GW end do print *, HNEW(NCOL,3,2) call GW_deallocate () + call PetscFinalize(ierr) end program test_GW

With that, the MPI error will be fixed. The code could run to gw_deallocate () before abort. There are other memory errors. You can install/use valgrind to fix them. Run it with valgrind ./GW.exe and look through the output

Thanks. --Junchao Zhang

On Thu, Jan 11, 2024 at 10:49 PM Shatanawi, Sawsan Muhammad < [email protected]> wrote:

Hello,

Thank you all for your help.

I have changed VecGetArray to VecGetArrayF90, and the location of destory call. but I want to make sure that VecGet ArrayF90 is to make a new array( vector) that I can use in the rest of my Fortran code?

when I run it and debugged it, I got

5.2000000E-03 50.00000 10.00000 0.0000000E+00 PETSC: Attaching gdb to /weka/data/lab/richey/sawsan/GW_CODE/code2024/SS_GWM/./GW.exe of pid 33065 on display :0.0 on machine sn16 Unable to start debugger in xterm: No such file or directory 0.0000000E+00 Attempting to use an MPI routine after finalizing MPICH srun: error: sn16: task 0: Exited with exit code 1 [sawsan.shatanawi@login-p2n02 SS_GWM]$ gdb ./GW/exe GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later < https://urldefense.us/v3/__http://gnu.org/licenses/gpl.html__;!!G_uCfscf7eWS... <https://urldefense.com/v3/__http://gnu.org/licenses/gpl.html__;!!JmPEgBY0HMszNaDT!o4qS1zaFLg2L8PlawWJVyYsJpnwYHuL6SIIZsmzTO98RCzbP26HwTi_0-ipCS_D2SBG0X4gtEnM13-nbzFEFKvtraKaM$>

...

This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <https://urldefense.us/v3/__http://www.gnu.org/software/gdb/bugs/__;!!G_uCfsc... <https://urldefense.com/v3/__http://www.gnu.org/software/gdb/bugs/__;!!JmPEgBY0HMszNaDT!o4qS1zaFLg2L8PlawWJVyYsJpnwYHuL6SIIZsmzTO98RCzbP26HwTi_0-ipCS_D2SBG0X4gtEnM13-nbzFEFKjO8jNfs$>

...

... ./GW/exe: No such file or directory. (gdb) run Starting program: No executable file specified. Use the "file" or "exec-file" command. (gdb) bt No stack. (gdb)

If the highlighted line is the error, I don't know why when I write gdb , it does not show me the location of error The code : sshatanawi/SS_GWM (github.com) <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!o4qS1zaFLg2L8PlawWJVyYsJpnwYHuL6SIIZsmzTO98RCzbP26HwTi_0-ipCS_D2SBG0X4gtEnM13-nbzFEFKp8i33ur$>

I really appreciate your helps

Sawsan ------------------------------ *From:* Barry Smith <[email protected]> *Sent:* Wednesday, January 10, 2024 5:35 PM *To:* Junchao Zhang <[email protected]> *Cc:* Shatanawi, Sawsan Muhammad <[email protected]>; Mark Adams < [email protected]>; [email protected] <[email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]*

On Jan 10, 2024, at 6:49 PM, Junchao Zhang <[email protected]> wrote:

Hi, Sawsan, I could build your code and I also could gdb it.

$ gdb ./GW.exe ... $ Thread 1 "GW.exe" received signal SIGSEGV, Segmentation fault. 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 257 *ierr = VecGetArray(*x, &lx); (gdb) bt #0 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 #1 0x000000000040b6e3 in gw_solver (t_s=1.40129846e-45, n=300) at GW_solver_try.F90:169 #2 0x000000000040c6a8 in test_gw () at test_main.F90:35

ierr=0x0 caused the segfault. See https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGetA... <https://urldefense.com/v3/__https://petsc.org/release/manualpages/Vec/VecGetArray/*vecgetarray__;Iw!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ9Km12jA$>, you should use VecGetArrayF90 instead.

BTW, Barry, the code https://urldefense.us/v3/__https://github.com/sshatanawi/SS_GWM/blob/main/GW... <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM/blob/main/GW_solver_try.F90*L169__;Iw!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZh2eAi4o$> has "call VecGetArray(temp_solution, H_vector, ierr)". I don't find petsc Fortran examples doing VecGetArray. Do we still support it?

This is not the correct calling sequence for VecGetArray() from Fortran.

Regardless, definitely should not be writing any new code that uses VecGetArray() from Fortran. Should use VecGetArrayF90().

--Junchao Zhang

On Wed, Jan 10, 2024 at 2:38 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected]> wrote:

Hello all,

I hope you are doing well.

Generally, I use gdb <the name of my exe.file> to debug the code. I got the attached error message.

I have tried to add the flag -start_in_debugger in the make file, but it didn't work, so it seems I was doing it in the wrong way

This is the link for the whole code: sshatanawi/SS_GWM (github.com) <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ1Veab3M$>

<https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ1Veab3M$> GitHub - sshatanawi/SS_GWM <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ1Veab3M$> Contribute to sshatanawi/SS_GWM development by creating an account on GitHub. github.com <https://urldefense.com/v3/__http://github.com/__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ8rcrPiA$> **

You can read the description of the code in " Model Desprciption.pdf" the compiling file is makefile_f90 where you can find the linked code files

I really appreciate your help

Bests, Sawsan ------------------------------ *From:* Mark Adams <[email protected]> *Sent:* Friday, January 5, 2024 4:53 AM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* Matthew Knepley <[email protected]>; [email protected] < [email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]* This is a segv. As Matt said, you need to use a debugger for this or add print statements to narrow down the place where this happens.

You will need to learn how to use debuggers to do your project so you might as well start now.

If you have a machine with a GUI debugger that is easier but command line debuggers are good to learn anyway.

I tend to run debuggers directly (eg, lldb ./a.out -- program-args ...) and use a GUI debugger (eg, Totalview or DDT) if available.

Mark

On Wed, Dec 20, 2023 at 10:02 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected]> wrote:

Hello Matthew,

Thank you for your help. I am sorry that I keep coming back with my error messages, but I reached a point that I don't know how to fix them, and I don't understand them easily. The list of errors is getting shorter, now I am getting the attached error messages

Thank you again,

Sawsan ------------------------------ *From:* Matthew Knepley <[email protected]> *Sent:* Wednesday, December 20, 2023 6:54 PM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* Barry Smith <[email protected]>; [email protected] < [email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]* On Wed, Dec 20, 2023 at 9:49 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected]> wrote:

Hello Barry,

Thank you a lot for your help, Now I am getting the attached error message.

Do not destroy the PC from KSPGetPC()

THanks,

Matt

Bests, Sawsan ------------------------------ *From:* Barry Smith <[email protected]> *Sent:* Wednesday, December 20, 2023 6:32 PM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* Mark Adams <[email protected]>; [email protected] < [email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]*

Instead of

call PCCreate(PETSC_COMM_WORLD, pc, ierr) call PCSetType(pc, PCILU,ierr) ! Choose a preconditioner type (ILU) call KSPSetPC(ksp, pc,ierr) ! Associate the preconditioner with the KSP solver

do

call KSPGetPC(ksp,pc,ierr) call PCSetType(pc, PCILU,ierr)

Do not call KSPSetUp(). It will be taken care of automatically during the solve

On Dec 20, 2023, at 8:52 PM, Shatanawi, Sawsan Muhammad via petsc-users < [email protected]> wrote:

Hello, I don't think that I set preallocation values when I created the matrix, would you please have look at my code. It is just the petsc related part from my code. I was able to fix some of the error messages. Now I have a new set of error messages related to the KSP solver (attached)

I appreciate your help

Sawsan ------------------------------ *From:* Mark Adams <[email protected]> *Sent:* Wednesday, December 20, 2023 6:44 AM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* [email protected] <[email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]* Did you set preallocation values when you created the matrix? Don't do that.

On Wed, Dec 20, 2023 at 9:36 AM Shatanawi, Sawsan Muhammad < [email protected]> wrote:

Hello,

I am trying to create a sparse matrix( which is as I believe a zero matrix) then adding some nonzero elements to it over a loop, then assembling it

Get Outlook for iOS <https://urldefense.com/v3/__https://aka.ms/o0ukef__;!!JmPEgBY0HMszNaDT!uUJ_jeYf45gcXDGR_PeMjhU7hbd_fKcXJPn0pM9eb-YQihKNYuXMYM9x-hglsbXsCFIwNBWgHXdetHODupsOloE$> ------------------------------ *From:* Mark Adams <[email protected]> *Sent:* Wednesday, December 20, 2023 2:48 AM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* [email protected] <[email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]* I am guessing that you are creating a matrix, adding to it, finalizing it ("assembly"), and then adding to it again, which is fine, but you are adding new non-zeros to the sparsity pattern. If this is what you want then you can tell the matrix to let you do that. Otherwise you have a bug.

Mark

On Tue, Dec 19, 2023 at 9:50 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected]> wrote:

Hello everyone,

I hope this email finds you well.

My Name is Sawsan Shatanawi, and I am currently working on developing a Fortran code for simulating groundwater flow in a 3D system. The code involves solving a nonlinear system, and I have created the matrix to be solved using the PCG solver and Picard iteration. However, when I tried to assign it as a PETSc matrix I started getting a lot of error messages.

I am kindly asking if someone can help me, I would be happy to share my code with him/her.

Please find the attached file contains a list of errors I have gotten

Thank you in advance for your time and assistance.

Best regards,

Sawsan

<Matrix_RHS.F90><out.txt><solver.F90>

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.com/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!JmPEgBY0HMszNaDT!uskvAyF0pMMWDbMIexr9g4qN46V7Rea17GQdNIVG2vH_HMaX7mXgie4ZYgusmPpss_DS7H1_8vn8arGQNSkC$>

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.com/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!JmPEgBY0HMszNaDT!uQ_CqaGbVrhAwdolHoSQABYOvUa_lcGbqSBo0R5vr6xf8cZbIdMji6gOsfHzJJg2Rl4pB9RLvcxAdNkAkI3m$>

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >

On Sat, May 18, 2024 at 7:29 PM Shatanawi, Sawsan Muhammad < [email protected]> wrote:

Hello everyone,

Thank you all for your feedback, it helped me a lot.

I read the PETSc document and examples related to the Jacobian and modified my code. Now I am getting errors related to the memory access. I tried to debug but couldn't find out how to fix it.

I think the right way to find the bug is to start with either a) Valgrind (https://urldefense.us/v3/__http://www.valgrind.org__;!!G_uCfscf7eWS!an0a5TBb... ) or b) Address Sanitizer, which is a feature of the clang compiler (-fsanitize=address) You can also run with -malloc_debug in PETSc, which can detect some errors. Thanks, Matt

sshatanawi/SawSim: Sawsan's Simulation (SawSim) is a groundwater dynamics scheme to be integrated into Noah MP land surface model (github.com) <https://urldefense.us/v3/__https://github.com/sshatanawi/SawSim__;!!G_uCfscf... > is the link to the code in the GitHub, I would appreciate it if you could have a look at it and guide me how to fix it. I believe the problem is in the memory of LHS and res_vector because they are new vectors I created.

Thank you in advance for your help, I really appreciate it

Bests, Sawsan ------------------------------ *From:* Matthew Knepley <[email protected]> *Sent:* Saturday, May 11, 2024 1:56 AM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* Barry Smith <[email protected]>; [email protected] < [email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]* On Fri, May 10, 2024 at 6:30 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected]> wrote:

Good afternoon, I have tried SNESComputeJacobianDefaultColor(), but the arguments needed are confusing me. Would you please have a look at my code and the error messages I am getting? I didn't understand what the nonzero values of the sparse ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization.

ZjQcmQRYFpfptBannerEnd Good afternoon,

I have tried SNESComputeJacobianDefaultColor(), but the arguments needed are confusing me.

Would you please have a look at my code and the error messages I am getting? I didn't understand what the nonzero values of the sparse Jacobian would be.

1) You are not checking any of the return values from PETSc calls. Look at the PETSc Fortran examples. You should wrap all PETSc calls in PetscCall() or PetscCallA().

2) You are not intended to directly call SNESComputeJacobianDefaultColor(). PETSc will do this automatically if you do not set of Jacobian function.

3) As Barry points out, coloring will not work unless we understand the nonzero structure of your Jacobian. This can happen by either:

a) Using a DM: This is the easiest. Find the type that matches your grid, or

b) Preallocating your Jacobian: Here you give the nonzero structure of your Jacobian, but not the values. Currently you do not do this. Instead you just give the size, not the nonzero structure (columns for each row).

Thanks,

Matt

Thank you for your patience and help Bests, Sawsan

------------------------------ *From:* Barry Smith <[email protected]> *Sent:* Thursday, May 9, 2024 12:05 PM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* [email protected] <[email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]*

On May 9, 2024, at 2:52 PM, Shatanawi, Sawsan Muhammad via petsc-users < [email protected]> wrote:

This Message Is From an External Sender This message came from outside your organization. Hello everyone,

Thank you for your responses and feedback,

I checked PFLOTRAN and found that it is a model to simulate groundwater flow, contaminant transport, and other subsurface processes. my goal is not to simulate the groundwater flow, my goal is to develop a code from scratch to simulate the groundwater flow with specific conditions, and then integrate this code with land surface models. Later, the simulation of this code will be on a large scale.

I want PETSc to calculate the Jacobian because the system is large and has complex nonlinear behavior, and I don’t risk calculating the derivative by myself. My A-Matrix has parts of source terms that depend on the flow fields, and independent parts will be in the RHS vector.

With coloring SNESComputeJacobianDefaultColor() PETSc can compute Jacobian's pretty efficiently. You do not to provide the residual function and you need to provide the nonzero pattern of the sparse Jacobian; that is what residual components f_i are coupled to what input variables in the array x_i. This information comes from your PDE and discretization and appears implicitly in your residual function.

Barry

I hope I have answered your questions, and I apologize that I wasn’t clear from the beginning, I was trying to keep my descriptions brief.

Bests, Sawsan ------------------------------ *From:* Matthew Knepley <[email protected]> *Sent:* Tuesday, May 7, 2024 5:17 PM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* [email protected] <[email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]* On Tue, May 7, 2024 at 2:23 PM Shatanawi, Sawsan Muhammad via petsc-users < [email protected]> wrote:

This Message Is From an External Sender This message came from outside your organization.

Hello everyone,

I hope this email finds you well.

My Name is Sawsan Shatanawi, and I was developing a Fortran code for simulating groundwater flow in a 3D system with nonlinear behavior. I solved the nonlinear system using the PCG solver and Picard iteration, but I did not get good results although I checked my matrix and RHS and everything, I decided to change my solver to Newton Rapson method. I checked PETSc documents but I have a few questions: 1) My groundwater system is time-dependent, so should I use TS only instead of SNES?

You could use TS, but it is not necessary. You could use SNES and your own timestepping. THe advantage of TS is that you can try many different timesteppers without recoding (just like you can try many different linear and nonlinear solvers).

2) My system has its deltaT, would using deltaT as dt affect my solver, or is it better to use TS-PETSc dt? Also, would using PETSc dt affect the simulation of the groundwater system

It sounds like your dt comes from your timestepper. If you use TS, you would use the dt from that.

3) I want my Jacobian matrix to be calculated by PETSc automatically

PETSc can calculate a full Jacobian for smaller problems, or a finite-difference Jacobian for any problem (but this impacts the solver). It should be straightfoward to code up the analytic Jacobian. Is there a reason it would be a problem?

4) Do I need to define and calculate the residual vector?

Yes.

My A-Matrix contains coefficients and external sources and my RHS vector includes the boundary conditions

It is strange that your matrix would contain source terms. Do they depend on the flow fields?

Barry is right, you should consider PFlotran, and at least know why it would not work for your problem if you don't use it.

Thanks,

Matt

Please find the attached file contains a draft of my code

Thank you in advance for your time and help.

Best regards,

Sawsan

------------------------------ *From:* Shatanawi, Sawsan Muhammad <[email protected]> *Sent:* Tuesday, January 16, 2024 10:43 AM *To:* Junchao Zhang <[email protected]> *Cc:* Barry Smith <[email protected]>; Matthew Knepley <[email protected]>; Mark Adams <[email protected]>; [email protected] < [email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

Hello all,

Thank you for your valuable help. I will do your recommendations and hope it will run without any issues.

Bests, Sawsan

------------------------------ *From:* Junchao Zhang <[email protected]> *Sent:* Friday, January 12, 2024 8:46 AM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* Barry Smith <[email protected]>; Matthew Knepley <[email protected]>; Mark Adams <[email protected]>; [email protected] < [email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]* Hi, Sawsan, First in test_main.F90, you need to call VecGetArrayF90(temp_solution, H_vector, ierr) and VecRestoreArrayF90 (temp_solution, H_vector, ierr) as Barry mentioned. Secondly, in the loop of test_main.F90, it calls GW_solver(). Within it, it calls PetscInitialize()/PetscFinalize(). But without MPI being initialized, PetscInitialize()/PetscFinalize()* can only be called once.* do timestep =2 , NTSP call GW_boundary_conditions(timestep-1) !print *,HNEW(1,1,1) call GW_elevation() ! print *, GWTOP(2,2,2) call GW_conductance() ! print *, CC(2,2,2) call GW_recharge() ! print *, B_Rech(5,4) call GW_pumping(timestep-1) ! print *, B_pump(2,2,2) call GW_SW(timestep-1) print *,B_RIVER (2,2,2) call GW_solver(timestep-1,N) call GW_deallocate_loop() end do

A solution is to delete PetscInitialize()/PetscFinalize() in GW_solver_try.F90 and move it to test_main.F90, outside the do loop.

diff --git a/test_main.F90 b/test_main.F90 index b5997c55..107bd3ee 100644 --- a/test_main.F90 +++ b/test_main.F90 @@ -1,5 +1,6 @@ program test_GW

+#include <petsc/finclude/petsc.h> use petsc use GW_constants use GW_param_by_user @@ -8,6 +9,9 @@ program test_GW implicit none integer :: N integer :: timestep + PetscErrorCode ierr + + call PetscInitialize(ierr) call GW_domain(N) !print *, "N=",N !print *, DELTAT @@ -37,4 +41,5 @@ program test_GW end do print *, HNEW(NCOL,3,2) call GW_deallocate () + call PetscFinalize(ierr) end program test_GW

With that, the MPI error will be fixed. The code could run to gw_deallocate () before abort. There are other memory errors. You can install/use valgrind to fix them. Run it with valgrind ./GW.exe and look through the output

Thanks. --Junchao Zhang

On Thu, Jan 11, 2024 at 10:49 PM Shatanawi, Sawsan Muhammad < [email protected]> wrote:

Hello,

Thank you all for your help.

I have changed VecGetArray to VecGetArrayF90, and the location of destory call. but I want to make sure that VecGet ArrayF90 is to make a new array( vector) that I can use in the rest of my Fortran code?

when I run it and debugged it, I got

5.2000000E-03 50.00000 10.00000 0.0000000E+00 PETSC: Attaching gdb to /weka/data/lab/richey/sawsan/GW_CODE/code2024/SS_GWM/./GW.exe of pid 33065 on display :0.0 on machine sn16 Unable to start debugger in xterm: No such file or directory 0.0000000E+00 Attempting to use an MPI routine after finalizing MPICH srun: error: sn16: task 0: Exited with exit code 1 [sawsan.shatanawi@login-p2n02 SS_GWM]$ gdb ./GW/exe GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later < https://urldefense.us/v3/__http://gnu.org/licenses/gpl.html__;!!G_uCfscf7eWS... <https://urldefense.com/v3/__http://gnu.org/licenses/gpl.html__;!!JmPEgBY0HMszNaDT!o4qS1zaFLg2L8PlawWJVyYsJpnwYHuL6SIIZsmzTO98RCzbP26HwTi_0-ipCS_D2SBG0X4gtEnM13-nbzFEFKvtraKaM$>

...

This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <https://urldefense.us/v3/__http://www.gnu.org/software/gdb/bugs/__;!!G_uCfsc... <https://urldefense.com/v3/__http://www.gnu.org/software/gdb/bugs/__;!!JmPEgBY0HMszNaDT!o4qS1zaFLg2L8PlawWJVyYsJpnwYHuL6SIIZsmzTO98RCzbP26HwTi_0-ipCS_D2SBG0X4gtEnM13-nbzFEFKjO8jNfs$>

...

... ./GW/exe: No such file or directory. (gdb) run Starting program: No executable file specified. Use the "file" or "exec-file" command. (gdb) bt No stack. (gdb)

If the highlighted line is the error, I don't know why when I write gdb , it does not show me the location of error The code : sshatanawi/SS_GWM (github.com) <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!o4qS1zaFLg2L8PlawWJVyYsJpnwYHuL6SIIZsmzTO98RCzbP26HwTi_0-ipCS_D2SBG0X4gtEnM13-nbzFEFKp8i33ur$>

I really appreciate your helps

Sawsan ------------------------------ *From:* Barry Smith <[email protected]> *Sent:* Wednesday, January 10, 2024 5:35 PM *To:* Junchao Zhang <[email protected]> *Cc:* Shatanawi, Sawsan Muhammad <[email protected]>; Mark Adams < [email protected]>; [email protected] <[email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]*

On Jan 10, 2024, at 6:49 PM, Junchao Zhang <[email protected]> wrote:

Hi, Sawsan, I could build your code and I also could gdb it.

$ gdb ./GW.exe ... $ Thread 1 "GW.exe" received signal SIGSEGV, Segmentation fault. 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 257 *ierr = VecGetArray(*x, &lx); (gdb) bt #0 0x00007ffff1e6d44f in vecgetarray_ (x=0x7fffffffa718, fa=0x0, ia=0x7fffffffa75c, ierr=0x0) at /scratch/jczhang/petsc/src/vec/vec/interface/ftn-custom/zvectorf.c:257 #1 0x000000000040b6e3 in gw_solver (t_s=1.40129846e-45, n=300) at GW_solver_try.F90:169 #2 0x000000000040c6a8 in test_gw () at test_main.F90:35

ierr=0x0 caused the segfault. See https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGetA... <https://urldefense.com/v3/__https://petsc.org/release/manualpages/Vec/VecGetArray/*vecgetarray__;Iw!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ9Km12jA$>, you should use VecGetArrayF90 instead.

BTW, Barry, the code https://urldefense.us/v3/__https://github.com/sshatanawi/SS_GWM/blob/main/GW... <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM/blob/main/GW_solver_try.F90*L169__;Iw!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZh2eAi4o$> has "call VecGetArray(temp_solution, H_vector, ierr)". I don't find petsc Fortran examples doing VecGetArray. Do we still support it?

This is not the correct calling sequence for VecGetArray() from Fortran.

Regardless, definitely should not be writing any new code that uses VecGetArray() from Fortran. Should use VecGetArrayF90().

--Junchao Zhang

On Wed, Jan 10, 2024 at 2:38 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected]> wrote:

Hello all,

I hope you are doing well.

Generally, I use gdb <the name of my exe.file> to debug the code. I got the attached error message.

I have tried to add the flag -start_in_debugger in the make file, but it didn't work, so it seems I was doing it in the wrong way

This is the link for the whole code: sshatanawi/SS_GWM (github.com) <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ1Veab3M$>

<https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ1Veab3M$> GitHub - sshatanawi/SS_GWM <https://urldefense.com/v3/__https://github.com/sshatanawi/SS_GWM__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ1Veab3M$> Contribute to sshatanawi/SS_GWM development by creating an account on GitHub. github.com <https://urldefense.com/v3/__http://github.com/__;!!JmPEgBY0HMszNaDT!tqBApprMfYxwNz4Zvnk8coNE5AeWjA9wSdAM7QJcIIVP1z0VDsVIalo4Sew2b0fW3bZtTAbPh-h0MUsZ8rcrPiA$> **

You can read the description of the code in " Model Desprciption.pdf" the compiling file is makefile_f90 where you can find the linked code files

I really appreciate your help

Bests, Sawsan ------------------------------ *From:* Mark Adams <[email protected]> *Sent:* Friday, January 5, 2024 4:53 AM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* Matthew Knepley <[email protected]>; [email protected] < [email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]* This is a segv. As Matt said, you need to use a debugger for this or add print statements to narrow down the place where this happens.

You will need to learn how to use debuggers to do your project so you might as well start now.

If you have a machine with a GUI debugger that is easier but command line debuggers are good to learn anyway.

I tend to run debuggers directly (eg, lldb ./a.out -- program-args ...) and use a GUI debugger (eg, Totalview or DDT) if available.

Mark

On Wed, Dec 20, 2023 at 10:02 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected]> wrote:

Hello Matthew,

Thank you for your help. I am sorry that I keep coming back with my error messages, but I reached a point that I don't know how to fix them, and I don't understand them easily. The list of errors is getting shorter, now I am getting the attached error messages

Thank you again,

Sawsan ------------------------------ *From:* Matthew Knepley <[email protected]> *Sent:* Wednesday, December 20, 2023 6:54 PM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* Barry Smith <[email protected]>; [email protected] < [email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]* On Wed, Dec 20, 2023 at 9:49 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected]> wrote:

Hello Barry,

Thank you a lot for your help, Now I am getting the attached error message.

Do not destroy the PC from KSPGetPC()

THanks,

Matt

Bests, Sawsan ------------------------------ *From:* Barry Smith <[email protected]> *Sent:* Wednesday, December 20, 2023 6:32 PM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* Mark Adams <[email protected]>; [email protected] < [email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]*

Instead of

call PCCreate(PETSC_COMM_WORLD, pc, ierr) call PCSetType(pc, PCILU,ierr) ! Choose a preconditioner type (ILU) call KSPSetPC(ksp, pc,ierr) ! Associate the preconditioner with the KSP solver

do

call KSPGetPC(ksp,pc,ierr) call PCSetType(pc, PCILU,ierr)

Do not call KSPSetUp(). It will be taken care of automatically during the solve

On Dec 20, 2023, at 8:52 PM, Shatanawi, Sawsan Muhammad via petsc-users < [email protected]> wrote:

Hello, I don't think that I set preallocation values when I created the matrix, would you please have look at my code. It is just the petsc related part from my code. I was able to fix some of the error messages. Now I have a new set of error messages related to the KSP solver (attached)

I appreciate your help

Sawsan ------------------------------ *From:* Mark Adams <[email protected]> *Sent:* Wednesday, December 20, 2023 6:44 AM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* [email protected] <[email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]* Did you set preallocation values when you created the matrix? Don't do that.

On Wed, Dec 20, 2023 at 9:36 AM Shatanawi, Sawsan Muhammad < [email protected]> wrote:

Hello,

I am trying to create a sparse matrix( which is as I believe a zero matrix) then adding some nonzero elements to it over a loop, then assembling it

Get Outlook for iOS <https://urldefense.com/v3/__https://aka.ms/o0ukef__;!!JmPEgBY0HMszNaDT!uUJ_jeYf45gcXDGR_PeMjhU7hbd_fKcXJPn0pM9eb-YQihKNYuXMYM9x-hglsbXsCFIwNBWgHXdetHODupsOloE$> ------------------------------ *From:* Mark Adams <[email protected]> *Sent:* Wednesday, December 20, 2023 2:48 AM *To:* Shatanawi, Sawsan Muhammad <[email protected]> *Cc:* [email protected] <[email protected]> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

*[EXTERNAL EMAIL]* I am guessing that you are creating a matrix, adding to it, finalizing it ("assembly"), and then adding to it again, which is fine, but you are adding new non-zeros to the sparsity pattern. If this is what you want then you can tell the matrix to let you do that. Otherwise you have a bug.

Mark

On Tue, Dec 19, 2023 at 9:50 PM Shatanawi, Sawsan Muhammad via petsc-users <[email protected]> wrote:

Hello everyone,

I hope this email finds you well.

My Name is Sawsan Shatanawi, and I am currently working on developing a Fortran code for simulating groundwater flow in a 3D system. The code involves solving a nonlinear system, and I have created the matrix to be solved using the PCG solver and Picard iteration. However, when I tried to assign it as a PETSc matrix I started getting a lot of error messages.

I am kindly asking if someone can help me, I would be happy to share my code with him/her.

Please find the attached file contains a list of errors I have gotten

Thank you in advance for your time and assistance.

Best regards,

Sawsan

<Matrix_RHS.F90><out.txt><solver.F90>

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.com/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!JmPEgBY0HMszNaDT!uskvAyF0pMMWDbMIexr9g4qN46V7Rea17GQdNIVG2vH_HMaX7mXgie4ZYgusmPpss_DS7H1_8vn8arGQNSkC$>

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.com/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!JmPEgBY0HMszNaDT!uQ_CqaGbVrhAwdolHoSQABYOvUa_lcGbqSBo0R5vr6xf8cZbIdMji6gOsfHzJJg2Rl4pB9RLvcxAdNkAkI3m$>

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.com/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!JmPEgBY0HMszNaDT!smICTGbSXrPHqMB6bHUNt1_rI9gtQR9Hw4rx7P_iIWaRDqYgwdFTHh8UL1N_9qHLFXbfPt2JZyat1djSVOuC$>

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >