CG+GAMG convergence issues in GHEP Krylov-Schur for some MPI runs
Dear all, I experience strange convergence problems in SLEPc for GHEP with Krylov-Schur and CG + GAMG. The issue appears to be contingent on the number of MPI cores used. Say for 8 cores there is no issue and for 4 cores there is an issue. When I substitute GAMG with Jacobi for the problematic number of cores -- all works. To be more specific, I solve Ax=\lambda Bx for a sequence of A’s where A is a function of eigenvectors. On each iteration step currently the eigensolver EPS is initialised from scratch. And thus should the underlying ST, KSP, PC objects: -st_ksp_type cg -st_pc_type gamg -st_ksp_rtol 1e-12. For these particular matrices the issue appears on the 4th iteration, even though the matrix to be inverted (mass/overlap matrix) is the same, does not change!!! From my debuging info the A matrix has the same norm for CG + GAMG vs CG + Jacobi cases: DEAL:: frobenius_norm = 365.7 DEAL:: linfty_norm = 19.87 DEAL:: l1_norm = 19.87 Just to be sure that there are no bugs on my side which would result in different mass matrices i check that it has the same norm for CG + GAMG vs CG + Jacobi BEFORE i start iteration: DEAL:: frobenius_norm = 166.4 DEAL:: linfty_norm = 8.342 DEAL:: l1_norm = 8.342 All the dependent scalar quantities I calculate on each iteration are identical for the two cases, which makes me believe that the solution path is the same up to the certain tolerance. The only output which is slightly different are the number iterations for convergence in EPS (e.g. 113 vs 108) and the resulting maxing EPSComputeResidualNorm : 4.1524e-07 vs 2.9639e-08. Any ideas what could be an issue, especially given the fact that it does work for some numbers of cores and does not for other? Perhaps some default settings in GAMG preconditioner? Although that does not explain why it works for the first 3 iterations and does not on 4th as the mass matrix is unchanged... Lastly, i suppose ideally i should keep the eigensolver context between the iterations and just update the matrices by EPSSetOperators. Is it correct to assume that since B matrix does not change between iterations and I use the default shift transformation with zero shift (operator is B^{-1)A ), the GAMG preconditioner will not be re-initialised and thus I should save some time? p.s. the relevant error message is below. I have the same issues on CentOS cluster, so it is not related to OS-X. Kind regards, Denis === [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] KSPSolve line 510 /private/tmp/petsc20151102-50378-1t7b3in/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [0] STMatSolve line 148 /private/tmp/slepc20151102-3081-1xln4h0/slepc-3.6.1/src/sys/classes/st/interface/stsles.c [0]PETSC ERROR: [0] STApply_Shift line 33 /private/tmp/slepc20151102-3081-1xln4h0/slepc-3.6.1/src/sys/classes/st/impls/shift/shift.c [0]PETSC ERROR: [0] STApply line 50 /private/tmp/slepc20151102-3081-1xln4h0/slepc-3.6.1/src/sys/classes/st/interface/stsolve.c [0]PETSC ERROR: [0] EPSGetStartVector line 726 /private/tmp/slepc20151102-3081-1xln4h0/slepc-3.6.1/src/eps/interface/epssolve.c [0]PETSC ERROR: [0] EPSSolve_KrylovSchur_Symm line 41 /private/tmp/slepc20151102-3081-1xln4h0/slepc-3.6.1/src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: [0] EPSSolve line 83 /private/tmp/slepc20151102-3081-1xln4h0/slepc-3.6.1/src/eps/interface/epssolve.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015 [0]PETSC ERROR: /Users/davydden/Desktop/work/C++/deal.ii-dft/build_debug~/dft on a real named MBP-Denis.fritz.box by davydden Tue Nov 3 07:02:47 2015 [0]PETSC ERROR: Configure options CC=/usr/local/bin/mpicc CXX=/usr/local/bin/mpicxx F77=/usr/local/bin/mpif77 FC=/usr/local/bin/mpif90 --with-shared-libraries=1 --with-pthread=0 --with-openmp=0 --with-debugging=1 --with-ssl=0 --with-superlu_dist-include=/usr/local/opt/superlu_dist/include/superlu_dist --with-superlu_dist-lib="-L/usr/local/opt/superlu_dist/lib -lsuperlu_dist" --with-superlu-include=/usr/local/Cellar/superlu43/4.3/include/superlu --with-superlu-lib="-L/usr/local/Cellar/superlu43/4.3/lib -lsuperlu" --with-fftw-dir=/usr/local/opt/fftw --with-netcdf-dir=/usr/local/opt/netcdf --with-suitesparse-dir=/usr/local/opt/suite-sparse --with-hdf5-dir=/usr/local/opt/hdf5 --with-metis-dir=/usr/local/opt/metis --with-parmetis-dir=/usr/local/opt/parmetis --with-scalapack-dir=/usr/local/opt/scalapack --with-mumps-dir=/usr/local/opt/mumps --with-x=0 --prefix=/usr/local/Cellar/petsc/3.6.2/real --with-scalar-type=real --with-hypre-dir=/usr/local/opt/hypre --with-sundials-dir=/usr/local/opt/sundials --with-hwloc-dir=/usr/local/opt/hwloc [0]PETSC ERROR: #8 User provided function() line 0 in unknown file -------------------------------------------------------------------------- mpirun noticed that process rank 3 with PID 96754 on node MBP-Denis exited on signal 6 (Abort trap: 6). --------------------------------------------------------------------------
I am answering the SLEPc-related questions: - Having different number of iterations when changing the number of processes is normal. - Yes, if you do not destroy the EPS solver, then the preconditioner would be reused. Regarding the segmentation fault, I have no clue. Not sure if this is related to GAMG or not. Maybe running under valgrind could provide more information. Jose On 3/11/2015, Denis Davydov <[email protected]> wrote:
Dear all,
I experience strange convergence problems in SLEPc for GHEP with Krylov-Schur and CG + GAMG. The issue appears to be contingent on the number of MPI cores used. Say for 8 cores there is no issue and for 4 cores there is an issue. When I substitute GAMG with Jacobi for the problematic number of cores -- all works.
To be more specific, I solve Ax=\lambda Bx for a sequence of A’s where A is a function of eigenvectors. On each iteration step currently the eigensolver EPS is initialised from scratch. And thus should the underlying ST, KSP, PC objects: -st_ksp_type cg -st_pc_type gamg -st_ksp_rtol 1e-12. For these particular matrices the issue appears on the 4th iteration, even though the matrix to be inverted (mass/overlap matrix) is the same, does not change!!! From my debuging info the A matrix has the same norm for CG + GAMG vs CG + Jacobi cases: DEAL:: frobenius_norm = 365.7 DEAL:: linfty_norm = 19.87 DEAL:: l1_norm = 19.87 Just to be sure that there are no bugs on my side which would result in different mass matrices i check that it has the same norm for CG + GAMG vs CG + Jacobi BEFORE i start iteration: DEAL:: frobenius_norm = 166.4 DEAL:: linfty_norm = 8.342 DEAL:: l1_norm = 8.342 All the dependent scalar quantities I calculate on each iteration are identical for the two cases, which makes me believe that the solution path is the same up to the certain tolerance. The only output which is slightly different are the number iterations for convergence in EPS (e.g. 113 vs 108) and the resulting maxing EPSComputeResidualNorm : 4.1524e-07 vs 2.9639e-08.
Any ideas what could be an issue, especially given the fact that it does work for some numbers of cores and does not for other? Perhaps some default settings in GAMG preconditioner? Although that does not explain why it works for the first 3 iterations and does not on 4th as the mass matrix is unchanged...
Lastly, i suppose ideally i should keep the eigensolver context between the iterations and just update the matrices by EPSSetOperators. Is it correct to assume that since B matrix does not change between iterations and I use the default shift transformation with zero shift (operator is B^{-1)A ), the GAMG preconditioner will not be re-initialised and thus I should save some time?
p.s. the relevant error message is below. I have the same issues on CentOS cluster, so it is not related to OS-X.
Kind regards, Denis
=== [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] KSPSolve line 510 /private/tmp/petsc20151102-50378-1t7b3in/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [0] STMatSolve line 148 /private/tmp/slepc20151102-3081-1xln4h0/slepc-3.6.1/src/sys/classes/st/interface/stsles.c [0]PETSC ERROR: [0] STApply_Shift line 33 /private/tmp/slepc20151102-3081-1xln4h0/slepc-3.6.1/src/sys/classes/st/impls/shift/shift.c [0]PETSC ERROR: [0] STApply line 50 /private/tmp/slepc20151102-3081-1xln4h0/slepc-3.6.1/src/sys/classes/st/interface/stsolve.c [0]PETSC ERROR: [0] EPSGetStartVector line 726 /private/tmp/slepc20151102-3081-1xln4h0/slepc-3.6.1/src/eps/interface/epssolve.c [0]PETSC ERROR: [0] EPSSolve_KrylovSchur_Symm line 41 /private/tmp/slepc20151102-3081-1xln4h0/slepc-3.6.1/src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: [0] EPSSolve line 83 /private/tmp/slepc20151102-3081-1xln4h0/slepc-3.6.1/src/eps/interface/epssolve.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015 [0]PETSC ERROR: /Users/davydden/Desktop/work/C++/deal.ii-dft/build_debug~/dft on a real named MBP-Denis.fritz.box by davydden Tue Nov 3 07:02:47 2015 [0]PETSC ERROR: Configure options CC=/usr/local/bin/mpicc CXX=/usr/local/bin/mpicxx F77=/usr/local/bin/mpif77 FC=/usr/local/bin/mpif90 --with-shared-libraries=1 --with-pthread=0 --with-openmp=0 --with-debugging=1 --with-ssl=0 --with-superlu_dist-include=/usr/local/opt/superlu_dist/include/superlu_dist --with-superlu_dist-lib="-L/usr/local/opt/superlu_dist/lib -lsuperlu_dist" --with-superlu-include=/usr/local/Cellar/superlu43/4.3/include/superlu --with-superlu-lib="-L/usr/local/Cellar/superlu43/4.3/lib -lsuperlu" --with-fftw-dir=/usr/local/opt/fftw --with-netcdf-dir=/usr/local/opt/netcdf --with-suitesparse-dir=/usr/local/opt/suite-sparse --with-hdf5-dir=/usr/local/opt/hdf5 --with-metis-dir=/usr/local/opt/metis --with-parmetis-dir=/usr/local/opt/parmetis --with-scalapack-dir=/usr/local/opt/scalapack --with-mumps-dir=/usr/local/opt/mumps --with-x=0 --prefix=/usr/local/Cellar/petsc/3.6.2/real --with-scalar-type=real --with-hypre-dir=/usr/local/opt/hypre --with-sundials-dir=/usr/local/opt/sundials --with-hwloc-dir=/usr/local/opt/hwloc [0]PETSC ERROR: #8 User provided function() line 0 in unknown file -------------------------------------------------------------------------- mpirun noticed that process rank 3 with PID 96754 on node MBP-Denis exited on signal 6 (Abort trap: 6). --------------------------------------------------------------------------
Hi Jose,
On 3 Nov 2015, at 12:20, Jose E. Roman <[email protected]> wrote:
I am answering the SLEPc-related questions: - Having different number of iterations when changing the number of processes is normal. the change in iterations i mentioned are for different preconditioners, but the same number of MPI processes.
- Yes, if you do not destroy the EPS solver, then the preconditioner would be reused.
Regarding the segmentation fault, I have no clue. Not sure if this is related to GAMG or not. Maybe running under valgrind could provide more information. will try that.
Denis.
BTW, I think that our advice for segv is use a debugger. DDT or Totalview, and gdb if need be, will get you right to the source code and will get 90% of bugs diagnosed. Valgrind is noisy and cumbersome to use but can diagnose 90% of the other 10%. On Tue, Nov 3, 2015 at 7:32 AM, Denis Davydov <[email protected]> wrote:
Hi Jose,
On 3 Nov 2015, at 12:20, Jose E. Roman <[email protected]> wrote:
I am answering the SLEPc-related questions: - Having different number of iterations when changing the number of processes is normal. the change in iterations i mentioned are for different preconditioners, but the same number of MPI processes.
- Yes, if you do not destroy the EPS solver, then the preconditioner would be reused.
Regarding the segmentation fault, I have no clue. Not sure if this is related to GAMG or not. Maybe running under valgrind could provide more information. will try that.
Denis.
I am more optimistic about valgrind than Mark. I first try valgrind and if that fails to be helpful then use the debugger. valgrind has the advantage that it finds the FIRST place that something is wrong, while in the debugger it is kind of late at the crash. Valgrind should not be noisy, if it is then the applications/libraries should be cleaned up so that they are valgrind clean and then valgrind is useful. Barry
On Nov 3, 2015, at 7:47 AM, Mark Adams <[email protected]> wrote:
BTW, I think that our advice for segv is use a debugger. DDT or Totalview, and gdb if need be, will get you right to the source code and will get 90% of bugs diagnosed. Valgrind is noisy and cumbersome to use but can diagnose 90% of the other 10%.
On Tue, Nov 3, 2015 at 7:32 AM, Denis Davydov <[email protected]> wrote: Hi Jose,
On 3 Nov 2015, at 12:20, Jose E. Roman <[email protected]> wrote:
I am answering the SLEPc-related questions: - Having different number of iterations when changing the number of processes is normal. the change in iterations i mentioned are for different preconditioners, but the same number of MPI processes.
- Yes, if you do not destroy the EPS solver, then the preconditioner would be reused.
Regarding the segmentation fault, I have no clue. Not sure if this is related to GAMG or not. Maybe running under valgrind could provide more information. will try that.
Denis.
I ran an 8 processor job on Edison of a small code for a short run (just a linear solve) and got 37 Mb of output! Here is a 'Petsc' grep. Perhaps we should build an ignore file for things that we believe is a false positive. On Tue, Nov 3, 2015 at 11:55 AM, Barry Smith <[email protected]> wrote:
I am more optimistic about valgrind than Mark. I first try valgrind and if that fails to be helpful then use the debugger. valgrind has the advantage that it finds the FIRST place that something is wrong, while in the debugger it is kind of late at the crash.
Valgrind should not be noisy, if it is then the applications/libraries should be cleaned up so that they are valgrind clean and then valgrind is useful.
Barry
On Nov 3, 2015, at 7:47 AM, Mark Adams <[email protected]> wrote:
BTW, I think that our advice for segv is use a debugger. DDT or Totalview, and gdb if need be, will get you right to the source code and will get 90% of bugs diagnosed. Valgrind is noisy and cumbersome to use but can diagnose 90% of the other 10%.
On Tue, Nov 3, 2015 at 7:32 AM, Denis Davydov <[email protected]> wrote: Hi Jose,
On 3 Nov 2015, at 12:20, Jose E. Roman <[email protected]> wrote:
I am answering the SLEPc-related questions: - Having different number of iterations when changing the number of processes is normal. the change in iterations i mentioned are for different preconditioners, but the same number of MPI processes.
- Yes, if you do not destroy the EPS solver, then the preconditioner would be reused.
Regarding the segmentation fault, I have no clue. Not sure if this is related to GAMG or not. Maybe running under valgrind could provide more information. will try that.
Denis.
valgrind on Edison does seem to give a lot of false positives. The line number are accurate (not always the case). "assert" triggers it, as does SETERRQ. On Tue, Nov 10, 2015 at 8:51 AM, Mark Adams <[email protected]> wrote:
I ran an 8 processor job on Edison of a small code for a short run (just a linear solve) and got 37 Mb of output!
Here is a 'Petsc' grep.
Perhaps we should build an ignore file for things that we believe is a false positive.
On Tue, Nov 3, 2015 at 11:55 AM, Barry Smith <[email protected]> wrote:
I am more optimistic about valgrind than Mark. I first try valgrind and if that fails to be helpful then use the debugger. valgrind has the advantage that it finds the FIRST place that something is wrong, while in the debugger it is kind of late at the crash.
Valgrind should not be noisy, if it is then the applications/libraries should be cleaned up so that they are valgrind clean and then valgrind is useful.
Barry
On Nov 3, 2015, at 7:47 AM, Mark Adams <[email protected]> wrote:
BTW, I think that our advice for segv is use a debugger. DDT or Totalview, and gdb if need be, will get you right to the source code and will get 90% of bugs diagnosed. Valgrind is noisy and cumbersome to use but can diagnose 90% of the other 10%.
On Tue, Nov 3, 2015 at 7:32 AM, Denis Davydov <[email protected]> wrote: Hi Jose,
On 3 Nov 2015, at 12:20, Jose E. Roman <[email protected]> wrote:
I am answering the SLEPc-related questions: - Having different number of iterations when changing the number of processes is normal. the change in iterations i mentioned are for different preconditioners, but the same number of MPI processes.
- Yes, if you do not destroy the EPS solver, then the preconditioner would be reused.
Regarding the segmentation fault, I have no clue. Not sure if this is related to GAMG or not. Maybe running under valgrind could provide more information. will try that.
Denis.
Please send me the full output. This is nuts and should be reported once we understand it better to NERSc as something to be fixed. When I pay $60 million in taxes to a computing center I expect something that works fine for free on my laptop to work also there. Barry
On Nov 10, 2015, at 7:51 AM, Mark Adams <[email protected]> wrote:
I ran an 8 processor job on Edison of a small code for a short run (just a linear solve) and got 37 Mb of output!
Here is a 'Petsc' grep.
Perhaps we should build an ignore file for things that we believe is a false positive.
On Tue, Nov 3, 2015 at 11:55 AM, Barry Smith <[email protected]> wrote:
I am more optimistic about valgrind than Mark. I first try valgrind and if that fails to be helpful then use the debugger. valgrind has the advantage that it finds the FIRST place that something is wrong, while in the debugger it is kind of late at the crash.
Valgrind should not be noisy, if it is then the applications/libraries should be cleaned up so that they are valgrind clean and then valgrind is useful.
Barry
On Nov 3, 2015, at 7:47 AM, Mark Adams <[email protected]> wrote:
BTW, I think that our advice for segv is use a debugger. DDT or Totalview, and gdb if need be, will get you right to the source code and will get 90% of bugs diagnosed. Valgrind is noisy and cumbersome to use but can diagnose 90% of the other 10%.
On Tue, Nov 3, 2015 at 7:32 AM, Denis Davydov <[email protected]> wrote: Hi Jose,
On 3 Nov 2015, at 12:20, Jose E. Roman <[email protected]> wrote:
I am answering the SLEPc-related questions: - Having different number of iterations when changing the number of processes is normal. the change in iterations i mentioned are for different preconditioners, but the same number of MPI processes.
- Yes, if you do not destroy the EPS solver, then the preconditioner would be reused.
Regarding the segmentation fault, I have no clue. Not sure if this is related to GAMG or not. Maybe running under valgrind could provide more information. will try that.
Denis.
<petsc_val.gz>
On Tue, Nov 10, 2015 at 11:15 AM, Barry Smith <[email protected]> wrote:
Please send me the full output. This is nuts and should be reported once we understand it better to NERSc as something to be fixed. When I pay $60 million in taxes to a computing center I expect something that works fine for free on my laptop to work also there.
Barry
On Nov 10, 2015, at 7:51 AM, Mark Adams <[email protected]> wrote:
I ran an 8 processor job on Edison of a small code for a short run (just a linear solve) and got 37 Mb of output!
Here is a 'Petsc' grep.
Perhaps we should build an ignore file for things that we believe is a false positive.
On Tue, Nov 3, 2015 at 11:55 AM, Barry Smith <[email protected]> wrote:
I am more optimistic about valgrind than Mark. I first try valgrind and if that fails to be helpful then use the debugger. valgrind has the advantage that it finds the FIRST place that something is wrong, while in the debugger it is kind of late at the crash.
Valgrind should not be noisy, if it is then the applications/libraries should be cleaned up so that they are valgrind clean and then valgrind is useful.
Barry
On Nov 3, 2015, at 7:47 AM, Mark Adams <[email protected]> wrote:
BTW, I think that our advice for segv is use a debugger. DDT or Totalview, and gdb if need be, will get you right to the source code and will get 90% of bugs diagnosed. Valgrind is noisy and cumbersome to use but can diagnose 90% of the other 10%.
On Tue, Nov 3, 2015 at 7:32 AM, Denis Davydov <[email protected]> wrote: Hi Jose,
On 3 Nov 2015, at 12:20, Jose E. Roman <[email protected]> wrote:
I am answering the SLEPc-related questions: - Having different number of iterations when changing the number of processes is normal. the change in iterations i mentioned are for different preconditioners, but the same number of MPI processes.
- Yes, if you do not destroy the EPS solver, then the preconditioner would be reused.
Regarding the segmentation fault, I have no clue. Not sure if this is related to GAMG or not. Maybe running under valgrind could provide more information. will try that.
Denis.
<petsc_val.gz>
Thanks do you use a petscrc file or any file with PETSc options in it for the run? Thanks please send me the exact PETSc commit you are built off so I can see the line numbers in our source when things go bad. Barry
On Nov 11, 2015, at 7:36 AM, Mark Adams <[email protected]> wrote:
On Tue, Nov 10, 2015 at 11:15 AM, Barry Smith <[email protected]> wrote:
Please send me the full output. This is nuts and should be reported once we understand it better to NERSc as something to be fixed. When I pay $60 million in taxes to a computing center I expect something that works fine for free on my laptop to work also there.
Barry
On Nov 10, 2015, at 7:51 AM, Mark Adams <[email protected]> wrote:
I ran an 8 processor job on Edison of a small code for a short run (just a linear solve) and got 37 Mb of output!
Here is a 'Petsc' grep.
Perhaps we should build an ignore file for things that we believe is a false positive.
On Tue, Nov 3, 2015 at 11:55 AM, Barry Smith <[email protected]> wrote:
I am more optimistic about valgrind than Mark. I first try valgrind and if that fails to be helpful then use the debugger. valgrind has the advantage that it finds the FIRST place that something is wrong, while in the debugger it is kind of late at the crash.
Valgrind should not be noisy, if it is then the applications/libraries should be cleaned up so that they are valgrind clean and then valgrind is useful.
Barry
On Nov 3, 2015, at 7:47 AM, Mark Adams <[email protected]> wrote:
BTW, I think that our advice for segv is use a debugger. DDT or Totalview, and gdb if need be, will get you right to the source code and will get 90% of bugs diagnosed. Valgrind is noisy and cumbersome to use but can diagnose 90% of the other 10%.
On Tue, Nov 3, 2015 at 7:32 AM, Denis Davydov <[email protected]> wrote: Hi Jose,
On 3 Nov 2015, at 12:20, Jose E. Roman <[email protected]> wrote:
I am answering the SLEPc-related questions: - Having different number of iterations when changing the number of processes is normal. the change in iterations i mentioned are for different preconditioners, but the same number of MPI processes.
- Yes, if you do not destroy the EPS solver, then the preconditioner would be reused.
Regarding the segmentation fault, I have no clue. Not sure if this is related to GAMG or not. Maybe running under valgrind could provide more information. will try that.
Denis.
<petsc_val.gz>
<outval.gz>
These are the only PETSc params that I used: -log_summary -options_left false -fp_trap I last update about 3 weeks ago and I am on a branch. I can redo this with a current master. My repo seems to have been polluted: 13:35 edison12 master> ~/petsc$ git status # On branch master # Your branch is ahead of 'origin/master' by 262 commits. # nothing to commit (working directory clean) I trust this is OK but let me know if you would like me to clone a fresh repo. Mark On Wed, Nov 11, 2015 at 11:21 AM, Barry Smith <[email protected]> wrote:
Thanks
do you use a petscrc file or any file with PETSc options in it for the run?
Thanks please send me the exact PETSc commit you are built off so I can see the line numbers in our source when things go bad.
Barry
On Nov 11, 2015, at 7:36 AM, Mark Adams <[email protected]> wrote:
On Tue, Nov 10, 2015 at 11:15 AM, Barry Smith <[email protected]> wrote:
Please send me the full output. This is nuts and should be reported once we understand it better to NERSc as something to be fixed. When I pay $60 million in taxes to a computing center I expect something that works fine for free on my laptop to work also there.
Barry
On Nov 10, 2015, at 7:51 AM, Mark Adams <[email protected]> wrote:
I ran an 8 processor job on Edison of a small code for a short run (just a linear solve) and got 37 Mb of output!
Here is a 'Petsc' grep.
Perhaps we should build an ignore file for things that we believe is a false positive.
On Tue, Nov 3, 2015 at 11:55 AM, Barry Smith <[email protected]> wrote:
I am more optimistic about valgrind than Mark. I first try valgrind and if that fails to be helpful then use the debugger. valgrind has the advantage that it finds the FIRST place that something is wrong, while in the debugger it is kind of late at the crash.
Valgrind should not be noisy, if it is then the applications/libraries should be cleaned up so that they are valgrind clean and then valgrind is useful.
Barry
On Nov 3, 2015, at 7:47 AM, Mark Adams <[email protected]> wrote:
BTW, I think that our advice for segv is use a debugger. DDT or Totalview, and gdb if need be, will get you right to the source code and will get 90% of bugs diagnosed. Valgrind is noisy and cumbersome to use but can diagnose 90% of the other 10%.
On Tue, Nov 3, 2015 at 7:32 AM, Denis Davydov <[email protected]> wrote: Hi Jose,
On 3 Nov 2015, at 12:20, Jose E. Roman <[email protected]> wrote:
I am answering the SLEPc-related questions: - Having different number of iterations when changing the number of processes is normal. the change in iterations i mentioned are for different preconditioners, but the same number of MPI processes.
- Yes, if you do not destroy the EPS solver, then the preconditioner would be reused.
Regarding the segmentation fault, I have no clue. Not sure if this is related to GAMG or not. Maybe running under valgrind could provide more information. will try that.
Denis.
<petsc_val.gz>
<outval.gz>
send the output from git log
On Nov 11, 2015, at 3:38 PM, Mark Adams <[email protected]> wrote:
These are the only PETSc params that I used:
-log_summary -options_left false -fp_trap
I last update about 3 weeks ago and I am on a branch. I can redo this with a current master. My repo seems to have been polluted:
13:35 edison12 master> ~/petsc$ git status # On branch master # Your branch is ahead of 'origin/master' by 262 commits. # nothing to commit (working directory clean)
I trust this is OK but let me know if you would like me to clone a fresh repo.
Mark
On Wed, Nov 11, 2015 at 11:21 AM, Barry Smith <[email protected]> wrote:
Thanks
do you use a petscrc file or any file with PETSc options in it for the run?
Thanks please send me the exact PETSc commit you are built off so I can see the line numbers in our source when things go bad.
Barry
On Nov 11, 2015, at 7:36 AM, Mark Adams <[email protected]> wrote:
On Tue, Nov 10, 2015 at 11:15 AM, Barry Smith <[email protected]> wrote:
Please send me the full output. This is nuts and should be reported once we understand it better to NERSc as something to be fixed. When I pay $60 million in taxes to a computing center I expect something that works fine for free on my laptop to work also there.
Barry
On Nov 10, 2015, at 7:51 AM, Mark Adams <[email protected]> wrote:
I ran an 8 processor job on Edison of a small code for a short run (just a linear solve) and got 37 Mb of output!
Here is a 'Petsc' grep.
Perhaps we should build an ignore file for things that we believe is a false positive.
On Tue, Nov 3, 2015 at 11:55 AM, Barry Smith <[email protected]> wrote:
I am more optimistic about valgrind than Mark. I first try valgrind and if that fails to be helpful then use the debugger. valgrind has the advantage that it finds the FIRST place that something is wrong, while in the debugger it is kind of late at the crash.
Valgrind should not be noisy, if it is then the applications/libraries should be cleaned up so that they are valgrind clean and then valgrind is useful.
Barry
On Nov 3, 2015, at 7:47 AM, Mark Adams <[email protected]> wrote:
BTW, I think that our advice for segv is use a debugger. DDT or Totalview, and gdb if need be, will get you right to the source code and will get 90% of bugs diagnosed. Valgrind is noisy and cumbersome to use but can diagnose 90% of the other 10%.
On Tue, Nov 3, 2015 at 7:32 AM, Denis Davydov <[email protected]> wrote: Hi Jose,
On 3 Nov 2015, at 12:20, Jose E. Roman <[email protected]> wrote:
I am answering the SLEPc-related questions: - Having different number of iterations when changing the number of processes is normal. the change in iterations i mentioned are for different preconditioners, but the same number of MPI processes.
- Yes, if you do not destroy the EPS solver, then the preconditioner would be reused.
Regarding the segmentation fault, I have no clue. Not sure if this is related to GAMG or not. Maybe running under valgrind could provide more information. will try that.
Denis.
<petsc_val.gz>
<outval.gz>
OK, here is an updated output: commit 4bf4127a5802dd8df0d302f4b0a83b52c238cccf Merge: 0d8bfb1 40b2df9 Author: Jason Sarich <[email protected]> Date: Wed Nov 11 09:47:21 2015 -0600 Merge branch 'sarich/jenkins' On Wed, Nov 11, 2015 at 4:53 PM, Barry Smith <[email protected]> wrote:
send the output from
git log
On Nov 11, 2015, at 3:38 PM, Mark Adams <[email protected]> wrote:
These are the only PETSc params that I used:
-log_summary -options_left false -fp_trap
I last update about 3 weeks ago and I am on a branch. I can redo this with a current master. My repo seems to have been polluted:
13:35 edison12 master> ~/petsc$ git status # On branch master # Your branch is ahead of 'origin/master' by 262 commits. # nothing to commit (working directory clean)
I trust this is OK but let me know if you would like me to clone a fresh repo.
Mark
On Wed, Nov 11, 2015 at 11:21 AM, Barry Smith <[email protected]> wrote:
Thanks
do you use a petscrc file or any file with PETSc options in it for the run?
Thanks please send me the exact PETSc commit you are built off so I can see the line numbers in our source when things go bad.
Barry
On Nov 11, 2015, at 7:36 AM, Mark Adams <[email protected]> wrote:
On Tue, Nov 10, 2015 at 11:15 AM, Barry Smith <[email protected]> wrote:
Please send me the full output. This is nuts and should be reported once we understand it better to NERSc as something to be fixed. When I pay $60 million in taxes to a computing center I expect something that works fine for free on my laptop to work also there.
Barry
On Nov 10, 2015, at 7:51 AM, Mark Adams <[email protected]> wrote:
I ran an 8 processor job on Edison of a small code for a short run (just a linear solve) and got 37 Mb of output!
Here is a 'Petsc' grep.
Perhaps we should build an ignore file for things that we believe is a false positive.
On Tue, Nov 3, 2015 at 11:55 AM, Barry Smith <[email protected]> wrote:
I am more optimistic about valgrind than Mark. I first try valgrind and if that fails to be helpful then use the debugger. valgrind has the advantage that it finds the FIRST place that something is wrong, while in the debugger it is kind of late at the crash.
Valgrind should not be noisy, if it is then the applications/libraries should be cleaned up so that they are valgrind clean and then valgrind is useful.
Barry
On Nov 3, 2015, at 7:47 AM, Mark Adams <[email protected]> wrote:
BTW, I think that our advice for segv is use a debugger. DDT or Totalview, and gdb if need be, will get you right to the source code and will get 90% of bugs diagnosed. Valgrind is noisy and cumbersome to use but can diagnose 90% of the other 10%.
On Tue, Nov 3, 2015 at 7:32 AM, Denis Davydov <[email protected]> wrote: Hi Jose,
On 3 Nov 2015, at 12:20, Jose E. Roman <[email protected]> wrote:
I am answering the SLEPc-related questions: - Having different number of iterations when changing the number of processes is normal. the change in iterations i mentioned are for different preconditioners, but the same number of MPI processes.
- Yes, if you do not destroy the EPS solver, then the preconditioner would be reused.
Regarding the segmentation fault, I have no clue. Not sure if this is related to GAMG or not. Maybe running under valgrind could provide more information. will try that.
Denis.
<petsc_val.gz>
<outval.gz>
Hmm, you absolutely must be using an options file otherwise it would never be doing all the stuff it is doing inside PetscOptionsInsertFile()! Please send me the options file. Barry Most of the reports are doing to vendor crimes but it possible that the PetscTokenFind() code has a memory issue though I don't see how. Seriously the NERSc people should be pressuring Cray to have valgrind clean code, this is disgraceful. Conditional jump or move depends on uninitialised value(s) ==2948== at 0x542EC7: PetscTokenFind (str.c:965) ==2948== by 0x4F00B9: PetscOptionsInsertString (options.c:390) ==2948== by 0x4F2F7B: PetscOptionsInsertFile (options.c:590) ==2948== by 0x4F4ED7: PetscOptionsInsert (options.c:721) ==2948== by 0x51A629: PetscInitialize (pinit.c:859) ==2948== by 0x47B98D: main (in /global/u2/m/madams/hpsr/src/hpsr.arch-xc30-dbg-intel.ex) ==2948== ==2948== Use of uninitialised value of size 8 ==2948== at 0x542ECD: PetscTokenFind (str.c:965) ==2948== by 0x4F00B9: PetscOptionsInsertString (options.c:390) ==2948== by 0x4F2F7B: PetscOptionsInsertFile (options.c:590) ==2948== by 0x4F4ED7: PetscOptionsInsert (options.c:721) ==2948== by 0x51A629: PetscInitialize (pinit.c:859) ==2948== by 0x47B98D: main (in /global/u2/m/madams/hpsr/src/hpsr.arch-xc30-dbg-intel.ex) ==2948== ==2948== Conditional jump or move depends on uninitialised value(s) ==2948== at 0x542F04: PetscTokenFind (str.c:966) ==2948== by 0x4F00B9: PetscOptionsInsertString (options.c:390) ==2948== by 0x4F2F7B: PetscOptionsInsertFile (options.c:590) ==2948== by 0x4F4ED7: PetscOptionsInsert (options.c:721) ==2948== by 0x51A629: PetscInitialize (pinit.c:859) ==2948== by 0x47B98D: main (in /global/u2/m/madams/hpsr/src/hpsr.arch-xc30-dbg-intel.ex) ==2948== ==2948== Use of uninitialised value of size 8 ==2948== at 0x542F0E: PetscTokenFind (str.c:967) ==2948== by 0x4F00B9: PetscOptionsInsertString (options.c:390) ==2948== by 0x4F2F7B: PetscOptionsInsertFile (options.c:590) ==2948== by 0x4F4ED7: PetscOptionsInsert (options.c:721) ==2948== by 0x51A629: PetscInitialize (pinit.c:859) ==2948== by 0x47B98D: main (in /global/u2/m/madams/hpsr/src/hpsr.arch-xc30-dbg-intel.ex) ==2948== ==2948== Use of uninitialised value of size 8 ==2948== at 0x542F77: PetscTokenFind (str.c:973) ==2948== by 0x4F00B9: PetscOptionsInsertString (options.c:390) ==2948== by 0x4F2F7B: PetscOptionsInsertFile (options.c:590) ==2948== by 0x4F4ED7: PetscOptionsInsert (options.c:721) ==2948== by 0x51A629: PetscInitialize (pinit.c:859) ==2948== by 0x47B98D: main (in /global/u2/m/madams/hpsr/src/hpsr.arch-xc30-dbg-intel.ex) ==2948== ==2948== Use of uninitialised value of size 8 ==2948== at 0x542F2D: PetscTokenFind (str.c:968) ==2948== by 0x4F00B9: PetscOptionsInsertString (options.c:390) ==2948== by 0x4F2F7B: PetscOptionsInsertFile (options.c:590) ==2948== by 0x4F4ED7: PetscOptionsInsert (options.c:721) ==2948== by 0x51A629: PetscInitialize (pinit.c:859) ==2948== by 0x47B98D: main (in /global/u2/m/madams/hpsr/src/hpsr.arch-xc30-dbg-intel.ex)
On Nov 11, 2015, at 3:38 PM, Mark Adams <[email protected]> wrote:
These are the only PETSc params that I used:
-log_summary -options_left false -fp_trap
I last update about 3 weeks ago and I am on a branch. I can redo this with a current master. My repo seems to have been polluted:
13:35 edison12 master> ~/petsc$ git status # On branch master # Your branch is ahead of 'origin/master' by 262 commits. # nothing to commit (working directory clean)
I trust this is OK but let me know if you would like me to clone a fresh repo.
Mark
On Wed, Nov 11, 2015 at 11:21 AM, Barry Smith <[email protected]> wrote:
Thanks
do you use a petscrc file or any file with PETSc options in it for the run?
Thanks please send me the exact PETSc commit you are built off so I can see the line numbers in our source when things go bad.
Barry
On Nov 11, 2015, at 7:36 AM, Mark Adams <[email protected]> wrote:
On Tue, Nov 10, 2015 at 11:15 AM, Barry Smith <[email protected]> wrote:
Please send me the full output. This is nuts and should be reported once we understand it better to NERSc as something to be fixed. When I pay $60 million in taxes to a computing center I expect something that works fine for free on my laptop to work also there.
Barry
On Nov 10, 2015, at 7:51 AM, Mark Adams <[email protected]> wrote:
I ran an 8 processor job on Edison of a small code for a short run (just a linear solve) and got 37 Mb of output!
Here is a 'Petsc' grep.
Perhaps we should build an ignore file for things that we believe is a false positive.
On Tue, Nov 3, 2015 at 11:55 AM, Barry Smith <[email protected]> wrote:
I am more optimistic about valgrind than Mark. I first try valgrind and if that fails to be helpful then use the debugger. valgrind has the advantage that it finds the FIRST place that something is wrong, while in the debugger it is kind of late at the crash.
Valgrind should not be noisy, if it is then the applications/libraries should be cleaned up so that they are valgrind clean and then valgrind is useful.
Barry
On Nov 3, 2015, at 7:47 AM, Mark Adams <[email protected]> wrote:
BTW, I think that our advice for segv is use a debugger. DDT or Totalview, and gdb if need be, will get you right to the source code and will get 90% of bugs diagnosed. Valgrind is noisy and cumbersome to use but can diagnose 90% of the other 10%.
On Tue, Nov 3, 2015 at 7:32 AM, Denis Davydov <[email protected]> wrote: Hi Jose,
On 3 Nov 2015, at 12:20, Jose E. Roman <[email protected]> wrote:
I am answering the SLEPc-related questions: - Having different number of iterations when changing the number of processes is normal. the change in iterations i mentioned are for different preconditioners, but the same number of MPI processes.
- Yes, if you do not destroy the EPS solver, then the preconditioner would be reused.
Regarding the segmentation fault, I have no clue. Not sure if this is related to GAMG or not. Maybe running under valgrind could provide more information. will try that.
Denis.
<petsc_val.gz>
<outval.gz>
On Wed, Nov 11, 2015 at 6:14 PM, Barry Smith <[email protected]> wrote:
Hmm, you absolutely must be using an options file otherwise it would never be doing all the stuff it is doing inside PetscOptionsInsertFile()!
Yes, here it is: -log_summary #-help -options_left false -damping 1.15 -fp_trap #-on_error_attach_debugger /usr/local/bin/gdb #-on_error_attach_debugger /Users/markadams/homebrew/bin/gdb #-start_in_debugger /Users/markadams/homebrew/bin/gdb -debugger_nodes 1 #-malloc_debug #-malloc_dump
Please send me the options file.
Barry
Most of the reports are doing to vendor crimes but it possible that the PetscTokenFind() code has a memory issue though I don't see how.
Seriously the NERSc people should be pressuring Cray to have valgrind clean code, this is disgraceful.
Conditional jump or move depends on uninitialised value(s) ==2948== at 0x542EC7: PetscTokenFind (str.c:965) ==2948== by 0x4F00B9: PetscOptionsInsertString (options.c:390) ==2948== by 0x4F2F7B: PetscOptionsInsertFile (options.c:590) ==2948== by 0x4F4ED7: PetscOptionsInsert (options.c:721) ==2948== by 0x51A629: PetscInitialize (pinit.c:859) ==2948== by 0x47B98D: main (in /global/u2/m/madams/hpsr/src/hpsr.arch-xc30-dbg-intel.ex) ==2948== ==2948== Use of uninitialised value of size 8 ==2948== at 0x542ECD: PetscTokenFind (str.c:965) ==2948== by 0x4F00B9: PetscOptionsInsertString (options.c:390) ==2948== by 0x4F2F7B: PetscOptionsInsertFile (options.c:590) ==2948== by 0x4F4ED7: PetscOptionsInsert (options.c:721) ==2948== by 0x51A629: PetscInitialize (pinit.c:859) ==2948== by 0x47B98D: main (in /global/u2/m/madams/hpsr/src/hpsr.arch-xc30-dbg-intel.ex) ==2948== ==2948== Conditional jump or move depends on uninitialised value(s) ==2948== at 0x542F04: PetscTokenFind (str.c:966) ==2948== by 0x4F00B9: PetscOptionsInsertString (options.c:390) ==2948== by 0x4F2F7B: PetscOptionsInsertFile (options.c:590) ==2948== by 0x4F4ED7: PetscOptionsInsert (options.c:721) ==2948== by 0x51A629: PetscInitialize (pinit.c:859) ==2948== by 0x47B98D: main (in /global/u2/m/madams/hpsr/src/hpsr.arch-xc30-dbg-intel.ex) ==2948== ==2948== Use of uninitialised value of size 8 ==2948== at 0x542F0E: PetscTokenFind (str.c:967) ==2948== by 0x4F00B9: PetscOptionsInsertString (options.c:390) ==2948== by 0x4F2F7B: PetscOptionsInsertFile (options.c:590) ==2948== by 0x4F4ED7: PetscOptionsInsert (options.c:721) ==2948== by 0x51A629: PetscInitialize (pinit.c:859) ==2948== by 0x47B98D: main (in /global/u2/m/madams/hpsr/src/hpsr.arch-xc30-dbg-intel.ex) ==2948== ==2948== Use of uninitialised value of size 8 ==2948== at 0x542F77: PetscTokenFind (str.c:973) ==2948== by 0x4F00B9: PetscOptionsInsertString (options.c:390) ==2948== by 0x4F2F7B: PetscOptionsInsertFile (options.c:590) ==2948== by 0x4F4ED7: PetscOptionsInsert (options.c:721) ==2948== by 0x51A629: PetscInitialize (pinit.c:859) ==2948== by 0x47B98D: main (in /global/u2/m/madams/hpsr/src/hpsr.arch-xc30-dbg-intel.ex) ==2948== ==2948== Use of uninitialised value of size 8 ==2948== at 0x542F2D: PetscTokenFind (str.c:968) ==2948== by 0x4F00B9: PetscOptionsInsertString (options.c:390) ==2948== by 0x4F2F7B: PetscOptionsInsertFile (options.c:590) ==2948== by 0x4F4ED7: PetscOptionsInsert (options.c:721) ==2948== by 0x51A629: PetscInitialize (pinit.c:859) ==2948== by 0x47B98D: main (in /global/u2/m/madams/hpsr/src/hpsr.arch-xc30-dbg-intel.ex)
On Nov 11, 2015, at 3:38 PM, Mark Adams <[email protected]> wrote:
These are the only PETSc params that I used:
-log_summary -options_left false -fp_trap
I last update about 3 weeks ago and I am on a branch. I can redo this with a current master. My repo seems to have been polluted:
13:35 edison12 master> ~/petsc$ git status # On branch master # Your branch is ahead of 'origin/master' by 262 commits. # nothing to commit (working directory clean)
I trust this is OK but let me know if you would like me to clone a fresh repo.
Mark
On Wed, Nov 11, 2015 at 11:21 AM, Barry Smith <[email protected]> wrote:
Thanks
do you use a petscrc file or any file with PETSc options in it for the run?
Thanks please send me the exact PETSc commit you are built off so I can see the line numbers in our source when things go bad.
Barry
On Nov 11, 2015, at 7:36 AM, Mark Adams <[email protected]> wrote:
On Tue, Nov 10, 2015 at 11:15 AM, Barry Smith <[email protected]> wrote:
Please send me the full output. This is nuts and should be reported once we understand it better to NERSc as something to be fixed. When I pay $60 million in taxes to a computing center I expect something that works fine for free on my laptop to work also there.
Barry
On Nov 10, 2015, at 7:51 AM, Mark Adams <[email protected]> wrote:
I ran an 8 processor job on Edison of a small code for a short run (just a linear solve) and got 37 Mb of output!
Here is a 'Petsc' grep.
Perhaps we should build an ignore file for things that we believe is a false positive.
On Tue, Nov 3, 2015 at 11:55 AM, Barry Smith <[email protected]> wrote:
I am more optimistic about valgrind than Mark. I first try valgrind and if that fails to be helpful then use the debugger. valgrind has the advantage that it finds the FIRST place that something is wrong, while in the debugger it is kind of late at the crash.
Valgrind should not be noisy, if it is then the applications/libraries should be cleaned up so that they are valgrind clean and then valgrind is useful.
Barry
On Nov 3, 2015, at 7:47 AM, Mark Adams <[email protected]> wrote:
BTW, I think that our advice for segv is use a debugger. DDT or Totalview, and gdb if need be, will get you right to the source code and will get 90% of bugs diagnosed. Valgrind is noisy and cumbersome to use but can diagnose 90% of the other 10%.
On Tue, Nov 3, 2015 at 7:32 AM, Denis Davydov <[email protected]> wrote: Hi Jose,
On 3 Nov 2015, at 12:20, Jose E. Roman <[email protected]> wrote:
I am answering the SLEPc-related questions: - Having different number of iterations when changing the number of processes is normal. the change in iterations i mentioned are for different preconditioners, but the same number of MPI processes.
- Yes, if you do not destroy the EPS solver, then the preconditioner would be reused.
Regarding the segmentation fault, I have no clue. Not sure if this is related to GAMG or not. Maybe running under valgrind could provide more information. will try that.
Denis.
<petsc_val.gz>
<outval.gz>
Thanks, I don't get any valgrind issues with this file so I have to conclude the valgrind issues all come from that damn Nersc machine. I highly recommend running the application code on some linux machine that is suitably valgrind clean to determine if the are any memory corruption issues with the application code. It is insane to try to debug application codes on damn Nersc machines directly. Barry
On Nov 12, 2015, at 9:35 AM, Mark Adams <[email protected]> wrote:
On Wed, Nov 11, 2015 at 6:14 PM, Barry Smith <[email protected]> wrote:
Hmm, you absolutely must be using an options file otherwise it would never be doing all the stuff it is doing inside PetscOptionsInsertFile()!
Yes, here it is:
-log_summary #-help -options_left false -damping 1.15 -fp_trap #-on_error_attach_debugger /usr/local/bin/gdb #-on_error_attach_debugger /Users/markadams/homebrew/bin/gdb #-start_in_debugger /Users/markadams/homebrew/bin/gdb -debugger_nodes 1 #-malloc_debug #-malloc_dump
Please send me the options file.
Barry
Most of the reports are doing to vendor crimes but it possible that the PetscTokenFind() code has a memory issue though I don't see how.
Seriously the NERSc people should be pressuring Cray to have valgrind clean code, this is disgraceful.
Conditional jump or move depends on uninitialised value(s) ==2948== at 0x542EC7: PetscTokenFind (str.c:965) ==2948== by 0x4F00B9: PetscOptionsInsertString (options.c:390) ==2948== by 0x4F2F7B: PetscOptionsInsertFile (options.c:590) ==2948== by 0x4F4ED7: PetscOptionsInsert (options.c:721) ==2948== by 0x51A629: PetscInitialize (pinit.c:859) ==2948== by 0x47B98D: main (in /global/u2/m/madams/hpsr/src/hpsr.arch-xc30-dbg-intel.ex) ==2948== ==2948== Use of uninitialised value of size 8 ==2948== at 0x542ECD: PetscTokenFind (str.c:965) ==2948== by 0x4F00B9: PetscOptionsInsertString (options.c:390) ==2948== by 0x4F2F7B: PetscOptionsInsertFile (options.c:590) ==2948== by 0x4F4ED7: PetscOptionsInsert (options.c:721) ==2948== by 0x51A629: PetscInitialize (pinit.c:859) ==2948== by 0x47B98D: main (in /global/u2/m/madams/hpsr/src/hpsr.arch-xc30-dbg-intel.ex) ==2948== ==2948== Conditional jump or move depends on uninitialised value(s) ==2948== at 0x542F04: PetscTokenFind (str.c:966) ==2948== by 0x4F00B9: PetscOptionsInsertString (options.c:390) ==2948== by 0x4F2F7B: PetscOptionsInsertFile (options.c:590) ==2948== by 0x4F4ED7: PetscOptionsInsert (options.c:721) ==2948== by 0x51A629: PetscInitialize (pinit.c:859) ==2948== by 0x47B98D: main (in /global/u2/m/madams/hpsr/src/hpsr.arch-xc30-dbg-intel.ex) ==2948== ==2948== Use of uninitialised value of size 8 ==2948== at 0x542F0E: PetscTokenFind (str.c:967) ==2948== by 0x4F00B9: PetscOptionsInsertString (options.c:390) ==2948== by 0x4F2F7B: PetscOptionsInsertFile (options.c:590) ==2948== by 0x4F4ED7: PetscOptionsInsert (options.c:721) ==2948== by 0x51A629: PetscInitialize (pinit.c:859) ==2948== by 0x47B98D: main (in /global/u2/m/madams/hpsr/src/hpsr.arch-xc30-dbg-intel.ex) ==2948== ==2948== Use of uninitialised value of size 8 ==2948== at 0x542F77: PetscTokenFind (str.c:973) ==2948== by 0x4F00B9: PetscOptionsInsertString (options.c:390) ==2948== by 0x4F2F7B: PetscOptionsInsertFile (options.c:590) ==2948== by 0x4F4ED7: PetscOptionsInsert (options.c:721) ==2948== by 0x51A629: PetscInitialize (pinit.c:859) ==2948== by 0x47B98D: main (in /global/u2/m/madams/hpsr/src/hpsr.arch-xc30-dbg-intel.ex) ==2948== ==2948== Use of uninitialised value of size 8 ==2948== at 0x542F2D: PetscTokenFind (str.c:968) ==2948== by 0x4F00B9: PetscOptionsInsertString (options.c:390) ==2948== by 0x4F2F7B: PetscOptionsInsertFile (options.c:590) ==2948== by 0x4F4ED7: PetscOptionsInsert (options.c:721) ==2948== by 0x51A629: PetscInitialize (pinit.c:859) ==2948== by 0x47B98D: main (in /global/u2/m/madams/hpsr/src/hpsr.arch-xc30-dbg-intel.ex)
On Nov 11, 2015, at 3:38 PM, Mark Adams <[email protected]> wrote:
These are the only PETSc params that I used:
-log_summary -options_left false -fp_trap
I last update about 3 weeks ago and I am on a branch. I can redo this with a current master. My repo seems to have been polluted:
13:35 edison12 master> ~/petsc$ git status # On branch master # Your branch is ahead of 'origin/master' by 262 commits. # nothing to commit (working directory clean)
I trust this is OK but let me know if you would like me to clone a fresh repo.
Mark
On Wed, Nov 11, 2015 at 11:21 AM, Barry Smith <[email protected]> wrote:
Thanks
do you use a petscrc file or any file with PETSc options in it for the run?
Thanks please send me the exact PETSc commit you are built off so I can see the line numbers in our source when things go bad.
Barry
On Nov 11, 2015, at 7:36 AM, Mark Adams <[email protected]> wrote:
On Tue, Nov 10, 2015 at 11:15 AM, Barry Smith <[email protected]> wrote:
Please send me the full output. This is nuts and should be reported once we understand it better to NERSc as something to be fixed. When I pay $60 million in taxes to a computing center I expect something that works fine for free on my laptop to work also there.
Barry
On Nov 10, 2015, at 7:51 AM, Mark Adams <[email protected]> wrote:
I ran an 8 processor job on Edison of a small code for a short run (just a linear solve) and got 37 Mb of output!
Here is a 'Petsc' grep.
Perhaps we should build an ignore file for things that we believe is a false positive.
On Tue, Nov 3, 2015 at 11:55 AM, Barry Smith <[email protected]> wrote:
I am more optimistic about valgrind than Mark. I first try valgrind and if that fails to be helpful then use the debugger. valgrind has the advantage that it finds the FIRST place that something is wrong, while in the debugger it is kind of late at the crash.
Valgrind should not be noisy, if it is then the applications/libraries should be cleaned up so that they are valgrind clean and then valgrind is useful.
Barry
On Nov 3, 2015, at 7:47 AM, Mark Adams <[email protected]> wrote:
BTW, I think that our advice for segv is use a debugger. DDT or Totalview, and gdb if need be, will get you right to the source code and will get 90% of bugs diagnosed. Valgrind is noisy and cumbersome to use but can diagnose 90% of the other 10%.
On Tue, Nov 3, 2015 at 7:32 AM, Denis Davydov <[email protected]> wrote: Hi Jose,
On 3 Nov 2015, at 12:20, Jose E. Roman <[email protected]> wrote:
I am answering the SLEPc-related questions: - Having different number of iterations when changing the number of processes is normal. the change in iterations i mentioned are for different preconditioners, but the same number of MPI processes.
- Yes, if you do not destroy the EPS solver, then the preconditioner would be reused.
Regarding the segmentation fault, I have no clue. Not sure if this is related to GAMG or not. Maybe running under valgrind could provide more information. will try that.
Denis.
<petsc_val.gz>
<outval.gz>
There is a valgrind for El Capitan now and I have it. It runs perfectly clean. Thanks, Mark On Thu, Nov 12, 2015 at 11:44 AM, Barry Smith <[email protected]> wrote:
Thanks, I don't get any valgrind issues with this file so I have to conclude the valgrind issues all come from that damn Nersc machine.
I highly recommend running the application code on some linux machine that is suitably valgrind clean to determine if the are any memory corruption issues with the application code. It is insane to try to debug application codes on damn Nersc machines directly.
Barry
On Nov 12, 2015, at 9:35 AM, Mark Adams <[email protected]> wrote:
On Wed, Nov 11, 2015 at 6:14 PM, Barry Smith <[email protected]> wrote:
Hmm, you absolutely must be using an options file otherwise it would never be doing all the stuff it is doing inside PetscOptionsInsertFile()!
Yes, here it is:
-log_summary #-help -options_left false -damping 1.15 -fp_trap #-on_error_attach_debugger /usr/local/bin/gdb #-on_error_attach_debugger /Users/markadams/homebrew/bin/gdb #-start_in_debugger /Users/markadams/homebrew/bin/gdb -debugger_nodes 1 #-malloc_debug #-malloc_dump
Please send me the options file.
Barry
Most of the reports are doing to vendor crimes but it possible that the PetscTokenFind() code has a memory issue though I don't see how.
Seriously the NERSc people should be pressuring Cray to have valgrind clean code, this is disgraceful.
Conditional jump or move depends on uninitialised value(s) ==2948== at 0x542EC7: PetscTokenFind (str.c:965) ==2948== by 0x4F00B9: PetscOptionsInsertString (options.c:390) ==2948== by 0x4F2F7B: PetscOptionsInsertFile (options.c:590) ==2948== by 0x4F4ED7: PetscOptionsInsert (options.c:721) ==2948== by 0x51A629: PetscInitialize (pinit.c:859) ==2948== by 0x47B98D: main (in /global/u2/m/madams/hpsr/src/hpsr.arch-xc30-dbg-intel.ex) ==2948== ==2948== Use of uninitialised value of size 8 ==2948== at 0x542ECD: PetscTokenFind (str.c:965) ==2948== by 0x4F00B9: PetscOptionsInsertString (options.c:390) ==2948== by 0x4F2F7B: PetscOptionsInsertFile (options.c:590) ==2948== by 0x4F4ED7: PetscOptionsInsert (options.c:721) ==2948== by 0x51A629: PetscInitialize (pinit.c:859) ==2948== by 0x47B98D: main (in /global/u2/m/madams/hpsr/src/hpsr.arch-xc30-dbg-intel.ex) ==2948== ==2948== Conditional jump or move depends on uninitialised value(s) ==2948== at 0x542F04: PetscTokenFind (str.c:966) ==2948== by 0x4F00B9: PetscOptionsInsertString (options.c:390) ==2948== by 0x4F2F7B: PetscOptionsInsertFile (options.c:590) ==2948== by 0x4F4ED7: PetscOptionsInsert (options.c:721) ==2948== by 0x51A629: PetscInitialize (pinit.c:859) ==2948== by 0x47B98D: main (in /global/u2/m/madams/hpsr/src/hpsr.arch-xc30-dbg-intel.ex) ==2948== ==2948== Use of uninitialised value of size 8 ==2948== at 0x542F0E: PetscTokenFind (str.c:967) ==2948== by 0x4F00B9: PetscOptionsInsertString (options.c:390) ==2948== by 0x4F2F7B: PetscOptionsInsertFile (options.c:590) ==2948== by 0x4F4ED7: PetscOptionsInsert (options.c:721) ==2948== by 0x51A629: PetscInitialize (pinit.c:859) ==2948== by 0x47B98D: main (in /global/u2/m/madams/hpsr/src/hpsr.arch-xc30-dbg-intel.ex) ==2948== ==2948== Use of uninitialised value of size 8 ==2948== at 0x542F77: PetscTokenFind (str.c:973) ==2948== by 0x4F00B9: PetscOptionsInsertString (options.c:390) ==2948== by 0x4F2F7B: PetscOptionsInsertFile (options.c:590) ==2948== by 0x4F4ED7: PetscOptionsInsert (options.c:721) ==2948== by 0x51A629: PetscInitialize (pinit.c:859) ==2948== by 0x47B98D: main (in /global/u2/m/madams/hpsr/src/hpsr.arch-xc30-dbg-intel.ex) ==2948== ==2948== Use of uninitialised value of size 8 ==2948== at 0x542F2D: PetscTokenFind (str.c:968) ==2948== by 0x4F00B9: PetscOptionsInsertString (options.c:390) ==2948== by 0x4F2F7B: PetscOptionsInsertFile (options.c:590) ==2948== by 0x4F4ED7: PetscOptionsInsert (options.c:721) ==2948== by 0x51A629: PetscInitialize (pinit.c:859) ==2948== by 0x47B98D: main (in /global/u2/m/madams/hpsr/src/hpsr.arch-xc30-dbg-intel.ex)
On Nov 11, 2015, at 3:38 PM, Mark Adams <[email protected]> wrote:
These are the only PETSc params that I used:
-log_summary -options_left false -fp_trap
I last update about 3 weeks ago and I am on a branch. I can redo this with a current master. My repo seems to have been polluted:
13:35 edison12 master> ~/petsc$ git status # On branch master # Your branch is ahead of 'origin/master' by 262 commits. # nothing to commit (working directory clean)
I trust this is OK but let me know if you would like me to clone a fresh repo.
Mark
On Wed, Nov 11, 2015 at 11:21 AM, Barry Smith <[email protected]> wrote:
Thanks
do you use a petscrc file or any file with PETSc options in it for the run?
Thanks please send me the exact PETSc commit you are built off so I can see the line numbers in our source when things go bad.
Barry
On Nov 11, 2015, at 7:36 AM, Mark Adams <[email protected]> wrote:
On Tue, Nov 10, 2015 at 11:15 AM, Barry Smith <[email protected]> wrote:
Please send me the full output. This is nuts and should be reported once we understand it better to NERSc as something to be fixed. When I pay $60 million in taxes to a computing center I expect something that works fine for free on my laptop to work also there.
Barry
On Nov 10, 2015, at 7:51 AM, Mark Adams <[email protected]> wrote:
I ran an 8 processor job on Edison of a small code for a short run (just a linear solve) and got 37 Mb of output!
Here is a 'Petsc' grep.
Perhaps we should build an ignore file for things that we believe is a false positive.
On Tue, Nov 3, 2015 at 11:55 AM, Barry Smith <[email protected]> wrote:
I am more optimistic about valgrind than Mark. I first try valgrind and if that fails to be helpful then use the debugger. valgrind has the advantage that it finds the FIRST place that something is wrong, while in the debugger it is kind of late at the crash.
Valgrind should not be noisy, if it is then the applications/libraries should be cleaned up so that they are valgrind clean and then valgrind is useful.
Barry
On Nov 3, 2015, at 7:47 AM, Mark Adams <[email protected]> wrote:
BTW, I think that our advice for segv is use a debugger. DDT or Totalview, and gdb if need be, will get you right to the source code and will get 90% of bugs diagnosed. Valgrind is noisy and cumbersome to use but can diagnose 90% of the other 10%.
On Tue, Nov 3, 2015 at 7:32 AM, Denis Davydov < [email protected]> wrote: Hi Jose,
On 3 Nov 2015, at 12:20, Jose E. Roman <[email protected]> wrote:
I am answering the SLEPc-related questions: - Having different number of iterations when changing the number of processes is normal. the change in iterations i mentioned are for different preconditioners, but the same number of MPI processes.
- Yes, if you do not destroy the EPS solver, then the preconditioner would be reused.
Regarding the segmentation fault, I have no clue. Not sure if this is related to GAMG or not. Maybe running under valgrind could provide more information. will try that.
Denis.
<petsc_val.gz>
<outval.gz>
Hi Mark,
On 12 Nov 2015, at 21:16, Mark Adams <[email protected]> wrote:
There is a valgrind for El Capitan now and I have it. It runs perfectly clean. Do you compile it yourself or use Homebrew / MacPorts? I always seem to have some noise it valgrind at least from OpenMPI (even with suppression file), perhaps it’s better with MPICH.
Kind regards, Denis
On Tue, Nov 17, 2015 at 1:38 AM, Denis Davydov <[email protected]> wrote:
Hi Mark,
On 12 Nov 2015, at 21:16, Mark Adams <[email protected]> wrote:
There is a valgrind for El Capitan now and I have it. It runs perfectly clean. Do you compile it yourself or use Homebrew / MacPorts?
Sorry for the delay. I ended up downloading it, but Homebrew works also (now, I think it did not work and I had to download it). Note, the web site said El Capitan support was partial so I don't know if they cover everything now.
I always seem to have some noise it valgrind at least from OpenMPI (even with suppression file), perhaps it’s better with MPICH.
Kind regards, Denis
Jose, Even when I have PETSc --with-debugging=1 and SLEPc picks it up during configure, i don’t seem to have debug symbols in resulting SLEPc lib (make stage): warning: no debug symbols in executable (-arch x86_64) Same when starting a debugger: warning: (x86_64) /usr/local/opt/slepc/real/lib/libslepc.3.6.dylib empty dSYM file detected, dSYM was created with an executable with no debug info. C/Fortran flags seems to have debug flags: Using C/C++ linker: /usr/local/bin/mpicc Using C/C++ flags: -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0 Using Fortran linker: /usr/local/bin/mpif90 Using Fortran flags: -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first -fPIC -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -g -O0 Any ideas? Kind regards, Denis
On 3 Nov 2015, at 12:20, Jose E. Roman <[email protected]> wrote:
I am answering the SLEPc-related questions: - Having different number of iterations when changing the number of processes is normal. - Yes, if you do not destroy the EPS solver, then the preconditioner would be reused.
Regarding the segmentation fault, I have no clue. Not sure if this is related to GAMG or not. Maybe running under valgrind could provide more information.
Jose
In MacOSX you have to keep the *.o files, and not delete them. With PETSc's makefiles, this can be done easily with e.g. $ make ex1 RM=echo Jose
El 3/11/2015, a las 19:46, Denis Davydov <[email protected]> escribió:
Jose,
Even when I have PETSc --with-debugging=1 and SLEPc picks it up during configure, i don’t seem to have debug symbols in resulting SLEPc lib (make stage):
warning: no debug symbols in executable (-arch x86_64)
Same when starting a debugger: warning: (x86_64) /usr/local/opt/slepc/real/lib/libslepc.3.6.dylib empty dSYM file detected, dSYM was created with an executable with no debug info.
C/Fortran flags seems to have debug flags:
Using C/C++ linker: /usr/local/bin/mpicc Using C/C++ flags: -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0 Using Fortran linker: /usr/local/bin/mpif90 Using Fortran flags: -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first -fPIC -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -g -O0
Any ideas?
Kind regards, Denis
After running in debug mode it seems that the GAMG solver indeed did not converge, however throwing the error leads to SIGABRT (backtrace and frames are below). It is still very suspicious why would solving for (unchanged) mass matrix wouldn't converge inside SLEPc's spectral transformation. p.s. valgrind takes enormous amount of time on this problem, will try to leave it over the weekend. Denis. =============== Program received signal SIGABRT, Aborted. 0x00007fffea87fcc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt #0 0x00007fffea87fcc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 #1 0x00007fffea8830d8 in __GI_abort () at abort.c:89 #2 0x00007fffeb790c91 in PetscTraceBackErrorHandler (comm=0x2a09bd0, line=798, fun=0x7fffed0e24b9 <__func__.20043> "KSPSolve", file=0x7fffed0e1620 "/home/davydden/.hashdist/tmp/petsc-hujktg3j6hq7/src/ksp/ksp/interface/itfunc.c", n=91, p=PETSC_ERROR_INITIAL, mess=0x7fffffffac30 "KSPSolve has not converged", ctx=0x0) at /home/davydden/.hashdist/tmp/petsc-hujktg3j6hq7/src/sys/error/errtrace.c:243 #3 0x00007fffeb78b8b9 in PetscError (comm=0x2a09bd0, line=798, func=0x7fffed0e24b9 <__func__.20043> "KSPSolve", file=0x7fffed0e1620 "/home/davydden/.hashdist/tmp/petsc-hujktg3j6hq7/src/ksp/ksp/interface/itfunc.c", n=91, p=PETSC_ERROR_INITIAL, mess=0x7fffed0e1e7a "KSPSolve has not converged") at /home/davydden/.hashdist/tmp/petsc-hujktg3j6hq7/src/sys/error/err.c:377 #4 0x00007fffec75e1e7 in KSPSolve (ksp=0x367227d0, b=0x35b285c0, x=0x35d89250) at /home/davydden/.hashdist/tmp/petsc-hujktg3j6hq7/src/ksp/ksp/interface/itfunc.c:798 #5 0x00007fffe32a8657 in STMatSolve (st=0x3672d820, b=0x35b285c0, x=0x35d89250) at /home/davydden/.hashdist/tmp/slepc-22nb32nbgvhx/src/sys/classes/st/interface/stsles.c:166 ---Type <return> to continue, or q <return> to quit---q Quit (gdb) f 5 #5 0x00007fffe32a8657 in STMatSolve (st=0x3672d820, b=0x35b285c0, x=0x35d89250) at /home/davydden/.hashdist/tmp/slepc-22nb32nbgvhx/src/sys/classes/st/interface/stsles.c:166 166 ierr = KSPSolve(st->ksp,b,x);CHKERRQ(ierr); (gdb) f 4 #4 0x00007fffec75e1e7 in KSPSolve (ksp=0x367227d0, b=0x35b285c0, x=0x35d89250) at /home/davydden/.hashdist/tmp/petsc-hujktg3j6hq7/src/ksp/ksp/interface/itfunc.c:798 798 if (ksp->errorifnotconverged && ksp->reason < 0) SETERRQ(comm,PETSC_ERR_NOT_CONVERGED,"KSPSolve has not converged"); (gdb) f 3 #3 0x00007fffeb78b8b9 in PetscError (comm=0x2a09bd0, line=798, func=0x7fffed0e24b9 <__func__.20043> "KSPSolve", file=0x7fffed0e1620 "/home/davydden/.hashdist/tmp/petsc-hujktg3j6hq7/src/ksp/ksp/interface/itfunc.c", n=91, p=PETSC_ERROR_INITIAL, mess=0x7fffed0e1e7a "KSPSolve has not converged") at /home/davydden/.hashdist/tmp/petsc-hujktg3j6hq7/src/sys/error/err.c:377 377 if (!eh) ierr = PetscTraceBackErrorHandler(comm,line,func,file,n,p,lbuf,0); (gdb) f 2 #2 0x00007fffeb790c91 in PetscTraceBackErrorHandler (comm=0x2a09bd0, line=798, fun=0x7fffed0e24b9 <__func__.20043> "KSPSolve", file=0x7fffed0e1620 "/home/davydden/.hashdist/tmp/petsc-hujktg3j6hq7/src/ksp/ksp/interface/itfunc.c", n=91, p=PETSC_ERROR_INITIAL, mess=0x7fffffffac30 "KSPSolve has not converged", ctx=0x0) at /home/davydden/.hashdist/tmp/petsc-hujktg3j6hq7/src/sys/error/errtrace.c:243 243 abort(); (gdb) f 1 #1 0x00007fffea8830d8 in __GI_abort () at abort.c:89 89 abort.c: No such file or directory.
Denis: Do you use shift-and-invert method for solving eigenvalue problem? If so, the linear problems would be extremely ill-conditioned, for which the direct solver, such LU or Cholesky are usually the only working option. You may run your petsc/slepc code with option '-ksp_monitor' to observe convergence behavior. Hong After running in debug mode it seems that the GAMG solver indeed did not
converge, however throwing the error leads to SIGABRT (backtrace and frames are below). It is still very suspicious why would solving for (unchanged) mass matrix wouldn't converge inside SLEPc's spectral transformation.
p.s. valgrind takes enormous amount of time on this problem, will try to leave it over the weekend.
Denis.
=============== Program received signal SIGABRT, Aborted. 0x00007fffea87fcc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt #0 0x00007fffea87fcc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 #1 0x00007fffea8830d8 in __GI_abort () at abort.c:89 #2 0x00007fffeb790c91 in PetscTraceBackErrorHandler (comm=0x2a09bd0, line=798, fun=0x7fffed0e24b9 <__func__.20043> "KSPSolve", file=0x7fffed0e1620 "/home/davydden/.hashdist/tmp/petsc-hujktg3j6hq7/src/ksp/ksp/interface/itfunc.c", n=91, p=PETSC_ERROR_INITIAL, mess=0x7fffffffac30 "KSPSolve has not converged", ctx=0x0) at /home/davydden/.hashdist/tmp/petsc-hujktg3j6hq7/src/sys/error/errtrace.c:243 #3 0x00007fffeb78b8b9 in PetscError (comm=0x2a09bd0, line=798, func=0x7fffed0e24b9 <__func__.20043> "KSPSolve", file=0x7fffed0e1620 "/home/davydden/.hashdist/tmp/petsc-hujktg3j6hq7/src/ksp/ksp/interface/itfunc.c", n=91, p=PETSC_ERROR_INITIAL, mess=0x7fffed0e1e7a "KSPSolve has not converged") at /home/davydden/.hashdist/tmp/petsc-hujktg3j6hq7/src/sys/error/err.c:377 #4 0x00007fffec75e1e7 in KSPSolve (ksp=0x367227d0, b=0x35b285c0, x=0x35d89250) at /home/davydden/.hashdist/tmp/petsc-hujktg3j6hq7/src/ksp/ksp/interface/itfunc.c:798 #5 0x00007fffe32a8657 in STMatSolve (st=0x3672d820, b=0x35b285c0, x=0x35d89250) at /home/davydden/.hashdist/tmp/slepc-22nb32nbgvhx/src/sys/classes/st/interface/stsles.c:166 ---Type <return> to continue, or q <return> to quit---q Quit (gdb) f 5 #5 0x00007fffe32a8657 in STMatSolve (st=0x3672d820, b=0x35b285c0, x=0x35d89250) at /home/davydden/.hashdist/tmp/slepc-22nb32nbgvhx/src/sys/classes/st/interface/stsles.c:166 166 ierr = KSPSolve(st->ksp,b,x);CHKERRQ(ierr); (gdb) f 4 #4 0x00007fffec75e1e7 in KSPSolve (ksp=0x367227d0, b=0x35b285c0, x=0x35d89250) at /home/davydden/.hashdist/tmp/petsc-hujktg3j6hq7/src/ksp/ksp/interface/itfunc.c:798 798 if (ksp->errorifnotconverged && ksp->reason < 0) SETERRQ(comm,PETSC_ERR_NOT_CONVERGED,"KSPSolve has not converged"); (gdb) f 3 #3 0x00007fffeb78b8b9 in PetscError (comm=0x2a09bd0, line=798, func=0x7fffed0e24b9 <__func__.20043> "KSPSolve", file=0x7fffed0e1620 "/home/davydden/.hashdist/tmp/petsc-hujktg3j6hq7/src/ksp/ksp/interface/itfunc.c", n=91, p=PETSC_ERROR_INITIAL, mess=0x7fffed0e1e7a "KSPSolve has not converged") at /home/davydden/.hashdist/tmp/petsc-hujktg3j6hq7/src/sys/error/err.c:377 377 if (!eh) ierr = PetscTraceBackErrorHandler(comm,line,func,file,n,p,lbuf,0); (gdb) f 2 #2 0x00007fffeb790c91 in PetscTraceBackErrorHandler (comm=0x2a09bd0, line=798, fun=0x7fffed0e24b9 <__func__.20043> "KSPSolve", file=0x7fffed0e1620 "/home/davydden/.hashdist/tmp/petsc-hujktg3j6hq7/src/ksp/ksp/interface/itfunc.c", n=91, p=PETSC_ERROR_INITIAL, mess=0x7fffffffac30 "KSPSolve has not converged", ctx=0x0) at /home/davydden/.hashdist/tmp/petsc-hujktg3j6hq7/src/sys/error/errtrace.c:243 243 abort(); (gdb) f 1 #1 0x00007fffea8830d8 in __GI_abort () at abort.c:89 89 abort.c: No such file or directory.
Hi Hong,
On 6 Nov 2015, at 16:09, Hong <[email protected]> wrote:
Denis: Do you use shift-and-invert method for solving eigenvalue problem? no, it’s just shift with zero value. So for GHEP one inverts B-matrix.
If so, the linear problems would be extremely ill-conditioned, for which the direct solver, such LU or Cholesky are usually the only working option. Depends on the shift, i would say. In any case the same problem works with jacobi preconditioner no no other changes, so i would not relate it to any settings on SLEPc part.
You may run your petsc/slepc code with option '-ksp_monitor' to observe convergence behavior. Will do, thanks.
Regards, Denis.
On Fri, Nov 6, 2015 at 9:15 AM, Denis Davydov <[email protected]> wrote:
Hi Hong,
On 6 Nov 2015, at 16:09, Hong <[email protected]> wrote:
Denis: Do you use shift-and-invert method for solving eigenvalue problem? no, it’s just shift with zero value. So for GHEP one inverts B-matrix.
If so, the linear problems would be extremely ill-conditioned, for which the direct solver, such LU or Cholesky are usually the only working option. Depends on the shift, i would say. In any case the same problem works with jacobi preconditioner no no other changes, so i would not relate it to any settings on SLEPc part.
Is it possible that the matrix is rank deficient? Jacobi will just chug along and sometimes work, but AMG will fail spectacularly in that case. Matt
You may run your petsc/slepc code with option '-ksp_monitor' to observe convergence behavior. Will do, thanks.
Regards, Denis.
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
On 6 Nov 2015, at 16:22, Matthew Knepley <[email protected]> wrote:
Is it possible that the matrix is rank deficient? Jacobi will just chug along and sometimes work, but AMG will fail spectacularly in that case.
It should not. It is just a mass (overlap) matrix coming from linear FEs with zero Dirichlet BC assembled in deal.II. Due to elimination of some algebraic constraints on DoFs there are lines with only diagonal element, but it should still be SPD. More interestingly is that it does not fail immediately (i.e. the first time it’s used in SLEPc solvers), but only on the 4th step. So 3 times SLEPc worked just fine to solve GHEP with Gamg and zero shift. Regards, Denis.
On Fri, Nov 6, 2015 at 9:29 AM, Denis Davydov <[email protected]> wrote:
On 6 Nov 2015, at 16:22, Matthew Knepley <[email protected]> wrote:
Is it possible that the matrix is rank deficient? Jacobi will just chug along and sometimes work, but AMG will fail spectacularly in that case.
It should not. It is just a mass (overlap) matrix coming from linear FEs with zero Dirichlet BC assembled in deal.II. Due to elimination of some algebraic constraints on DoFs there are lines with only diagonal element, but it should still be SPD.
More interestingly is that it does not fail immediately (i.e. the first time it’s used in SLEPc solvers), but only on the 4th step. So 3 times SLEPc worked just fine to solve GHEP with Gamg and zero shift.
Then I think it is not doing what you suppose. I am not inclined to believe that it behaves differently on the same matrix. Matt
Regards, Denis.
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
If it is a true mass matrix in the finite element sense of the word then it should be very well conditioned and one definitely would not use something like GAMG on. Jacobi + CG or maybe SSOR + CG should converge rapidly Barry
On Nov 6, 2015, at 9:29 AM, Denis Davydov <[email protected]> wrote:
On 6 Nov 2015, at 16:22, Matthew Knepley <[email protected]> wrote:
Is it possible that the matrix is rank deficient? Jacobi will just chug along and sometimes work, but AMG will fail spectacularly in that case.
It should not. It is just a mass (overlap) matrix coming from linear FEs with zero Dirichlet BC assembled in deal.II. Due to elimination of some algebraic constraints on DoFs there are lines with only diagonal element, but it should still be SPD.
More interestingly is that it does not fail immediately (i.e. the first time it’s used in SLEPc solvers), but only on the 4th step. So 3 times SLEPc worked just fine to solve GHEP with Gamg and zero shift.
Regards, Denis.
On 6 Nov 2015, at 17:39, Barry Smith <[email protected]> wrote:
If it is a true mass matrix in the finite element sense of the word then it should be very well conditioned and one definitely would not use something like GAMG on. Jacobi + CG or maybe SSOR + CG should converge rapidly
That I understand and absolutely agree. It just does not explain why GAMG would fail, especially on 4 cores and not on 8. Regards, Denis.
You can run with -info and grep on GAMG, and send this. If you are shifting a matrix then it can/will get indefinite. If it is just a mass matrix then Jacobi should converge quickly - does it? On Fri, Nov 6, 2015 at 9:35 AM, Denis Davydov <[email protected]> wrote:
On 6 Nov 2015, at 17:39, Barry Smith <[email protected]> wrote:
If it is a true mass matrix in the finite element sense of the word then it should be very well conditioned and one definitely would not use something like GAMG on. Jacobi + CG or maybe SSOR + CG should converge rapidly
That I understand and absolutely agree. It just does not explain why GAMG would fail, especially on 4 cores and not on 8.
Regards, Denis.
On Tue, Nov 3, 2015 at 2:07 AM, Denis Davydov <[email protected]> wrote:
Dear all,
I experience strange convergence problems in SLEPc for GHEP with Krylov-Schur and CG + GAMG. The issue appears to be contingent on the number of MPI cores used. Say for 8 cores there is no issue and for 4 cores there is an issue. When I substitute GAMG with Jacobi for the problematic number of cores -- all works.
To be more specific, I solve Ax=\lambda Bx for a sequence of A’s where A is a function of eigenvectors. On each iteration step currently the eigensolver EPS is initialised from scratch. And thus should the underlying ST, KSP, PC objects: -st_ksp_type cg -st_pc_type gamg -st_ksp_rtol 1e-12. For these particular matrices the issue appears on the 4th iteration, even though the matrix to be inverted (mass/overlap matrix) is the same, does not change!!!'
I assume the issue is the SEGV below? I agree with Jose that you need to run valgrind. An SEGV can result from memory corruption in a distant part of the code. This seems very likely to me since it is the same matrix coming in. Thanks, Matt
From my debuging info the A matrix has the same norm for CG + GAMG vs CG + Jacobi cases: DEAL:: frobenius_norm = 365.7 DEAL:: linfty_norm = 19.87 DEAL:: l1_norm = 19.87 Just to be sure that there are no bugs on my side which would result in different mass matrices i check that it has the same norm for CG + GAMG vs CG + Jacobi BEFORE i start iteration: DEAL:: frobenius_norm = 166.4 DEAL:: linfty_norm = 8.342 DEAL:: l1_norm = 8.342 All the dependent scalar quantities I calculate on each iteration are identical for the two cases, which makes me believe that the solution path is the same up to the certain tolerance. The only output which is slightly different are the number iterations for convergence in EPS (e.g. 113 vs 108) and the resulting maxing EPSComputeResidualNorm : 4.1524e-07 vs 2.9639e-08.
Any ideas what could be an issue, especially given the fact that it does work for some numbers of cores and does not for other? Perhaps some default settings in GAMG preconditioner? Although that does not explain why it works for the first 3 iterations and does not on 4th as the mass matrix is unchanged...
Lastly, i suppose ideally i should keep the eigensolver context between the iterations and just update the matrices by EPSSetOperators. Is it correct to assume that since B matrix does not change between iterations and I use the default shift transformation with zero shift (operator is B^{-1)A ), the GAMG preconditioner will not be re-initialised and thus I should save some time?
p.s. the relevant error message is below. I have the same issues on CentOS cluster, so it is not related to OS-X.
Kind regards, Denis
=== [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] KSPSolve line 510 /private/tmp/petsc20151102-50378-1t7b3in/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [0] STMatSolve line 148 /private/tmp/slepc20151102-3081-1xln4h0/slepc-3.6.1/src/sys/classes/st/interface/stsles.c [0]PETSC ERROR: [0] STApply_Shift line 33 /private/tmp/slepc20151102-3081-1xln4h0/slepc-3.6.1/src/sys/classes/st/impls/shift/shift.c [0]PETSC ERROR: [0] STApply line 50 /private/tmp/slepc20151102-3081-1xln4h0/slepc-3.6.1/src/sys/classes/st/interface/stsolve.c [0]PETSC ERROR: [0] EPSGetStartVector line 726 /private/tmp/slepc20151102-3081-1xln4h0/slepc-3.6.1/src/eps/interface/epssolve.c [0]PETSC ERROR: [0] EPSSolve_KrylovSchur_Symm line 41 /private/tmp/slepc20151102-3081-1xln4h0/slepc-3.6.1/src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: [0] EPSSolve line 83 /private/tmp/slepc20151102-3081-1xln4h0/slepc-3.6.1/src/eps/interface/epssolve.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015 [0]PETSC ERROR: /Users/davydden/Desktop/work/C++/deal.ii-dft/build_debug~/dft on a real named MBP-Denis.fritz.box by davydden Tue Nov 3 07:02:47 2015 [0]PETSC ERROR: Configure options CC=/usr/local/bin/mpicc CXX=/usr/local/bin/mpicxx F77=/usr/local/bin/mpif77 FC=/usr/local/bin/mpif90 --with-shared-libraries=1 --with-pthread=0 --with-openmp=0 --with-debugging=1 --with-ssl=0 --with-superlu_dist-include=/usr/local/opt/superlu_dist/include/superlu_dist --with-superlu_dist-lib="-L/usr/local/opt/superlu_dist/lib -lsuperlu_dist" --with-superlu-include=/usr/local/Cellar/superlu43/4.3/include/superlu --with-superlu-lib="-L/usr/local/Cellar/superlu43/4.3/lib -lsuperlu" --with-fftw-dir=/usr/local/opt/fftw --with-netcdf-dir=/usr/local/opt/netcdf --with-suitesparse-dir=/usr/local/opt/suite-sparse --with-hdf5-dir=/usr/local/opt/hdf5 --with-metis-dir=/usr/local/opt/metis --with-parmetis-dir=/usr/local/opt/parmetis --with-scalapack-dir=/usr/local/opt/scalapack --with-mumps-dir=/usr/local/opt/mumps --with-x=0 --prefix=/usr/local/Cellar/petsc/3.6.2/real --with-scalar-type=real --with-hypre-dir=/usr/local/opt/hypre --with-sundials-dir=/usr/local/opt/sundials --with-hwloc-dir=/usr/local/opt/hwloc [0]PETSC ERROR: #8 User provided function() line 0 in unknown file -------------------------------------------------------------------------- mpirun noticed that process rank 3 with PID 96754 on node MBP-Denis exited on signal 6 (Abort trap: 6). --------------------------------------------------------------------------
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
participants (6)
-
Barry Smith -
Denis Davydov -
Hong -
Jose E. Roman -
Mark Adams -
Matthew Knepley