June 2015 - petsc-users - mailman3.cels.anl.gov

Re: [petsc-users] Varying TAO optimization solve iterations using BLMVM
by Jason Sarich 23 Jun '15

23 Jun '15

BLMVM doesn't use a KSP or preconditioner, it updates using the L-BFGS-B formula On Thu, Jun 18, 2015 at 1:45 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > On Thu, Jun 18, 2015 at 12:15 PM, Jason Sarich <jason.sarich(a)gmail.com> > wrote: > >> Hi Justin, >> >> I can't tell for sure why this is happening, have you tried using quad >> precision to make sure that numerical cutoffs isn't the problem? >> >> 1 The Hessian being approximate and the resulting implicit computation >> is the source of the cutoff, but would not be causing different convergence >> rates in infinite precision. >> >> 2 the local size may affect load balancing but not the resulting >> norms/convergence rate. >> > > This sounds to be like the preconditioner is dependent on the partition. > Can you send -tao_view -snes_view > > Matt > > >> Jason >> >> >> On Thu, Jun 18, 2015 at 10:44 AM, Justin Chang <jychang48(a)gmail.com> >> wrote: >> >>> I solved a transient diffusion across multiple cores using TAO BLMVM. >>> When I simulate the same problem but on different numbers of processing >>> cores, the number of solve iterations change quite drastically. The >>> numerical solution is the same, but these changes are quite vast. I >>> attached a PDF showing a comparison between KSP and TAO. KSP remains >>> largely invariant with number of processors but TAO (with bounded >>> constraints) fluctuates. >>> >>> My question is, why is this happening? I understand that accumulation of >>> numerical round-offs may attribute to this, but the differences seem quite >>> vast to me. My initial thought was that >>> >>> 1) the Hessian is only projected and not explicitly computed, which >>> may have something to do with the rate of convergence >>> >>> 2) local problem size. Certain regions of my domain have different >>> number of "violations" which need to be corrected by the bounded >>> constraints so the rate of convergence depends on how these regions are >>> partitioned? >>> >>> Any thoughts? >>> >>> Thanks, >>> Justin >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener >

3 3

June 24th DMPlex Webinar
by Matthew Knepley 23 Jun '15

23 Jun '15

Michael Lange will give a webinar tomorrow at 9:00am CDT which covers the DMPlex interface for unstructured mesh management. Thanks, Matt ------------------------------------------------------------------------------ ARCHER Webinar: Flexible, Scalable Mesh and Data Management using PETSc DMPlex ------------------------------------------------------------------------------ 15:00 BST, Wed 24th June 2015 http://www.archer.ac.uk/training/virtual/. Michael Lange, Imperial College, London Scalable file I/O and efficient domain topology management present important challenges for many scientific applications if they are to fully utilise HPC resources. The use of composable abstractions allows a wide variety of application codes to utilise a range of mesh file formats while automatically inheriting the benefits of well-known performance optimisations. In this talk we give an overview of PETSc's DMPlex domain topology abstraction and it's recent integration with the Fluidity CFD application and the Firedrake automated finite element system. We highlight how both applications utilise DMPlex's mesh management capabilities to perform common mesh management tasks and data layout optimisation, while promoting application interoperability through shared file I/O routines. To join, click "Launch" at http://www.archer.ac.uk/training/virtual/. For more detailed information, including software requirements, seehttp://www.archer.ac.uk/training/virtual/blackboard_collaborate.php ----------------------------------------------------------------- -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-users] MUMPS error and superLU error
by venkatesh g 22 Jun '15

22 Jun '15

Hi Matt, users, Thanks for the info. Do you also use Petsc and Slepc with MUMPS ? I get into the segmentation error if I increase my matrix size. Can you suggest other software for direct solver for QR in parallel since as LU may not be good for a singular B matrix in Ax=lambda Bx ? I am attaching the working version mumps log. My matrix size here is around 47000x47000. If I am not wrong, the memory usage per core is 272MB. Can you tell me if I am wrong ? or really if its light on memory for this matrix ? Thanks cheers, Venkatesh On Fri, May 29, 2015 at 4:00 PM, Matt Landreman <matt.landreman(a)gmail.com> wrote: > Dear Venkatesh, > > As you can see in the error log, you are now getting a segmentation fault, > which is almost certainly a separate issue from the info(1)=-9 memory > problem you had previously. Here is one idea which may or may not help. > I've used mumps on the NERSC Edison system, and I found that I sometimes > get segmentation faults when using the default Intel compiler. When I > switched to the cray compiler the problem disappeared. So you could perhaps > try a different compiler if one is available on your system. > > Matt > On May 29, 2015 4:04 AM, "venkatesh g" <venkateshgk.j(a)gmail.com> wrote: > >> Hi Matt, >> >> I did what you told and read the manual of that CNTL parameters. I solve >> for that with CNTL(1)=1e-4. It is working. >> >> But it was a test matrix with size 46000x46000. Actual matrix size is >> 108900x108900 and will increase in the future. >> >> I get this error of memory allocation failed. And the binary matrix size >> of A is 20GB and B is 5 GB. >> >> Now I submit this in 240 processors each 4 GB RAM and also in 128 >> Processors with total 512 GB RAM. >> >> In both the cases, it fails with the following error like memory is not >> enough. But for 90000x90000 size it had run serially in Matlab with <256 GB >> RAM. >> >> Kindly let me know. >> >> Venkatesh >> >> On Tue, May 26, 2015 at 8:02 PM, Matt Landreman <matt.landreman(a)gmail.com >> > wrote: >> >>> Hi Venkatesh, >>> >>> I've struggled a bit with mumps memory allocation too. I think the >>> behavior of mumps is roughly the following. First, in the "analysis step", >>> mumps computes a minimum memory required based on the structure of nonzeros >>> in the matrix. Then when it actually goes to factorize the matrix, if it >>> ever encounters an element smaller than CNTL(1) (default=0.01) in the >>> diagonal of a sub-matrix it is trying to factorize, it modifies the >>> ordering to avoid the small pivot, which increases the fill-in (hence >>> memory needed). ICNTL(14) sets the margin allowed for this unanticipated >>> fill-in. Setting ICNTL(14)=200000 as in your email is not the solution, >>> since this means mumps asks for a huge amount of memory at the start. >>> Better would be to lower CNTL(1) or (I think) use static pivoting >>> (CNTL(4)). Read the section in the mumps manual about these CNTL >>> parameters. I typically set CNTL(1)=1e-6, which eliminated all the >>> INFO(1)=-9 errors for my problem, without having to modify ICNTL(14). >>> >>> Also, I recommend running with ICNTL(4)=3 to display diagnostics. Look >>> for the line in standard output that says "TOTAL space in MBYTES for IC >>> factorization". This is the amount of memory that mumps is trying to >>> allocate, and for the default ICNTL(14), it should be similar to matlab's >>> need. >>> >>> Hope this helps, >>> -Matt Landreman >>> University of Maryland >>> >>> On Tue, May 26, 2015 at 10:03 AM, venkatesh g <venkateshgk.j(a)gmail.com> >>> wrote: >>> >>>> I posted a while ago in MUMPS forums but no one seems to reply. >>>> >>>> I am solving a large generalized Eigenvalue problem. >>>> >>>> I am getting the following error which is attached, after giving the >>>> command: >>>> >>>> /cluster/share/venkatesh/petsc-3.5.3/linux-gnu/bin/mpiexec -np 64 >>>> -hosts compute-0-4,compute-0-6,compute-0-7,compute-0-8 ./ex7 -f1 a72t -f2 >>>> b72t -st_type sinvert -eps_nev 3 -eps_target 0.5 -st_ksp_type preonly >>>> -st_pc_type lu -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14 >>>> 200000 >>>> >>>> IT IS impossible to allocate so much memory per processor.. it is >>>> asking like around 70 GB per processor. >>>> >>>> A serial job in MATLAB for the same matrices takes < 60GB. >>>> >>>> After trying out superLU_dist, I have attached the error there also >>>> (segmentation error). >>>> >>>> Kindly help me. >>>> >>>> Venkatesh >>>> >>>> >>>> >>> >>

4 7

Very low CFL number for PETSC's GS
by Orxan Shibliyev 22 Jun '15

22 Jun '15

I wanted to compare my own GS and the one of PETSC's. I used KSPRICHARDSON with PCSOR to obtain GS. I tested my GS with CFL=40 and solved Ax=b problem successfully and fast. However, PETSC failed to solve at CFL=40 and it gives an answer only for very low CFL numbers such as 0.1. Of course, the convergence was very slow. My question is that if A and b and the implementations are the same why PETSC fails to solve with same CFL number as for my GS solver? PS1: My GS solver is a plain GS, no fancy stuff. PS2: A is a block matrix (MATSEQBAIJ) for PETSC. Also, the process is sequential.

2 1

Petsc and cmake
by Michele Rosso 21 Jun '15

21 Jun '15

Hi, I am trying to move to cmake to build my code. How would you suggest to handle the dependency on PETSc? Currently my makefile relies on the PETSc variables for building the all code, namely FLINKER, CLINKER and so on. I found the FindPETSc.cmake module and I successfully used it but it does not import the aforementioned variables. Any suggestion on the matter is greatly appreciated. Thanks, Michele

2 3

Re: [petsc-users] About MatView
by Barry Smith 21 Jun '15

21 Jun '15

Eric, > On Jun 20, 2015, at 1:42 AM, Longyin Cui <cuilongyin(a)gmail.com> wrote: > > OMG you are real!!! > OK, All my error message looks like this: > PETSc Error ... exiting > -------------------------------------------------------------------------- > mpirun has exited due to process rank 13 with PID 1816 on > node cnode174.local exiting improperly. There are two reasons this could occur: > > 1. this process did not call "init" before exiting, but others in > the job did. This can cause a job to hang indefinitely while it waits > for all processes to call "init". By rule, if one process calls "init", > then ALL processes must call "init" prior to termination. > > 2. this process called "init", but exited without calling "finalize". > By rule, all processes that call "init" MUST call "finalize" prior to > exiting or it will be considered an "abnormal termination" > > This may have caused other processes in the application to be > terminated by signals sent by mpirun (as reported here). This crash doesn't seem to have anything to do in particular with code below. Do the PETSc examples run in parallel? Does your code that you ran have a PetscInitialize() in it? What about running on two processors, does that work? > > You are right, I did use PETSC_COMM_SELE, and when I used PETSC_COMM_WORLD alone I could get the entire matrix printed. But, this is one whole matrix in one file. The reason I used: PetscViewerASCIIOpen ( PETSC_COMM_SELF, "mat.output", &viewer); and MatView (matrix,viewer); was because it says "Each processor can instead write its own independent output by specifying the communicator PETSC_COMM_SELF". Yikes, this is completely untrue and has been for decades. We have no way of saving the matrix in its parts; you cannot use use a PETSC_COMM_SELF viewer with a parallel matrix. Sorry about the wrong information in the documentation; I have fixed it. Why can't you just save the matrix in one file and then compare it? We don't provide a way to save objects one part per process because we think it is a bad model for parallel computing since the result depends on the number of processors you are using. Barry > > Also, I tried this as well, which failed, same error message : > PetscMPIInt my_rank; > MPI_Comm_rank(MPI_COMM_WORLD, &my_rank); > string str = KILLME(my_rank); // KILLME is a int to string function... > const char * c = str.c_str(); > PetscViewer viewer; > PetscViewerASCIIOpen(PETSC_COMM_WORLD, c , &viewer); > MatView(impOP,viewer); //impOP is the huge matrix. > PetscViewerDestroy(&viewer); > > I was trying to generate 16 files recording each matrix hold by each processor so I can conpare them with the big matrix...so, what do you think > > Thank you very muuuch. > > Longyin Cui (or you know me as Eric); > Student from C.S. division; > Cell: 7407047169; > return 0; > > > On Sat, Jun 20, 2015 at 1:34 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > You need to cut and paste and send the entire error message: "not working" makes it very difficult for us to know what has gone wrong. > Based on the code fragment you sent I guess one of your problems is that the viewer communicator is not the same as the matrix communicator. Since the matrix is on 16 processors (I am guessing PETSC_COMM_WORLD) the viewer communicator must also be the same (also PETSC_COMM_WORLD). > The simplest code you can do is > > > PetscViewerASCIIOpen(PETSC_COMM_WORLD,"stdout",&viewer); > > MatView(impOP,viewer); > > but you can get a similar effect with the command line option -mat_view and not write any code at all (the less code you have to write the better). > > Barry > > > > On Jun 19, 2015, at 10:42 PM, Longyin Cui <cuilongyin(a)gmail.com> wrote: > > > > Hi dear whoever reads this: > > > > I have a quick question: > > After matrix assembly, suppouse I have matrix A. Assuming I used 16 processors, if I want each processor to print out their local contents of the A, how do I proceed? (I simply want to know how the matrix is stored from generating to communicating to solving, so I get to display all the time to get better undersdtanding) > > > > I read the examples, and I tried things like below and sooooo many different codes from examples, it still is not working. > > PetscViewer viewer; > > PetscMPIInt my_rank; > > MPI_Comm_rank(PETSC_COMM_WORLD,&my_rank); > > PetscPrintf(MPI_COMM_SELF,"[%d] rank\n",my_rank); > > PetscViewerASCIIOpen(MPI_COMM_SELF,NULL,&viewer); > > PetscViewerPushFormat(viewer,PETSC_VIEWER_ASCII_INFO); > > MatView(impOP,viewer); > > > > Plea......se give me some hints > > > > Thank you so very much! > > > > > > Longyin Cui (or you know me as Eric); > > Student from C.S. division; > > Cell: 7407047169; > > return 0; > > > >

3 2

About MatView
by Longyin Cui 20 Jun '15

20 Jun '15

Hi dear whoever reads this: I have a quick question: After matrix assembly, suppouse I have matrix A. Assuming I used 16 processors, if I want each processor to print out their local contents of the A, how do I proceed? (I simply want to know how the matrix is stored from generating to communicating to solving, so I get to display all the time to get better undersdtanding) I read the examples, and I tried things like below and sooooo many different codes from examples, it still is not working. PetscViewer viewer; PetscMPIInt my_rank; MPI_Comm_rank(PETSC_COMM_WORLD,&my_rank); PetscPrintf(MPI_COMM_SELF,"[%d] rank\n",my_rank); PetscViewerASCIIOpen(MPI_COMM_SELF,NULL,&viewer); PetscViewerPushFormat(viewer,PETSC_VIEWER_ASCII_INFO); MatView(impOP,viewer); Plea......se give me some hints Thank you so very much! Longyin Cui (or you know me as Eric); Student from C.S. division; Cell: 7407047169; return 0;

2 1

from standard fortran program to PETSc
by Anthony Haas 19 Jun '15

19 Jun '15

Hi, I have a Fortran90 program that solves a complex linear generalized eigenvalue problem (GEVP) using standard fortran 90 programming: Subroutines, modules, allocatable arrays, real(8), int,... This program uses Lapack to solve the GEVP. The program is mainly made off: 1) set dimensions of problem and initialize arrays,... 2) compute the baseflow (for instance boundary layer flow) 3) build the (stability) complex generalized eigenvalue problem ==> build (dense) matrices A and B 4) solve the GEVP with Lapack Now I want to use PETSc + SLEPc to use sparse matrices. Do I need to rewrite/modify everything in terms of PETSc variables as follows: - int -> PetscInt - real(8) -> PetscScalar - complex*16 -> PetscScalar or is it possible to reuse all that F90 code? For instance I have a similarity solver that computes Blasius solution. If that similarity solver provides me with u and v velocities in terms of standard fortran90 real(8) variables, how should I do to use these variables to build my complex matrix? Should I convert them to Petsc variables? How? what should I do with my Fortran90 allocatable arrays? real(dp),allocatable,dimension(:,:) :: u--> PetscScalar,allocatable,dimension(:,:) :: u ???? Thanks a lot, Anthony

3 3

Re: [petsc-users] Reg : Petsc installation failed
by Satish Balay 19 Jun '15

19 Jun '15

Please keep the conversation on list. [or use petsc-maint] On Fri, 19 Jun 2015, Manish Kumar K wrote: > > Hi Satish, > > I did as per your following instructions mentioned below for PETSC to install on my system using MS compilers , > For some reasons its flagging me the same error . > I have attached configure.log file with this email and snap shot of error in bash. > > Steps followed > 1)I installed Cygwin freshly and used latest PETSc release 3.6. Hm - I see latest cygwin as: CYGWIN_NT-6.1 ps4 2.0.4(0.287/5/3) 2015-06-09 12:22 x86_64 Cygwin However configure.log has: ('CYGWIN_NT-6.1', 'EESBLRW106', '2.0.2(0.287/5/3)', '2015-05-08 17:00', 'x86_64', '') a slightly older version. Are you sure you've install a fresh cygwin? Also can you try the following? Copy/paste and e-mail the complete output. cd src/benchmarks /cygdrive/d/Software/Cygwin64/package/petsc/bin/win32fe/win32fe cl -c sizeof.c echo $? /cygdrive/d/Software/Cygwin64/package/petsc/bin/win32fe/win32fe cl -o sizeof.exe sizeof.c echo $? ./sizeof.exe Satish ----------- balay@ps4 ~/petsc/src/benchmarks $ ~/petsc/bin/win32fe/win32fe.exe cl -c sizeof.c sizeof.c balay@ps4 ~/petsc/src/benchmarks $ echo $? 0 balay@ps4 ~/petsc/src/benchmarks $ ~/petsc/bin/win32fe/win32fe.exe cl -o sizeof.exe sizeof.o balay@ps4 ~/petsc/src/benchmarks $ echo $? 0 balay@ps4 ~/petsc/src/benchmarks $ ./sizeof.exe long double : 8 double : 8 int : 4 char : 1 short : 2 long : 4 long long : 8 int * : 8 size_t : 8 balay@ps4 ~/petsc/src/benchmarks $ > And followed same command as per the instructions mentioned at this link > > http://www.mcs.anl.gov/petsc/documentation/installation.html#windows > > Kindly help me out in this . > > Let me know if I missed anything. > > > Regards > Manish K > _______________________________________________________________________________________________________________________________________ > > -----Original Message----- > From: Satish Balay [mailto:[email protected]] > Sent: Thursday, June 18, 2015 7:45 PM > To: Manish Kumar K > Cc: petsc-users(a)mcs.anl.gov > Subject: Re: [petsc-users] Reg : Petsc installation failed > > >>>>>>> > Executing: /package/petsc/petsc-3.5.3/bin/win32fe/win32fe cl -c -o /cygdrive/c/Users/20055335/AppData/Local/Temp/petsc-leuQLz/config.setCompilers/conftest.o -I/cygdrive/c/Users/20055335/AppData/Local/Temp/petsc-leuQLz/config.setCompilers /cygdrive/c/Users/20055335/AppData/Local/Temp/petsc-leuQLz/config.setCompilers/conftest.c > Possible ERROR while running compiler: exit code 50176 <<<<< > > For some reason the compiler is returning error codes. Just to eliminate cygwin from the equation - can you do the following: > > 1. reboot the machine > 2. [make sure you don't start any cygwin processes] 3. run cygwin setup [it defaults to update mode] - and run it to completion. > It should run rebaseall at the end. > 4. Use petsc-3.6 > > And rerun configure [from compiler-cmd,bash per instructions] > > Satish > > On Thu, 18 Jun 2015, Manish Kumar K wrote: > > > Dear PETSc Team, > > > > > > > > I am configuring and trying to install PETSc libraries using > > Cygwin shell for windows > > > > > > > > I checked instructions at: > > > > http://www.mcs.anl.gov/petsc/documentation/installation.html#windows > > > > And invoked following command > > > > > > > > Step 1: > > > > ./configure --with-cc='win32fe cl' --with-fc=0 > > --download-f2cblaslapack=1 > > > > > > C compiler you provided with -with-cc=win32fe cl does not work. > > Cannot compile C with /package/petsc/petsc-3.5.3/bin/win32fe/win32fe cl. > > > > > > > > I am sending you the log file for this . > > > > > > > > Please help me out in this . > > > > > > > > Regards > > > > Manish K > > > > [https://relayq.larsentoubro.com/DigitalSignature2015.jpg] > > > > L&T Technology Services Ltd > > > > www.LntTechservices.com<http://www.lnttechservices.com/> > > > > This Email may contain confidential or privileged information for the intended recipient (s). If you are not the intended recipient, please do not use or disseminate the information, notify the sender and delete it from your system. > > > > [https://relayq.larsentoubro.com/DigitalSignature2015.jpg] > L&T Technology Services Ltd > > www.LntTechservices.com<http://www.lnttechservices.com/> > > This Email may contain confidential or privileged information for the intended recipient (s). If you are not the intended recipient, please do not use or disseminate the information, notify the sender and delete it from your system. >

1 0

arkimex rejecting all dt in petsc 3.6.0
by Italo Tasso 18 Jun '15

18 Jun '15

I just upgraded to 3.6.0 and my code stopped working. All dt are rejected. I used the same configure line, same code, same everything. With 3.5.4 I get: 0 TS dt 1e-06 time 0 0 SNES Function norm 2.549981005316e+05 1 SNES Function norm 6.107056905987e-03 2 SNES Function norm 1.483881932064e-10 3 SNES Function norm 9.122873794272e-11 Nonlinear solve converged due to CONVERGED_FNORM_ABS iterations 3 0 SNES Function norm 7.790429171165e+04 1 SNES Function norm 7.289068747803e-04 2 SNES Function norm 8.227639633330e-11 Nonlinear solve converged due to CONVERGED_FNORM_ABS iterations 2 0 SNES Function norm 1.154356516184e+05 1 SNES Function norm 2.309925413255e-03 2 SNES Function norm 6.382141981406e-11 Nonlinear solve converged due to CONVERGED_FNORM_ABS iterations 2 TSAdapt 'basic': step 0 accepted t=0 + 1.000e-06 wlte=0.000654 family='arkimex' scheme=0:'3' dt=1.000e-05 With 3.6.0 I get: 0 TS dt 1e-06 time 0 0 SNES Function norm 2.549981005316e+05 1 SNES Function norm 6.107056925316e-03 2 SNES Function norm 1.519319591792e-10 3 SNES Function norm 9.070104116945e-11 Nonlinear solve converged due to CONVERGED_FNORM_ABS iterations 3 0 SNES Function norm 7.790429171165e+04 1 SNES Function norm 6.942541792651e-04 2 SNES Function norm 8.458781909516e-11 Nonlinear solve converged due to CONVERGED_FNORM_ABS iterations 2 0 SNES Function norm 1.154356516184e+05 1 SNES Function norm 2.287202942961e-03 2 SNES Function norm 6.585201377573e-11 Nonlinear solve converged due to CONVERGED_FNORM_ABS iterations 2 TSAdapt 'basic': step 0 rejected t=0 + 1.000e-06 wlte= 324 family='arkimex' scheme=0:'3' dt=1.311e-07 Any ideas? I attached the full output. Options I use: -ts_view -ts_type arkimex -ts_arkimex_fully_implicit -ts_adapt_monitor -ts_monitor -snes_monitor -snes_converged_reason -ksp_type preonly -pc_type lu -pc_factor_mat_solver_package mumps -snes_rtol 0 -snes_atol 1e-10 -snes_stol 0

4 6