June 2015 - petsc-users - mailman3.cels.anl.gov

Parallel IO of DMPlex
by Justin Chang 29 Jun '15

29 Jun '15

Hi everyone, I see that parallel IO of various input mesh formats for DMPlex is not currently supported. However, is there a way to read a custom mesh file in parallel? For instance, I have a binary data file formatted like this: <No. of elements> <No. of vertices> <No. of nodes per element> <Spatial dimension> <Connectivity array ... ....> <Coordinates array ... ....> <No. of exterior nodes> <List of exterior nodes ... ....> Reading this will allow me to create a DMPlex with DMPlexCreateFromDAG(). Currently, only the root process reads this binary mesh file and creates the DMPlex whereas the other processes create an empty DMPlex. Then all processes invoke DMPlexDistribute(). This one-to-all distribution seems to be a bottleneck, and I want to know if it's possible to have each process read a local portion of the connectivity and coordinates arrays and let DMPlex/METIS/ParMETIS handle the load balancing and redistribution. Intuitively this would be easy to write, but again I want to know how to do this through leveraging the functions and routines within DMPlex. Thanks, Justin

2 2

PetscBinaryRead and quad precision
by Justin Chang 27 Jun '15

27 Jun '15

Hi all, I need to run simulations that rely on several of my custom binary datafiles (written in 32 bit int and 64 bit doubles). These date files were generated from MATLAB. In my PETSc code I invoke PetscBinaryRead(...) into these binary files, which gives me mesh data, auxiliaries, etc. However, when I now configure with quad precision (--with-precision=__float128 and --download-f2cblaslapack) my PetscBinaryRead() functions give me segmentation violation errors. I am guessing this is because I have binary files written in double precision but have PETSc which reads in quad precision, meaning I will be reading past the end of these files due to the larger strides. So my question is, is there a way to circumvent this issue? That is, to read double-precision binary data into a PETSc program configured with quad-precision? Otherwise I would have to rewrite or redo all of my datafiles, which I would prefer not to do if possible. Thanks, Justin

3 4

DMDASetInterpolationType
by Michele Rosso 26 Jun '15

26 Jun '15

Hi, I am wondering if there is a way to use a custom interpolation routine for DMDA. The reason is that DMDA_Q0 does not support periodic BCs and I need that. Thanks, Michele

2 1

Parallel efficiency of the gmres solver with ASM
by Lei Shi 25 Jun '15

25 Jun '15

Hello, I'm trying to improve the parallel efficiency of gmres solve in my. In my CFD solver, Petsc gmres is used to solve the linear system generated by the Newton's method. To test its efficiency, I started with a very simple inviscid subsonic 3D flow as the first testcase. The parallel efficiency of gmres solve with asm as the preconditioner is very bad. The results are from our latest cluster. Right now, I'm only looking at the wclock time of the ksp_solve. 1. First I tested ASM with gmres and ilu 0 for the sub domain , the cpu time of 2 cores is almost the same as the serial run. Here is the options for this case -ksp_type gmres -ksp_max_it 100 -ksp_rtol 1e-5 -ksp_atol 1e-50 -ksp_gmres_restart 30 -ksp_pc_side right -pc_type asm -sub_ksp_type gmres -sub_ksp_rtol 0.001 -sub_ksp_atol 1e-30 -sub_ksp_max_it 1000 -sub_pc_type ilu -sub_pc_factor_levels 0 -sub_pc_factor_fill 1.9 The iteration numbers increase a lot for parallel run. coresiterationserrpetsc solve wclock timespeedupefficiency121.15E-0411.95125 2.05E-0210.51.010.50462.19E-027.641.390.34 2. Then I tested ASM with ilu 0 as the preconditoner only, the cpu time of 2 cores is better than the 1st test, but the speedup is still very bad. Here is the options i'm using -ksp_type gmres -ksp_max_it 100 -ksp_rtol 1e-5 -ksp_atol 1e-50 -ksp_gmres_restart 30 -ksp_pc_side right -pc_type asm -sub_pc_type ilu -sub_pc_factor_levels 0 -sub_pc_factor_fill 1.9 coresiterationserrpetsc solve cpu timespeedupefficiency1104.54E-0410.681211 9.55E-048.21.300.654123.59E-045.262.030.50 Those results are from a third order "DG" scheme with a very coarse 3D mesh (480 elements). I believe I should get some speedups for this test even on this coarse mesh. My question is why does the asm with a local solve take much longer time than the asm as a preconditioner only? Also the accuracy is very bad too I have tested changing the overlap of asm to 2, but make it even worse. If I used a larger mesh ~4000 elements, the 2nd case with asm as the preconditioner gives me a better speedup, but still not very good. coresiterationserrpetsc solve cpu timespeedupefficiency171.91E-0297.32127 2.07E-0264.941.50.74472.61E-0236.972.60.65 Attached are the log_summary dumped from petsc, any suggestions are welcome. I really appreciate it. Sincerely Yours, Lei Shi ---------

4 8

Re: [petsc-users] Varying TAO optimization solve iterations using BLMVM
by Jason Sarich 25 Jun '15

25 Jun '15

Hi Justin, I don't see anything obviously wrong that would be causing this variation in iterations due to number of processors. Is it at all feasible to send be an example code that reproduces the problem (perhaps a smaller version)? I'm still guessing the problem lies in numerical precision, it would be nice to find a way to avoid them. I don't think the job scheduling or compute nodes would affect this. > Basically I call TaoSolve at each time level. What I find strange is that > the number of TAO solve iterations vary at each time level for a given > number of processors Just for clarification, do you mean that for a given problem, you run the same problem several times the only difference being the number of processors, and that on each time step you get (close to) the same solution for each run, just with a different number of TAO iterations? If you run the same problem twice using the same number of processors, is the output identical? No OpenMP threads or GPU's? > 2) Sometimes, I get Tao Termination reason of -5, and from what I see from > the online documentation, it means the number of function evaluations > exceeds the maximum number of function evaluations. I only get this at > certain time levels, and it also varies when I change the number of > processors. This looks like the same issue, unless the number of iterations is hugely different (say it converges on some number of processors after 200 iterations, but still hasn't converged after 2000 on a different number). Jason On Tue, Jun 23, 2015 at 6:36 PM, Justin Chang <jychang48(a)gmail.com> wrote: > I was unable to do quad precision or even with 64 bit integers > because my data files rely on intricate binary files that have been written > in 32 bit. > > However, I noticed a couple things which are puzzling to me: > > 1) I am solving a transient problem using my own backward euler function. > Basically I call TaoSolve at each time level. What I find strange is that > the number of TAO solve iterations vary at each time level for a given > number of processors. The solution is roughly the same when I change the > number of processors. Any idea why this is happening, or might this have > more to do with the job scheduling/compute nodes on my HPC machine? > > 2) Sometimes, I get Tao Termination reason of -5, and from what I see > from the online documentation, it means the number of function evaluations > exceeds the maximum number of function evaluations. I only get this at > certain time levels, and it also varies when I change the number of > processors. > > I can understand the number of iterations going down the further in time > i go (this is due to the nature of my problem), but I am not sure why the > above two observations are happening. Any thoughts? > > Thanks, > Justin > > On Fri, Jun 19, 2015 at 11:52 AM, Justin Chang <jychang48(a)gmail.com> > wrote: > >> My code sort of requires HDF5 so installing quad precision might be a >> little difficult. I could try to work around this but that might take some >> effort. >> >> In the mean time, is there any other potential explanation or >> alternative to figuring this out? >> >> Thanks, >> Justin >> >> >> On Thursday, June 18, 2015, Matthew Knepley <knepley(a)gmail.com> wrote: >> >>> On Thu, Jun 18, 2015 at 1:52 PM, Jason Sarich <jason.sarich(a)gmail.com> >>> wrote: >>> >>>> BLMVM doesn't use a KSP or preconditioner, it updates using the >>>> L-BFGS-B formula >>>> >>> >>> Then this sounds like a bug, unless one of the constants is partition >>> dependent. >>> >>> Matt >>> >>> >>>> On Thu, Jun 18, 2015 at 1:45 PM, Matthew Knepley <knepley(a)gmail.com> >>>> wrote: >>>> >>>>> On Thu, Jun 18, 2015 at 12:15 PM, Jason Sarich < >>>>> jason.sarich(a)gmail.com> wrote: >>>>> >>>>>> Hi Justin, >>>>>> >>>>>> I can't tell for sure why this is happening, have you tried using >>>>>> quad precision to make sure that numerical cutoffs isn't the problem? >>>>>> >>>>>> 1 The Hessian being approximate and the resulting implicit >>>>>> computation is the source of the cutoff, but would not be causing different >>>>>> convergence rates in infinite precision. >>>>>> >>>>>> 2 the local size may affect load balancing but not the resulting >>>>>> norms/convergence rate. >>>>>> >>>>> >>>>> This sounds to be like the preconditioner is dependent on the >>>>> partition. Can you send -tao_view -snes_view >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Jason >>>>>> >>>>>> >>>>>> On Thu, Jun 18, 2015 at 10:44 AM, Justin Chang <jychang48(a)gmail.com> >>>>>> wrote: >>>>>> >>>>>>> I solved a transient diffusion across multiple cores using TAO >>>>>>> BLMVM. When I simulate the same problem but on different numbers of >>>>>>> processing cores, the number of solve iterations change quite drastically. >>>>>>> The numerical solution is the same, but these changes are quite vast. I >>>>>>> attached a PDF showing a comparison between KSP and TAO. KSP remains >>>>>>> largely invariant with number of processors but TAO (with bounded >>>>>>> constraints) fluctuates. >>>>>>> >>>>>>> My question is, why is this happening? I understand that >>>>>>> accumulation of numerical round-offs may attribute to this, but the >>>>>>> differences seem quite vast to me. My initial thought was that >>>>>>> >>>>>>> 1) the Hessian is only projected and not explicitly computed, >>>>>>> which may have something to do with the rate of convergence >>>>>>> >>>>>>> 2) local problem size. Certain regions of my domain have different >>>>>>> number of "violations" which need to be corrected by the bounded >>>>>>> constraints so the rate of convergence depends on how these regions are >>>>>>> partitioned? >>>>>>> >>>>>>> Any thoughts? >>>>>>> >>>>>>> Thanks, >>>>>>> Justin >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >

2 1

petsc4py Build Problem
by Mikhail Khodak 24 Jun '15

24 Jun '15

Hello, I am trying to build petsc4py-3.5.1 using Cygwin on 64-bit Windows 7. I have built PETSc 3.5.4 with shared and dynamic libraries using mpich2-1.2.1 and successfully ran the installation tests. I am using Python 2.7 and NumPy 1.9.2 and have installed mpi4py. However, when I attempt to install petsc4py (both with pip and distutils) I get a mpicc compiler error due to undefined references/symbols. I have attached the output of running pip install petsc petsc4py --allow-external petsc --allow-external petsc4py Thank you in advance for any help, Mikhail Khodak

4 13

DMDAVecGetArray vs DMDAVecGetArrayRead
by Gideon Simpson 24 Jun '15

24 Jun '15

Is there any difference between these two routines? Is there a reason to use one over the other? -gideon

2 1

nan after VecDuplicate
by Orxan Shibliyev 24 Jun '15

24 Jun '15

I get a nan for solution vector, 'x' right after VecDuplicate (x, &b) where, 'b' is a Vec. This happens only with multiple processes while sequential one works perfectly. I even tried to set 'x' with VecZeroEntries(x) but didn't help.

2 1

jacobians for multicomponent problems
by Gideon Simpson 24 Jun '15

24 Jun '15

Suppose I have a problem for which the field variable has multiple components, in this case real and imaginary parts, and I have created a DM with two degrees of freedom. When assembling a Jacobian associated with a nonlinear problem on this data, is there any data management accounting for the different components that makes this “easier?” -gideon

3 5

双喜软件外贸工作上的好帮手您值得拥有
by Tayloruz9 Herrerazid89（通过 Google 表单） 24 Jun '15

24 Jun '15

各位外贸朋友们好，我们双喜外贸客户搜索与开发软件集合了全世界的主要搜索引擎200个国家下的 700多个当地的语言搜索引擎，软件主要通过您的产品关键词进入网络上快速有效地搜索到您行业的目标客户，软件原理完全模拟手工在google等搜索引擎上搜索客户户一样的，软件件一天可以搜索到上万的客户信息，通过软件的邮件群发模块投递开发信，群发功能采用了多邮件主题多邮件内容多发信账号，随机组合完全模拟手工一对一的投递开发信。全球通用搜索(18个世界主流引擎)+全球700多个国家搜索引擎 1880元/季 2880元/年详细软件信息请咨询QQ：61737441---双喜软件加QQ后可在线利用贵司产品关键词，直接演示搜索效果我已邀请您填写表单未命名的表单。要填写此表单，请访问： https://docs.google.com/forms/d/1sOCrOpzEdb1oWM-2JR09ecMGQPArX2OM6cYJEYg1lH…

1 0