April 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Regression in manualpages generation?
by Satish Balay 06 Apr '21

06 Apr '21

https://www.mcs.anl.gov/petsc/petsc-current/ is from petsc-3.15.0 - and is normally gets updated with the next patch release - i.e petsc-3.15.1. [ok - I'll update now with current release snapshot] Also dev pages should get updated daily [there is a bug here that I should fix] With the latest snapshots - both the pages appear to list MatGetRow_MPIDense https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetRow.… https://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatGetRow.html Satish On Tue, 6 Apr 2021, Pierre Jolivet wrote: > Thanks for the recent MR. > This sadly only partially addresses the issue. > https://www.mcs.anl.gov/petsc/petsc-3.14.6/docs/manualpages/Mat/MatGetRow.h… <https://www.mcs.anl.gov/petsc/petsc-3.14.6/docs/manualpages/Mat/MatGetRow.h…> correctly has MatGetRow_MPIDense while https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetRow.… <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetRow.…> hasn’t. > > Thanks, > Pierre > > > On 3 Apr 2021, at 1:24 PM, Patrick Sanan <patrick.sanan(a)gmail.com> wrote: > > > > I think this was likely a non-portable regex that was being used to match the function signatures, > > causing it to miss anything with an internal "s" in the function name. Here's a quick fix for the release branch: > > > > https://gitlab.com/petsc/petsc/-/merge_requests/3813 <https://gitlab.com/petsc/petsc/-/merge_requests/3813> > > > > (This sort of thing will become easier to work with, soon, as we'll have an automated doc build which can be examined for each MR.) > > > > P.S. nice email address! > > > >> Am 03.04.2021 um 11:19 schrieb Patrick Sanan <patrick.sanan(a)gmail.com <mailto:[email protected]>>: > >> > >> That isn't expected - thanks for pointing this out! I can reproduce locally so will take a look. > >> > >>> Am 03.04.2021 um 11:10 schrieb Pierre Jolivet <pierre(a)joliv.et <mailto:[email protected]>>: > >>> > >>> Hello, > >>> https://www.mcs.anl.gov/petsc/petsc-3.14/docs/manualpages/Mat/MatTranspose.… <https://www.mcs.anl.gov/petsc/petsc-3.14/docs/manualpages/Mat/MatTranspose.…> lists available implementations while https://www.mcs.anl.gov/petsc/petsc-3.15/docs/manualpages/Mat/MatTranspose.… <https://www.mcs.anl.gov/petsc/petsc-3.15/docs/manualpages/Mat/MatTranspose.…> doesn’t. > >>> Is this expected? > >>> > >>> Thanks, > >>> Pierre > >> > > > >

1 0

Re: [petsc-dev] Regression in manualpages generation?
by Pierre Jolivet 06 Apr '21

06 Apr '21

Thanks for the recent MR. This sadly only partially addresses the issue. https://www.mcs.anl.gov/petsc/petsc-3.14.6/docs/manualpages/Mat/MatGetRow.h… <https://www.mcs.anl.gov/petsc/petsc-3.14.6/docs/manualpages/Mat/MatGetRow.h…> correctly has MatGetRow_MPIDense while https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetRow.… <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetRow.…> hasn’t. Thanks, Pierre > On 3 Apr 2021, at 1:24 PM, Patrick Sanan <patrick.sanan(a)gmail.com> wrote: > > I think this was likely a non-portable regex that was being used to match the function signatures, > causing it to miss anything with an internal "s" in the function name. Here's a quick fix for the release branch: > > https://gitlab.com/petsc/petsc/-/merge_requests/3813 <https://gitlab.com/petsc/petsc/-/merge_requests/3813> > > (This sort of thing will become easier to work with, soon, as we'll have an automated doc build which can be examined for each MR.) > > P.S. nice email address! > >> Am 03.04.2021 um 11:19 schrieb Patrick Sanan <patrick.sanan(a)gmail.com <mailto:[email protected]>>: >> >> That isn't expected - thanks for pointing this out! I can reproduce locally so will take a look. >> >>> Am 03.04.2021 um 11:10 schrieb Pierre Jolivet <pierre(a)joliv.et <mailto:[email protected]>>: >>> >>> Hello, >>> https://www.mcs.anl.gov/petsc/petsc-3.14/docs/manualpages/Mat/MatTranspose.… <https://www.mcs.anl.gov/petsc/petsc-3.14/docs/manualpages/Mat/MatTranspose.…> lists available implementations while https://www.mcs.anl.gov/petsc/petsc-3.15/docs/manualpages/Mat/MatTranspose.… <https://www.mcs.anl.gov/petsc/petsc-3.15/docs/manualpages/Mat/MatTranspose.…> doesn’t. >>> Is this expected? >>> >>> Thanks, >>> Pierre >> >

1 0

回复： Petsc: Error code 1
by Chen Gang 06 Apr '21

06 Apr '21

See the attachements. alltest.log is on a machine with 96 cores, ARM, with FC,gcc9.3.5,mpich3.4.1,fblaslapack; 6 failures                                  alltest2.log is on an intel machine with 40  cores,x86, without FC; icc&mkl& intel mpi; only 1 failure ------------------ 原始邮件 ------------------ 发件人: "petsc-dev" <balay(a)mcs.anl.gov>; 发送时间: 2021年4月6日(星期二) 中午12:38 收件人: "petsc-dev"<petsc-dev(a)mcs.anl.gov>; 抄送: "Chen Gang"<569615491(a)qq.com>;"cglwdm"<cglwdm(a)scu.edu.cn>; 主题: Re: [petsc-dev] Petsc: Error code 1 Note: do not use '-j' with alltests. And run alltests on both machines [but *not* at the same time on machines] and send us logs from both the runs. Satish On Mon, 5 Apr 2021, Satish Balay wrote: > Try: > > make alltests TIMEOUT=600 > > And send us the complete log (alltests.log) > > Satish > > On Tue, 6 Apr 2021, Chen Gang wrote: > > > Dear sir, > > > > > > The result of make check is OK. And I do set the timeout to a larger value, which keeps me from getting timeout error. The thing is I have two machines. And I get the error code 1 in different tests on different machines.I don’t know what is error code1. What case this? How can I fix the failure tests. > > > > > > ------------------ Original ------------------ > > From: Satish Balay <balay(a)mcs.anl.gov&gt; > > Date: Tue,Apr 6,2021 0:18 PM > > To: Chen Gang <569615491(a)qq.com&gt; > > Cc: petsc-dev <petsc-dev(a)mcs.anl.gov&gt;, cglwdm <cglwdm(a)scu.edu.cn&gt; > > Subject: Re: [petsc-dev] Petsc: Error code 1 >

1 0

回复： Petsc: Error code 1
by Chen Gang 06 Apr '21

06 Apr '21

Dear Professor， Attachment is the alltests.log file. Best, Gang ------------------ 原始邮件 ------------------ 发件人: "petsc-dev" <balay(a)mcs.anl.gov>; 发送时间: 2021年4月6日(星期二) 中午12:38 收件人: "petsc-dev"<petsc-dev(a)mcs.anl.gov>; 抄送: "Chen Gang"<569615491(a)qq.com>;"cglwdm"<cglwdm(a)scu.edu.cn>; 主题: Re: [petsc-dev] Petsc: Error code 1 Note: do not use '-j' with alltests. And run alltests on both machines [but *not* at the same time on machines] and send us logs from both the runs. Satish On Mon, 5 Apr 2021, Satish Balay wrote: > Try: > > make alltests TIMEOUT=600 > > And send us the complete log (alltests.log) > > Satish > > On Tue, 6 Apr 2021, Chen Gang wrote: > > > Dear sir, > > > > > > The result of make check is OK. And I do set the timeout to a larger value, which keeps me from getting timeout error. The thing is I have two machines. And I get the error code 1 in different tests on different machines.I don’t know what is error code1. What case this? How can I fix the failure tests. > > > > > > ------------------ Original ------------------ > > From: Satish Balay <balay(a)mcs.anl.gov&gt; > > Date: Tue,Apr 6,2021 0:18 PM > > To: Chen Gang <569615491(a)qq.com&gt; > > Cc: petsc-dev <petsc-dev(a)mcs.anl.gov&gt;, cglwdm <cglwdm(a)scu.edu.cn&gt; > > Subject: Re: [petsc-dev] Petsc: Error code 1 >

1 0

Re: [petsc-dev] -with-kokkos-cuda-arch=AMPERE80 nonsense
by Jeff Hammond 06 Apr '21

06 Apr '21

> > > Generically, independent of Kokkos, ideally I would run a single > precompiled NVIDIA program that gave me all the information about the > current hardware I was running and that would provide in simple format > exactly the information I needed to configure PETSc, Kokkos etc for THAT > system. > I will try to write something for you tomorrow. For NVIDIA hardware, the sole dependency will be nvcc. Jeff -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [petsc-dev] Petsc: Error code 1
by Satish Balay 05 Apr '21

05 Apr '21

Note: do not use '-j' with alltests. And run alltests on both machines [but *not* at the same time on machines] and send us logs from both the runs. Satish On Mon, 5 Apr 2021, Satish Balay wrote: > Try: > > make alltests TIMEOUT=600 > > And send us the complete log (alltests.log) > > Satish > > On Tue, 6 Apr 2021, Chen Gang wrote: > > > Dear sir, > > > > > > The result of make check is OK. And I do set the timeout to a larger value, which keeps me from getting timeout error. The thing is I have two machines. And I get the error code 1 in different tests on different machines.I don’t know what is error code1. What case this? How can I fix the failure tests. > > > > > > ------------------ Original ------------------ > > From: Satish Balay <balay(a)mcs.anl.gov> > > Date: Tue,Apr 6,2021 0:18 PM > > To: Chen Gang <569615491(a)qq.com> > > Cc: petsc-dev <petsc-dev(a)mcs.anl.gov>, cglwdm <cglwdm(a)scu.edu.cn> > > Subject: Re: [petsc-dev] Petsc: Error code 1 >

1 0

Re: [petsc-dev] Petsc: Error code 1
by Satish Balay 05 Apr '21

05 Apr '21

Try: make alltests TIMEOUT=600 And send us the complete log (alltests.log) Satish On Tue, 6 Apr 2021, Chen Gang wrote: > Dear sir, > > > The result of make check is OK. And I do set the timeout to a larger value, which keeps me from getting timeout error. The thing is I have two machines. And I get the error code 1 in different tests on different machines.I don’t know what is error code1. What case this? How can I fix the failure tests. > > > ------------------ Original ------------------ > From: Satish Balay <balay(a)mcs.anl.gov> > Date: Tue,Apr 6,2021 0:18 PM > To: Chen Gang <569615491(a)qq.com> > Cc: petsc-dev <petsc-dev(a)mcs.anl.gov>, cglwdm <cglwdm(a)scu.edu.cn> > Subject: Re: [petsc-dev] Petsc: Error code 1

1 0

Re: [petsc-dev] Petsc: Error code 1
by Chen Gang 05 Apr '21

05 Apr '21

Dear sir, The result of make check is OK. And I do set the timeout to a larger value, which keeps me from getting timeout error. The thing is I have two machines. And I get the error code 1 in different tests on different machines.I don’t know what is error code1. What case this? How can I fix the failure tests. ------------------ Original ------------------ From: Satish Balay <balay(a)mcs.anl.gov> Date: Tue,Apr 6,2021 0:18 PM To: Chen Gang <569615491(a)qq.com> Cc: petsc-dev <petsc-dev(a)mcs.anl.gov>, cglwdm <cglwdm(a)scu.edu.cn> Subject: Re: [petsc-dev] Petsc: Error code 1

1 0

Re: [petsc-dev] Petsc: Error code 1
by Satish Balay 05 Apr '21

05 Apr '21

To check the install - we suggest: make check 'make test' runs the full test suite. Note the test suite is parallel - so if using '-j' the multiple parallel jobs can overload the machine and slow things down. So best to use a much smaller -j value, and with a larger timeout. make -j4 test TIMEOUT=600 Satish On Tue, 6 Apr 2021, Chen Gang wrote: > Dear Professor, > > > > > > > I was running the petsc test using  > > > # make all test -j > > > AND I got the error code 1 message of six examples. > > > What is error code 1 and how can location the error. > > > I went the exactly file and make, it seems the examples can be running well. I don't understand what is error code 1. > > > The below is the summary. > > > Looking forward to hearing you! > > > ----- > > #   Summary > > # ------------- > > # FAILED diff-tao_complementarity_tutorials-blackscholes_2 diff-tao_bound_tutorials-plate2f_2 diff-tao_complementarity_tutorials-blackscholes_6 diff-tao_bound_tutorials-plate2f_1 diff-vec_is_is_tutorials-ex2f_1 diff-tao_complementarity_tutorials-blackscholes_7 > > # success 7544/9528 tests (79.2%) > > # failed 6/9528 tests (0.1%) > > # todo 225/9528 tests (2.4%) > > # skip 1753/9528 tests (18.4%) > > # > > # Wall clock time for tests: 1552 sec > > # Approximate CPU time (not incl. build time): 106566.87999999993 sec > > # > > # To rerun failed tests: > > #     /opt/rh/devtoolset-9/root/usr/bin/gmake -f gmakefile test test-fail=1 > > # > > # Timing summary (actual test time / total CPU time): > > #   mat_tests-ex44_mpiio_15: 220.10 sec / 221.33 sec > > #   mat_tests-ex44_stdio_15: 220.07 sec / 221.15 sec > > #   dm_impls_stag_tests-ex1_basic_2: 209.29 sec / 211.16 sec > > #   dm_tutorials-ex15_3: 177.42 sec / 178.46 sec > > #   dm_impls_stag_tutorials-ex6_4: 177.10 sec / 178.31 sec > > Best, > > > Gang Chen > > > Sichuan Univertsity, > School of Mathematics, > Chengdu, China

1 0

Re: [petsc-dev] -with-kokkos-cuda-arch=AMPERE80 nonsense
by Barry Smith 05 Apr '21

05 Apr '21

Junchao, I hope my latest MRs manages that for the current generation of those values. If not, we need refinement. Barry > On Apr 5, 2021, at 9:30 PM, Junchao Zhang <junchao.zhang(a)gmail.com> wrote: > > > > > On Mon, Apr 5, 2021 at 7:33 PM Jeff Hammond <jeff.science(a)gmail.com <mailto:[email protected]>> wrote: > NVCC has supported multi-versioned "fat" binaries since I worked for Argonne. Libraries should figure out what the oldest hardware they are about is and then compile for everything from that point forward. Kepler (3.5) is oldest version any reasonable person should be thinking about at this point. The oldest thing I know of in the DOE HPC fleet is Pascal (6.x). Volta and Turing are 7.x and Ampere is 8.x. > > The biggest architectural changes came with unified memory (https://developer.nvidia.com/blog/unified-memory-in-cuda-6/ <https://developer.nvidia.com/blog/unified-memory-in-cuda-6/>) and cooperative (https://developer.nvidia.com/blog/cooperative-groups/ <https://developer.nvidia.com/blog/cooperative-groups/> in CUDA 9) but Kokkos doesn't use the latter. Both features can be used on quite old GPU architectures, although the performance is better on newer ones. > > I haven't dug into what Kokkos and PETSc are doing but the direct use of this stuff in CUDA is well-documented, certainly as well as the CPU switches for x86 binaries in the Intel compiler are. > > https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#compute-ca… <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#compute-ca…> > > Devices with the same major revision number are of the same core architecture. The major revision number is 8 for devices based on the NVIDIA Ampere GPU architecture, 7 for devices based on the Volta architecture, 6 for devices based on the Pascal architecture, 5 for devices based on the Maxwell architecture, 3 for devices based on the Kepler architecture, 2 for devices based on the Fermi architecture, and 1 for devices based on the Tesla architecture. > Kokkos has config options Kokkos_ARCH_TURING75, Kokkos_ARCH_VOLTA70, Kokkos_ARCH_VOLTA72. Any idea how one can map compute capability versions to arch names? > > > > https://docs.nvidia.com/cuda/pascal-compatibility-guide/index.html#building… <https://docs.nvidia.com/cuda/pascal-compatibility-guide/index.html#building…> > https://docs.nvidia.com/cuda/volta-compatibility-guide/index.html#building-… <https://docs.nvidia.com/cuda/volta-compatibility-guide/index.html#building-…> > https://docs.nvidia.com/cuda/turing-compatibility-guide/index.html#building… <https://docs.nvidia.com/cuda/turing-compatibility-guide/index.html#building…> > https://docs.nvidia.com/cuda/ampere-compatibility-guide/index.html#building… <https://docs.nvidia.com/cuda/ampere-compatibility-guide/index.html#building…> > > Programmatic querying can be done with the following (https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__DEVICE.html <https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__DEVICE.html>): > > cudaDeviceGetAttribute > cudaDevAttrComputeCapabilityMajor <https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#gro…>: Major compute capability version number; > cudaDevAttrComputeCapabilityMinor <https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#gro…>: Minor compute capability version number; > The compiler help tells me this, which can be cross-referenced with CUDA documentation above. > > $ /usr/local/cuda-10.0/bin/nvcc -h > > Usage : nvcc [options] <inputfile> > > ... > > Options for steering GPU code generation. > ========================================= > > --gpu-architecture <arch> (-arch) > Specify the name of the class of NVIDIA 'virtual' GPU architecture for which > the CUDA input files must be compiled. > With the exception as described for the shorthand below, the architecture > specified with this option must be a 'virtual' architecture (such as compute_50). > Normally, this option alone does not trigger assembly of the generated PTX > for a 'real' architecture (that is the role of nvcc option '--gpu-code', > see below); rather, its purpose is to control preprocessing and compilation > of the input to PTX. > For convenience, in case of simple nvcc compilations, the following shorthand > is supported. If no value for option '--gpu-code' is specified, then the > value of this option defaults to the value of '--gpu-architecture'. In this > situation, as only exception to the description above, the value specified > for '--gpu-architecture' may be a 'real' architecture (such as a sm_50), > in which case nvcc uses the specified 'real' architecture and its closest > 'virtual' architecture as effective architecture values. For example, 'nvcc > --gpu-architecture=sm_50' is equivalent to 'nvcc --gpu-architecture=compute_50 > --gpu-code=sm_50,compute_50'. > Allowed values for this option: 'compute_30','compute_32','compute_35', > 'compute_37','compute_50','compute_52','compute_53','compute_60','compute_61', > 'compute_62','compute_70','compute_72','compute_75','sm_30','sm_32','sm_35', > 'sm_37','sm_50','sm_52','sm_53','sm_60','sm_61','sm_62','sm_70','sm_72', > 'sm_75'. > > --gpu-code <code>,... (-code) > Specify the name of the NVIDIA GPU to assemble and optimize PTX for. > nvcc embeds a compiled code image in the resulting executable for each specified > <code> architecture, which is a true binary load image for each 'real' architecture > (such as sm_50), and PTX code for the 'virtual' architecture (such as compute_50). > During runtime, such embedded PTX code is dynamically compiled by the CUDA > runtime system if no binary load image is found for the 'current' GPU. > Architectures specified for options '--gpu-architecture' and '--gpu-code' > may be 'virtual' as well as 'real', but the <code> architectures must be > compatible with the <arch> architecture. When the '--gpu-code' option is > used, the value for the '--gpu-architecture' option must be a 'virtual' PTX > architecture. > For instance, '--gpu-architecture=compute_35' is not compatible with '--gpu-code=sm_30', > because the earlier compilation stages will assume the availability of 'compute_35' > features that are not present on 'sm_30'. > Allowed values for this option: 'compute_30','compute_32','compute_35', > 'compute_37','compute_50','compute_52','compute_53','compute_60','compute_61', > 'compute_62','compute_70','compute_72','compute_75','sm_30','sm_32','sm_35', > 'sm_37','sm_50','sm_52','sm_53','sm_60','sm_61','sm_62','sm_70','sm_72', > 'sm_75'. > > --generate-code <specification>,... (-gencode) > This option provides a generalization of the '--gpu-architecture=<arch> --gpu-code=<code>, > ...' option combination for specifying nvcc behavior with respect to code > generation. Where use of the previous options generates code for different > 'real' architectures with the PTX for the same 'virtual' architecture, option > '--generate-code' allows multiple PTX generations for different 'virtual' > architectures. In fact, '--gpu-architecture=<arch> --gpu-code=<code>, > ...' is equivalent to '--generate-code arch=<arch>,code=<code>,...'. > '--generate-code' options may be repeated for different virtual architectures. > Allowed keywords for this option: 'arch','code'. > > On Mon, Apr 5, 2021 at 1:19 PM Satish Balay via petsc-dev <petsc-dev(a)mcs.anl.gov <mailto:[email protected]>> wrote: > This is nvidia mess-up. Why isn't there a command that give me these values [if they insist on this interface for nvcc] > > I see Barry want configure to do something here - but whatever we do - we would be shifting the problem around. > [even if we detect stuff - build box might not have the GPU used for runs.] > > We have --with-cuda-arch - which I tried to remove from configure - but its come back in a different form (--with-cuda-gencodearch) > > And I see other packages: > > --with-kokkos-cuda-arch > > Wrt spack - I'm having to do: > > spack install xsdk+cuda ^magma cuda_arch=60 > > [magma uses CudaPackage() infrastructure in spack] > > Satish > > On Mon, 5 Apr 2021, Mills, Richard Tran via petsc-dev wrote: > > > You raise a good point, Barry. I've been completely mystified by what some of these names even mean. What does "PASCAL60" vs. "PASCAL61" even mean? Do you know of where this is even documented? I can't really find anything about it in the Kokkos documentation. The only thing I can really find is an issue or two about "hey, shouldn't our CMake stuff figure this out automatically" and then some posts about why it can't really do that. Not encouraging. > > > > --Richard > > > > On 4/3/21 8:42 PM, Barry Smith wrote: > > > > > > It would be very nice to NOT require PETSc users to provide this flag, how the heck will they know what it should be when we cannot automate it ourselves? > > > > Any ideas of how this can be determined based on the current system? NVIDIA does not help since these "advertising" names don't seem to trivially map to information you can get from a particular GPU when you logged into it. For example nvidia-smi doesn't use these names directly. Is there some mapping from nvidia-smi to these names we could use? If we are serious about having a non-trivial number of users utilizing GPUs, which we need to be for future, we cannot have this absurd demands in our installation process. > > > > Barry > > > > Does spack have some magic for this we could use? > > > > > > > > > > > > -- > Jeff Hammond > jeff.science(a)gmail.com <mailto:[email protected]> > http://jeffhammond.github.io/ <http://jeffhammond.github.io/>

1 0