hi, We have our PETSc-based code compiled on a Cray XC-50 machine, and it has just recently started running about 2.5 times slower on there. Neither the code nor PETSc has been recompiled lately. Turning the PETSc logging on, it appears to be spending more time on I/O than it used to. The cluster admins have suggested we rebuild with the Cray "perftools" module loaded to get profiling info. It's a slight hassle to rebuild everything, so I wondered, would this actually tell us anything that we don't already know from the PETSc logs? - Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: [email protected] tel: +64 (0)9 923 4611
If you can share before/after output from -log_view, it would likely help localize. Another unintrusive thing (if you're allowed to run Linux perf) is to $ perf record --call-graph dwarf -F99 ./app [... runs ...] $ perf script | stackcollapse-perf | flamegraph > flame.svg and open flame.svg in a browser (it's interactive). This uses the flamegraph tools (https://github.com/brendangregg/FlameGraph). You can direct `perf script` to a file and share that if you can't/won't install flamegraph. This doesn't require compiling any special way and yet helps understand where time is spent. Adrian Croucher <[email protected]> writes:
hi,
We have our PETSc-based code compiled on a Cray XC-50 machine, and it has just recently started running about 2.5 times slower on there. Neither the code nor PETSc has been recompiled lately.
Turning the PETSc logging on, it appears to be spending more time on I/O than it used to.
The cluster admins have suggested we rebuild with the Cray "perftools" module loaded to get profiling info. It's a slight hassle to rebuild everything, so I wondered, would this actually tell us anything that we don't already know from the PETSc logs?
- Adrian
-- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: [email protected] tel: +64 (0)9 923 4611
hi Jed, On 2/22/22 11:45, Jed Brown wrote:
If you can share before/after output from -log_view, it would likely help localize.
Unfortunately there aren't any "before" logs, because -log_view wasn't used while everything was going well. It looks like it might be related to I/O partly based on comparing logs with those from runs on other machines (not very comparable ones, admittedly) and also experimenting with things like turning file (HDF5) output from the code off.
Another unintrusive thing (if you're allowed to run Linux perf) is to
Looks like that is not available on the cluster, but will check and see if some module can be loaded to get it. On 2/22/22 12:10, Barry Smith wrote:
Have any of the modules loaded changed? Or the shared libraries that are in any of the modules? Will check. We build most of the important dependencies (e.g. blaslapack, netcdf, exodusii, hdf5, scotch, chaco) ourselves using --download-xxx in the PETSc build.
- Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: [email protected] tel: +64 (0)9 923 4611
Have any of the modules loaded changed? Or the shared libraries that are in any of the modules?
On Feb 21, 2022, at 5:38 PM, Adrian Croucher <[email protected]> wrote:
hi,
We have our PETSc-based code compiled on a Cray XC-50 machine, and it has just recently started running about 2.5 times slower on there. Neither the code nor PETSc has been recompiled lately.
Turning the PETSc logging on, it appears to be spending more time on I/O than it used to.
The cluster admins have suggested we rebuild with the Cray "perftools" module loaded to get profiling info. It's a slight hassle to rebuild everything, so I wondered, would this actually tell us anything that we don't already know from the PETSc logs?
- Adrian
-- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: [email protected] tel: +64 (0)9 923 4611
participants (3)
-
Adrian Croucher -
Barry Smith -
Jed Brown