Hi Barry, Thanks for the explanation. On Wed, Oct 5, 2022 at 4:11 PM Barry Smith <[email protected]> wrote:

It prints Nan to indicate that the time for that event is not known accurately. But the times for the larger events that contain these events are known. So for example the time for KSPSolve is know but not the time for VecNorm. The other numbers in the events, like number of times called etc that are not Nan are correct as displayed.

This is done because correctly tracking the times of the individual events requires synchronizations that slow down the entire calculation a bit; for example the time for the KSPSolve will register a longer time then it registers if the smaller events are not timed.

To display the times of the smaller events use -log_view_gpu_time also but note this will increase the times of the larger events a bit.

Barry

On Oct 5, 2022, at 4:47 PM, Sajid Ali <[email protected]> wrote:

Hi PETSc-developers,

I'm having trouble with getting performance logs from an application that uses PETSc. There are no issues when I run it on a CPU, but every time a GPU is used there is no timing data and almost all times are replaced by times that are just `nan` (on two different clusters). I am attaching the log files for both cases with this email. Could someone explain what is happening here ?

In case it helps, here are the routines used to initialize/finalize the application that also handle initializing/finalizing PETSc and printing the PETSc performance logs to PETSC_VIEWER_STDOUT_WORLD : https://github.com/fnalacceleratormodeling/synergia2/blob/devel3/src/synergi...

Thank You, Sajid Ali (he/him) | Research Associate Scientific Computing Division Fermi National Accelerator Laboratory s-sajid-ali.github.io <log-gpu><log-cpu>

-- Thank You, Sajid Ali (he/him) | Research Associate Scientific Computing Division Fermi National Accelerator Laboratory s-sajid-ali.github.io