The tests are run in parallel - but most other tests are sequential. When I tried to reproduce I ran sequentially [as in 'make -f gmakefile' without -j] - and some of them hung.. Satish On Fri, 6 Apr 2018, Karl Rupp wrote:

Hi again,

the reason for the higher number of timeouts is likely to be due to the higher number of GPU tests. GPU builds that formerly only used CUSP now run against the CUDA backend, which has a higher number of tests. Also, the CUDA backend uses CUBLAS and CUSPARSE, whereas CUSP used its own kernels. As far as I know, CUBLAS and CUSPARSE initialization is fairly slow on the M2090.

Best regards, Karli

On 04/06/2018 09:13 PM, Karl Rupp wrote:

...

Hi,

...

The CUDA tests are hanging/timing-out more often now. For eg: http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/06/examples_nex...

And I did see some build where they didn't get killed due to timeout. For eg: http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/05/examples_nex...

This is on M2090.ￜ I can see them getting stuck on es.mcs [when I run manually - and check with nvidia-smi]

When i run these tests manually on GTX1050 (frog.mcs) - they zip through.. Any idea why they get stuck on M2090? [more frequently than random hangs..]

no, I don't know why this is the case. All my local tests finish quickly, too. I noticed last summer that there is higher startup overhead on the M2090 than on more recent GPUs, but that was in the seconds regime, not in minutes.

Are the tests run in parallel? If so, then maybe the parallel initialization of GPUs is slowing things down.

Best regards, Karli