Added this w/o luck:

#if defined(PETSC_HAVE_CUDA)
ierr = PetscOptionsCheckCUDA(logView);CHKERRQ(ierr);
#if defined(PETSC_HAVE_THREADSAFETY)
ierr = PetscCUPMInitializeCheck();CHKERRQ(ierr);
#endif
#endif

Do you think I should keep this in or take it out? Seems like a good idea and when it all works we can see if we can make it lazy.

1) Calling PetscOptions inside threads. I looked quickly at the code and it seems like it should be ok but perhaps not. This is one reason why having stuff like PetscOptionsBegin inside a low-level creation VecCreate_SeqCUDA_Private is normally not done in PETSc. Eventually this needs to be moved or reworked.

I will try this next. It is hard to see the stack here. I think I will put it in ddt and put a breakpoint PetscOptionsEnd_Private. Other ideas welcome.

Mark

2) PetscCUDAInitializeCheck is not thread safe.If it is being call for the first timeby multiple threads there can be trouble. So edit init.c and under

#if defined(PETSC_HAVE_CUDA)
ierr = PetscOptionsCheckCUDA(logView);CHKERRQ(ierr);
#endif

#if defined(PETSC_HAVE_HIP)
ierr = PetscOptionsCheckHIP(logView);CHKERRQ(ierr);
#endif

put in
#if defined thread safety
PetscCUPMInitializeCheck
#endif

this will force the initialize to be done before any threads are used