Mark Carpenter,  NASA Langley.

I am a novice PETSC user of about 10 years.  I’ve build  a DG-FEM code with petsc as one of the solver paths (I have my own as well).  Furthermore, I use petsc for MPI communication.

I’m running the DG-FEM code on our NAS supercomputer.  Everything works when my integer sizes are small.  When I exceed the 2^32 limit of integer arithmetic the code fails in very strange ways.

The users that originally set up the petsc infrastructure in the code are no longer at NASA and I’m “dead in the water”.

I think I’ve promoted all the integers that  are problematic in my code (F95).  On PETSC side:  I’ve tried

1. Reinstall petsc with –with-64-bit-integers  (no luck)
2. Reinstall petsc with –with-64-bit-integers and –with-64-bit-indices  (code will not compile with these options.  Additional variables on F90 side require promotion and then the errors cascade through code  when making PETSC calls.
3. It’s possible that I’ve missed offending integers, but the petsc error messages are so cryptic that I can’t even tell where it is failing.

Further complicating matters:

The problem by definition needs to be HUGE.  Problem sizes requiring 1000 cores (10^6 elements at P5) are needed to experience the errors, which involves waiting in queues for ½ day at least.

Attached are the

1. Install script used to install PETSC on our machine
2. The Makefile used on the fortran side
3. A data dump from an offending simulation (which is huge and I can’t see any useful information.)

How do I attack this problem.

(I’ve never gotten debugging working properly).

Mark