Are there newer versions of the Gnu compiler for this system?

Yes:

-------------------------------------------------------------------------------------------------- /opt/modulefiles --------------------------------------------------------------------------------------------------
gcc/4.8.1          gcc/4.8.2          gcc/4.9.3          gcc/5.3.0          gcc/6.1.0          gcc/6.2.0          gcc/6.3.0(default) gcc/7.1.0          gcc/7.2.0          gcc/7.3.0

 
Are there any other compilers on the system that would likely be less buggy? IBM compilers? If this simple code generates a gross error with optimization who's to say how many more subtle bugs may be induced in the library by the buggy optimizer (there may be none but IMHO probability says there will be others).

Let me ask them what they recommend to use with cuda codes.
 

    Is there any chance that valgrind runs on this machine; you could run the optimized version through it and see what it says.


Valgrind works but tons of output and I could not see anything interesting in there.

And, this test does work with 1 processor! 

I think this is only a problem when GMRES is uses as the eigen estimator in Cheby. GMRES solvers work and Cheby works with -mg_levels_esteig_ksp_type cg.