Peter and Glenn, You had asked about using parallel direct solvers with PFLOTRAN. If you install a package such as mumps or superlu_dist with the corresponding PETSc interfaces, you can easily try using a direct solver using command line options such as the following: -flow_mat_type mpiaij -flow_ksp_type preonly -flow_pc_type lu -flow_pc_factor_mat_solver_package mumps You'll need to pull and update to include the push I made tonight that allows you to specify the matrix type on the command line. (Things default to BAIJ, which isn't supported for the direct solvers.) The above looks pretty messy but we can worry about making a cleaner way to do this if this turns out to be something that you might want to do on a routine basis. I have tested PFLOTRAN with both MUMPS and SuperLU_dist on a Linux workstation using the very simple 'TAO/100_10_10' problem in the examples directory. This works fine. I was having some problems getting things to work properly on Jaguar (no surprise) and wasn't quite able to get things resolved before the machine went down for most of today. I will try again tomorrow. In your own experiments, I would start with trying out MUMPS, as anecdotal evidence suggests that it is fairly robust. --Richard