I noticed that GCC mispredicts the conditional in CHKERRQ. At high optimization levels it still relocates most of the PetscError call, but the non-error path is cleaner with #define CHKERRQ(n) if (UNLIKELY(n)) {return PetscError(__LINE__,__FUNCT__,__FILE__,__SDIR__,n,0," ");} where the following definitions are typical #define UNLIKELY(c) __builtin_expect(!!(c),0) #define LIKELY(c) __builtin_expect(!!(c),1) I'm not familiar with the analogues for other compilers. I don't suggest cluttering the code with such hints, but just putting it in CHKERRQ would decrease code size for the non-error path which seems like a good thing. Jed