Filesystem performance is a red herring. Don't compile on the gold-plated filesystem. Use /tmp or /scratch, as suggested by https://www.alcf.anl.gov/user-guides/compiling-and-linking-faq. Every Linux login node has /tmp.
1. System headers and libraries are still often on a remote filesystem. The workflow is a lot more complicated if you mirror that locally.
The workflow does not change at all. E.g. "#include <stdio.h>" works irrespective of where you build.
2. /tmp isn't shared between login nodes, let alone compute nodes, so you have to copy back to global storage. With multiple login nodes, when you background/nohup a compilation task and log out or are disconnected, you have to remember the login node number to get back to the result. The cognitive load of doing all compilation on /tmp, especially with multiple packages, is not trivial.
This is not complicated. You just ssh back to the login node you were building on before or you rsync between them. People that don't know how to use ssh and rsync shouldn't be building MPICH from source.
3. Try teaching this to a novice. The fact is that if HPC is going to grow its user base, it has to be easier to use. If you want to take the elitist stance that expensive machines should only be used by experts, stop giving INCITE awards to people that don't know how to use compilers, debuggers, or the shell. Until then, hiccups along the way land in our lap, and a complicated workflow only serves to intimidate and to create more opportunity for human error, which again comes back in the form of support mail.
Novices don't build MPICH from source. I like fish, but I refuse to swallow all the red herrings you keep feeding me. Jeff -- Jeff Hammond [email protected]