See papers by Sameer Kumar and coworkers on Google Scholar for details. You need to look at the pamid device code in MPICH to see the collectives that are actually used on BGQ, but that won't actually help since the all the BGQ collective magic is usually inside of PAMI. PAMI is open-source but not exactly easy to understand. Unlike previous generation of Blue Gene, BGQ collective optimizations are much more general and often appear for communicator sizes other than 2^n.