Hi Bob, Thanks for your suggestions. Here are more tests. We actually have three clusters. Cluster 1 and 2: 8 nodes, (2 Processors, 4 cores/processor, no HT – Total 8 Threads)/node Cluster 3: 8 nodes, (1 Processors, 4 cores /processor, HT – Total 8 Threads )/node We also have a standalone machine: 2 processors, 6 cores/processor, HT – total 24 threads. For one particular case: Cluster 1 and 2 take 48 min to finish with 8 nodes, 8 threads/node, 60% CPU usage; 53 min to finish with 3 nodes, 8 threads/node, 90% CPU usage; Cluster 3 takes 227 min to finish with 8 nodes, 8 threads/node, 20% CPU usage; 207 min to finish with 3 nodes, 8 threads/node, 50% CPU usage; Standalone machine takes 24 min to finish with 24 threads, 100% CPU usage. It looks like with 24 threads, they should be pretty busy? Could the above phenomena be a hardware issue more than software? Qiguo From: Bob Ilgner [mailto:[email protected]] Sent: Thursday, October 23, 2014 1:11 AM To: [email protected] Subject: Re: [mpich-discuss] CPU usage versus Nodes, Threads Hi Qiguo,