Jan, Here is some output from your code on a cluster that uses FDR Infiniband instead of 1Gbit Ethernet. I also used the vendor MPI which is SMP aware. 2 nodes each with 2 sandybridge sockets (no process placement) ./a.out START: NchTtot=201600 NchTXtot=40320; oneCPU: NchTX=254016000 tot_size/B=508032000, kB=496125, MB=484, nCyc=1 Ncpu=32 , block: NchT_b=6300 NchX_b=1260 size/B=15876000, kB=15503, MB=15 #### Summary: Ncpu=32 avrT/sec=8.05 , maxT/sec=8.05, sumT/sec=257.68 totMB=16257.0 avrSpeed=2018.9(MB/sec) 4 nodes each with 2 sandybridge sockets (no process placement) mpiexec -np 32 ./a.out ./a.out START: NchTtot=201600 NchTXtot=40320; oneCPU: NchTX=254016000 tot_size/B=508032000, kB=496125, MB=484, nCyc=1 Ncpu=32 , block: NchT_b=6300 NchX_b=1260 size/B=15876000, kB=15503, MB=15 #### Summary: Ncpu=32 avrT/sec=6.94 , maxT/sec=6.94, sumT/sec=221.97 totMB=16257.0 avrSpeed=2343.7(MB/sec) Same 4 nodes, but with process placement (4 ranks/socket) mpiexec -np 32 ./a.out ./a.out START: NchTtot=201600 NchTXtot=40320; oneCPU: NchTX=254016000 tot_size/B=508032000, kB=496125, MB=484, nCyc=1 Ncpu=32 , block: NchT_b=6300 NchX_b=1260 size/B=15876000, kB=15503, MB=15 #### Summary: Ncpu=32 avrT/sec=6.86 , maxT/sec=6.86, sumT/sec=219.50 totMB=16257.0 avrSpeed=2370.0(MB/sec) ________________________________________ From: [email protected] [[email protected]] on behalf of Jan T. Balewski [[email protected]] Sent: Monday, May 19, 2014 10:19 PM To: [email protected] Subject: [mpich-discuss] optimization of MPI_Alltoall(..) Hi, I want to use MPI to transpose a big NxM matrix. Currently the size is: N=20k, M=40k, type=ushort, total size =16GB (it will be larger in the future) To evaluate the code I report the processing speed in MB/sec, it is dominated by the cost of MPI_Alltoall(...) between 32 MPI processes. By changing the order in which MPI jobs are assigned to the cores on the blades I was able to increase the overall speed by ~15% , from 349 MB/sec to 398 MB/sec. My question is are there more tricks I can play to accelerate MPI_Alltoall(...) further ? (For now using the existing hardware) Thanks for any suggestions Jan P.S. Below are gory details: The transposition is done in 2 stages on 32 MPI jobs running on 4 8-core Dell 1950 blades. The 4 blades are connected via eth0 to the same card in the Cisco 1 GBit switch. This is the order of operations: - big matrix is divided on 32x32 non-square blocks - each of 32 blocks is transposed individually on CPU in each MPI job - the 32x32 blocks are exchanged between MPI jobs and transposed using MPI_Alltoall(...) command - the time is measured using MPI_Wtime() FYI, the whole code is visible at: https://bitbucket.org/balewski/kruk-mpi/src/8d3c4b7deb566f2768f132135b12e58f... ***** mode 1: uses 8 consecutive cores per blade, next fill 2nd blade, etc $ mpiexec -f machinefile -n 32 ./kruk2 where machinefile is: oswrk139.lns.mit.edu:8 oswrk140.lns.mit.edu:8 oswrk145.lns.mit.edu:8 oswrk150.lns.mit.edu:8 Summary: Ncpu=32 avrT/sec=40.83, totMB=16257.0 avrSpeed=398.2(MB/sec) ***** mode 2: uses 1st core from all blades, next use 2nd core from all, etc: $ mpiexec -f machinefile -n 32 ./kruk2 where machinefile is: oswrk139.lns.mit.edu:1 oswrk140.lns.mit.edu:1 oswrk145.lns.mit.edu:1 oswrk150.lns.mit.edu:1 Summary: Ncpu=32 avrT/sec=46.61, totMB=16257.0 avrSpeed=348.8(MB/sec) (3) _______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss