Hi nek5000-users & developers, I wonder if anyone has experience with the performance of nek5000 on a Cray ? In particular what scaling could one expect compared to e.g. a Blue Gene? We have a new Cray XT6m system, based on the AMD Opteron 12-core “Magny-Cours” <http://www.cray.com/Products/XT/Specifications.aspx> (2.1 GHz) processors and the Cray SeaStar2 interconnect technology. It consists of 11040 compute cores and 10 service cores. The scaling is not too encouraging (see attached pdf-files). The attached files show three different runs, and the scaling is apparently highly dependent on where in the torus the job happens to be. This is maybe ok, but is the leveling off after 2048 cores consistent with what people observe on a Cray ? Thanks for your suggestions and experience! Best regards, Johan -- Johan Malm Department of Mechanics, KTH SE-100 44, Stockholm, Sweden Phone: +46 8 7906876 E-mail: [email protected]
Hi Johan, we can definitely do better on Cray systems but I guess we need to do some fine tuning first. The BG architecture is quite different and you need much more grid points per core to scale on the Cray (4-8x more). - Can you post your logfiles? - Did you try to enable Nek's internal MPI profiling where you spend most of the communication cost? - Did you try to use an optimized MPI rank mapping? Is it possible to get an account on that machine to do some experiments? Cheers, Stefan ----- Original Message ----- From: [email protected] To: [email protected] Sent: Tue, 19 Oct 2010 05:18:41 -0600 (GMT-06:00) Subject: [Nek5000-users] scaling of nek5000 on a Cray Hi nek5000-users & developers, I wonder if anyone has experience with the performance of nek5000 on a Cray ? In particular what scaling could one expect compared to e.g. a Blue Gene? We have a new Cray XT6m system, based on the AMD Opteron 12-core “Magny-Cours” <http://www.cray.com/Products/XT/Specifications.aspx> (2.1 GHz) processors and the Cray SeaStar2 interconnect technology. It consists of 11040 compute cores and 10 service cores. The scaling is not too encouraging (see attached pdf-files). The attached files show three different runs, and the scaling is apparently highly dependent on where in the torus the job happens to be. This is maybe ok, but is the leveling off after 2048 cores consistent with what people observe on a Cray ? Thanks for your suggestions and experience! Best regards, Johan -- Johan Malm Department of Mechanics, KTH SE-100 44, Stockholm, Sweden Phone: +46 8 7906876 E-mail: [email protected]
I'm hoping to include our ping-pong and mxm timers in a future source release so that we can get a handle on the platform parameters via a couple of quick calls from userchk. With luck, we'll have this in a few days. -- Paul On Wed, 20 Oct 2010, [email protected] wrote:
Hi Johan,
we can definitely do better on Cray systems but I guess we need to do some fine tuning first. The BG architecture is quite different and you need much more grid points per core to scale on the Cray (4-8x more). - Can you post your logfiles? - Did you try to enable Nek's internal MPI profiling where you spend most of the communication cost? - Did you try to use an optimized MPI rank mapping? Is it possible to get an account on that machine to do some experiments? Cheers, Stefan ----- Original Message ----- From: [email protected] To: [email protected] Sent: Tue, 19 Oct 2010 05:18:41 -0600 (GMT-06:00) Subject: [Nek5000-users] scaling of nek5000 on a Cray Hi nek5000-users & developers, I wonder if anyone has experience with the performance of nek5000 on a Cray ? In particular what scaling could one expect compared to e.g. a Blue Gene? We have a new Cray XT6m system, based on the AMD Opteron 12-core “Magny-Cours” <http://www.cray.com/Products/XT/Specifications.aspx> (2.1 GHz) processors and the Cray SeaStar2 interconnect technology. It consists of 11040 compute cores and 10 service cores. The scaling is not too encouraging (see attached pdf-files). The attached files show three different runs, and the scaling is apparently highly dependent on where in the torus the job happens to be. This is maybe ok, but is the leveling off after 2048 cores consistent with what people observe on a Cray ? Thanks for your suggestions and experience! Best regards, Johan -- Johan Malm Department of Mechanics, KTH SE-100 44, Stockholm, Sweden Phone: +46 8 7906876 E-mail: [email protected] _______________________________________________ Nek5000-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
Hi Johan, I've added some performance tests to the repo (v. 570) There are a couple of calling options - I would suggest, in usrdat, within your .usr file to have iverbose = 1 ! fully verbose call platform_timer(iverbose) call exitti('timer done$',iverbose) and run this with a reasonably large 3D case on your parallel platform. If you can send us the output of this run off list we'll have a pretty good idea of what's going on. Thanks! Paul On Wed, 20 Oct 2010, [email protected] wrote:
I'm hoping to include our ping-pong and mxm timers in a future source release so that we can get a handle on the platform parameters via a couple of quick calls from userchk. With luck, we'll have this in a few days. -- Paul
On Wed, 20 Oct 2010, [email protected] wrote:
Hi Johan,
we can definitely do better on Cray systems but I guess we need to do some fine tuning first. The BG architecture is quite different and you need much more grid points per core to scale on the Cray (4-8x more).
- Can you post your logfiles? - Did you try to enable Nek's internal MPI profiling where you spend most of the communication cost? - Did you try to use an optimized MPI rank mapping?
Is it possible to get an account on that machine to do some experiments?
Cheers, Stefan
----- Original Message ----- From: [email protected] To: [email protected] Sent: Tue, 19 Oct 2010 05:18:41 -0600 (GMT-06:00) Subject: [Nek5000-users] scaling of nek5000 on a Cray
Hi nek5000-users & developers,
I wonder if anyone has experience with the performance of nek5000 on a Cray ? In particular what scaling could one expect compared to e.g. a Blue Gene?
We have a new Cray XT6m system, based on the AMD Opteron 12-core “Magny-Cours” <http://www.cray.com/Products/XT/Specifications.aspx> (2.1 GHz) processors and the Cray SeaStar2 interconnect technology. It consists of 11040 compute cores and 10 service cores.
The scaling is not too encouraging (see attached pdf-files). The attached files show three different runs, and the scaling is apparently highly dependent on where in the torus the job happens to be. This is maybe ok, but is the leveling off after 2048 cores consistent with what people observe on a Cray ?
Thanks for your suggestions and experience!
Best regards,
Johan
-- Johan Malm Department of Mechanics, KTH SE-100 44, Stockholm, Sweden Phone: +46 8 7906876 E-mail: [email protected]
_______________________________________________ Nek5000-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
Thanks Paul, Our system is currently down, but I'll do as you suggest as soon as the system is up and running again. Johan [email protected] wrote:
Hi Johan,
I've added some performance tests to the repo (v. 570)
There are a couple of calling options -
I would suggest, in usrdat, within your .usr file to have
iverbose = 1 ! fully verbose call platform_timer(iverbose) call exitti('timer done$',iverbose)
and run this with a reasonably large 3D case on your parallel platform.
If you can send us the output of this run off list we'll have a pretty good idea of what's going on.
Thanks!
Paul
On Wed, 20 Oct 2010, [email protected] wrote:
I'm hoping to include our ping-pong and mxm timers in a future source release so that we can get a handle on the platform parameters via a couple of quick calls from userchk. With luck, we'll have this in a few days. -- Paul
On Wed, 20 Oct 2010, [email protected] wrote:
Hi Johan,
we can definitely do better on Cray systems but I guess we need to do some fine tuning first. The BG architecture is quite different and you need much more grid points per core to scale on the Cray (4-8x more).
- Can you post your logfiles? - Did you try to enable Nek's internal MPI profiling where you spend most of the communication cost? - Did you try to use an optimized MPI rank mapping?
Is it possible to get an account on that machine to do some experiments?
Cheers, Stefan
----- Original Message ----- From: [email protected] To: [email protected] Sent: Tue, 19 Oct 2010 05:18:41 -0600 (GMT-06:00) Subject: [Nek5000-users] scaling of nek5000 on a Cray
Hi nek5000-users & developers,
I wonder if anyone has experience with the performance of nek5000 on a Cray ? In particular what scaling could one expect compared to e.g. a Blue Gene?
We have a new Cray XT6m system, based on the AMD Opteron 12-core “Magny-Cours” <http://www.cray.com/Products/XT/Specifications.aspx> (2.1 GHz) processors and the Cray SeaStar2 interconnect technology. It consists of 11040 compute cores and 10 service cores.
The scaling is not too encouraging (see attached pdf-files). The attached files show three different runs, and the scaling is apparently highly dependent on where in the torus the job happens to be. This is maybe ok, but is the leveling off after 2048 cores consistent with what people observe on a Cray ?
Thanks for your suggestions and experience!
Best regards,
Johan
-- Johan Malm Department of Mechanics, KTH SE-100 44, Stockholm, Sweden Phone: +46 8 7906876 E-mail: [email protected]
_______________________________________________ Nek5000-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
------------------------------------------------------------------------
_______________________________________________ Nek5000-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
-- Johan Malm Department of Mechanics, KTH SE-100 44, Stockholm, Sweden Phone: +46 8 7906876 E-mail: [email protected]
Hello Stefan, Here is a (shortened) log file pertaining to a simulation on 4096 processors of `our' Cray XT6m. The case is as follows: * Nek5000, rev. 565 * 220248 elements in total, i.e. 54 elements/core * Restart from a nearly converged initial condition To your hints given below: How do I enable Nek's internal MPI profiling? We got a long list of runtime statistics and core-wise times; maybe the internal profiling was actually active... Could you give a brief explanation of the most important numbers, please? How can I invoke the optimized MPI rank mapping? Thank you in advance for some clarification! Lars (who collaborates with Johan) [email protected] wrote:
Hi Johan,
we can definitely do better on Cray systems but I guess we need to do some fine tuning first. The BG architecture is quite different and you need much more grid points per core to scale on the Cray (4-8x more).
- Can you post your logfiles? - Did you try to enable Nek's internal MPI profiling where you spend most of the communication cost? - Did you try to use an optimized MPI rank mapping?
Is it possible to get an account on that machine to do some experiments?
Cheers, Stefan
----- Original Message ----- From: [email protected] To: [email protected] Sent: Tue, 19 Oct 2010 05:18:41 -0600 (GMT-06:00) Subject: [Nek5000-users] scaling of nek5000 on a Cray
Hi nek5000-users & developers,
I wonder if anyone has experience with the performance of nek5000 on a Cray ? In particular what scaling could one expect compared to e.g. a Blue Gene?
We have a new Cray XT6m system, based on the AMD Opteron 12-core “Magny-Cours” <http://www.cray.com/Products/XT/Specifications.aspx> (2.1 GHz) processors and the Cray SeaStar2 interconnect technology. It consists of 11040 compute cores and 10 service cores.
The scaling is not too encouraging (see attached pdf-files). The attached files show three different runs, and the scaling is apparently highly dependent on where in the torus the job happens to be. This is maybe ok, but is the leveling off after 2048 cores consistent with what people observe on a Cray ?
Thanks for your suggestions and experience!
Best regards,
Johan
/----------------------------------------------------------\ | _ __ ______ __ __ ______ ____ ____ ____ | | / | / // ____// //_/ / ____/ / __ \ / __ \ / __ \ | | / |/ // __/ / ,< /___ \ / / / // / / // / / / | | / /| // /___ / /| | ____/ / / /_/ // /_/ // /_/ / | | /_/ |_//_____//_/ |_|/_____/ \____/ \____/ \____/ | | | |----------------------------------------------------------| | | | NEK5000: Open Source Spectal Element Solver | | COPYRIGHT (c) 2008-2010 UCHICAGO ARGONNE, LLC | | Version: 1.0rc1 / SVN r560 | | Web: http://nek5000.mcs.anl.gov | | | \----------------------------------------------------------/ Number of processors: 4096 REAL wdsize : 8 INTEGER wdsize : 4 ... 103 Parameters from file:/mnt/lustre_server/pdc/johanoh/nlf/doublerough/4096/nlfRGH_N11_200x10x18.rea 1 1.0 DENSITY 2 -10010.90830427619 VISCOS (<0 -> Reynolds number else 1/Re) 7 1.0 RHOCP 8 5.0e-3 CONDUCT 11 5.0e1 NSTEPS 12 -2.5e-5 DT (negative value -> forcing) 15 5.0e1 IOSTEP 18 0.025 GRID 19 -1.0 INTYPE 20 7.0 NORDER (only used by prenek) 21 3.0E-14 DIVERGENCE 22 3.0E-14 HELMHOLTZ 24 1.0E-06 TOLREL 25 1.0E-10 TOLABS 26 0.5 COURANT 27 3.0 TORDER 28 0.000000E+00 NABMSH (0: extrapolation >0: AB) 33 1.0 p33, LUS: BCs from file(s) acc. to bc.in 49 0.000000 mixing length factor 65 1.0 p65 number of i/o nodes (< 0 multi subdirectories) 66 6.0 write format (<0: ASCII else BINARY) 0.0 67 6.0 read form 0.0 74 1.0 >0: print Helmholz solver iterations 93 20.0 number of previous sol'ns to save 95 5.0 start saving pressure soln's after this 99 3.0 <0: turn off dealiasing 102 1.0 dump out divergence at each time st 103 0.2 filter weight (< 0 no filtering) IFTRAN = T IFFLOW = T IFHEAT = F IFSPLIT = F IFLOMACH = F IFUSERVP = F IFUSERMV = F IFSTRS = F IFCHAR = F IFCYCLIC = F IFAXIS = F IFMVBD = F IFMELT = F IFMODEL = F IFKEPS = F IFMOAB = F IFSYNC = T IFVCOR = F IFINTQ = F IFCWUZ = F IFSWALL = F IFGEOM = F IFSURT = F IFWCNO = F IFTMSH for field 1 = F IFADVC for field 1 = T IFNONL for field 1 = F Dealiasing enabled, lxd= 12 ... set initial conditions Checking restart options: /mnt/lustre_server/pdc/johanoh/nlf/doublerough/4096/nlfRGH_N11_200x10x18.restart Reading checkpoint data 0 0 OPEN: /mnt/lustre_server/pdc/johanoh/nlf/doublerough/4096/nlfRGH_N11_200x10x18.restart byte swap: F 6.543210 -2.9312772E+35 0 5.0125E-01 done :: Read checkpoint data avg data-throughput = -55.7MBps io-nodes = 1 ... Step 50, t= 5.0250000E-01, DT= 2.5000000E-05, C= 0.008 7.3555E+02 1.0015E+01 Solving for fluid 50 Hmholtz VELX: 4 2.2558E-16 6.8947E-01 3.0000E-14 50 Hmholtz VELY: 4 1.3317E-16 3.4072E-01 3.0000E-14 50 Hmholtz VELZ: 4 3.1077E-16 4.5643E-01 3.0000E-14 50 5 alpha: 6.3514E-06 6.2921E-07 4.3961E-08 1.6760E-09 7.4077E-10 50 5 4.2647E-08 1.1361E-12 3.7538E+04 alph12 50 U-PRES gmres: 33 2.8138E-14 3.0000E-14 1.1361E-12 6.5668E+00 9.2376E+00 50 DNORM, DIVEX 3.1629080853163553E-014 2.8137819833890836E-014 50 5.0250E-01 1.0296E+01 Fluid done 50 5.0250E-01 Write checkpoint: 0 50 OPEN: /mnt/lustre_server/pdc/johanoh/nlf/doublerough/4096/nlfRGH_N11_200x10x180.f00001 50 5.0250E-01 done :: Write checkpoint file size = 302.E+01MB avg data-throughput = 123.3MB/s io-nodes = 1 end of time-step loop call usrdat5 done :: usrdat5 runtime statistics: total time 770.4385341213775 inv3 time 2610 1.118097305297852 1.4512479007464892E-003 invc time 150 3.3671140670776367E-002 4.3703863682228151E-005 mltd time 7980 8.620663166046143 1.1189293868689094E-002 cdtp time 7980 10.11404132843018 1.3127642090182336E-002 eslv time 50 689.6867790222168 0.8951872842247545 pres time 50 696.9211962223053 0.9045772834001446 crsl time 2439 74.85183858871460 9.7154847886829848E-002 crsl min 59.82716941833496 crsl max 94.41280984878540 crsl avg 81.66899811057374 hmhz time 150 31.97629642486572 4.1504020124502321E-002 spro time 51 6.5573453903198242E-002 8.5111856428598075E-005 usbc time 50 8.785777330398560 1.1403605792405002E-002 usbc min 8.560441017150879 usbc max 8.964121341705322 usb avg 8.682757482863963 axhm time 1116 1.593900442123413 2.0688223284951951E-003 advc time 150 1.006965398788452 1.3070029005452254E-003 vdss time 2611 91.34421586990356 0.1185613281584808 vdss min 63.92344927787781 vdss max 129.4121160507202 vdss avg 82.33194673579419 dsum time 8427 246.5665671825409 0.3200340536753266 dsum min 232.8616766929626 dsum max 261.6403193473816 dsum avg 240.4033431065618 dadd time 0 257.9166781902313 0.3347660673338001 ddsl time 2439 489.3997883796692 0.6352223658410204 solv time 2439 2.981695413589478 3.8701275722012769E-003 prep time 50 24.58435845375061 3.1909564961969450E-002 # nid tusbc tdadd tcrsl tvdss tdsum tgop qqq 0 8.7858E+00 2.5792E+02 7.4852E+01 9.1344E+01 2.4657E+02 0.0000E+00 qqq 1 8.7861E+00 2.5781E+02 7.4042E+01 9.2054E+01 2.4592E+02 0.0000E+00 qqq 2 8.7840E+00 2.5784E+02 7.1168E+01 9.5010E+01 2.4603E+02 0.0000E+00 qqq 3 8.7852E+00 2.5781E+02 7.2827E+01 9.3698E+01 2.4596E+02 0.0000E+00 qqq 4 8.7802E+00 2.5818E+02 7.1552E+01 9.4182E+01 2.4665E+02 0.0000E+00 qqq 5 8.7807E+00 2.5796E+02 7.0593E+01 9.5107E+01 2.4639E+02 0.0000E+00 qqq 6 8.7793E+00 2.5795E+02 7.2363E+01 9.3033E+01 2.4657E+02 0.0000E+00 qqq 7 8.7803E+00 2.5792E+02 7.2035E+01 9.3591E+01 2.4643E+02 0.0000E+00 qqq 8 8.7804E+00 2.5809E+02 7.3022E+01 9.2647E+01 2.4679E+02 0.0000E+00 qqq 9 8.7803E+00 2.5790E+02 7.2361E+01 9.3156E+01 2.4646E+02 0.0000E+00 qqq 10 8.7806E+00 2.5791E+02 7.3857E+01 9.2019E+01 2.4641E+02 0.0000E+00 qqq 11 8.7814E+00 2.5792E+02 7.0931E+01 9.4789E+01 2.4648E+02 0.0000E+00 qqq 12 8.7808E+00 2.5805E+02 7.4226E+01 9.1674E+01 2.4666E+02 0.0000E+00 qqq 13 8.7810E+00 2.5786E+02 7.2993E+01 9.2708E+01 2.4638E+02 0.0000E+00 qqq 14 8.7805E+00 2.5787E+02 7.3823E+01 9.1775E+01 2.4647E+02 0.0000E+00 qqq 15 8.7815E+00 2.5786E+02 7.2894E+01 9.2895E+01 2.4635E+02 0.0000E+00 qqq 16 8.7866E+00 2.5803E+02 7.5833E+01 9.0600E+01 2.4654E+02 0.0000E+00 qqq ... 4079 8.6580E+00 2.6142E+02 8.9165E+01 7.0576E+01 2.4050E+02 0.0000E+00 qqq 4080 8.6531E+00 2.6155E+02 8.5569E+01 7.4002E+01 2.4057E+02 0.0000E+00 qqq 4081 8.6530E+00 2.6148E+02 8.3350E+01 7.5916E+01 2.4050E+02 0.0000E+00 qqq 4082 8.6534E+00 2.6143E+02 8.7352E+01 7.1907E+01 2.4048E+02 0.0000E+00 qqq 4083 8.6540E+00 2.6142E+02 8.7341E+01 7.1970E+01 2.4048E+02 0.0000E+00 qqq 4084 8.6564E+00 2.6153E+02 8.5829E+01 7.4169E+01 2.4042E+02 0.0000E+00 qqq 4085 8.6561E+00 2.6144E+02 8.7598E+01 7.2145E+01 2.4021E+02 0.0000E+00 qqq 4086 8.6584E+00 2.6147E+02 8.6941E+01 7.2734E+01 2.4035E+02 0.0000E+00 qqq 4087 8.6569E+00 2.6150E+02 8.8861E+01 7.0998E+01 2.4038E+02 0.0000E+00 qqq 4088 8.6589E+00 2.6154E+02 8.7007E+01 7.3074E+01 2.4044E+02 0.0000E+00 qqq 4089 8.6586E+00 2.6141E+02 8.6709E+01 7.3017E+01 2.4039E+02 0.0000E+00 qqq 4090 8.6579E+00 2.6143E+02 8.8299E+01 7.1693E+01 2.4028E+02 0.0000E+00 qqq 4091 8.6589E+00 2.6142E+02 8.9661E+01 7.0449E+01 2.4021E+02 0.0000E+00 qqq 4092 8.6611E+00 2.6217E+02 8.0231E+01 7.9393E+01 2.4175E+02 0.0000E+00 qqq 4093 8.6611E+00 2.6215E+02 8.1451E+01 7.8101E+01 2.4170E+02 0.0000E+00 qqq 4094 8.6610E+00 2.6214E+02 8.0526E+01 7.9058E+01 2.4179E+02 0.0000E+00 qqq 4095 8.6616E+00 2.6211E+02 8.2362E+01 7.7343E+01 2.4172E+02 0.0000E+00 qqq call exitt: dying ... backtrace(): obtained 6 stack frames. [0x4d46f8] [0x55300b] [0x40b7f4] [0x400360] [0x6633b0] [0x400229] total elapsed time : 1.14048E+03 sec total solver time incl. I/O : 7.70439E+02 sec time/timestep : 1.54088E+01 sec CPU seconds/timestep/gridpt : 8.28603E-04 sec Application 57063 resources: utime 113705, stime 358
participants (1)
-
nek5000-users＠lists.mcs.anl.gov