July 2016 - petsc-dev - mailman3.cels.anl.gov

Question about DMCreateDomainDecomposition()
by Lulu Liu 18 Jul '16

18 Jul '16

Hi, I add the following lines in the example /petsc/src/snes/examples/tutorials/ex19. When I run this example using different numbers of processors, n always equals to 1. However, according to the manual, n is the number of subproblems in the domain decomposition. I feel confused why n does not return the number of processors, could you explain it? Thanks! PetscInt n; DM *subdms; ierr =DMCreateDomainDecomposition(da,&n,NULL,NULL,NULL,&subdms);CHKERRQ(ierr); PetscPrintf(PETSC_COMM_WORLD,"### n==%d\n", n); Output: ### n==1 -- ------------------------------ This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.

1 0

Re: [petsc-dev] Nested event logging and human friendly output
by Barry Smith 16 Jul '16

16 Jul '16

Thank you very much for fixing the errors I introduced and improving the code. I have put it into branch barry/fix-xml-logging and merged to next for testing. I do have one concern, when I run on src/snes/examples/tutorials/ex19.c with no options (see attached result) it lists a MatAssemblyEnd() as a top-level event (but not a MatAssemblyBegin()) but there is no MatAssemblyEnd() at the top level, they should all be occurring inside the SNESSolve() and when I run in the debugger all the MatAssemblyEnd() occur within the SNESSolve(). I am not sure why it incorrectly locates the MatAssemblyEnd() as a top level event? Thanks again Barry > On Jul 8, 2016, at 7:59 AM, Koos Huijssen <koos.huijssen(a)vortech.nl> wrote: > > Dear Barry, > > I found some time to fix the issues with the nested timers. The attached patch file should work on commit > > c03b0cd 2016-07-07 | Merge branch 'stefano_zampini/hypre-ams-zerointerior-feature' > > What I have found is the following: > > - There were some issues with the merging of the code into the Petsc code base. I have reviewed the merge and fixed this (mainly the section around depth/maxdepth determination). > - There was indeed a fundamental issue, concerning a wrong assumption that the PetscLogEvent id =0 was reserved for the overall 'awake' root event. As it was also used for a normal event, this normal event was mistakenly considered as the root event which caused some trouble. I fixed the code. The issue of the single-event example giving a crash is now resolved as well. > - When running snes/examples/tutorial/ex56, I found that a timer in DMPlexDistribute was not properly ended. If fixed this. > > With regard to the latter I wonder why the 'standard' timer events did not notice the fact that the DMPLEX_Distribute was not closed and that it should count as the event with the longest evaluation time. Could it be that the logging disregards any timers that are still open? I was thinking of including a warning message in the nested timers for any open timers at the time of the log generation, but I haven't done that for now. > > One thing that is still open is the fact that the nested timers are only considering the timings of the events as far as they are logged in the Main Stage. Timings in other stages are simply ignored. For instance, with the example snes/examples/tutorial/ex56.c, we will only get meaningful nested timing information from the ascii_xml viewer if we set log_stages = PETSC_FALSE on line 13. For now, I see three possible approaches to resolve this: > - Tell the users that they should turn off stages in their code if they would like to use the nested timers. Given the fact that the nested timer functionality basically makes the stages obsolete, this could be a future option. > - After the Main Stage is pushed in PetscLogInitialize, disable the PetscLogStagePush() and PetscLogStagePop() functionality if the ascii_xml viewer is selected for output. > - Adjust the xmllogevent functions to consider the eventPerfInfo array of all stages that are in use. This is however a complex code adjustment which I would not prefer to do, also given the fact that the staging functionality is not relevant for the nested timers. > > Please let me know what you think of the above. Could you please include the patch in the Petsc code? > > With kind regards, > > Koos Huijssen > __________________________________________ > > VORtech BV - Scientific software engineers > __________________________________________ > > Dr.ir. Koos Huijssen > > P.O. Box 260 > 2600 AG Delft > The Netherlands > > phone +31(0)15-285 0125 > mobile +31(0)6-3333 0803 > email koos.huijssen(a)vortech.nl > web www.vortech.nl > > -----Oorspronkelijk bericht----- > Van: Barry Smith [mailto:[email protected]] > Verzonden: maandag 6 juni 2016 17:21 > Aan: Koos Huijssen > CC: Richard Mills; petsc-dev(a)mcs.anl.gov; Klaij, Christiaan > Onderwerp: Re: [petsc-dev] Nested event logging and human friendly output > > > Whenever you can get to it is fine. > > Thanks > > Barry > >> On Jun 6, 2016, at 8:14 AM, Koos Huijssen <koos.huijssen(a)vortech.nl> wrote: >> >> Dear Barry, >> >> Thanks for notifying us of these issues. There seems to be quite a few fundamental issues with the nested timer functionality and the xml output. We will have to check the case, reproduce the problems and start looking into the code. However, currently we are unable to do so. We may be able to look into the issue in a month or so. Would that be okay for you? If so, then we will come back on the issue in the beginning of July. >> >> With kind regards, >> >> Koos >> >> From: Barry Smith [mailto:[email protected]] >> Sent: zondag 5 juni 2016 1:19 >> To: Koos Huijssen <koos.huijssen(a)vortech.nl>; Richard Mills >> <richardtmills(a)gmail.com> >> Cc: petsc-dev(a)mcs.anl.gov; Klaij, Christiaan <C.Klaij(a)marin.nl> >> Subject: Re: [petsc-dev] Nested event logging and human friendly >> output >> >> >> We are having some major problems with your xml nested logging on a slightly more complicated example and I've been trying to debug it with no success. So I went back to my original commit bb1d7374b64f295b2ed5ff23b89435d65e905a54 and found something I was not expecting. When I run src/snes/examples/tutorials/ex19 with logging it generates the attached image. Which is wrong, note that SNESJacobianEvaluate, KSPSolve etc are embedded in the SNES solver but this is not properly displayed. Shouldn't they be in one level from the SNESSolve? Is this a bug, a feature? Or ...? >> >> Thanks for any information, >> >> Barry >> >> The major problem we are seeing with the nested logging is in the >> branch mark/snes-ex56c when we run src/snes/examples/tutorials/ex56 >> with >> >> petscmpiexec -n 1 ./ex56 -dm_refine 2 -ne 8 -alpha 1.e-3 -two_solves >> false -petscspace_poly_tensor -petscspace_order 1 -ksp_type cg >> -ksp_monitor_short -ksp_rtol 1.e-8 -pc_type gamg -pc_gamg_type agg >> -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 100 >> -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1 >> -pc_gamg_threshold 0.0 -ksp_converged_reason -use_mat_nearnullspace >> true -mg_levels_ksp_max_it 2 -mg_levels_ksp_type chebyshev >> -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10 >> -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -mg_levels_pc_type sor >> -mat_block_size 3 -petscpartitioner_type chaco -log_view >> ascii:ex56-intel2016_knl_fast_64ranks_ne8_dmrefine3_log.xml:ascii_xml >> >> it messes up the nesting and has total nonsense for the numerical values of time, for example in the different events, while the traditional -log_summary prints out reasonable results. It seems somehow either to be not gathering the data properly into the nested event data structures you have or not properly processing the data to generate the xml. I tried debugging but the logic is unclear to me. >> >> Simple programs such as: >> ierr = PetscLogEventRegister("Event1",0,&event1);CHKERRQ(ierr); >> >> ierr = PetscLogEventBegin(event1,0,0,0,0);CHKERRQ(ierr); >> ierr = PetscSleep(1.0);CHKERRQ(ierr); >> ierr = PetscLogEventEnd(event1,0,0,0,0);CHKERRQ(ierr); >> >> produce: >> >> [0]PETSC ERROR: Petsc has generated inconsistent data [0]PETSC ERROR: >> Depth 2 > maxdepth + 1 1 [0]PETSC ERROR: See >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.7.1-405-gbb23584 >> GIT Date: 2016-06-04 11:37:36 -0500 [0]PETSC ERROR: ./ex30 on a >> arch-xmllog named Barrys-MacBook-Pro.local by barrysmith Sat Jun 4 >> 18:13:00 2016 [0]PETSC ERROR: Configure options --download-chaco >> --with-mpi-dir=/Users/barrysmith/libraries PETSC_ARCH=arch-xmllog >> [0]PETSC ERROR: #1 PetscCreateLogTreeNested() line 719 in >> /Users/barrysmith/Src/petsc/src/sys/logging/xmllogevent.c >> >> ierr = PetscLogEventRegister("Event1",0,&event1);CHKERRQ(ierr); >> ierr = PetscLogEventRegister("Event2",0,&event2);CHKERRQ(ierr); >> ierr = PetscLogEventRegister("Event3",0,&event3);CHKERRQ(ierr); >> >> ierr = PetscLogEventBegin(event1,0,0,0,0);CHKERRQ(ierr); >> ierr = PetscSleep(1.0);CHKERRQ(ierr); >> ierr = PetscLogEventBegin(event2,0,0,0,0);CHKERRQ(ierr); >> ierr = PetscSleep(1.0);CHKERRQ(ierr); >> ierr = PetscLogEventBegin(event3,0,0,0,0);CHKERRQ(ierr); >> ierr = PetscSleep(1.0);CHKERRQ(ierr); >> ierr = PetscLogEventEnd(event3,0,0,0,0);CHKERRQ(ierr); >> ierr = PetscLogEventEnd(event2,0,0,0,0);CHKERRQ(ierr); >> ierr = PetscLogEventEnd(event1,0,0,0,0);CHKERRQ(ierr); >> >> doesn't crash but doesn't nest event2 and 3 in one but does nest 3 into 2. >> >> If seems that the ordering of the event values mater, if I change the >> registration order to >> >> ierr = PetscLogEventRegister("Event2",0,&event2);CHKERRQ(ierr); >> ierr = PetscLogEventRegister("Event1",0,&event1);CHKERRQ(ierr); >> ierr = PetscLogEventRegister("Event3",0,&event3);CHKERRQ(ierr); >> >> then it crashes with >> >> [0]PETSC ERROR: Petsc has generated inconsistent data [0]PETSC ERROR: >> Depth 2 > maxdepth + 1 1 [0]PETSC ERROR: See >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.7.1-405-gbb23584 >> GIT Date: 2016-06-04 11:37:36 -0500 [0]PETSC ERROR: ./ex30 on a >> arch-xmllog named Barrys-MacBook-Pro.local by barrysmith Sat Jun 4 >> 18:17:40 2016 [0]PETSC ERROR: Configure options --download-chaco >> --with-mpi-dir=/Users/barrysmith/libraries PETSC_ARCH=arch-xmllog >> [0]PETSC ERROR: #1 PetscCreateLogTreeNested() line 719 in >> /Users/barrysmith/Src/petsc/src/sys/logging/xmllogevent.c >> [0]PETSC ERROR: #2 PetscLogView_Nested() line 1399 in >> /Users/barrysmith/Src/petsc/src/sys/logging/xmllogevent.c >> >> >> >> Possibly related issue: If you run an example that has nothing that is actually logged but attempt to use the -log_view it crashes, there is some implicit assumption in your generation of the xml that some values will be non-empty. >> >> >> >> >> >> >> On Sep 18, 2<image002.png>015, at 10:09 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >> >> >> Thank you for contributing the nested logging. I have incorporated >> into the PETSc branch barry/xml-nested-logging if you look at >> https://bitbucket.org/petsc/petsc/commits/bb1d7374b64f295b2ed5ff23b894 >> 35d65e905a54?at=masteryou can see exactly what I have incorporated >> into PETSc. I will merge it into next for portability testing in the >> next couple of days. I expect over time as I understand it better I >> will be able to improve its integration with PETSc. Currently to >> generate the nested logging it is as simple to use as -log_view >> :filename.xml:ascii_xml >> >> Thanks again, >> >> Barry >> >> >> >> >> >> >> On Sep 14, 2015, at 7:45 AM, Koos Huijssen <koos.huijssen(a)vortech.nl> wrote: >> >> Dear PETSc development team, >> >> We have developed an extension of the PETSc event logging facilities that has the following advanced features: >> >> - It allows logging of events in the form of a nested tree. So if some function is called from multiple locations in the code, these instances are distinguished. This in contrast with the standard event logger, which only logs the amount of total call time. >> - It allows the output report to be formatted in XML format. This >> output can then be viewed in a human-friendly form in a web browser with the use of the XSL Transformation script performance_xml2html.xsl. The html features an nested timings tree that can be expanded and collapsed as desired. >> >> This tool has been very useful for us to analyze the code and pinpoint performance bottle necks. We think that it can be useful for others as well, and therefore we are providing the code here for integration in the open source distribution of PETSc. >> >> For more information I refer to the included manual. We have also provided a test program and a makefile for convenience. The test program can be run using MPI with for instance 3-6 processes. >> >> I apologize for not using the git repo to submit the developed code. I also apologize for not adhering to the PETSc coding standards (or at least not as far as I know), but I hope that it is not too far off.. Apart from the whole capital/underscore standardization stuff one issue may require special attention, namely the (ab)use of the format PETSc_VIEWER_ASCII_IMPL for signaling the XML format in XMLViewer.c. I couldn't find an already existing and better fitting format, but it could be necessary to add a new format here for this purpose. >> >> Can you take it up from here and realize the integration of the code in the PETSc distribution? >> >> With kind regards, >> >> Koos Huijssen >> >> -- >> ____________________________________________________________________ >> >> VORtech BV - Scientific software engineers >> ____________________________________________________________________ >> >> Dr.ir. Koos Huijssen >> >> P.O. Box 260 >> 2600 AG Delft >> The Netherlands >> >> phone +31(0)15-285 0125 >> mobile +31(0)6-3333 0803 >> email koos.huijssen(a)vortech.nl >> web www.vortech.nl >> ____________________________________________________________________ >> >> <timers.tar.gz> >> >> > > <0001-ascii_xml-logging-fixes-to-nested-tree-generation-an.patch>

1 0

Re: [petsc-dev] DMGetOutputDM hangs on multiple procs
by Sander Arens 13 Jul '16

13 Jul '16

Ah, great! Thanks, Matt. On 13 July 2016 at 04:44, Matthew Knepley <knepley(a)gmail.com> wrote: > On Thu, Jun 23, 2016 at 8:33 AM, Matthew Knepley <knepley(a)gmail.com> > wrote: > >> On Thu, Jun 23, 2016 at 5:46 AM, Sander Arens <Sander.Arens(a)ugent.be> >> wrote: >> >>> I noticed that DMGetOutputDM hangs when using multiple procs. The >>> culprit is >>> https://bitbucket.org/petsc/petsc/commits/c26acbdef0eedda92db6423afb8b38a15…. >>> >>> This doesn't work when one or more procs don't have any constraints >>> while others have, because the ones with constraints call DMClone >>> (collective) while the other procs don't. >>> >> >> Yep, will have to make the constraint logic global. I can do this at the >> PETSc Meeting >> next week in Vienna. >> > > My timing was off, but the fix is in 'next' if you want to try it. > > Thanks, > > Matt > > >> Thanks, >> >> Matt >> >> >>> Thanks, >>> Sander >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener >

1 0

Re: [petsc-dev] DMGetOutputDM hangs on multiple procs
by Matthew Knepley 12 Jul '16

12 Jul '16

On Thu, Jun 23, 2016 at 8:33 AM, Matthew Knepley <knepley(a)gmail.com> wrote: > On Thu, Jun 23, 2016 at 5:46 AM, Sander Arens <Sander.Arens(a)ugent.be> > wrote: > >> I noticed that DMGetOutputDM hangs when using multiple procs. The culprit >> is >> https://bitbucket.org/petsc/petsc/commits/c26acbdef0eedda92db6423afb8b38a15…. >> >> This doesn't work when one or more procs don't have any constraints while >> others have, because the ones with constraints call DMClone (collective) >> while the other procs don't. >> > > Yep, will have to make the constraint logic global. I can do this at the > PETSc Meeting > next week in Vienna. > My timing was off, but the fix is in 'next' if you want to try it. Thanks, Matt > Thanks, > > Matt > > >> Thanks, >> Sander >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] p4est configure error
by Mark Adams 09 Jul '16

09 Jul '16

On Fri, Jul 8, 2016 at 1:30 PM, Tobin Isaac <tisaac(a)uchicago.edu> wrote: > > What is `libtool --version` on Cori? > 2.4.2 on Edison: 07:59 edison09 maint= ~/petsc_install/petsc$ libtool --version libtool (GNU libtool) 2.4.6 On Fri, Jul 08, 2016 at 08:37:15AM -0400, Mark Adams wrote: > > And here is the good configure on Cori, > > > > On Fri, Jul 8, 2016 at 8:23 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > As I said, I need two passes to get this to work, here is the first > > > (failed) configure on Cori. With any luck my current configure will > work > > > and I can send that along. > > > Note, this used to work on Edison, so it might be something not worth, > or > > > even possible, to fix with NERSC's help. > > > Thanks, > > > > > > On Thu, Jul 7, 2016 at 9:12 PM, Tobin Isaac <tisaac(a)uchicago.edu> > wrote: > > > > > >> I'd be happy to help some more if you send a successful log from Cori. > > >> > > >> On July 7, 2016 7:18:30 PM CDT, Mark Adams <mfadams(a)lbl.gov> wrote: > > >> >Yes, if fails on debug=0. This used to work on Edison and it does > work > > >> >now > > >> >on Cori. > > >> > > > >> >On Fri, Jul 8, 2016 at 1:07 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > >> > > > >> >> whoops, that did not have p4est > > >> >> > > >> >> > > >> >> On Fri, Jul 8, 2016 at 1:04 AM, Mark Adams <mfadams(a)lbl.gov> > wrote: > > >> >> > > >> >>> > > >> >>> > > >> >>> On Fri, Jul 8, 2016 at 12:18 AM, Tobin Isaac <tisaac(a)uchicago.edu > > > > >> >wrote: > > >> >>> > > >> >>>> Interesting, you'd think that --disable-shared would be all that > > >> >>>> matters. Could you please send that configure.log for > comparison? > > >> >>>> > > >> >>>> On July 7, 2016 5:14:13 PM CDT, Mark Adams <mfadams(a)lbl.gov> > wrote: > > >> >>>> >Well, this did not work as you suspected ... FYI, it seems to > work > > >> >>>> >with-debugging=0. > > >> >>>> > > > >> >>>> >On Thu, Jul 7, 2016 at 11:13 PM, Satish Balay < > balay(a)mcs.anl.gov> > > >> >>>> >wrote: > > >> >>>> > > > >> >>>> >> > > >> >>>> >> > > >> >>>> >> >>>>>>> > > >> >>>> >> /bin/sh ./libtool --tag=CC --mode=link cc -no-ipo -g -O0 > > >> >-o > > >> >>>> >> libb64/sc_b64enc libb64/libb64_sc_b64enc-b64enc.o ./src/ > libsc.la > > >> >>>> >-lgomp > > >> >>>> >> libtool: link: cc -no-ipo -g -O0 -o libb64/sc_b64enc > > >> >>>> >> libb64/libb64_sc_b64enc-b64enc.o ./src/.libs/libsc.a > > >> >>>> >> /opt/gcc/5.2.0/snos/lib/../lib64/libgomp.so -lrt -ldl > -Wl,-rpath > > >> >>>> >> -Wl,/opt/gcc/5.2.0/snos/lib/../lib64 -Wl,-rpath > > >> >>>> >> -Wl,/opt/gcc/5.2.0/snos/lib/../lib64 > > >> >>>> >> <<<<<<< > > >> >>>> >> > > >> >>>> >> For some reason libtool is converting "-lgomp" into > > >> >>>> >> "/opt/gcc/5.2.0/snos/lib/../lib64/libgomp.so" - and giving an > ld > > >> >>>> >error. > > >> >>>> >> > > >> >>>> >> On my linux box - I see: > > >> >>>> >> >>>>>>> > > >> >>>> >> /bin/sh ./libtool --tag=CC --mode=link mpicc -g3 -o > > >> >>>> >libb64/sc_b64enc > > >> >>>> >> libb64/libb64_sc_b64enc-b64enc.o ./src/libsc.la -lgomp > -lpthread > > >> >-lz > > >> >>>> >-lm > > >> >>>> >> libtool: link: mpicc -g3 -o libb64/sc_b64enc > > >> >>>> >> libb64/libb64_sc_b64enc-b64enc.o ./src/.libs/libsc.a -lgomp > > >> >>>> >-lpthread -lz > > >> >>>> >> -lm > > >> >>>> >> <<<<<< > > >> >>>> >> > > >> >>>> >> I don't really understand libtool - or this error.. > > >> >>>> >> > > >> >>>> >> Satish > > >> >>>> >> > > >> >>>> >> On Thu, 7 Jul 2016, Tobin Isaac wrote: > > >> >>>> >> > > >> >>>> >> > Nope, different error: p4est autotools found an openmp > library > > >> >and > > >> >>>> >tried > > >> >>>> >> to link it, but there's a shared/static mismatch. I'll try to > > >> >get to > > >> >>>> >the > > >> >>>> >> bottom of this. > > >> >>>> >> > > > >> >>>> >> > > > >> >>>> >> > On July 7, 2016 3:27:23 PM CDT, Mark Adams <mfadams(a)lbl.gov > > > > >> >wrote: > > >> >>>> >> > >I nuked and ran this twice: > > >> >>>> >> > > > > >> >>>> >> > >1013 rm -fr arch-xc30-dbg64-intel > > >> >>>> >> > > 1014 ../arch-xc30-dbg64-intel.py > > >> >>>> >> > > 1015 h > > >> >>>> >> > > 1016 ../arch-xc30-dbg64-intel.py > > >> >>>> >> > > > > >> >>>> >> > >Same error. > > >> >>>> >> > > > > >> >>>> >> > > > > >> >>>> >> > >On Thu, Jul 7, 2016 at 9:36 PM, Satish Balay > > >> ><balay(a)mcs.anl.gov> > > >> >>>> >wrote: > > >> >>>> >> > > > > >> >>>> >> > >> perhaps configure was not rerun - so a stale p4est was > > >> >being > > >> >>>> >used.. > > >> >>>> >> > >> > > >> >>>> >> > >> Satish > > >> >>>> >> > >> > > >> >>>> >> > >> On Thu, 7 Jul 2016, Tobin Isaac wrote: > > >> >>>> >> > >> > > >> >>>> >> > >> > p4est.py automatically pulls from whatever p4est > branch I > > >> >call > > >> >>>> >> > >'petsc' > > >> >>>> >> > >> > in my repo, so I don't see how this problem persists if > > >> >you > > >> >>>> >really > > >> >>>> >> > >> > nuked it. > > >> >>>> >> > >> > > > >> >>>> >> > >> > - Did you nuke the right directory? > > >> >>>> >> > >> > - Did you send me the right configure.log? > > >> >>>> >> > >> > - Is the new failure the same (the same stale p4est > > >> >commit is > > >> >>>> >being > > >> >>>> >> > >> > used)? > > >> >>>> >> > >> > > > >> >>>> >> > >> > On Thu, Jul 07, 2016 at 08:33:57PM +0200, Mark Adams > > >> >wrote: > > >> >>>> >> > >> > > It sounds like I have not merged your fix. Should I > > >> >pull > > >> >>>> >from > > >> >>>> >> > >master? > > >> >>>> >> > >> > > > > >> >>>> >> > >> > > I have nuked this many times. > > >> >>>> >> > >> > > > > >> >>>> >> > >> > > On Thu, Jul 7, 2016 at 4:26 PM, Tobin Isaac > > >> >>>> ><tisaac(a)uchicago.edu> > > >> >>>> >> > >> wrote: > > >> >>>> >> > >> > > > > >> >>>> >> > >> > > > > > >> >>>> >> > >> > > > On Thu, Jul 07, 2016 at 03:16:40PM +0200, Mark > Adams > > >> >>>> >wrote: > > >> >>>> >> > >> > > > > p4est on Edison configures in w/o debugging, > > >> >although I > > >> >>>> >have > > >> >>>> >> > >to run > > >> >>>> >> > >> > > > > configure twice. > > >> >>>> >> > >> > > > > But with debugging it fails. I can not see any > > >> >>>> >difference > > >> >>>> >> > >between > > >> >>>> >> > >> the > > >> >>>> >> > >> > > > two > > >> >>>> >> > >> > > > > configures other than debugging. > > >> >>>> >> > >> > > > > Mark > > >> >>>> >> > >> > > > > > >> >>>> >> > >> > > > The log shows that you are configuring a p4est > commit > > >> >>>> >(3d3e1) > > >> >>>> >> > >that is > > >> >>>> >> > >> > > > before the one I pushed specifically to avoid the > > >> >fortran > > >> >>>> >> > >> > > > configuration error (f54bb3). Please clear the > > >> >git.p4est > > >> >>>> >> > >directory, > > >> >>>> >> > >> > > > and confirm that you are configuring with the > current > > >> >>>> >'petsc' > > >> >>>> >> > >branch > > >> >>>> >> > >> > > > of p4est (`grep 'checking Point version' > > >> >configure.log` > > >> >>>> >should > > >> >>>> >> > >show > > >> >>>> >> > >> > > > X.XXX-a9f22). > > >> >>>> >> > >> > > > > > >> >>>> >> > >> > > > Cheers, > > >> >>>> >> > >> > > > Toby > > >> >>>> >> > >> > > > > > >> >>>> >> > >> > > > >> >>>> >> > >> > > >> >>>> >> > >> > > >> >>>> >> > > > >> >>>> >> > > > >> >>>> >> > > >> >>>> >> > > >> >>>> > > >> >>>> > > >> >>> > > >> >> > > >> > > >> > > > > > >

1 0

Re: [petsc-dev] Soliciting suggestions for linear solver work under SciDAC 4 Institutes
by Jeff Hammond 08 Jul '16

08 Jul '16

On Friday, July 8, 2016, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > On Jul 8, 2016, at 12:17 PM, Jeff Hammond <jeff.science(a)gmail.com > <javascript:;>> wrote: > > > > > > > > On Fri, Jul 8, 2016 at 9:48 AM, Richard Mills <richardtmills(a)gmail.com > <javascript:;>> wrote: > > > > > > On Fri, Jul 8, 2016 at 9:40 AM, Jeff Hammond <jeff.science(a)gmail.com > <javascript:;>> wrote: > > > > > 1) How do we run at bandwidth peak on new architectures like Cori or > Aurora? > > > > Huh, there is a how here, not a why? > > > > > > Patrick and Rich have good suggestions here. Karl and Rich showed some > promising numbers for KNL at the PETSc meeting. > > > > > > > > > Future systems from multiple vendors basically move from 2-tier memory > hierarchy of shared LLC and DRAM to a 3-tier hierarchy of fast memory (e.g. > HBM), regular memory (e.g. DRAM), and slow (likely nonvolatile) memory on > a node. > > > > Jeff, > > > > Would Intel sell me a system that had essentially no regular memory > DRAM (which is too slow anyway) and no slow memory (which is absurdly too > slow)? What cost savings would I get in $ and power usage compared to say > what is going in the theta? 10% and 20%, 5% and 30%, 5% and 5 %? If it is a > significant savings then get the cut down machine, if it is insignificant > than realize the cost of not using it (the DRAM you paid so little for) is > insignificant and not worth worrying about, just like cruise control when > you don't use the highway. Actually I could use the DRAM to store the > history needed for the adjoints; so maybe it is ok to keep, but surely not > useful for data that is continuously involved in the computation. > > > > Disclaimer: All of the following data is pulled off of the Internet, > which in some cases is horribly unreliable. My comments are strictly for > academic discussion and not meant to be authoritative or have any influence > on purchasing or design decisions. Do not equate quoted TDP to measured > power during any workload, or assume that different measurements can be > compared directly. > > > > Your thinking is in line with > http://www.nextplatform.com/2015/08/03/future-systems-intel-ponders-breakin…. > .. > > > > Intel sells KNL packages as parts ( > http://ark.intel.com/products/family/92650/Intel-Xeon-Phi-Product-Family-x2…) > that don't have any DRAM in them, just MCDRAM. It's the decision of the > integrator what goes into the system, which of course is correlated to what > the intended customer wants. While you might not need a node with DRAM, > many users do, and the systems that DOE buys are designed to meet the needs > of their broad user base. > > > > I don't know if KNL is bootable without no DRAM at all - this is likely > more to do with what motherboard, BIOS, etc. expect than the processor > package itself. However, the KNL alltoall mode addresses the case where > DRAM channels are underpopulated (with fully populated channels, one should > use quadrant, hemisphere, SNC-2 or SNC-4), so if DRAM is necessary, you > should be able to boot it with only one channel populated. Of course, if > you do this, you'll get 1/6 of the DDR4 bandwidth. > > > > Just FYI: I have run on KNL systems with no DRAM, only MCDRAM. This was > on an internal lab machine and not a commercially available system, but I > see no reason why one couldn't buy systems this way. > > > > > > It puts quite a bit of pressure on the system software footprint if one > does not have DDR4. Blue Gene CNK had a very small memory footprint, but > commodity Linux is certainly much larger. The memory footprint of MPI at > scale depends on the fabric HW/SW. It's probably cheaper to buy one 32 GB > stick per node than pay for someone to write the system software that gives > you at least 15.5 GB of usable MCDRAM. > > > Sure. > > But this doesn't help with the question of whether the DRAM uses > significant power. Say it does use significant power then it might make > sense to be able to turn off most of the DRAM via software when running > PETSc programs. If it doesn't use significant power then it doesn't matter > if we don't use it. > > Like I said, it's approximately 0.37 W/GB and in the neighborhood of 15-20% of total system power in supercomputers. On the other hand, CPU is approx 50-60% of system power and, unlike DRAM, amenable to a range of power optimizations, at least in theory. Whether operators give users C- and P-state control is a policy question. Jeff > Barry > > > Jeff > > > > -- > > Jeff Hammond > > jeff.science(a)gmail.com <javascript:;> > > http://jeffhammond.github.io/ > > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [petsc-dev] Soliciting suggestions for linear solver work under SciDAC 4 Institutes
by Barry Smith 08 Jul '16

08 Jul '16

> On Jul 8, 2016, at 12:17 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > > > On Fri, Jul 8, 2016 at 9:48 AM, Richard Mills <richardtmills(a)gmail.com> wrote: > > > On Fri, Jul 8, 2016 at 9:40 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > > 1) How do we run at bandwidth peak on new architectures like Cori or Aurora? > > Huh, there is a how here, not a why? > > > > Patrick and Rich have good suggestions here. Karl and Rich showed some promising numbers for KNL at the PETSc meeting. > > > > > > Future systems from multiple vendors basically move from 2-tier memory hierarchy of shared LLC and DRAM to a 3-tier hierarchy of fast memory (e.g. HBM), regular memory (e.g. DRAM), and slow (likely nonvolatile) memory on a node. > > Jeff, > > Would Intel sell me a system that had essentially no regular memory DRAM (which is too slow anyway) and no slow memory (which is absurdly too slow)? What cost savings would I get in $ and power usage compared to say what is going in the theta? 10% and 20%, 5% and 30%, 5% and 5 %? If it is a significant savings then get the cut down machine, if it is insignificant than realize the cost of not using it (the DRAM you paid so little for) is insignificant and not worth worrying about, just like cruise control when you don't use the highway. Actually I could use the DRAM to store the history needed for the adjoints; so maybe it is ok to keep, but surely not useful for data that is continuously involved in the computation. > > Disclaimer: All of the following data is pulled off of the Internet, which in some cases is horribly unreliable. My comments are strictly for academic discussion and not meant to be authoritative or have any influence on purchasing or design decisions. Do not equate quoted TDP to measured power during any workload, or assume that different measurements can be compared directly. > > Your thinking is in line with http://www.nextplatform.com/2015/08/03/future-systems-intel-ponders-breakin…... > > Intel sells KNL packages as parts (http://ark.intel.com/products/family/92650/Intel-Xeon-Phi-Product-Family-x2…) that don't have any DRAM in them, just MCDRAM. It's the decision of the integrator what goes into the system, which of course is correlated to what the intended customer wants. While you might not need a node with DRAM, many users do, and the systems that DOE buys are designed to meet the needs of their broad user base. > > I don't know if KNL is bootable without no DRAM at all - this is likely more to do with what motherboard, BIOS, etc. expect than the processor package itself. However, the KNL alltoall mode addresses the case where DRAM channels are underpopulated (with fully populated channels, one should use quadrant, hemisphere, SNC-2 or SNC-4), so if DRAM is necessary, you should be able to boot it with only one channel populated. Of course, if you do this, you'll get 1/6 of the DDR4 bandwidth. > > Just FYI: I have run on KNL systems with no DRAM, only MCDRAM. This was on an internal lab machine and not a commercially available system, but I see no reason why one couldn't buy systems this way. > > > It puts quite a bit of pressure on the system software footprint if one does not have DDR4. Blue Gene CNK had a very small memory footprint, but commodity Linux is certainly much larger. The memory footprint of MPI at scale depends on the fabric HW/SW. It's probably cheaper to buy one 32 GB stick per node than pay for someone to write the system software that gives you at least 15.5 GB of usable MCDRAM. > Sure. But this doesn't help with the question of whether the DRAM uses significant power. Say it does use significant power then it might make sense to be able to turn off most of the DRAM via software when running PETSc programs. If it doesn't use significant power then it doesn't matter if we don't use it. Barry > Jeff > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/

1 0

Re: [petsc-dev] p4est configure error
by Tobin Isaac 08 Jul '16

08 Jul '16

What is `libtool --version` on Cori? On Fri, Jul 08, 2016 at 08:37:15AM -0400, Mark Adams wrote: > And here is the good configure on Cori, > > On Fri, Jul 8, 2016 at 8:23 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > As I said, I need two passes to get this to work, here is the first > > (failed) configure on Cori. With any luck my current configure will work > > and I can send that along. > > Note, this used to work on Edison, so it might be something not worth, or > > even possible, to fix with NERSC's help. > > Thanks, > > > > On Thu, Jul 7, 2016 at 9:12 PM, Tobin Isaac <tisaac(a)uchicago.edu> wrote: > > > >> I'd be happy to help some more if you send a successful log from Cori. > >> > >> On July 7, 2016 7:18:30 PM CDT, Mark Adams <mfadams(a)lbl.gov> wrote: > >> >Yes, if fails on debug=0. This used to work on Edison and it does work > >> >now > >> >on Cori. > >> > > >> >On Fri, Jul 8, 2016 at 1:07 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > >> > > >> >> whoops, that did not have p4est > >> >> > >> >> > >> >> On Fri, Jul 8, 2016 at 1:04 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > >> >> > >> >>> > >> >>> > >> >>> On Fri, Jul 8, 2016 at 12:18 AM, Tobin Isaac <tisaac(a)uchicago.edu> > >> >wrote: > >> >>> > >> >>>> Interesting, you'd think that --disable-shared would be all that > >> >>>> matters. Could you please send that configure.log for comparison? > >> >>>> > >> >>>> On July 7, 2016 5:14:13 PM CDT, Mark Adams <mfadams(a)lbl.gov> wrote: > >> >>>> >Well, this did not work as you suspected ... FYI, it seems to work > >> >>>> >with-debugging=0. > >> >>>> > > >> >>>> >On Thu, Jul 7, 2016 at 11:13 PM, Satish Balay <balay(a)mcs.anl.gov> > >> >>>> >wrote: > >> >>>> > > >> >>>> >> > >> >>>> >> > >> >>>> >> >>>>>>> > >> >>>> >> /bin/sh ./libtool --tag=CC --mode=link cc -no-ipo -g -O0 > >> >-o > >> >>>> >> libb64/sc_b64enc libb64/libb64_sc_b64enc-b64enc.o ./src/libsc.la > >> >>>> >-lgomp > >> >>>> >> libtool: link: cc -no-ipo -g -O0 -o libb64/sc_b64enc > >> >>>> >> libb64/libb64_sc_b64enc-b64enc.o ./src/.libs/libsc.a > >> >>>> >> /opt/gcc/5.2.0/snos/lib/../lib64/libgomp.so -lrt -ldl -Wl,-rpath > >> >>>> >> -Wl,/opt/gcc/5.2.0/snos/lib/../lib64 -Wl,-rpath > >> >>>> >> -Wl,/opt/gcc/5.2.0/snos/lib/../lib64 > >> >>>> >> <<<<<<< > >> >>>> >> > >> >>>> >> For some reason libtool is converting "-lgomp" into > >> >>>> >> "/opt/gcc/5.2.0/snos/lib/../lib64/libgomp.so" - and giving an ld > >> >>>> >error. > >> >>>> >> > >> >>>> >> On my linux box - I see: > >> >>>> >> >>>>>>> > >> >>>> >> /bin/sh ./libtool --tag=CC --mode=link mpicc -g3 -o > >> >>>> >libb64/sc_b64enc > >> >>>> >> libb64/libb64_sc_b64enc-b64enc.o ./src/libsc.la -lgomp -lpthread > >> >-lz > >> >>>> >-lm > >> >>>> >> libtool: link: mpicc -g3 -o libb64/sc_b64enc > >> >>>> >> libb64/libb64_sc_b64enc-b64enc.o ./src/.libs/libsc.a -lgomp > >> >>>> >-lpthread -lz > >> >>>> >> -lm > >> >>>> >> <<<<<< > >> >>>> >> > >> >>>> >> I don't really understand libtool - or this error.. > >> >>>> >> > >> >>>> >> Satish > >> >>>> >> > >> >>>> >> On Thu, 7 Jul 2016, Tobin Isaac wrote: > >> >>>> >> > >> >>>> >> > Nope, different error: p4est autotools found an openmp library > >> >and > >> >>>> >tried > >> >>>> >> to link it, but there's a shared/static mismatch. I'll try to > >> >get to > >> >>>> >the > >> >>>> >> bottom of this. > >> >>>> >> > > >> >>>> >> > > >> >>>> >> > On July 7, 2016 3:27:23 PM CDT, Mark Adams <mfadams(a)lbl.gov> > >> >wrote: > >> >>>> >> > >I nuked and ran this twice: > >> >>>> >> > > > >> >>>> >> > >1013 rm -fr arch-xc30-dbg64-intel > >> >>>> >> > > 1014 ../arch-xc30-dbg64-intel.py > >> >>>> >> > > 1015 h > >> >>>> >> > > 1016 ../arch-xc30-dbg64-intel.py > >> >>>> >> > > > >> >>>> >> > >Same error. > >> >>>> >> > > > >> >>>> >> > > > >> >>>> >> > >On Thu, Jul 7, 2016 at 9:36 PM, Satish Balay > >> ><balay(a)mcs.anl.gov> > >> >>>> >wrote: > >> >>>> >> > > > >> >>>> >> > >> perhaps configure was not rerun - so a stale p4est was > >> >being > >> >>>> >used.. > >> >>>> >> > >> > >> >>>> >> > >> Satish > >> >>>> >> > >> > >> >>>> >> > >> On Thu, 7 Jul 2016, Tobin Isaac wrote: > >> >>>> >> > >> > >> >>>> >> > >> > p4est.py automatically pulls from whatever p4est branch I > >> >call > >> >>>> >> > >'petsc' > >> >>>> >> > >> > in my repo, so I don't see how this problem persists if > >> >you > >> >>>> >really > >> >>>> >> > >> > nuked it. > >> >>>> >> > >> > > >> >>>> >> > >> > - Did you nuke the right directory? > >> >>>> >> > >> > - Did you send me the right configure.log? > >> >>>> >> > >> > - Is the new failure the same (the same stale p4est > >> >commit is > >> >>>> >being > >> >>>> >> > >> > used)? > >> >>>> >> > >> > > >> >>>> >> > >> > On Thu, Jul 07, 2016 at 08:33:57PM +0200, Mark Adams > >> >wrote: > >> >>>> >> > >> > > It sounds like I have not merged your fix. Should I > >> >pull > >> >>>> >from > >> >>>> >> > >master? > >> >>>> >> > >> > > > >> >>>> >> > >> > > I have nuked this many times. > >> >>>> >> > >> > > > >> >>>> >> > >> > > On Thu, Jul 7, 2016 at 4:26 PM, Tobin Isaac > >> >>>> ><tisaac(a)uchicago.edu> > >> >>>> >> > >> wrote: > >> >>>> >> > >> > > > >> >>>> >> > >> > > > > >> >>>> >> > >> > > > On Thu, Jul 07, 2016 at 03:16:40PM +0200, Mark Adams > >> >>>> >wrote: > >> >>>> >> > >> > > > > p4est on Edison configures in w/o debugging, > >> >although I > >> >>>> >have > >> >>>> >> > >to run > >> >>>> >> > >> > > > > configure twice. > >> >>>> >> > >> > > > > But with debugging it fails. I can not see any > >> >>>> >difference > >> >>>> >> > >between > >> >>>> >> > >> the > >> >>>> >> > >> > > > two > >> >>>> >> > >> > > > > configures other than debugging. > >> >>>> >> > >> > > > > Mark > >> >>>> >> > >> > > > > >> >>>> >> > >> > > > The log shows that you are configuring a p4est commit > >> >>>> >(3d3e1) > >> >>>> >> > >that is > >> >>>> >> > >> > > > before the one I pushed specifically to avoid the > >> >fortran > >> >>>> >> > >> > > > configuration error (f54bb3). Please clear the > >> >git.p4est > >> >>>> >> > >directory, > >> >>>> >> > >> > > > and confirm that you are configuring with the current > >> >>>> >'petsc' > >> >>>> >> > >branch > >> >>>> >> > >> > > > of p4est (`grep 'checking Point version' > >> >configure.log` > >> >>>> >should > >> >>>> >> > >show > >> >>>> >> > >> > > > X.XXX-a9f22). > >> >>>> >> > >> > > > > >> >>>> >> > >> > > > Cheers, > >> >>>> >> > >> > > > Toby > >> >>>> >> > >> > > > > >> >>>> >> > >> > > >> >>>> >> > >> > >> >>>> >> > >> > >> >>>> >> > > >> >>>> >> > > >> >>>> >> > >> >>>> >> > >> >>>> > >> >>>> > >> >>> > >> >> > >> > >> > >

1 0

Re: [petsc-dev] Soliciting suggestions for linear solver work under SciDAC 4 Institutes
by Jeff Hammond 08 Jul '16

08 Jul '16

On Fri, Jul 8, 2016 at 9:48 AM, Richard Mills <richardtmills(a)gmail.com> wrote: > > > On Fri, Jul 8, 2016 at 9:40 AM, Jeff Hammond <jeff.science(a)gmail.com> > wrote: > >> >>> > 1) How do we run at bandwidth peak on new architectures like Cori or >>> Aurora? >>> >>> Huh, there is a how here, not a why? >>> > >>> > Patrick and Rich have good suggestions here. Karl and Rich showed some >>> promising numbers for KNL at the PETSc meeting. >>> > >>> > >>> > Future systems from multiple vendors basically move from 2-tier memory >>> hierarchy of shared LLC and DRAM to a 3-tier hierarchy of fast memory (e.g. >>> HBM), regular memory (e.g. DRAM), and slow (likely nonvolatile) memory on >>> a node. >>> >>> Jeff, >>> >>> Would Intel sell me a system that had essentially no regular memory >>> DRAM (which is too slow anyway) and no slow memory (which is absurdly too >>> slow)? What cost savings would I get in $ and power usage compared to say >>> what is going in the theta? 10% and 20%, 5% and 30%, 5% and 5 %? If it is a >>> significant savings then get the cut down machine, if it is insignificant >>> than realize the cost of not using it (the DRAM you paid so little for) is >>> insignificant and not worth worrying about, just like cruise control when >>> you don't use the highway. Actually I could use the DRAM to store the >>> history needed for the adjoints; so maybe it is ok to keep, but surely not >>> useful for data that is continuously involved in the computation. >>> >> >> *Disclaimer: All of the following data is pulled off of the Internet, >> which in some cases is horribly unreliable. My comments are strictly for >> academic discussion and not meant to be authoritative or have any influence >> on purchasing or design decisions. Do not equate quoted TDP to measured >> power during any workload, or assume that different measurements can be >> compared directly.* >> >> Your thinking is in line with >> http://www.nextplatform.com/2015/08/03/future-systems-intel-ponders-breakin…. >> .. >> >> Intel sells KNL packages as parts ( >> http://ark.intel.com/products/family/92650/Intel-Xeon-Phi-Product-Family-x2…) >> that don't have any DRAM in them, just MCDRAM. It's the decision of the >> integrator what goes into the system, which of course is correlated to what >> the intended customer wants. While you might not need a node with DRAM, >> many users do, and the systems that DOE buys are designed to meet the needs >> of their broad user base. >> >> I don't know if KNL is bootable without no DRAM at all - this is likely >> more to do with what motherboard, BIOS, etc. expect than the processor >> package itself. However, the KNL alltoall mode addresses the case where >> DRAM channels are underpopulated (with fully populated channels, one should >> use quadrant, hemisphere, SNC-2 or SNC-4), so if DRAM is necessary, you >> should be able to boot it with only one channel populated. Of course, if >> you do this, you'll get 1/6 of the DDR4 bandwidth. >> > > Just FYI: I have run on KNL systems with no DRAM, only MCDRAM. This was > on an internal lab machine and not a commercially available system, but I > see no reason why one couldn't buy systems this way. > > It puts quite a bit of pressure on the system software footprint if one does not have DDR4. Blue Gene CNK had a very small memory footprint, but commodity Linux is certainly much larger. The memory footprint of MPI at scale depends on the fabric HW/SW. It's probably cheaper to buy one 32 GB stick per node than pay for someone to write the system software that gives you at least 15.5 GB of usable MCDRAM. Jeff -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [petsc-dev] Soliciting suggestions for linear solver work under SciDAC 4 Institutes
by Richard Mills 08 Jul '16

08 Jul '16

On Fri, Jul 8, 2016 at 9:40 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > >> > 1) How do we run at bandwidth peak on new architectures like Cori or >> Aurora? >> >> Huh, there is a how here, not a why? >> > >> > Patrick and Rich have good suggestions here. Karl and Rich showed some >> promising numbers for KNL at the PETSc meeting. >> > >> > >> > Future systems from multiple vendors basically move from 2-tier memory >> hierarchy of shared LLC and DRAM to a 3-tier hierarchy of fast memory (e.g. >> HBM), regular memory (e.g. DRAM), and slow (likely nonvolatile) memory on >> a node. >> >> Jeff, >> >> Would Intel sell me a system that had essentially no regular memory >> DRAM (which is too slow anyway) and no slow memory (which is absurdly too >> slow)? What cost savings would I get in $ and power usage compared to say >> what is going in the theta? 10% and 20%, 5% and 30%, 5% and 5 %? If it is a >> significant savings then get the cut down machine, if it is insignificant >> than realize the cost of not using it (the DRAM you paid so little for) is >> insignificant and not worth worrying about, just like cruise control when >> you don't use the highway. Actually I could use the DRAM to store the >> history needed for the adjoints; so maybe it is ok to keep, but surely not >> useful for data that is continuously involved in the computation. >> > > *Disclaimer: All of the following data is pulled off of the Internet, > which in some cases is horribly unreliable. My comments are strictly for > academic discussion and not meant to be authoritative or have any influence > on purchasing or design decisions. Do not equate quoted TDP to measured > power during any workload, or assume that different measurements can be > compared directly.* > > Your thinking is in line with > http://www.nextplatform.com/2015/08/03/future-systems-intel-ponders-breakin…. > .. > > Intel sells KNL packages as parts ( > http://ark.intel.com/products/family/92650/Intel-Xeon-Phi-Product-Family-x2…) > that don't have any DRAM in them, just MCDRAM. It's the decision of the > integrator what goes into the system, which of course is correlated to what > the intended customer wants. While you might not need a node with DRAM, > many users do, and the systems that DOE buys are designed to meet the needs > of their broad user base. > > I don't know if KNL is bootable without no DRAM at all - this is likely > more to do with what motherboard, BIOS, etc. expect than the processor > package itself. However, the KNL alltoall mode addresses the case where > DRAM channels are underpopulated (with fully populated channels, one should > use quadrant, hemisphere, SNC-2 or SNC-4), so if DRAM is necessary, you > should be able to boot it with only one channel populated. Of course, if > you do this, you'll get 1/6 of the DDR4 bandwidth. > Just FYI: I have run on KNL systems with no DRAM, only MCDRAM. This was on an internal lab machine and not a commercially available system, but I see no reason why one couldn't buy systems this way. --Richard > As to the question of DRAM power, there is a lot of detailed information > available (e.g. > https://www.micron.com/~/media/documents/products/power-calculator/ddr4_pow…, > > https://www.micron.com/~/media/Documents/Products/Technical%20Note/DRAM/TN4…, > https://lenovopress.com/lp0083.pdf) but since I am lazy, I'll use the > numbers reported on > http://www.tomshardware.com/reviews/intel-core-i7-5960x-haswell-e-cpu,3918-… > for client memory (i.e. not server memory, hence probably not providing > ECC, but ECC doesn't change power consumption much), which works out to > 0.37 W/GB for DDR4-2133, hence 71 W for 192 GB [ > http://www.nextplatform.com/2015/11/30/inside-future-knights-landing-xeon-p…] > That 71W is ~1/3 of the processor package power (215W). The network > adapter draws some power, and the cables and switches (especially optics) > are a nontrivial power draw. So DRAM is at most 25% of the node power, and > perhaps ~17% of system power based upon what I can derive from Shaheen II. > > Shaheen II Cray XC40 > 1.96 MW = 6174 * (2 sockets * 135 W/socket + 128 GB * 0.37 W/GB) > 2.83 MW total > = 69% from CPU+DRAM > > Again, *these are not the exact numbers* but what I can derive from > https://www.top500.org/system/178515, > https://www.hpc.kaust.edu.sa/content/shaheen-ii and > http://ark.intel.com/products/81060/Intel-Xeon-Processor-E5-2698-v3-40M-Cac… > . > > Back to the higher level analysis, what is unfortunate about DRAM is that > it needs power to hold data even if the data isn't used, because it is not > persistent. I don't know how well it powers down when the physical memory > isn't mapped but it seems that power is not gated today [ > http://digitalpiglet.org/research/sion2014socc.pdf] The advantage of > nonvolatile memory is that it doesn't require power when not being > accessed, whether or not the data is preserved. > > I suspect that nonvolatile memory (NVM) is the right place to put your > adjoint matrices, provided the NVM bandwidth is sufficient. > > *Disclaimer: All of these are academic comments. Do not use them to try > to influence others or make any decisions. Do your own research and be > skeptical of everything I derived from the Internet.* > > Jeff > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ >

1 0