November 2009 - petsc-dev - mailman3.cels.anl.gov

Re: -malign-double
by Jed Brown 17 Nov '09

17 Nov '09

Barry Smith wrote: > Does it? I ask myself everyday! Well, rdtsc just gives one measurement, maybe if I was familiar with how to read the others I would have no use for PAPI. It's a truly spectacular quantity of code for just reading counters (sloccount tells me it's as big as PETSc!). Of course interrupt-based profilers (e.g. Oprofile and Sun's tool) can be the right thing, but -log_summary has a much lower entry cost. Jed

1 0

Re: -malign-double
by Barry Smith 17 Nov '09

17 Nov '09

On Nov 17, 2009, at 3:13 PM, Jed Brown wrote: > Barry Smith wrote: >> >> On Nov 16, 2009, at 2:50 PM, Jed Brown wrote: >> >>> >>> so. BTW, should we put PAPI support into PETSc, it no longer needs >>> kernel patches with 2.6.31. >>> >> The basics are there in config/PETSc/packages/PAPI.py and > > This file does not exist in the repo. sorry it is papi.py I haven't added the --download-papi support. >> > > Note that I'm totally new to PAPI and the interface does seem screwy > (not to mention all the private nonsense dumped into the header and > the > documentation is in MS-Word), but it offers a lot more than just > calling > rdtsc myself. Does it? I ask myself everyday! Barry > > Jed >

1 0

Re: -malign-double
by Jed Brown 17 Nov '09

17 Nov '09

Barry Smith wrote: > > On Nov 16, 2009, at 2:50 PM, Jed Brown wrote: > >> >> so. BTW, should we put PAPI support into PETSc, it no longer needs >> kernel patches with 2.6.31. >> > The basics are there in config/PETSc/packages/PAPI.py and This file does not exist in the repo. > initialization in plog.c and eventLog.c It would be good to add counts > for cache misses etc. Cache misses would be neat, but we have the problem of where to display it (there is limited space for extra columns in -log_summary). But it would be very slick to --download-papi and get flops for external libraries and user code. I think this can be made to work reliably on Linux from kernel 2.6.31 onward. > I got frustrated because with intel there are multiple instructions to > do similar things that have separate counts so the results you get out > are not really what you want. I see what you mean. What do we want? Some candidates L1_ICM - L1I miss L1_DCM - L1D miss L2_DCM - L2D miss L1_DCH (L1D hits) or LST_INS (load/store inst.) to be able to scale miss rates For flops we have FP_INS - floating point instructions, would count FMA as one, probably packed vector ins. are one FP_OPS - derived quantity, not clear how they distinguish between single/double SP_OPS/DP_OPS - probably better, use whichever agrees with PETSc's precision VEC_SP/VEC_DP - vector instructions, not sure this is relevant because everything is a vector instruction these days (compilers rarely use x87). Note that I'm totally new to PAPI and the interface does seem screwy (not to mention all the private nonsense dumped into the header and the documentation is in MS-Word), but it offers a lot more than just calling rdtsc myself. Jed

1 0

Re: -malign-double
by Barry Smith 17 Nov '09

17 Nov '09

Thanks On Nov 17, 2009, at 1:08 PM, Jed Brown wrote: > I think this is complete. > > http://petsc.cs.iit.edu/petsc/petsc-dev/rev/a36906938641 > > This was my first time messing with configure. It works in all > cases I > have tested, let me know if I screwed up. The default alignment is > now > 16, configurable via --with-memalign. Alignment of pointers > compiles to > (gcc-4.4.2/x86-64 at all positive optimization levels) > > lea r9,[r8+rbx*1+0xf] > and r9,0xfffffffffffffff0 > > so there should be no concern over runtime cost of alignment. If the > user provides their own malloc (via PetscMallocSet) which does not > provide the desired alignment then the first array will not have that > alignment (obviously) but every other will. It costs a few extra > bytes > per allocation but this should only add up to a few kB in real > applications. > > Jed >

1 0

Re: -malign-double
by Barry Smith 17 Nov '09

17 Nov '09

On Nov 16, 2009, at 2:50 PM, Jed Brown wrote: > > so. BTW, should we put PAPI support into PETSc, it no longer needs > kernel patches with 2.6.31. > The basics are there in config/PETSc/packages/PAPI.py and initialization in plog.c and eventLog.c It would be good to add counts for cache misses etc. I got frustrated because with intel there are multiple instructions to do similar things that have separate counts so the results you get out are not really what you want. Plus, since Jack is involved in PAPI , it pretty much sucks. Barry > Jed >

1 0

Re: -malign-double
by Jed Brown 17 Nov '09

17 Nov '09

I think this is complete. http://petsc.cs.iit.edu/petsc/petsc-dev/rev/a36906938641 This was my first time messing with configure. It works in all cases I have tested, let me know if I screwed up. The default alignment is now 16, configurable via --with-memalign. Alignment of pointers compiles to (gcc-4.4.2/x86-64 at all positive optimization levels) lea r9,[r8+rbx*1+0xf] and r9,0xfffffffffffffff0 so there should be no concern over runtime cost of alignment. If the user provides their own malloc (via PetscMallocSet) which does not provide the desired alignment then the first array will not have that alignment (obviously) but every other will. It costs a few extra bytes per allocation but this should only add up to a few kB in real applications. Jed

1 0

Re: -malign-double
by Jed Brown 16 Nov '09

16 Nov '09

Barry Smith wrote: > > What about have configure look for uintptr_t if it does not exist > then have configure check if unsigned long long is large enough, if not > try size_t otherwise barf? You're right, the compiler will warn about casting the pointer to an integer of different size, as in char *Align(char *ptr) { unsigned low = 0xf & (unsigned)ptr; return ptr - low + ((low+0xf) & ~0xf); } although I think this is actually okay (we are explicitly truncating it further, and I think it is guaranteed that the truncation preserves the low byte). By barf I assume you mean just skip the alignment fix-up. Jed

1 0

Re: -malign-double
by Barry Smith 16 Nov '09

16 Nov '09

What about have configure look for uintptr_t if it does not exist then have configure check if unsigned long long is large enough, if not try size_t otherwise barf? Barry On Nov 16, 2009, at 1:55 PM, Jed Brown wrote: > Barry Smith wrote: >> >> Jed, >> >> Seems ok to me. Feel free to update PETSc. (But I think PETSc style >> would detect that your PETSC_ALIGN() macro would be called >> PetscMemAlign() or something, we only use caps for macros that are >> fixed >> values, not for macros that like functions.) > > Indeed (though there are a few exceptions). > > Although stdint.h (which contains uintptr_t) is required by C99, I > suppose it's likely that it's broken somewhere. On Win64, unsigned > long > is not sufficient so we need unsigned long long (also required by C99, > but not in C++). This is messy enough that we may as well work around > it by truncating the pointer into an unsigned, computing the shift, > and > adding the shift through a (char*) cast. Please speak up if uintptr_t > is actually reliable. (I just require a C99 compiler and don't > tolerate > non-compliance, but PETSc can't do that). > > Jed >

1 0

Re: -malign-double
by Jed Brown 16 Nov '09

16 Nov '09

Jed Brown wrote: > malloc4 1000000 loops in 1412541359 cycles, average 1412.541359 > malloc1 1000000 loops in 298137728 cycles, average 298.137728 > > If there is an extra allocation between malloc and free, we get > > malloc4 1000000 loops in 1667959560 cycles, average 1667.959560 > malloc1 1000000 loops in 538137819 cycles, average 538.137819 > > Clearly glibc is optimized for freeing the very last thing allocated. I should elaborate. What I had actually done is to do an extra malloc/free on every other iteration. So 596 cycles for two malloc/frees in two iterations, 1076 for three malloc/frees in two iterations, suggesting that this extra malloc/free costs significantly more than the one on top of the stack. That should explain why 4 per loop takes more than 4 times as long as 1 per loop. Jed

1 0

Re: -malign-double
by Jed Brown 16 Nov '09

16 Nov '09

Satish Balay wrote: > The whole reason for having PetscMalloc2() PetscMalloc3() etc is to > reduce the number of calls to malloc() - thus hoping for a performance > boost. > > For one - this usage was triggered by old SunOS boxes [where malloc > took a long time]. We don't know if this is still true for any of the > current OSes. > > With the increased complexity of managing alignment here - is it still > worth keeping these merged mallocs? [do we still get any payoff from > this complexity? - instead of relying directly on malloc() for the > necessary alignment?] I agree with Barry that the macros are good even if they were always doing separate macros underneath, it's much easier to see what is allocated together, and I frequently look for the PetscFree6 that must be associated with the PetscMalloc6 I'm looking at. Here are some benchmarks. In the following, I compare doing four mallocs in each loop (calling malloc directly) and doing a single malloc (and fixing up alignment). If there is nothing between the malloc and free, we get malloc4 1000000 loops in 1412541359 cycles, average 1412.541359 malloc1 1000000 loops in 298137728 cycles, average 298.137728 If there is an extra allocation between malloc and free, we get malloc4 1000000 loops in 1667959560 cycles, average 1667.959560 malloc1 1000000 loops in 538137819 cycles, average 538.137819 Clearly glibc is optimized for freeing the very last thing allocated. Fixing the alignment takes only a couple cycles. Remarkably, these results are reproducible to 6 significant figures or so. BTW, should we put PAPI support into PETSc, it no longer needs kernel patches with 2.6.31. Jed

1 0