Yes. Its fairly flat which I think is due to BoomerAMG. [8, 8, 9, 11, 12] On Fri, May 18, 2012 at 5:06 PM, Matthew Knepley <[email protected]> wrote:
On Fri, May 18, 2012 at 8:02 PM, Mohammad Mirzadeh <[email protected]>wrote:
Yes, I'm looking at weak scalability right now. I'm using BiCGSTAB with BoomerAMG (all default options except for rtol = 1e-12). I've not looked into MF/s yet but I'll surely do to see if I'm having any problem there. So far, just timing the KSPSolve, I get [0.231, 0.238, 0.296, 0.451, 0.599] seconds/KSP iteration for p=[1, 4, 16, 64, 256] with almost 93K nodes (matrix-row) per proc. Which is not bad I guess but still increased by a factor of 3 for 256 proc. Problem is, I don't know how good/bad this is. In fact I'm not even sure that is a valid question to ask since it may be very problem dependent.
Something I just though about, how crucial is the matrix structure for KSP solvers? The nodes have bad numbering and I do partitioning to get a better one here.
Did you look at the number of iterates?
Matt
On Fri, May 18, 2012 at 4:47 PM, Matthew Knepley <[email protected]>wrote:
On Fri, May 18, 2012 at 7:43 PM, Mohammad Mirzadeh <[email protected]>wrote:
I see; well that's a fair point. So i have my timing results obtained via -log_summary; what should I be looking into for MatMult? Should I be looking at wall timings? Or do I need to look into MFlops/s? I'm sorry but I'm not sure what measure I should be looking into to determine scalability.
Time is only meaningful in isolation if I know how big your matrix is, but you obviously take the ratio to look how it is scaling. I am assuming you are looking at weak scalability so it should remain constant. MF/s will let you know how the routine is performing independent of size, and thus is an easy way to see what is happening. It should scale like P, and when that drops off you have insufficient bandwidth. VecMDot is a good way to look at the latency of reductions (assuming you use GMRES). There is indeed no good guide to this. Barry should write one.
Matt
Also, is there any general meaningful advice one could give? in terms of using the resources, compiler flags (beyond -O3), etc?
Thanks, Mohammad
On Fri, May 18, 2012 at 4:18 PM, Matthew Knepley <[email protected]>wrote:
On Fri, May 18, 2012 at 7:06 PM, Mohammad Mirzadeh <[email protected]
wrote:
Hi guys,
I'm trying to generate scalability plots for my code and do profiling and fine tuning. In doing so I have noticed that some of the factors affecting my results are sort of subtle. For example, I figured, the other day, that using all of the cores on a single node is somewhat (50-60%) slower when compared to using only half of the cores which I suspect is due to memory bandwidth and/or other hardware-related issues.
So I thought to ask and see if there is any example in petsc that has been tested for scalability and has been documented? Basically I want to use this test example as a benchmark to compare my results with. My own test code is currently a linear Poisson solver on an adaptive quadtree grid and involves non-trivial geometry (well basically a circle for the boundary but still not a simple box).
Unfortunately, I do not even know what that means. We can't guarantee a certain level of performance because it not only depends on the hardware, but how you use it (as evident in your case). In a perfect world, we would have an abstract model of the computation (available for MatMult) and your machine (not available anywhere) and we would automatically work out the consequences and tell you what to expect. Instead today, we tell you to look at a few key indicators like the MatMult event, to see what is going on. When MatMult stops scaling, you have run out of bandwidth.
Matt
Thanks, Mohammad
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener