Thanks for your response Jed! I've been doing some other investigations using this example. I made some small modifications: 1. Added preallocation as Jed Brown suggested in a previous email (http://lists.mcs.anl.gov/pipermail/petsc-users/2011-June/009054.html). 2. Added a small VTK viewer. 3. Set the initial guess to zero. 4. Changed the entries in the element stiffness matrix to the following:   Ke[ 0] =  2./3.; Ke[ 1] = -1./6.; Ke[ 2] = -1./3.; Ke[ 3] = -1./6.;   Ke[ 4] = -1./6.; Ke[ 5] =  2./3.; Ke[ 6] = -1./6.; Ke[ 7] = -1./3.;   Ke[ 8] = -1./3.; Ke[ 9] = -1./6.; Ke[10] =  2./3.; Ke[11] = -1./6.;   Ke[12] = -1./6.; Ke[13] = -1./3.; Ke[14] = -1./6.; Ke[15] =  2./3.; I computed these by evaluating $K^e_{ij} = \int_{\Omega_e} \nabla \psi^e_i \cdot \nabla \psi^e_j \, \mathrm{d}\Omega$ with the shape functions $\psi^e$ corresponding to a bilinear quadratic finite element denoted by $\Omega_e$. This is different to what was originally in the code and I'm not sure where the original code comes from. This isn't important so you can just ignore it if you like, I get the same solution using both matrices. --- I am running on a two compute node clusters which each look as follows: 2 quad-core Intel Xeon 5345 processors, 16GB memory. The node clusters are connected with the following interconnect: Mellanox InfiniScale 2400. I computed my results using a machine file which specifies that (up to) the first 8 processes are computed on node1 and the second group of 8 processes are computed on node2. My timing results are shown in the table below, I'm running each test using Bi-CGStab with no preconditioning (-ksp_type bcgs -pc_type none) on a computational grid of 800 x 800 cells, so 641601 DOFs. I have attached my modified source code (you could look at my changes using diff) and the -log_summary output for each of the tests. # number of processes | time for KSPsolve() | iterations to convergence | norm of error 1 64.008 692 0.00433961 2 36.2767 626 0.00611835 4 35.9989 760 0.00311053 8 30.5215 664 0.00599148 16 14.1164 710 0.00792162 Why is the scaling so poor? I have read the FAQ (http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#computers), am I experiencing the problem described? I think my machine has a bandwidth of 2GB/s per process as suggested. Also, how can you tell if a computation is memory bound by looking at the -log_summary? Many thanks, Matija On Tue, Sep 20, 2011 at 11:44 AM, Jed Brown <[email protected]> wrote:

On Tue, Sep 20, 2011 at 11:45, Matija Kecman <[email protected]> wrote:

...

$ mpirun -np 1 ./ex3 -ksp_type gmres -pc_type none -m 100 Norm of error 0.570146 Iterations 0

This uses a nonzero initial guess so the initial residual norm is compared to the right hand side. $ ./ex3 -ksp_type gmres -ksp_monitor -m 100 -pc_type none -ksp_converged_reason -info |grep Converged [0] KSPDefaultConverged(): user has provided nonzero initial guess, computing 2-norm of preconditioned RHS [0] KSPDefaultConverged(): Linear solver has converged. Residual norm 1.113646413065e-04 is less than relative tolerance 1.000000000000e-05 times initial right hand side norm 1.291007358616e+01 at iteration 0 You can use the true residual, it just costs something so it's not enabled by default: $ ./ex3 -ksp_type gmres -ksp_monitor -m 100 -pc_type none -ksp_converged_reason -ksp_converged_use_initial_residual_norm [many iterations] Linear solve converged due to CONVERGED_RTOL iterations 1393 Norm of error 0.000664957 Iterations 1393

Thanks Barry, I'm using PETSc 3.1.0 Patch 8, is there something similar I can do that would work for this version? On Thu, Sep 29, 2011 at 1:42 PM, Barry Smith <[email protected]> wrote:

 If you are using petsc-3.2 you can also run "make streams" in the $PETSC_DIR and that will run the streams benchmark giving you a pretty good idea of how the memory bandwidth scales with the number of cores. If the streams benchmark does not scale with the cores then no iterative solver will scale with more cores.

  Barry

On Sep 29, 2011, at 6:28 AM, Matija Kecman wrote:

...

Thanks for your response Jed! I've been doing some other investigations using this example. I made some small modifications:

1. Added preallocation as Jed Brown suggested in a previous email (http://lists.mcs.anl.gov/pipermail/petsc-users/2011-June/009054.html). 2. Added a small VTK viewer. 3. Set the initial guess to zero. 4. Changed the entries in the element stiffness matrix to the following:

  Ke[ 0] =  2./3.; Ke[ 1] = -1./6.; Ke[ 2] = -1./3.; Ke[ 3] = -1./6.;   Ke[ 4] = -1./6.; Ke[ 5] =  2./3.; Ke[ 6] = -1./6.; Ke[ 7] = -1./3.;   Ke[ 8] = -1./3.; Ke[ 9] = -1./6.; Ke[10] =  2./3.; Ke[11] = -1./6.;   Ke[12] = -1./6.; Ke[13] = -1./3.; Ke[14] = -1./6.; Ke[15] =  2./3.;

I computed these by evaluating $K^e_{ij} = \int_{\Omega_e} \nabla \psi^e_i \cdot \nabla \psi^e_j \, \mathrm{d}\Omega$ with the shape functions $\psi^e$ corresponding to a bilinear quadratic finite element denoted by $\Omega_e$. This is different to what was originally in the code and I'm not sure where the original code comes from. This isn't important so you can just ignore it if you like, I get the same solution using both matrices.

---

I am running on a two compute node clusters which each look as follows: 2 quad-core Intel Xeon 5345 processors, 16GB memory. The node clusters are connected with the following interconnect: Mellanox InfiniScale 2400. I computed my results using a machine file which specifies that (up to) the first 8 processes are computed on node1 and the second group of 8 processes are computed on node2. My timing results are shown in the table below, I'm running each test using Bi-CGStab with no preconditioning (-ksp_type bcgs -pc_type none) on a computational grid of 800 x 800 cells, so 641601 DOFs. I have attached my modified source code (you could look at my changes using diff) and the -log_summary output for each of the tests.

# number of processes | time for KSPsolve() | iterations to convergence | norm of error 1 64.008 692 0.00433961 2 36.2767 626 0.00611835 4 35.9989 760 0.00311053 8 30.5215 664 0.00599148 16 14.1164 710 0.00792162

Why is the scaling so poor? I have read the FAQ (http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#computers), am I experiencing the problem described? I think my machine has a bandwidth of 2GB/s per process as suggested. Also, how can you tell if a computation is memory bound by looking at the -log_summary?

Many thanks,

Matija

On Tue, Sep 20, 2011 at 11:44 AM, Jed Brown <[email protected]> wrote:

...

On Tue, Sep 20, 2011 at 11:45, Matija Kecman <[email protected]> wrote:

...

$ mpirun -np 1 ./ex3 -ksp_type gmres -pc_type none -m 100 Norm of error 0.570146 Iterations 0

This uses a nonzero initial guess so the initial residual norm is compared to the right hand side. $ ./ex3 -ksp_type gmres -ksp_monitor -m 100 -pc_type none -ksp_converged_reason -info |grep Converged [0] KSPDefaultConverged(): user has provided nonzero initial guess, computing 2-norm of preconditioned RHS [0] KSPDefaultConverged(): Linear solver has converged. Residual norm 1.113646413065e-04 is less than relative tolerance 1.000000000000e-05 times initial right hand side norm 1.291007358616e+01 at iteration 0 You can use the true residual, it just costs something so it's not enabled by default: $ ./ex3 -ksp_type gmres -ksp_monitor -m 100 -pc_type none -ksp_converged_reason -ksp_converged_use_initial_residual_norm [many iterations] Linear solve converged due to CONVERGED_RTOL iterations 1393 Norm of error 0.000664957 Iterations 1393

<ex3_log_summary_1process><ex3_log_summary_2process><ex3_log_summary_4process><ex3_log_summary_8process><ex3_log_summary_16process><ex3_modified.c>

On Thu, Sep 29, 2011 at 11:28 AM, Matija Kecman <[email protected]>wrote:

Thanks for your response Jed! I've been doing some other investigations using this example. I made some small modifications:

1. Added preallocation as Jed Brown suggested in a previous email (http://lists.mcs.anl.gov/pipermail/petsc-users/2011-June/009054.html). 2. Added a small VTK viewer. 3. Set the initial guess to zero. 4. Changed the entries in the element stiffness matrix to the following:

Ke[ 0] = 2./3.; Ke[ 1] = -1./6.; Ke[ 2] = -1./3.; Ke[ 3] = -1./6.; Ke[ 4] = -1./6.; Ke[ 5] = 2./3.; Ke[ 6] = -1./6.; Ke[ 7] = -1./3.; Ke[ 8] = -1./3.; Ke[ 9] = -1./6.; Ke[10] = 2./3.; Ke[11] = -1./6.; Ke[12] = -1./6.; Ke[13] = -1./3.; Ke[14] = -1./6.; Ke[15] = 2./3.;

I computed these by evaluating $K^e_{ij} = \int_{\Omega_e} \nabla \psi^e_i \cdot \nabla \psi^e_j \, \mathrm{d}\Omega$ with the shape functions $\psi^e$ corresponding to a bilinear quadratic finite element denoted by $\Omega_e$. This is different to what was originally in the code and I'm not sure where the original code comes from. This isn't important so you can just ignore it if you like, I get the same solution using both matrices.

---

I am running on a two compute node clusters which each look as follows: 2 quad-core Intel Xeon 5345 processors, 16GB memory. The node clusters are connected with the following interconnect: Mellanox InfiniScale 2400. I computed my results using a machine file which specifies that (up to) the first 8 processes are computed on node1 and the second group of 8 processes are computed on node2. My timing results are shown in the table below, I'm running each test using Bi-CGStab with no preconditioning (-ksp_type bcgs -pc_type none) on a computational grid of 800 x 800 cells, so 641601 DOFs. I have attached my modified source code (you could look at my changes using diff) and the -log_summary output for each of the tests.

The way I read these numbers is that there is bandwidth for about 3 cores on this machine, and non-negligible synchronization penalty: 1 proc 2 proc 4 proc 8 proc VecAXPBYCZ 496 857 1064 1070 VecDot 451 724 1089 736 MatMult 434 638 701 703 The bandwidth tops out between 2 and 4 cores (The 5345 should have 10.6 GB/s but you should runs streams as Barry says to see what is achievable). There is obviously a penalty for VecDot against VecAXPYCZ, which is the sync penalty which also seems to affect MatMult. Maybe Jed can explain that. Matt # number of processes | time for KSPsolve() | iterations to

convergence | norm of error 1 64.008 692 0.00433961 2 36.2767 626 0.00611835 4 35.9989 760 0.00311053 8 30.5215 664 0.00599148 16 14.1164 710 0.00792162

Why is the scaling so poor? I have read the FAQ (http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#computers), am I experiencing the problem described? I think my machine has a bandwidth of 2GB/s per process as suggested. Also, how can you tell if a computation is memory bound by looking at the -log_summary?

Many thanks,

Matija

On Tue, Sep 20, 2011 at 11:44 AM, Jed Brown <[email protected]> wrote:

...

On Tue, Sep 20, 2011 at 11:45, Matija Kecman <[email protected]>

wrote:

...

...

$ mpirun -np 1 ./ex3 -ksp_type gmres -pc_type none -m 100 Norm of error 0.570146 Iterations 0

This uses a nonzero initial guess so the initial residual norm is compared to the right hand side. $ ./ex3 -ksp_type gmres -ksp_monitor -m 100 -pc_type none -ksp_converged_reason -info |grep Converged [0] KSPDefaultConverged(): user has provided nonzero initial guess, computing 2-norm of preconditioned RHS [0] KSPDefaultConverged(): Linear solver has converged. Residual norm 1.113646413065e-04 is less than relative tolerance 1.000000000000e-05 times initial right hand side norm 1.291007358616e+01 at iteration 0 You can use the true residual, it just costs something so it's not enabled by default: $ ./ex3 -ksp_type gmres -ksp_monitor -m 100 -pc_type none -ksp_converged_reason -ksp_converged_use_initial_residual_norm [many iterations] Linear solve converged due to CONVERGED_RTOL iterations 1393 Norm of error 0.000664957 Iterations 1393

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

Thanks for your useful replies.

...

The bandwidth tops out between 2 and 4 cores (The 5345 should have 10.6 GB/s but you should runs streams as Barry says to see what is achievable).

I will try to get the STREAM benchmark running

Note that your naive 1D partition is getting worse as you add processes. The MatMult should scale out somewhat better if you use a 2D decomposition, as is done by any of the examples using a DA (DMDA in 3.2) for grid management.

I am able to recompute the problem using a 2D decomposition and I will look at the log-summary output for those tests also Best, Matija On Thu, Sep 29, 2011 at 2:09 PM, Jed Brown <[email protected]> wrote:

On Thu, Sep 29, 2011 at 07:44, Matthew Knepley <[email protected]> wrote:

...

The way I read these numbers is that there is bandwidth for about 3 cores on this machine, and non-negligible synchronization penalty:                         1 proc   2 proc   4 proc    8 proc VecAXPBYCZ    496        857      1064      1070 VecDot               451        724      1089        736 MatMult              434        638        701        703

Matt, thanks for pulling out this summary. The synchronization for the dot product is clearly expensive here. I'm surprised it's so significant on such a small problem, but it is a common problem for scalability. I think that MPI is putting one process per socket when you go from 1 to 2 processes. This gives you pretty good speedup despite the fact that memory traffic for both sockets is routed through the "Blackford" chipset. Intel fixed this in later generations by throwing out the notion of uniform memory access by having independent memory banks for each socket (which AMD had at the time these chips came out). If not odd hardware issues such as having enough memory streams and imbalances, one process per socket can saturate the memory bandwidth, so the speed-up you get from 2 procs (1 per socket) to 4 procs (2 per socket) is minimal. On this architecture, you typically see the STREAM bandwidth go down slightly as you add more than 2 procs per socket, so it's no surprise that it doesn't help here. Note that your naive 1D partition is getting worse as you add processes. The MatMult should scale out somewhat better if you use a 2D decomposition, as is done by any of the examples using a DA (DMDA in 3.2) for grid management.

...

The bandwidth tops out between 2 and 4 cores (The 5345 should have 10.6 GB/s but you should runs streams as Barry says to see what is achievable). There is obviously a penalty for VecDot against VecAXPYCZ, which is the sync penalty which also seems to affect MatMult. Maybe Jed can explain that.