Still need the -ksp_view output. It is spending most of the time in the LU factorization and solve. I suspect the coarse problem is way to big (like you are using two levels of multigrid) and since it is solved redundantly that takes all the time. Run with say 5 levels. Barry On Feb 23, 2012, at 3:03 PM, Francis Poulin wrote:

Hello again,

I am using v3.1 of PETSc.

I changed the grid sizes slightly and I'm including 4 log_summary files.

The times are shown below. I have not modified the example at all except in specifying the matrix size. Could it be that I need much larger? When I tried much larger matrices I think I might have got an error because I was using too much memory.

n time 2 22s 4 29.8s 8 33.7s 16 28.3s

Sorry for my first email but I hope this has more information.

Cheers, Francis

<saw_log_summary_n2.txt> <saw_log_summary_n4.txt> <saw_log_summary_n8.txt> <saw_log_summary_n16.txt>

...

On 2012-02-23, at 3:27 PM, Jed Brown wrote:

...

Always send output with -log_summary for each run that you do. On Thu, Feb 23, 2012 at 14:16, Francis Poulin <[email protected]> wrote: Hello,

I am learning to use PetSc but am just a notice. I have a rather basic question to ask and couldn't not find it on the achieves.

I am wanting to test the scalability of a Multigrid solver to the 3D Poisson equation. I found ksp/ex22.c that seems to solve the problem that I'm interested in. I ran it on a large server using different processors.

The syntax that I use to run using MPI was

./ex22 -da_grid_x 64 -da_grid_y 64 -da_grid_z 32

Which version of PETSc?

I tested it using 2, 4, 8, 16 cpus and found that the time increases. See below. Clearly there is something that I don't understand since the time should be reduced.

n wtime --------------------- 2 3m58s 4 3m54s 8 5m51s 16 7m23s

Any advice would be greatly appreciated.

Best regrads, Francis

On Thu, Feb 23, 2012 at 18:58, Francis Poulin <[email protected]> wrote:

I can do it for each of them if that helps but I suspect the method is the same so I'm sending the information for the first 3, n = 2, 4, 8. In the mean time I will figure out how to change the number of levels..

There are only three levels. The coarsest level has 32k degrees of freedom, which is very expensive to solve (redundantly) with a direct solver. Run this, it's higher resolution and will be much faster than what you had. mpiexec -n 2 ./ex22 -da_grid_x 5 -da_grid_y 5 -da_grid_z 5 -dmmg_nlevels 6 -ksp_monitor -mg_levels_ksp_type richardson -mg_levels_pc_type sor -log_summary Also, please switch to a more recent release of PETSc as soon as possible and do not develop new code using DMMG since that component has been removed (and its functionality incorporated into SNES and KSP).

Hello Barry and Jed, Thanks a lot for the responses, Barry's in particular made me laugh. In my defence I did install v3.2 on my desktop but the server that I'm using only has v3.1 installed. I've asked them nicely if they can update it. If not I'll figure out a way of doing it myself. In the mean time I will do the testing in my desktop to set things up before I do serious runs on the big cluster. I will use ex45.c and the options that you both suggested. Thanks again for the help, Francis On 2012-02-23, at 11:30 PM, Barry Smith wrote:

On Feb 23, 2012, at 8:24 PM, Jed Brown wrote:

...

On Thu, Feb 23, 2012 at 18:58, Francis Poulin <[email protected]> wrote: I can do it for each of them if that helps but I suspect the method is the same so I'm sending the information for the first 3, n = 2, 4, 8. In the mean time I will figure out how to change the number of levels..

There are only three levels. The coarsest level has 32k degrees of freedom, which is very expensive to solve (redundantly) with a direct solver.

Run this, it's higher resolution and will be much faster than what you had.

mpiexec -n 2 ./ex22 -da_grid_x 5 -da_grid_y 5 -da_grid_z 5 -dmmg_nlevels 6 -ksp_monitor -mg_levels_ksp_type richardson -mg_levels_pc_type sor -log_summary

Also, please switch to a more recent release of PETSc as soon as possible and do not develop new code using DMMG since that component has been removed (and its functionality incorporated into SNES and KSP).

Using Petsc Release Version 3.1.0, Patch 4, Fri Jul 30 14:42:02 CDT 2010

Goodness gracious, you won't run performance tests on your grandpa's desk calculator would you? Switch to petsc-dev immediately http://www.mcs.anl.gov/petsc/developers/index.html and use src/ksp/ksp/examples/tutorials/ex45.c and use the options -ksp_monitor -mg_levels_ksp_type richardson -mg_levels_pc_type sor -log_summary -da_refine 6

Barry

On Feb 24, 2012, at 9:54 AM, Francis Poulin wrote:

Hello again,

I am now running v3.2p6 and working with ksp/ex45.c, as suggested.

When I try running ex22 like Jed suggested it works fine.

When I try running ex45 like Barry suggested it runs but it does not seem to recognize two options.

WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-mg_levels_ksp_type value: richardson Option left: name:-mg_levels_pc_type value: sor

You likely need -pc_type mg Also run with -ksp_view to see what solver it is using NEVER assume you know what solver it is using without explicitly checking with -ksp_view

Also, I am getting an message saying that it is running it with the debugger and I should rerun ./configure to turn it off and it will run 2 or 3 times faster. Does that mean that the installation always uses the debugger or never uses it? I thought I would like it keep it for testing, assuming i figure out how it works, but then turn it off for serious runs.

By default we build the debug version which has more error checking and all the debug symbols. You should use this for ALL code development and testing of correctness. For production runs and testing of SPEED you should run ./configure WITH A DIFFERENT string name for PETSC_ARCH with the ./configure option --with-debugging=0 Barry

Thanks again, Francis

On 2012-02-23, at 11:30 PM, Barry Smith wrote:

...

On Feb 23, 2012, at 8:24 PM, Jed Brown wrote:

...

On Thu, Feb 23, 2012 at 18:58, Francis Poulin <[email protected]> wrote: I can do it for each of them if that helps but I suspect the method is the same so I'm sending the information for the first 3, n = 2, 4, 8. In the mean time I will figure out how to change the number of levels..

There are only three levels. The coarsest level has 32k degrees of freedom, which is very expensive to solve (redundantly) with a direct solver.

Run this, it's higher resolution and will be much faster than what you had.

mpiexec -n 2 ./ex22 -da_grid_x 5 -da_grid_y 5 -da_grid_z 5 -dmmg_nlevels 6 -ksp_monitor -mg_levels_ksp_type richardson -mg_levels_pc_type sor -log_summary

Also, please switch to a more recent release of PETSc as soon as possible and do not develop new code using DMMG since that component has been removed (and its functionality incorporated into SNES and KSP).

Using Petsc Release Version 3.1.0, Patch 4, Fri Jul 30 14:42:02 CDT 2010

Goodness gracious, you won't run performance tests on your grandpa's desk calculator would you? Switch to petsc-dev immediately http://www.mcs.anl.gov/petsc/developers/index.html and use src/ksp/ksp/examples/tutorials/ex45.c and use the options -ksp_monitor -mg_levels_ksp_type richardson -mg_levels_pc_type sor -log_summary -da_refine 6

Barry

I don't seem to have the hydra in my bin folder but I do have petscmpiexec that I've been using.

Use these, it will run the same method and sizes as the options I gave for ex22 before.

mpiexec.hydra -n 2 ./ex45 -da_grid_x 5 -da_grid_y 5 -da_grid_z 5 -da_refine 5 -ksp_monitor -pc_type mg -mg_levels_ksp_type richardson -mg_levels_pc_type sor -log_summary

Also, I want to install PetSc on an SGI machine that I have access to. I have been told that using MPT would give better performance compared to mpich2. When I configure petsc on this server I don't suppose -with-mpi-dir=/opt/sgi/mpt the above would work because of the different name. Do you have a suggestion as to what I could try? Francis

Hello, I wanted to thank everyone for the help and say that I managed to get it running on the cluster and I have done some efficiency calculations. To run the code I used, mpirun -np # ./ex45 -da_grid_x 5 -da_grid_y 5 -da_grid_z 5 -da_refine 6 -ksp_monitor -pc_type mg -mg_levels_ksp_type richardson -mg_levels_pc_type sor -log_summary as suggested and found the following p(#cpu) Tp (parallel) T1 (serial) Efficiency [ = Tp/(p*T1) ] -------------------------------------------------------------------------------------- 1 904 904 1.00 2 553 904 0.82 4 274 904 0.83 8 138 904 0.82 16 70 904 0.81 32 36 904 0.78 It seems to scale beautifully starting at 2 but there is a big drop from 1 to 2. I suspect there's a very good reason for this, I just don't know what. Thanks again for all of your help, Francis On 2012-02-24, at 12:59 PM, Jed Brown wrote:

On Fri, Feb 24, 2012 at 11:49, Francis Poulin <[email protected]> wrote: I don't seem to have the hydra in my bin folder but I do have petscmpiexec that I've been using.

That's just the name of my mpiexec. Use whichever one is used with your build of PETSc.

...

Use these, it will run the same method and sizes as the options I gave for ex22 before.

mpiexec.hydra -n 2 ./ex45 -da_grid_x 5 -da_grid_y 5 -da_grid_z 5 -da_refine 5 -ksp_monitor -pc_type mg -mg_levels_ksp_type richardson -mg_levels_pc_type sor -log_summary

Also, I want to install PetSc on an SGI machine that I have access to. I have been told that using MPT would give better performance compared to mpich2. When I configure petsc on this server I don't suppose

-with-mpi-dir=/opt/sgi/mpt

the above would work because of the different name. Do you have a suggestion as to what I could try?

It's just the way you launch parallel jobs.

Thanks for the link. I did the tests on an SGI SMP machine that, if I understand correctly, shares the memory much better than most other systems. I will read about these different components and see what I can figure out. Thanks, Francis

You need to understand the issues of memory bandwidth shared between cores and between processors, memory affinity and thread affinity ("binding") see http://www.mcs.anl.gov/petsc/documentation/faq.html#computers

Barry

On Fri, Feb 24, 2012 at 14:43, Francis Poulin <[email protected]> wrote:

It seems to scale beautifully starting at 2 but there is a big drop from 1 to 2. I suspect there's a very good reason for this, I just don't know what.

Can you send the full output? The smoother is slightly different, so the number of iterations could be different by 1 (for example). The -log_summary part of the output will show us where the time is being spent, so we'll be able to say what did not scale well between 1 and 2 procs.

You need to understand FOR THIS SPECIFIC MACHINE ARCHITECTURE what happens when it gos from using one processor to two? Is the second processor another core that shares common memory with the first processor? Or is it a completely separate core on a different node that has its own memory? Consider MatSOR 48 1.0 3.8607e+01 1.0 3.27e+09 1.0 0.0e+00 0.0e+00 0.0e+00 24 45 0 0 0 24 45 0 0 0 85 MatSOR 48 1.0 2.5600e+01 1.1 1.64e+09 1.0 4.8e+01 1.2e+05 4.8e+01 22 45 19 30 8 22 45 19 30 8 128 this computation across the two cores is embarrassing parallel, hence the flop rate for two processes should be 170, not 128 (the ratio is .75 very close to the .74 efficiency you get on two processes). So why is it not 170? The most likely answer is that the two cores shared a common memory and that memory is not fast enough (does not have enough memory bandwidth) to server both cores at the individual speed (85) that each of them can run. This is the curse of virtually any shared memory systems (that most people like to gloss over). Unless the memory bandwidth grows linearly with the number of cores you use the performance cannot grow linearly with the number of cores. On almost no system with shared memory does the memory bandwidth grow in that direction. You need to find out for this machine how to direct the executable to be run so that each of the two processes runs on different NODES of the system so they don't shared memory, then you will see a number better than the .74 for the two processes. But note that once you want to use all the cores on the system you will have cores shared memory and your parallel efficiency will go down. This is all material that should be presented in the first week of a parallel computing class and is crucial to understand if one plans to do "parallel computing". Barry On Feb 24, 2012, at 9:20 PM, Francis Poulin wrote:

Hello Barry,

Thanks for offering to look at this.

I configured a different version that does not have the debugger and the results are a lot faster but the efficiency is still similar. Below you'll see the results.

n Tp Efficiency ------------------------------------- 1 162 1 2 110 0.74 4 47 0.86 8 24 0.84 16 12 0.84 32 6 0.84

I'm also including the output of the log_summary for n =1 and 2.

As I said before, this is an SMP machine so I would expect it to be better than the typical cluster. That being said I am still learning how this works.

I am very happy that I managed to get some encouraging results in a day.

Cheers, Francis

<output_n1.txt> <output_n2.txt>

On 2012-02-24, at 9:08 PM, Jed Brown wrote:

...

On Fri, Feb 24, 2012 at 14:43, Francis Poulin <[email protected]> wrote: It seems to scale beautifully starting at 2 but there is a big drop from 1 to 2. I suspect there's a very good reason for this, I just don't know what.

Can you send the full output? The smoother is slightly different, so the number of iterations could be different by 1 (for example). The -log_summary part of the output will show us where the time is being spent, so we'll be able to say what did not scale well between 1 and 2 procs.