Re: [petsc-users] Varying TAO optimization solve iterations using BLMVM
Hi Justin, I don't see anything obviously wrong that would be causing this variation in iterations due to number of processors. Is it at all feasible to send be an example code that reproduces the problem (perhaps a smaller version)? I'm still guessing the problem lies in numerical precision, it would be nice to find a way to avoid them. I don't think the job scheduling or compute nodes would affect this.
Basically I call TaoSolve at each time level. What I find strange is that the number of TAO solve iterations vary at each time level for a given number of processors
Just for clarification, do you mean that for a given problem, you run the same problem several times the only difference being the number of processors, and that on each time step you get (close to) the same solution for each run, just with a different number of TAO iterations? If you run the same problem twice using the same number of processors, is the output identical? No OpenMP threads or GPU's?
2) Sometimes, I get Tao Termination reason of -5, and from what I see from the online documentation, it means the number of function evaluations exceeds the maximum number of function evaluations. I only get this at certain time levels, and it also varies when I change the number of processors.
This looks like the same issue, unless the number of iterations is hugely different (say it converges on some number of processors after 200 iterations, but still hasn't converged after 2000 on a different number). Jason On Tue, Jun 23, 2015 at 6:36 PM, Justin Chang <[email protected]> wrote:
I was unable to do quad precision or even with 64 bit integers because my data files rely on intricate binary files that have been written in 32 bit.
However, I noticed a couple things which are puzzling to me:
1) I am solving a transient problem using my own backward euler function. Basically I call TaoSolve at each time level. What I find strange is that the number of TAO solve iterations vary at each time level for a given number of processors. The solution is roughly the same when I change the number of processors. Any idea why this is happening, or might this have more to do with the job scheduling/compute nodes on my HPC machine?
2) Sometimes, I get Tao Termination reason of -5, and from what I see from the online documentation, it means the number of function evaluations exceeds the maximum number of function evaluations. I only get this at certain time levels, and it also varies when I change the number of processors.
I can understand the number of iterations going down the further in time i go (this is due to the nature of my problem), but I am not sure why the above two observations are happening. Any thoughts?
Thanks, Justin
On Fri, Jun 19, 2015 at 11:52 AM, Justin Chang <[email protected]> wrote:
My code sort of requires HDF5 so installing quad precision might be a little difficult. I could try to work around this but that might take some effort.
In the mean time, is there any other potential explanation or alternative to figuring this out?
Thanks, Justin
On Thursday, June 18, 2015, Matthew Knepley <[email protected]> wrote:
On Thu, Jun 18, 2015 at 1:52 PM, Jason Sarich <[email protected]> wrote:
BLMVM doesn't use a KSP or preconditioner, it updates using the L-BFGS-B formula
Then this sounds like a bug, unless one of the constants is partition dependent.
Matt
On Thu, Jun 18, 2015 at 1:45 PM, Matthew Knepley <[email protected]> wrote:
On Thu, Jun 18, 2015 at 12:15 PM, Jason Sarich < [email protected]> wrote:
Hi Justin,
I can't tell for sure why this is happening, have you tried using quad precision to make sure that numerical cutoffs isn't the problem?
1 The Hessian being approximate and the resulting implicit computation is the source of the cutoff, but would not be causing different convergence rates in infinite precision.
2 the local size may affect load balancing but not the resulting norms/convergence rate.
This sounds to be like the preconditioner is dependent on the partition. Can you send -tao_view -snes_view
Matt
Jason
On Thu, Jun 18, 2015 at 10:44 AM, Justin Chang <[email protected]> wrote:
I solved a transient diffusion across multiple cores using TAO BLMVM. When I simulate the same problem but on different numbers of processing cores, the number of solve iterations change quite drastically. The numerical solution is the same, but these changes are quite vast. I attached a PDF showing a comparison between KSP and TAO. KSP remains largely invariant with number of processors but TAO (with bounded constraints) fluctuates.
My question is, why is this happening? I understand that accumulation of numerical round-offs may attribute to this, but the differences seem quite vast to me. My initial thought was that
1) the Hessian is only projected and not explicitly computed, which may have something to do with the rate of convergence
2) local problem size. Certain regions of my domain have different number of "violations" which need to be corrected by the bounded constraints so the rate of convergence depends on how these regions are partitioned?
Any thoughts?
Thanks, Justin
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
Jason, I was experimenting with some smaller steady-state problems and I still get the same issue: every time I run the same problem on the same number of processors the number of iterations differs but the solutions remain the same. I think this is the root to why I get such erratic behavior in iterations in the transient case. Attached are the log files for the two runs I did. Is it possible to tell what's going on from these? Or should I reinstall PETSc? Because I only noticed this problem once I switched over to the master version of 3.6 (although it's possible I might have screwed up the installation a bit because I had to scp everything due to the firewall). I can send you the example code and working files if needed Thanks, Justin On Thu, Jun 25, 2015 at 9:51 AM, Jason Sarich <[email protected]> wrote:
Hi Justin,
I don't see anything obviously wrong that would be causing this variation in iterations due to number of processors. Is it at all feasible to send be an example code that reproduces the problem (perhaps a smaller version)? I'm still guessing the problem lies in numerical precision, it would be nice to find a way to avoid them. I don't think the job scheduling or compute nodes would affect this.
Basically I call TaoSolve at each time level. What I find strange is that the number of TAO solve iterations vary at each time level for a given number of processors
Just for clarification, do you mean that for a given problem, you run the same problem several times the only difference being the number of processors, and that on each time step you get (close to) the same solution for each run, just with a different number of TAO iterations?
If you run the same problem twice using the same number of processors, is the output identical? No OpenMP threads or GPU's?
2) Sometimes, I get Tao Termination reason of -5, and from what I see from the online documentation, it means the number of function evaluations exceeds the maximum number of function evaluations. I only get this at certain time levels, and it also varies when I change the number of processors.
This looks like the same issue, unless the number of iterations is hugely different (say it converges on some number of processors after 200 iterations, but still hasn't converged after 2000 on a different number).
Jason
On Tue, Jun 23, 2015 at 6:36 PM, Justin Chang <[email protected]> wrote:
I was unable to do quad precision or even with 64 bit integers because my data files rely on intricate binary files that have been written in 32 bit.
However, I noticed a couple things which are puzzling to me:
1) I am solving a transient problem using my own backward euler function. Basically I call TaoSolve at each time level. What I find strange is that the number of TAO solve iterations vary at each time level for a given number of processors. The solution is roughly the same when I change the number of processors. Any idea why this is happening, or might this have more to do with the job scheduling/compute nodes on my HPC machine?
2) Sometimes, I get Tao Termination reason of -5, and from what I see from the online documentation, it means the number of function evaluations exceeds the maximum number of function evaluations. I only get this at certain time levels, and it also varies when I change the number of processors.
I can understand the number of iterations going down the further in time i go (this is due to the nature of my problem), but I am not sure why the above two observations are happening. Any thoughts?
Thanks, Justin
On Fri, Jun 19, 2015 at 11:52 AM, Justin Chang <[email protected]> wrote:
My code sort of requires HDF5 so installing quad precision might be a little difficult. I could try to work around this but that might take some effort.
In the mean time, is there any other potential explanation or alternative to figuring this out?
Thanks, Justin
On Thursday, June 18, 2015, Matthew Knepley <[email protected]> wrote:
On Thu, Jun 18, 2015 at 1:52 PM, Jason Sarich <[email protected]> wrote:
BLMVM doesn't use a KSP or preconditioner, it updates using the L-BFGS-B formula
Then this sounds like a bug, unless one of the constants is partition dependent.
Matt
On Thu, Jun 18, 2015 at 1:45 PM, Matthew Knepley <[email protected]> wrote:
On Thu, Jun 18, 2015 at 12:15 PM, Jason Sarich < [email protected]> wrote:
Hi Justin,
I can't tell for sure why this is happening, have you tried using quad precision to make sure that numerical cutoffs isn't the problem?
1 The Hessian being approximate and the resulting implicit computation is the source of the cutoff, but would not be causing different convergence rates in infinite precision.
2 the local size may affect load balancing but not the resulting norms/convergence rate.
This sounds to be like the preconditioner is dependent on the partition. Can you send -tao_view -snes_view
Matt
Jason
On Thu, Jun 18, 2015 at 10:44 AM, Justin Chang <[email protected]> wrote:
I solved a transient diffusion across multiple cores using TAO BLMVM. When I simulate the same problem but on different numbers of processing cores, the number of solve iterations change quite drastically. The numerical solution is the same, but these changes are quite vast. I attached a PDF showing a comparison between KSP and TAO. KSP remains largely invariant with number of processors but TAO (with bounded constraints) fluctuates.
My question is, why is this happening? I understand that accumulation of numerical round-offs may attribute to this, but the differences seem quite vast to me. My initial thought was that
1) the Hessian is only projected and not explicitly computed, which may have something to do with the rate of convergence
2) local problem size. Certain regions of my domain have different number of "violations" which need to be corrected by the bounded constraints so the rate of convergence depends on how these regions are partitioned?
Any thoughts?
Thanks, Justin
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
participants (2)
-
Jason Sarich -
Justin Chang