Barry Smith schrieb:

On May 29, 2009, at 4:34 AM, Andreas Grassl wrote:

...

Hello,

I'm working with the PCNN preconditioner and hence with ISLocalToGlobalMapping. After solving I want to write the solution to an ASCII-file where only the values belonging to the "external" global numbering are given and not followed ^^^^^^^^^^^^^^^^^^^^^^^

by the zeros.

What do you mean? What parts of the vector do you want?

I want the first actdof entries actdof is the number of DOF the system has. the values of indices is in the range of 0 to actdof-1. I create the mapping by ISLocalToGlobalMappingCreate(commw,ind_length,indices,&gridmapping); Due to the "existence" of interface DOF's the sum over all ind_length is greather than actdof, namely the size of the Vectors, but only actdof entries of this Vector are nonzero, if I view it.

...

Currently I'm giving this commands:

ierr = PetscViewerSetFormat(viewer,PETSC_VIEWER_ASCII_SYMMODU);CHKERRQ(ierr); ierr = VecView(X,viewer);CHKERRQ(ierr);

I hope you got an idea, what problem I have. Cheers, ando -- /"\ \ / ASCII Ribbon X against HTML email / \

Barry Smith schrieb:

Hmm, it sounds like the difference between local "ghosted" vectors and the global parallel vectors. But I do not understand why any of the local vector entries would be zero. Doesn't the vector X that is passed into KSP (or SNES) have the global entries and uniquely define the solution? Why is viewing that not right?

I still don't understand fully the underlying processes of the whole PCNN solution procedure, but trying around I substituted MatCreateIS(commw, ind_length, ind_length, PETSC_DECIDE, PETSC_DECIDE, gridmapping, &A); by MatCreateIS(commw, PETSC_DECIDE, PETSC_DECIDE, actdof, actdof, gridmapping, &A); and received the needed results. Furthermore it seems, that the load balance is now better, although I still don't reach the expected values, e.g. ilu-cg 320 iterations, condition 4601 cg only 1662 iterations, condition 84919 nn-cg on 2 nodes 229 iterations, condition 6285 nn-cg on 4 nodes 331 iterations, condition 13312 or is it not to expect, that nn-cg is faster than ilu-cg? cheers, ando -- /"\ Grassl Andreas \ / ASCII Ribbon Campaign Uni Innsbruck Institut f. Mathematik X against HTML email Technikerstr. 13 Zi 709 / \ +43 (0)512 507 6091

Thank you for the explanation, first I'll try the null space set up and then I come back to your hints. Jed Brown schrieb:

Andreas Grassl wrote:

...

Barry Smith schrieb:

...

Hmm, it sounds like the difference between local "ghosted" vectors and the global parallel vectors. But I do not understand why any of the local vector entries would be zero. Doesn't the vector X that is passed into KSP (or SNES) have the global entries and uniquely define the solution? Why is viewing that not right?

I still don't understand fully the underlying processes of the whole PCNN solution procedure, but trying around I substituted

MatCreateIS(commw, ind_length, ind_length, PETSC_DECIDE, PETSC_DECIDE, gridmapping, &A);

This creates a matrix that is bigger than you want, and gives you the dead values at the end (global dofs that are not in the range of the LocalToGlobalMapping.

This from the note on MatCreateIS:

| m and n are NOT related to the size of the map, they are the size of the part of the vector owned | by that process. m + nghosts (or n + nghosts) is the length of map since map maps all local points | plus the ghost points to global indices.

...

by

MatCreateIS(commw, PETSC_DECIDE, PETSC_DECIDE, actdof, actdof, gridmapping, &A);

This creates a matrix of the correct size, but it looks like it could easily end up with the "wrong" dofs owned locally. What you probably want to do is:

1. Resolve ownership just like with any other DD method. This partitions your dofs into n owned dofs and ngh ghosted dofs on each process. The global sum of n is N, the size of the global vectors that the solver will interact with.

2. Make an ISLocalToGlobalMapping where all the owned dofs come first, mapping (0..n-1) to (rstart..rstart+n-1), followed by the ghosted dofs (local index n..ngh-1) which map to remote processes. (rstart is the global index of the first owned dof)

One way to do this is to use MPI_Scan to find rstart, then number all the owned dofs and scatter the result. The details will be dependent on how you store your mesh. (I'm assuming it's unstructured, this step is trivial if you use a DA.)

3. Call MatCreateIS(comm,n,n,PETSC_DECIDE,PETSC_DECIDE,mapping,&A);

...

Furthermore it seems, that the load balance is now better, although I still don't reach the expected values, e.g. ilu-cg 320 iterations, condition 4601 cg only 1662 iterations, condition 84919

nn-cg on 2 nodes 229 iterations, condition 6285 nn-cg on 4 nodes 331 iterations, condition 13312

or is it not to expect, that nn-cg is faster than ilu-cg?

It depends a lot on the problem. As you probably know, for a second order elliptic problem with exact subdomain solves, the NN preconditioned operator (without a coarse component) has condition number that scales as

(1/H^2)(1 + log(H/h))^2

where H is the subdomain diameter and h is the element size. In contrast, overlapping additive Schwarz is

1/H^2

and block Jacobi is

1/(Hh)

(the original problem was 1/h^2)

In particular, there is no reason to expect that NN is uniformly better than ASM, although it may be for certain problems. When a coarse solve is used, NN becomes

(1 + log(H/h))^2

which is quasi-optimal (these methods are known as BDDC, which is essentially equivalent to FETI-DP). The key advantage over multigrid (or multilivel Schwarz) is improved robustness with variable coefficients. My understanding is that PCNN is BDDC, and uses direct subdomain solves by default, but I could have missed something. In particular, if the coarse solve is missing or inexact solves are used, you could easily see relatively poor scaling. AFAIK, it's not for vector problems at this time.

Good luck.

Jed

-- /"\ \ / ASCII Ribbon X against HTML email / \

Andreas Grassl wrote:

Hi Jed,

the BNN-Algorithm in the literature distinguishes always between inner nodes and interface nodes. The short question arising from your explanation for me is, if owned DOF's is a synonym for the inner DOF's and ghosted DOF's for the interface DOF's?

No, every degree of freedom (interior and interface) must be owned by exactly one process. You want every process to own their interior degrees of freedom, but I don't think there is a way to guarantee this without using a process like I described.

Below you find more extended thoughts and an example.

Jed Brown schrieb:

...

Andreas Grassl wrote:

...

Barry Smith schrieb:

...

Hmm, it sounds like the difference between local "ghosted" vectors and the global parallel vectors. But I do not understand why any of the local vector entries would be zero. Doesn't the vector X that is passed into KSP (or SNES) have the global entries and uniquely define the solution? Why is viewing that not right?

I still don't understand fully the underlying processes of the whole PCNN solution procedure, but trying around I substituted

MatCreateIS(commw, ind_length, ind_length, PETSC_DECIDE, PETSC_DECIDE, gridmapping, &A); This creates a matrix that is bigger than you want, and gives you the dead values at the end (global dofs that are not in the range of the LocalToGlobalMapping.

This from the note on MatCreateIS:

| m and n are NOT related to the size of the map, they are the size of the part of the vector owned | by that process. m + nghosts (or n + nghosts) is the length of map since map maps all local points | plus the ghost points to global indices.

...

by

MatCreateIS(commw, PETSC_DECIDE, PETSC_DECIDE, actdof, actdof, gridmapping, &A); This creates a matrix of the correct size, but it looks like it could easily end up with the "wrong" dofs owned locally. What you probably want to do is:

1. Resolve ownership just like with any other DD method. This partitions your dofs into n owned dofs and ngh ghosted dofs on each process. The global sum of n is N, the size of the global vectors that the solver will interact with.

do I understand right, that owned dofs are the inner nodes and the ghosted dofs are the interface dofs?

No, a dof is ghosted on processes that reference it, but do not own it.

...

2. Make an ISLocalToGlobalMapping where all the owned dofs come first, mapping (0..n-1) to (rstart..rstart+n-1), followed by the ghosted dofs (local index n..ngh-1) which map to remote processes. (rstart is the global index of the first owned dof)

currently I set up my ISLocalToGlobalMapping by giving the processes all the dofs in arbitrary order having the effect, that the interface dofs appear more times. Attached I give you a small example with 2 subdomains and 270 DOF's.

I think you're ending up with a lot of interior dofs owned by remote processes (this is bad). I'll try to explain for the 24 dof example below.

...

One way to do this is to use MPI_Scan to find rstart, then number all the owned dofs and scatter the result. The details will be dependent on how you store your mesh. (I'm assuming it's unstructured, this step is trivial if you use a DA.)

Yes, the mesh is unstructured, I read out from the FE-package the partitioning at element-basis, loop over all elements to find the belonging DOF's and assemble the index vector for the ISLocalToGlobalMapping this way, without regarding interface DOF's, thinking this would be done automatically by setting up the mapping because by this some global DOF's appear more times.

...

3. Call MatCreateIS(comm,n,n,PETSC_DECIDE,PETSC_DECIDE,mapping,&A);

Seeing this function call and interpreting the owned DOF's as the subdomain inner DOF's the Matrix A has not the full size?!

Given a 4x6 grid with 1 DOF per node divided into 4 subdomains I get 9 interface DOF's.

0 o o O o 5 | 6 o o O o o | O--O--O--O--O--O | o o o O o 23

My first approach to create the Matrix would give a Matrix size of 35x35, with 11 dead entries at the end of the vector.

My second approach would give the "correct" Matrix size of 24x24.

By splitting up in n owned values and some ghosted values I would expect to receive a Matrix of size 15x15. Otherwise I don't see how I could partition the grid in a consistent way.

I would really appreciate, if you could show me, how the partition and ownership of the DOF's in this little example work out.

I see 4 subdomains with interior dofs rank 0: 0 1 2 6 7 8 rank 1: 4 5 10 11 rank 2: 18 19 20 rank 3: 22 23 I'll continue to use this "natural ordering" to describe the dofs, but you don't normally want to use it because it is not compatible with the decomposition you are actually using. Suppose we resolve ownership by assigning it to the lowest rank touching it. Then the global vector (seen by the solver) is rank 0: 0 1 2 3 6 7 8 9 12 13 14 15 (global indices 0:12) rank 1: 4 5 10 11 16 17 (global indices 12:18) rank 2: 18 19 20 21 (global indices 18:22) rank 3: 22 23 (global indices 22:24) Your local-to-global map should be with respect to the global indices in the ordering compatible with the decomposition. With respect to the natural ordering, it is rank 0: 0 1 2 3 6 7 8 9 12 13 14 15 rank 1: 3 4 5 9 10 11 15 16 17 rank 2: 12 13 14 15 18 19 20 21 rank 3: 15 16 17 21 22 23 Converting this to the ordering compatible with your decomposition, we have rank 0: 0 1 2 3 4 5 6 7 8 9 10 11 rank 1: 3 12 13 7 14 15 11 16 17 rank 2: 8 9 10 11 18 19 20 21 rank 3: 11 16 17 21 22 23 When you create the matrix, 'n' is the number of owned dofs on each process [12,6,4,2] and you want to use this final form of the local to global mapping. If you just give the total size (24), the partition will balance the number of owned dofs, but interior dofs won't end up being owned on the correct process. If you use the natural ordering, it's hopeless to end up with correct interior ownership. (Note that assigning ownership to the highest touching rank would have been better balanced in this case.) Does this help? Jed

The promised mapping for the last post -- /"\ Grassl Andreas \ / ASCII Ribbon Campaign Uni Innsbruck Institut f. Mathematik X against HTML email Technikerstr. 13 Zi 709 / \ +43 (0)512 507 6091 [0] 0 0 [0] 1 1 [0] 2 2 [0] 3 3 [0] 4 4 [0] 5 5 [0] 6 6 [0] 7 7 [0] 8 8 [0] 9 9 [0] 10 10 [0] 11 11 [0] 12 12 [0] 13 13 [0] 14 14 [0] 15 15 [0] 16 16 [0] 17 17 [0] 18 18 [0] 19 19 [0] 20 20 [0] 21 21 [0] 22 22 [0] 23 23 [0] 24 24 [0] 25 25 [0] 26 26 [0] 27 27 [0] 28 28 [0] 29 29 [0] 30 30 [0] 31 31 [0] 32 32 [0] 33 33 [0] 34 34 [0] 35 35 [0] 36 36 [0] 37 37 [0] 38 38 [0] 39 39 [0] 40 40 [0] 41 41 [0] 42 42 [0] 43 43 [0] 44 44 [0] 45 45 [0] 46 46 [0] 47 47 [0] 48 48 [0] 49 49 [0] 50 50 [0] 51 51 [0] 52 52 [0] 53 53 [0] 54 54 [0] 55 55 [0] 56 56 [0] 57 57 [0] 58 58 [0] 59 59 [0] 60 60 [0] 61 61 [0] 62 62 [0] 63 63 [0] 64 64 [0] 65 65 [0] 66 66 [0] 67 67 [0] 68 68 [0] 69 69 [0] 70 70 [0] 71 71 [0] 72 72 [0] 73 73 [0] 74 74 [0] 75 75 [0] 76 76 [0] 77 77 [0] 78 78 [0] 79 79 [0] 80 80 [0] 81 81 [0] 82 82 [0] 83 83 [0] 84 84 [0] 85 85 [0] 86 86 [0] 87 87 [0] 88 88 [0] 89 89 [0] 90 90 [0] 91 91 [0] 92 92 [0] 93 93 [0] 94 94 [0] 95 95 [0] 96 96 [0] 97 97 [0] 98 98 [0] 99 99 [0] 100 100 [0] 101 101 [0] 102 102 [0] 103 103 [0] 104 104 [0] 105 105 [0] 106 106 [0] 107 107 [0] 108 108 [0] 109 109 [0] 110 110 [0] 111 111 [0] 112 112 [0] 113 113 [0] 114 114 [0] 115 115 [0] 116 116 [0] 117 117 [0] 118 118 [0] 119 119 [0] 120 120 [0] 121 121 [0] 122 122 [0] 123 123 [0] 124 124 [0] 125 125 [0] 126 126 [0] 127 127 [0] 128 128 [0] 129 129 [0] 130 130 [0] 131 131 [0] 132 132 [0] 133 133 [0] 134 134 [0] 135 135 [0] 136 136 [0] 137 137 [0] 138 138 [0] 139 139 [0] 140 140 [0] 141 141 [0] 142 142 [0] 143 143 [0] 144 144 [0] 145 145 [0] 146 146 [0] 147 147 [0] 148 148 [0] 149 149 [0] 150 150 [0] 151 151 [0] 152 152 [0] 153 153 [0] 154 154 [1] 0 97 [1] 1 98 [1] 2 99 [1] 3 100 [1] 4 101 [1] 5 102 [1] 6 108 [1] 7 109 [1] 8 110 [1] 9 111 [1] 10 112 [1] 11 113 [1] 12 116 [1] 13 117 [1] 14 118 [1] 15 119 [1] 16 120 [1] 17 121 [1] 18 125 [1] 19 126 [1] 20 127 [1] 21 128 [1] 22 131 [1] 23 132 [1] 24 133 [1] 25 134 [1] 26 135 [1] 27 136 [1] 28 140 [1] 29 141 [1] 30 142 [1] 31 143 [1] 32 146 [1] 33 147 [1] 34 148 [1] 35 149 [1] 36 150 [1] 37 151 [1] 38 152 [1] 39 155 [1] 40 156 [1] 41 157 [1] 42 158 [1] 43 159 [1] 44 160 [1] 45 161 [1] 46 162 [1] 47 163 [1] 48 164 [1] 49 165 [1] 50 166 [1] 51 167 [1] 52 168 [1] 53 169 [1] 54 170 [1] 55 171 [1] 56 172 [1] 57 173 [1] 58 174 [1] 59 175 [1] 60 176 [1] 61 177 [1] 62 178 [1] 63 179 [1] 64 180 [1] 65 181 [1] 66 182 [1] 67 183 [1] 68 184 [1] 69 185 [1] 70 186 [1] 71 187 [1] 72 188 [1] 73 189 [1] 74 190 [1] 75 191 [1] 76 192 [1] 77 193 [1] 78 194 [1] 79 195 [1] 80 196 [1] 81 197 [1] 82 198 [1] 83 199 [1] 84 200 [1] 85 201 [1] 86 202 [1] 87 203 [1] 88 204 [1] 89 205 [1] 90 206 [1] 91 207 [1] 92 208 [1] 93 209 [1] 94 210 [1] 95 211 [1] 96 212 [1] 97 213 [1] 98 214 [1] 99 215 [1] 100 216 [1] 101 217 [1] 102 218 [1] 103 219 [1] 104 220 [1] 105 221 [1] 106 222 [1] 107 223 [1] 108 224 [1] 109 225 [1] 110 226 [1] 111 227 [1] 112 228 [1] 113 229 [1] 114 230 [1] 115 231 [1] 116 232 [1] 117 233 [1] 118 234 [1] 119 235 [1] 120 236 [1] 121 237 [1] 122 238 [1] 123 239 [1] 124 240 [1] 125 241 [1] 126 242 [1] 127 243 [1] 128 244 [1] 129 245 [1] 130 246 [1] 131 247 [1] 132 248 [1] 133 249 [1] 134 250 [1] 135 251 [1] 136 252 [1] 137 253 [1] 138 254 [1] 139 255 [1] 140 256 [1] 141 257 [1] 142 258 [1] 143 259 [1] 144 260 [1] 145 261 [1] 146 262 [1] 147 263 [1] 148 264 [1] 149 265 [1] 150 266 [1] 151 267 [1] 152 268 [1] 153 269 [1] 154 270

Barry Smith schrieb:

When properly running nn-cg (are you sure everything is symmetric?) should require 10-30 iterations (certainly for model problems)

ok, this was the number I expected.

...

nn-cg on 2 nodes 229 iterations, condition 6285 nn-cg on 4 nodes 331 iterations, condition 13312

Are you sure that your operator has the null space of only constants?

no, I didn't touch anything regarding the null space since I thought it would be done inside the NN-preconditioner. Does this mean I have to set up a null space of the size of the Schur complement system, i.e. the number of interface DOF's? cheers, ando

Barry

On Jun 3, 2009, at 5:29 AM, Andreas Grassl wrote:

...

Barry Smith schrieb:

...

Hmm, it sounds like the difference between local "ghosted" vectors and the global parallel vectors. But I do not understand why any of the local vector entries would be zero. Doesn't the vector X that is passed into KSP (or SNES) have the global entries and uniquely define the solution? Why is viewing that not right?

I still don't understand fully the underlying processes of the whole PCNN solution procedure, but trying around I substituted

MatCreateIS(commw, ind_length, ind_length, PETSC_DECIDE, PETSC_DECIDE, gridmapping, &A);

by

MatCreateIS(commw, PETSC_DECIDE, PETSC_DECIDE, actdof, actdof, gridmapping, &A);

and received the needed results.

Furthermore it seems, that the load balance is now better, although I still don't reach the expected values, e.g. ilu-cg 320 iterations, condition 4601 cg only 1662 iterations, condition 84919

nn-cg on 2 nodes 229 iterations, condition 6285 nn-cg on 4 nodes 331 iterations, condition 13312

or is it not to expect, that nn-cg is faster than ilu-cg?

cheers,

ando

-- /"\ Grassl Andreas \ / ASCII Ribbon Campaign Uni Innsbruck Institut f. Mathematik X against HTML email Technikerstr. 13 Zi 709 / \ +43 (0)512 507 6091

-- /"\ \ / ASCII Ribbon X against HTML email / \

Barry Smith schrieb:

On Jun 3, 2009, at 5:29 PM, Andreas Grassl wrote:

...

Barry Smith schrieb:

...

When properly running nn-cg (are you sure everything is symmetric?) should require 10-30 iterations (certainly for model problems)

ok, this was the number I expected.

...

...

nn-cg on 2 nodes 229 iterations, condition 6285 nn-cg on 4 nodes 331 iterations, condition 13312

Are you sure that your operator has the null space of only constants?

no, I didn't touch anything regarding the null space since I thought it would be done inside the NN-preconditioner. Does this mean I have to set up a null space of the size of the Schur complement system, i.e. the number of interface DOF's?

No, I don't think you need to do anything about the null space. The code in PETSc for NN is for (and only for) a null space of constants. BTW: with 2 or 4 subdomains they all touch the boundary and likely don't have a null space anyways.

Run with -ksp_view and make sure the local solves are being done with LU

I don't find the anomalies... setting local_ksp-rtol to 1e-8 doesn't change anything the options passed are: -is_localD_ksp_type preonly -is_localD_pc_factor_shift_positive_definite -is_localD_pc_type lu -is_localN_ksp_type preonly -is_localN_pc_factor_shift_positive_definite -is_localN_pc_type lu -ksp_rtol 1e-8 -ksp_view #-is_localD_ksp_view #-is_localN_ksp_view #-nn_coarse_ksp_view # -pc_is_remove_nullspace_fixed this option doesn't produce any effect -log_summary -options_left and produce: -ksp_view: KSP Object: type: cg maximum iterations=10000 tolerances: relative=1e-08, absolute=1e-50, divergence=10000 left preconditioning PC Object: type: nn linear system matrix = precond matrix: Matrix Object: type=is, rows=28632, cols=28632 Matrix Object:(is) type=seqaij, rows=7537, cols=7537 total: nonzeros=359491, allocated nonzeros=602960 using I-node routines: found 4578 nodes, limit used is 5 Matrix Object:(is) type=seqaij, rows=7515, cols=7515 total: nonzeros=349347, allocated nonzeros=601200 using I-node routines: found 5159 nodes, limit used is 5 Matrix Object:(is) type=seqaij, rows=7533, cols=7533 total: nonzeros=357291, allocated nonzeros=602640 using I-node routines: found 4739 nodes, limit used is 5 Matrix Object:(is) type=seqaij, rows=7360, cols=7360 total: nonzeros=364390, allocated nonzeros=588800 using I-node routines: found 3602 nodes, limit used is 5 -is_local...: KSP Object:(is_localD_) type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning PC Object:(is_localD_) type: lu LU: out-of-place factorization matrix ordering: nd LU: tolerance for zero pivot 1e-12 LU: using Manteuffel shift LU: factor fill ratio needed 4.73566 Factored matrix follows Matrix Object: type=seqaij, rows=6714, cols=6714 package used to perform factorization: petsc total: nonzeros=1479078, allocated nonzeros=1479078 using I-node routines: found 2790 nodes, limit used is 5 linear system matrix = precond matrix: Matrix Object: type=seqaij, rows=6714, cols=6714 total: nonzeros=312328, allocated nonzeros=312328 using I-node routines: found 4664 nodes, limit used is 5 KSP Object:(is_localN_) type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning PC Object:(is_localN_) type: lu LU: out-of-place factorization matrix ordering: nd LU: tolerance for zero pivot 1e-12 LU: using Manteuffel shift LU: factor fill ratio needed 5.07571 Factored matrix follows Matrix Object: type=seqaij, rows=7537, cols=7537 package used to perform factorization: petsc total: nonzeros=1824671, allocated nonzeros=1824671 using I-node routines: found 2939 nodes, limit used is 5 linear system matrix = precond matrix: Matrix Object:(is) type=seqaij, rows=7537, cols=7537 total: nonzeros=359491, allocated nonzeros=602960 using I-node routines: found 4578 nodes, limit used is 5 -nn_coarse_ksp_view: KSP Object:(nn_coarse_) type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning PC Object:(nn_coarse_) type: redundant Redundant preconditioner: First (color=0) of 4 PCs follows KSP Object:(redundant_) type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning PC Object:(redundant_) type: lu LU: out-of-place factorization matrix ordering: nd LU: tolerance for zero pivot 1e-12 LU: factor fill ratio needed 1 Factored matrix follows Matrix Object: type=seqaij, rows=4, cols=4 package used to perform factorization: petsc total: nonzeros=4, allocated nonzeros=4 not using I-node routines linear system matrix = precond matrix: Matrix Object: type=seqaij, rows=4, cols=4 total: nonzeros=4, allocated nonzeros=4 not using I-node routines linear system matrix = precond matrix: Matrix Object: type=mpiaij, rows=4, cols=4 total: nonzeros=4, allocated nonzeros=68 not using I-node (on process 0) routines cheers, ando -- /"\ Grassl Andreas \ / ASCII Ribbon Campaign Uni Innsbruck Institut f. Mathematik X against HTML email Technikerstr. 13 Zi 709 / \ +43 (0)512 507 6091

Barry Smith schrieb:

Run with GMRES, what happens?

Same behaviour...

On Jun 4, 2009, at 11:07 AM, Andreas Grassl wrote:

...

Barry Smith schrieb:

...

On Jun 3, 2009, at 5:29 PM, Andreas Grassl wrote:

...

Barry Smith schrieb:

...

When properly running nn-cg (are you sure everything is symmetric?) should require 10-30 iterations (certainly for model problems)

ok, this was the number I expected.

...

...

nn-cg on 2 nodes 229 iterations, condition 6285 nn-cg on 4 nodes 331 iterations, condition 13312

Are you sure that your operator has the null space of only constants?

no, I didn't touch anything regarding the null space since I thought it would be done inside the NN-preconditioner. Does this mean I have to set up a null space of the size of the Schur complement system, i.e. the number of interface DOF's?

No, I don't think you need to do anything about the null space. The code in PETSc for NN is for (and only for) a null space of constants. BTW: with 2 or 4 subdomains they all touch the boundary and likely don't have a null space anyways.

Run with -ksp_view and make sure the local solves are being done with LU

I don't find the anomalies... setting local_ksp-rtol to 1e-8 doesn't change anything

the options passed are:

-is_localD_ksp_type preonly -is_localD_pc_factor_shift_positive_definite -is_localD_pc_type lu -is_localN_ksp_type preonly -is_localN_pc_factor_shift_positive_definite -is_localN_pc_type lu -ksp_rtol 1e-8 -ksp_view #-is_localD_ksp_view #-is_localN_ksp_view #-nn_coarse_ksp_view # -pc_is_remove_nullspace_fixed this option doesn't produce any effect -log_summary -options_left

and produce:

-ksp_view:

KSP Object: type: cg maximum iterations=10000 tolerances: relative=1e-08, absolute=1e-50, divergence=10000 left preconditioning PC Object: type: nn linear system matrix = precond matrix: Matrix Object: type=is, rows=28632, cols=28632 Matrix Object:(is) type=seqaij, rows=7537, cols=7537 total: nonzeros=359491, allocated nonzeros=602960 using I-node routines: found 4578 nodes, limit used is 5 Matrix Object:(is) type=seqaij, rows=7515, cols=7515 total: nonzeros=349347, allocated nonzeros=601200 using I-node routines: found 5159 nodes, limit used is 5 Matrix Object:(is) type=seqaij, rows=7533, cols=7533 total: nonzeros=357291, allocated nonzeros=602640 using I-node routines: found 4739 nodes, limit used is 5 Matrix Object:(is) type=seqaij, rows=7360, cols=7360 total: nonzeros=364390, allocated nonzeros=588800 using I-node routines: found 3602 nodes, limit used is 5

-is_local...:

KSP Object:(is_localD_) type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning PC Object:(is_localD_) type: lu LU: out-of-place factorization matrix ordering: nd LU: tolerance for zero pivot 1e-12 LU: using Manteuffel shift LU: factor fill ratio needed 4.73566 Factored matrix follows Matrix Object: type=seqaij, rows=6714, cols=6714 package used to perform factorization: petsc total: nonzeros=1479078, allocated nonzeros=1479078 using I-node routines: found 2790 nodes, limit used is 5 linear system matrix = precond matrix: Matrix Object: type=seqaij, rows=6714, cols=6714 total: nonzeros=312328, allocated nonzeros=312328 using I-node routines: found 4664 nodes, limit used is 5

KSP Object:(is_localN_) type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning PC Object:(is_localN_) type: lu LU: out-of-place factorization matrix ordering: nd LU: tolerance for zero pivot 1e-12 LU: using Manteuffel shift LU: factor fill ratio needed 5.07571 Factored matrix follows Matrix Object: type=seqaij, rows=7537, cols=7537 package used to perform factorization: petsc total: nonzeros=1824671, allocated nonzeros=1824671 using I-node routines: found 2939 nodes, limit used is 5 linear system matrix = precond matrix: Matrix Object:(is) type=seqaij, rows=7537, cols=7537 total: nonzeros=359491, allocated nonzeros=602960 using I-node routines: found 4578 nodes, limit used is 5

-nn_coarse_ksp_view:

KSP Object:(nn_coarse_) type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning PC Object:(nn_coarse_) type: redundant Redundant preconditioner: First (color=0) of 4 PCs follows KSP Object:(redundant_) type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning PC Object:(redundant_) type: lu LU: out-of-place factorization matrix ordering: nd LU: tolerance for zero pivot 1e-12 LU: factor fill ratio needed 1 Factored matrix follows Matrix Object: type=seqaij, rows=4, cols=4 package used to perform factorization: petsc total: nonzeros=4, allocated nonzeros=4 not using I-node routines linear system matrix = precond matrix: Matrix Object: type=seqaij, rows=4, cols=4 total: nonzeros=4, allocated nonzeros=4 not using I-node routines linear system matrix = precond matrix: Matrix Object: type=mpiaij, rows=4, cols=4 total: nonzeros=4, allocated nonzeros=68 not using I-node (on process 0) routines

cheers,

ando

-- /"\ Grassl Andreas \ / ASCII Ribbon Campaign Uni Innsbruck Institut f. Mathematik X against HTML email Technikerstr. 13 Zi 709 / \ +43 (0)512 507 6091

-- /"\ \ / ASCII Ribbon X against HTML email / \