Accessing Vector's ghost values
Hi all, I've never used a mailing list before, so I hope this message will reach PETSc users and experts and someone might be willing to help me. I am also novice in PETSc. I have developed an unstructured finite volume solver on top of PETSc libraries. In sequential, it works like a charm. For the parallel version, I do domain decomposition externally with Metis, and work out local and global numberings, as well as communication patterns between processor. (The latter don't seem to be needed for PETSc, though.) When I run my program in parallel, it also works, but I miss values in vectors' ghost points. I create vectors with command: VecCreate(PETSC_COMM_WORLD, &x); Is it possible to get the ghost values if a vector is created like this? I have tried to use VecCreateGhost, but for some reason which is beyond my comprehension, PETSc goes berserk when it reaches the command: VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, ifrom, &x) Can anyone help me? Either how to reach ghost values for vector created by VecCreate, or how to use VecCreateGhost properly? Kind regards, Bojan
On Thu, Feb 23, 2012 at 10:46 AM, Bojan Niceno <[email protected]> wrote:
Hi all,
I've never used a mailing list before, so I hope this message will reach PETSc users and experts and someone might be willing to help me. I am also novice in PETSc.
I have developed an unstructured finite volume solver on top of PETSc libraries. In sequential, it works like a charm. For the parallel version, I do domain decomposition externally with Metis, and work out local and global numberings, as well as communication patterns between processor. (The latter don't seem to be needed for PETSc, though.) When I run my program in parallel, it also works, but I miss values in vectors' ghost points.
I create vectors with command: VecCreate(PETSC_COMM_WORLD, &x);
Is it possible to get the ghost values if a vector is created like this?
I do not understand this question. By definition, "ghost values" are those not stored in the global vector.
I have tried to use VecCreateGhost, but for some reason which is beyond my comprehension, PETSc goes berserk when it reaches the command: VecCreateGhost(PETSC_COMM_**WORLD, n, PETSC_DECIDE, nghost, ifrom, &x)
I think you can understand that "berserk" tells me absolutely nothing. Error message? Stack trace? Did you try to run an example which uses VecGhost? Thanks, Matt
Can anyone help me? Either how to reach ghost values for vector created by VecCreate, or how to use VecCreateGhost properly?
Kind regards,
Bojan
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
Dear Matthew, thank you for your response. When I use VecCreateGhost, I get the following: [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] PetscCommDuplicate line 140 src/sys/objects/tagm.c [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 30 src/sys/objects/inherit.c [0]PETSC ERROR: [0] VecCreate line 32 src/vec/vec/interface/veccreate.c [0]PETSC ERROR: [0] VecCreateGhostWithArray line 567 src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: [0] VecCreateGhost line 647 src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./PSI-Flow on a arch-linu named lccfd06 by niceno Thu Feb 23 19:02:45 2012 [0]PETSC ERROR: Libraries linked from /homecfd/niceno/PETSc-3.2-p6/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Fri Feb 10 10:24:13 2012 [0]PETSC ERROR: Configure options [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file I don't understand what could be causing it. I took very good care to match the global numbers of ghost cells when calling VecCreateGhost Kind regards, Bojan On 2/23/2012 5:53 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 10:46 AM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
Hi all,
I've never used a mailing list before, so I hope this message will reach PETSc users and experts and someone might be willing to help me. I am also novice in PETSc.
I have developed an unstructured finite volume solver on top of PETSc libraries. In sequential, it works like a charm. For the parallel version, I do domain decomposition externally with Metis, and work out local and global numberings, as well as communication patterns between processor. (The latter don't seem to be needed for PETSc, though.) When I run my program in parallel, it also works, but I miss values in vectors' ghost points.
I create vectors with command: VecCreate(PETSC_COMM_WORLD, &x);
Is it possible to get the ghost values if a vector is created like this?
I do not understand this question. By definition, "ghost values" are those not stored in the global vector.
I have tried to use VecCreateGhost, but for some reason which is beyond my comprehension, PETSc goes berserk when it reaches the command: VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, ifrom, &x)
I think you can understand that "berserk" tells me absolutely nothing. Error message? Stack trace? Did you try to run an example which uses VecGhost?
Thanks,
Matt
Can anyone help me? Either how to reach ghost values for vector created by VecCreate, or how to use VecCreateGhost properly?
Kind regards,
Bojan
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
On Thu, Feb 23, 2012 at 12:05 PM, Bojan Niceno <[email protected]> wrote:
Dear Matthew,
thank you for your response. When I use VecCreateGhost, I get the following:
It appears that you passed a bad communicator. Did you not initialize a 'comm' variable? Matt
[0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] PetscCommDuplicate line 140 src/sys/objects/tagm.c [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 30 src/sys/objects/inherit.c [0]PETSC ERROR: [0] VecCreate line 32 src/vec/vec/interface/veccreate.c [0]PETSC ERROR: [0] VecCreateGhostWithArray line 567 src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: [0] VecCreateGhost line 647 src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./PSI-Flow on a arch-linu named lccfd06 by niceno Thu Feb 23 19:02:45 2012 [0]PETSC ERROR: Libraries linked from /homecfd/niceno/PETSc-3.2-p6/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Fri Feb 10 10:24:13 2012 [0]PETSC ERROR: Configure options [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
I don't understand what could be causing it. I took very good care to match the global numbers of ghost cells when calling VecCreateGhost
Kind regards,
Bojan
On 2/23/2012 5:53 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 10:46 AM, Bojan Niceno <[email protected]>wrote:
Hi all,
I've never used a mailing list before, so I hope this message will reach PETSc users and experts and someone might be willing to help me. I am also novice in PETSc.
I have developed an unstructured finite volume solver on top of PETSc libraries. In sequential, it works like a charm. For the parallel version, I do domain decomposition externally with Metis, and work out local and global numberings, as well as communication patterns between processor. (The latter don't seem to be needed for PETSc, though.) When I run my program in parallel, it also works, but I miss values in vectors' ghost points.
I create vectors with command: VecCreate(PETSC_COMM_WORLD, &x);
Is it possible to get the ghost values if a vector is created like this?
I do not understand this question. By definition, "ghost values" are those not stored in the global vector.
I have tried to use VecCreateGhost, but for some reason which is beyond my comprehension, PETSc goes berserk when it reaches the command: VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, ifrom, &x)
I think you can understand that "berserk" tells me absolutely nothing. Error message? Stack trace? Did you try to run an example which uses VecGhost?
Thanks,
Matt
Can anyone help me? Either how to reach ghost values for vector created by VecCreate, or how to use VecCreateGhost properly?
Kind regards,
Bojan
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
On 2/23/2012 7:24 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 12:05 PM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
Dear Matthew,
thank you for your response. When I use VecCreateGhost, I get the following:
It appears that you passed a bad communicator. Did you not initialize a 'comm' variable?
I pass PETSC_COMM_WORLD to VecCreateGhost. I don't know what you mean by 'comm' variable :-( I called all the routines to initialize PETSc. Cheers, Bojan
Matt
[0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] PetscCommDuplicate line 140 src/sys/objects/tagm.c [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 30 src/sys/objects/inherit.c [0]PETSC ERROR: [0] VecCreate line 32 src/vec/vec/interface/veccreate.c [0]PETSC ERROR: [0] VecCreateGhostWithArray line 567 src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: [0] VecCreateGhost line 647 src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./PSI-Flow on a arch-linu named lccfd06 by niceno Thu Feb 23 19:02:45 2012 [0]PETSC ERROR: Libraries linked from /homecfd/niceno/PETSc-3.2-p6/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Fri Feb 10 10:24:13 2012 [0]PETSC ERROR: Configure options [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
I don't understand what could be causing it. I took very good care to match the global numbers of ghost cells when calling VecCreateGhost
Kind regards,
Bojan
On 2/23/2012 5:53 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 10:46 AM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
Hi all,
I've never used a mailing list before, so I hope this message will reach PETSc users and experts and someone might be willing to help me. I am also novice in PETSc.
I have developed an unstructured finite volume solver on top of PETSc libraries. In sequential, it works like a charm. For the parallel version, I do domain decomposition externally with Metis, and work out local and global numberings, as well as communication patterns between processor. (The latter don't seem to be needed for PETSc, though.) When I run my program in parallel, it also works, but I miss values in vectors' ghost points.
I create vectors with command: VecCreate(PETSC_COMM_WORLD, &x);
Is it possible to get the ghost values if a vector is created like this?
I do not understand this question. By definition, "ghost values" are those not stored in the global vector.
I have tried to use VecCreateGhost, but for some reason which is beyond my comprehension, PETSc goes berserk when it reaches the command: VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, ifrom, &x)
I think you can understand that "berserk" tells me absolutely nothing. Error message? Stack trace? Did you try to run an example which uses VecGhost?
Thanks,
Matt
Can anyone help me? Either how to reach ghost values for vector created by VecCreate, or how to use VecCreateGhost properly?
Kind regards,
Bojan
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
On Thu, Feb 23, 2012 at 12:28 PM, Bojan Niceno <[email protected]> wrote:
On 2/23/2012 7:24 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 12:05 PM, Bojan Niceno <[email protected]>wrote:
Dear Matthew,
thank you for your response. When I use VecCreateGhost, I get the following:
It appears that you passed a bad communicator. Did you not initialize a 'comm' variable?
I pass PETSC_COMM_WORLD to VecCreateGhost.
I don't know what you mean by 'comm' variable :-( I called all the routines to initialize PETSc.
Send your code to [email protected]. Matt
Cheers,
Bojan
Matt
[0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] PetscCommDuplicate line 140 src/sys/objects/tagm.c [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 30 src/sys/objects/inherit.c [0]PETSC ERROR: [0] VecCreate line 32 src/vec/vec/interface/veccreate.c [0]PETSC ERROR: [0] VecCreateGhostWithArray line 567 src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: [0] VecCreateGhost line 647 src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./PSI-Flow on a arch-linu named lccfd06 by niceno Thu Feb 23 19:02:45 2012 [0]PETSC ERROR: Libraries linked from /homecfd/niceno/PETSc-3.2-p6/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Fri Feb 10 10:24:13 2012 [0]PETSC ERROR: Configure options [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
I don't understand what could be causing it. I took very good care to match the global numbers of ghost cells when calling VecCreateGhost
Kind regards,
Bojan
On 2/23/2012 5:53 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 10:46 AM, Bojan Niceno <[email protected]>wrote:
Hi all,
I've never used a mailing list before, so I hope this message will reach PETSc users and experts and someone might be willing to help me. I am also novice in PETSc.
I have developed an unstructured finite volume solver on top of PETSc libraries. In sequential, it works like a charm. For the parallel version, I do domain decomposition externally with Metis, and work out local and global numberings, as well as communication patterns between processor. (The latter don't seem to be needed for PETSc, though.) When I run my program in parallel, it also works, but I miss values in vectors' ghost points.
I create vectors with command: VecCreate(PETSC_COMM_WORLD, &x);
Is it possible to get the ghost values if a vector is created like this?
I do not understand this question. By definition, "ghost values" are those not stored in the global vector.
I have tried to use VecCreateGhost, but for some reason which is beyond my comprehension, PETSc goes berserk when it reaches the command: VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, ifrom, &x)
I think you can understand that "berserk" tells me absolutely nothing. Error message? Stack trace? Did you try to run an example which uses VecGhost?
Thanks,
Matt
Can anyone help me? Either how to reach ghost values for vector created by VecCreate, or how to use VecCreateGhost properly?
Kind regards,
Bojan
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
Dear Matt, are you sure? It is almost 4000 lines long! Shall I send only the function which bother me? If the entire code is what you need, shall I make a tarball and attach it? Kind regards, Bojan On 2/23/2012 7:44 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 12:28 PM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
On 2/23/2012 7:24 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 12:05 PM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
Dear Matthew,
thank you for your response. When I use VecCreateGhost, I get the following:
It appears that you passed a bad communicator. Did you not initialize a 'comm' variable?
I pass PETSC_COMM_WORLD to VecCreateGhost.
I don't know what you mean by 'comm' variable :-( I called all the routines to initialize PETSc.
Send your code to [email protected] <mailto:[email protected]>.
Matt
Cheers,
Bojan
Matt
[0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] PetscCommDuplicate line 140 src/sys/objects/tagm.c [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 30 src/sys/objects/inherit.c [0]PETSC ERROR: [0] VecCreate line 32 src/vec/vec/interface/veccreate.c [0]PETSC ERROR: [0] VecCreateGhostWithArray line 567 src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: [0] VecCreateGhost line 647 src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./PSI-Flow on a arch-linu named lccfd06 by niceno Thu Feb 23 19:02:45 2012 [0]PETSC ERROR: Libraries linked from /homecfd/niceno/PETSc-3.2-p6/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Fri Feb 10 10:24:13 2012 [0]PETSC ERROR: Configure options [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
I don't understand what could be causing it. I took very good care to match the global numbers of ghost cells when calling VecCreateGhost
Kind regards,
Bojan
On 2/23/2012 5:53 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 10:46 AM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
Hi all,
I've never used a mailing list before, so I hope this message will reach PETSc users and experts and someone might be willing to help me. I am also novice in PETSc.
I have developed an unstructured finite volume solver on top of PETSc libraries. In sequential, it works like a charm. For the parallel version, I do domain decomposition externally with Metis, and work out local and global numberings, as well as communication patterns between processor. (The latter don't seem to be needed for PETSc, though.) When I run my program in parallel, it also works, but I miss values in vectors' ghost points.
I create vectors with command: VecCreate(PETSC_COMM_WORLD, &x);
Is it possible to get the ghost values if a vector is created like this?
I do not understand this question. By definition, "ghost values" are those not stored in the global vector.
I have tried to use VecCreateGhost, but for some reason which is beyond my comprehension, PETSc goes berserk when it reaches the command: VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, ifrom, &x)
I think you can understand that "berserk" tells me absolutely nothing. Error message? Stack trace? Did you try to run an example which uses VecGhost?
Thanks,
Matt
Can anyone help me? Either how to reach ghost values for vector created by VecCreate, or how to use VecCreateGhost properly?
Kind regards,
Bojan
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
On Thu, Feb 23, 2012 at 12:51 PM, Bojan Niceno <[email protected]> wrote:
Dear Matt,
are you sure? It is almost 4000 lines long! Shall I send only the function which bother me?
If the entire code is what you need, shall I make a tarball and attach it?
Send something the builds and runs. Don't care how long it is. Matt
Kind regards,
Bojan
On 2/23/2012 7:44 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 12:28 PM, Bojan Niceno <[email protected]>wrote:
On 2/23/2012 7:24 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 12:05 PM, Bojan Niceno <[email protected]>wrote:
Dear Matthew,
thank you for your response. When I use VecCreateGhost, I get the following:
It appears that you passed a bad communicator. Did you not initialize a 'comm' variable?
I pass PETSC_COMM_WORLD to VecCreateGhost.
I don't know what you mean by 'comm' variable :-( I called all the routines to initialize PETSc.
Send your code to [email protected].
Matt
Cheers,
Bojan
Matt
[0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] PetscCommDuplicate line 140 src/sys/objects/tagm.c [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 30 src/sys/objects/inherit.c [0]PETSC ERROR: [0] VecCreate line 32 src/vec/vec/interface/veccreate.c [0]PETSC ERROR: [0] VecCreateGhostWithArray line 567 src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: [0] VecCreateGhost line 647 src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./PSI-Flow on a arch-linu named lccfd06 by niceno Thu Feb 23 19:02:45 2012 [0]PETSC ERROR: Libraries linked from /homecfd/niceno/PETSc-3.2-p6/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Fri Feb 10 10:24:13 2012 [0]PETSC ERROR: Configure options [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
I don't understand what could be causing it. I took very good care to match the global numbers of ghost cells when calling VecCreateGhost
Kind regards,
Bojan
On 2/23/2012 5:53 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 10:46 AM, Bojan Niceno <[email protected]>wrote:
Hi all,
I've never used a mailing list before, so I hope this message will reach PETSc users and experts and someone might be willing to help me. I am also novice in PETSc.
I have developed an unstructured finite volume solver on top of PETSc libraries. In sequential, it works like a charm. For the parallel version, I do domain decomposition externally with Metis, and work out local and global numberings, as well as communication patterns between processor. (The latter don't seem to be needed for PETSc, though.) When I run my program in parallel, it also works, but I miss values in vectors' ghost points.
I create vectors with command: VecCreate(PETSC_COMM_WORLD, &x);
Is it possible to get the ghost values if a vector is created like this?
I do not understand this question. By definition, "ghost values" are those not stored in the global vector.
I have tried to use VecCreateGhost, but for some reason which is beyond my comprehension, PETSc goes berserk when it reaches the command: VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, ifrom, &x)
I think you can understand that "berserk" tells me absolutely nothing. Error message? Stack trace? Did you try to run an example which uses VecGhost?
Thanks,
Matt
Can anyone help me? Either how to reach ghost values for vector created by VecCreate, or how to use VecCreateGhost properly?
Kind regards,
Bojan
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
Dear Matt, I sent the code as an attached tarball. I sent it with case I run, so is 2 MB big. It is now in the cue for moderator's approval. Thanks. Kind regards, Bojan On 2/23/2012 8:04 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 12:51 PM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
Dear Matt,
are you sure? It is almost 4000 lines long! Shall I send only the function which bother me?
If the entire code is what you need, shall I make a tarball and attach it?
Send something the builds and runs. Don't care how long it is.
Matt
Kind regards,
Bojan
On 2/23/2012 7:44 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 12:28 PM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
On 2/23/2012 7:24 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 12:05 PM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
Dear Matthew,
thank you for your response. When I use VecCreateGhost, I get the following:
It appears that you passed a bad communicator. Did you not initialize a 'comm' variable?
I pass PETSC_COMM_WORLD to VecCreateGhost.
I don't know what you mean by 'comm' variable :-( I called all the routines to initialize PETSc.
Send your code to [email protected] <mailto:[email protected]>.
Matt
Cheers,
Bojan
Matt
[0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] PetscCommDuplicate line 140 src/sys/objects/tagm.c [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 30 src/sys/objects/inherit.c [0]PETSC ERROR: [0] VecCreate line 32 src/vec/vec/interface/veccreate.c [0]PETSC ERROR: [0] VecCreateGhostWithArray line 567 src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: [0] VecCreateGhost line 647 src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./PSI-Flow on a arch-linu named lccfd06 by niceno Thu Feb 23 19:02:45 2012 [0]PETSC ERROR: Libraries linked from /homecfd/niceno/PETSc-3.2-p6/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Fri Feb 10 10:24:13 2012 [0]PETSC ERROR: Configure options [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
I don't understand what could be causing it. I took very good care to match the global numbers of ghost cells when calling VecCreateGhost
Kind regards,
Bojan
On 2/23/2012 5:53 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 10:46 AM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
Hi all,
I've never used a mailing list before, so I hope this message will reach PETSc users and experts and someone might be willing to help me. I am also novice in PETSc.
I have developed an unstructured finite volume solver on top of PETSc libraries. In sequential, it works like a charm. For the parallel version, I do domain decomposition externally with Metis, and work out local and global numberings, as well as communication patterns between processor. (The latter don't seem to be needed for PETSc, though.) When I run my program in parallel, it also works, but I miss values in vectors' ghost points.
I create vectors with command: VecCreate(PETSC_COMM_WORLD, &x);
Is it possible to get the ghost values if a vector is created like this?
I do not understand this question. By definition, "ghost values" are those not stored in the global vector.
I have tried to use VecCreateGhost, but for some reason which is beyond my comprehension, PETSc goes berserk when it reaches the command: VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, ifrom, &x)
I think you can understand that "berserk" tells me absolutely nothing. Error message? Stack trace? Did you try to run an example which uses VecGhost?
Thanks,
Matt
Can anyone help me? Either how to reach ghost values for vector created by VecCreate, or how to use VecCreateGhost properly?
Kind regards,
Bojan
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
On Thu, Feb 23, 2012 at 1:33 PM, Bojan Niceno <[email protected]> wrote:
Dear Matt,
I sent the code as an attached tarball. I sent it with case I run, so is 2 MB big. It is now in the cue for moderator's approval.
No, you HAVE to send it to [email protected], as I said last time, for exactly this reason. Matt
Thanks.
Kind regards,
Bojan
On 2/23/2012 8:04 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 12:51 PM, Bojan Niceno <[email protected]>wrote:
Dear Matt,
are you sure? It is almost 4000 lines long! Shall I send only the function which bother me?
If the entire code is what you need, shall I make a tarball and attach it?
Send something the builds and runs. Don't care how long it is.
Matt
Kind regards,
Bojan
On 2/23/2012 7:44 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 12:28 PM, Bojan Niceno <[email protected]>wrote:
On 2/23/2012 7:24 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 12:05 PM, Bojan Niceno <[email protected]>wrote:
Dear Matthew,
thank you for your response. When I use VecCreateGhost, I get the following:
It appears that you passed a bad communicator. Did you not initialize a 'comm' variable?
I pass PETSC_COMM_WORLD to VecCreateGhost.
I don't know what you mean by 'comm' variable :-( I called all the routines to initialize PETSc.
Send your code to [email protected].
Matt
Cheers,
Bojan
Matt
[0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] PetscCommDuplicate line 140 src/sys/objects/tagm.c [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 30 src/sys/objects/inherit.c [0]PETSC ERROR: [0] VecCreate line 32 src/vec/vec/interface/veccreate.c [0]PETSC ERROR: [0] VecCreateGhostWithArray line 567 src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: [0] VecCreateGhost line 647 src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./PSI-Flow on a arch-linu named lccfd06 by niceno Thu Feb 23 19:02:45 2012 [0]PETSC ERROR: Libraries linked from /homecfd/niceno/PETSc-3.2-p6/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Fri Feb 10 10:24:13 2012 [0]PETSC ERROR: Configure options [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
I don't understand what could be causing it. I took very good care to match the global numbers of ghost cells when calling VecCreateGhost
Kind regards,
Bojan
On 2/23/2012 5:53 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 10:46 AM, Bojan Niceno <[email protected]>wrote:
Hi all,
I've never used a mailing list before, so I hope this message will reach PETSc users and experts and someone might be willing to help me. I am also novice in PETSc.
I have developed an unstructured finite volume solver on top of PETSc libraries. In sequential, it works like a charm. For the parallel version, I do domain decomposition externally with Metis, and work out local and global numberings, as well as communication patterns between processor. (The latter don't seem to be needed for PETSc, though.) When I run my program in parallel, it also works, but I miss values in vectors' ghost points.
I create vectors with command: VecCreate(PETSC_COMM_WORLD, &x);
Is it possible to get the ghost values if a vector is created like this?
I do not understand this question. By definition, "ghost values" are those not stored in the global vector.
I have tried to use VecCreateGhost, but for some reason which is beyond my comprehension, PETSc goes berserk when it reaches the command: VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, ifrom, &x)
I think you can understand that "berserk" tells me absolutely nothing. Error message? Stack trace? Did you try to run an example which uses VecGhost?
Thanks,
Matt
Can anyone help me? Either how to reach ghost values for vector created by VecCreate, or how to use VecCreateGhost properly?
Kind regards,
Bojan
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
Dear Matt, I have a new insight, although is not the full resolution. If I change my code in PETScSolver.cpp from: /*-------------------------------------------------+ | Make necessary PETSc intializations for vetors | +-------------------------------------------------*/ Int nghost = N - n; Int * ghosts = new Int(nghost); for(Int n=0; n<M.mesh.nodes.size(); n++) { assert( M.mesh.nodes[n].global_number >= 0); assert( M.mesh.nodes[n].global_number < 14065); } for(Int i=n; i<N; i++) { assert( M.mesh.nodes[i].global_number >= 0); assert( M.mesh.nodes[i].global_number < 14065); assert( ! (M.mesh.nodes[i].global_number >= n_start && M.mesh.nodes[i].global_number < n_end) ); ghosts[i] = M.mesh.nodes[i].global_number; } VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, &ghosts[0], &x); to: /*-------------------------------------------------+ | Make necessary PETSc intializations for vetors | +-------------------------------------------------*/ Int nghost = N - n; Indices ghosts; // <---= NEW! for(Int n=0; n<M.mesh.nodes.size(); n++) { assert( M.mesh.nodes[n].global_number >= 0); assert( M.mesh.nodes[n].global_number < 14065); } for(Int i=n; i<N; i++) { assert( M.mesh.nodes[i].global_number >= 0); assert( M.mesh.nodes[i].global_number < 14065); assert( ! (M.mesh.nodes[i].global_number >= n_start && M.mesh.nodes[i].global_number < n_end) ); ghosts.push_back( M.mesh.nodes[i].global_number ); // <---= NEW! } assert( ghosts.size() == nghost ); // <---= NEW! VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, &ghosts[0], &x); I pass the VecCreateGhost phase. "Indices" is an STL container of integers. It seems it works better than classical C array for this case. However, I still do not see the ghost values, i.e. I get the following error: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: Can only get local values, trying 3529! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Argument out of range! [1]PETSC ERROR: Can only get local values, trying 22! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message ------------------------------------ [2]PETSC ERROR: Argument out of range! [2]PETSC ERROR: Can only get local values, trying 86! [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 when I am trying to access values in ghost cells. What do I have to use to see them ghosts? I reckon VecGhostGetLocalForm should be used, right? Kind regards, Bojan On 2/23/2012 8:36 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 1:33 PM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
Dear Matt,
I sent the code as an attached tarball. I sent it with case I run, so is 2 MB big. It is now in the cue for moderator's approval.
No, you HAVE to send it to [email protected] <mailto:[email protected]>, as I said last time, for exactly this reason.
Matt
Thanks.
Kind regards,
Bojan
On 2/23/2012 8:04 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 12:51 PM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
Dear Matt,
are you sure? It is almost 4000 lines long! Shall I send only the function which bother me?
If the entire code is what you need, shall I make a tarball and attach it?
Send something the builds and runs. Don't care how long it is.
Matt
Kind regards,
Bojan
On 2/23/2012 7:44 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 12:28 PM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
On 2/23/2012 7:24 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 12:05 PM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
Dear Matthew,
thank you for your response. When I use VecCreateGhost, I get the following:
It appears that you passed a bad communicator. Did you not initialize a 'comm' variable?
I pass PETSC_COMM_WORLD to VecCreateGhost.
I don't know what you mean by 'comm' variable :-( I called all the routines to initialize PETSc.
Send your code to [email protected] <mailto:[email protected]>.
Matt
Cheers,
Bojan
Matt
[0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] PetscCommDuplicate line 140 src/sys/objects/tagm.c [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 30 src/sys/objects/inherit.c [0]PETSC ERROR: [0] VecCreate line 32 src/vec/vec/interface/veccreate.c [0]PETSC ERROR: [0] VecCreateGhostWithArray line 567 src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: [0] VecCreateGhost line 647 src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./PSI-Flow on a arch-linu named lccfd06 by niceno Thu Feb 23 19:02:45 2012 [0]PETSC ERROR: Libraries linked from /homecfd/niceno/PETSc-3.2-p6/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Fri Feb 10 10:24:13 2012 [0]PETSC ERROR: Configure options [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
I don't understand what could be causing it. I took very good care to match the global numbers of ghost cells when calling VecCreateGhost
Kind regards,
Bojan
On 2/23/2012 5:53 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 10:46 AM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
Hi all,
I've never used a mailing list before, so I hope this message will reach PETSc users and experts and someone might be willing to help me. I am also novice in PETSc.
I have developed an unstructured finite volume solver on top of PETSc libraries. In sequential, it works like a charm. For the parallel version, I do domain decomposition externally with Metis, and work out local and global numberings, as well as communication patterns between processor. (The latter don't seem to be needed for PETSc, though.) When I run my program in parallel, it also works, but I miss values in vectors' ghost points.
I create vectors with command: VecCreate(PETSC_COMM_WORLD, &x);
Is it possible to get the ghost values if a vector is created like this?
I do not understand this question. By definition, "ghost values" are those not stored in the global vector.
I have tried to use VecCreateGhost, but for some reason which is beyond my comprehension, PETSc goes berserk when it reaches the command: VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, ifrom, &x)
I think you can understand that "berserk" tells me absolutely nothing. Error message? Stack trace? Did you try to run an example which uses VecGhost?
Thanks,
Matt
Can anyone help me? Either how to reach ghost values for vector created by VecCreate, or how to use VecCreateGhost properly?
Kind regards,
Bojan
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
Are you using local numbering when accessing the local part of ghost nodes? On Thu, Feb 23, 2012 at 12:43 PM, Bojan Niceno <[email protected]> wrote:
Dear Matt,
I have a new insight, although is not the full resolution. If I change my code in PETScSolver.cpp from:
/*-------------------------------------------------+ | Make necessary PETSc intializations for vetors | +-------------------------------------------------*/ Int nghost = N - n; Int * ghosts = new Int(nghost); for(Int n=0; n<M.mesh.nodes.size(); n++) { assert( M.mesh.nodes[n].global_number >= 0); assert( M.mesh.nodes[n].global_number < 14065); } for(Int i=n; i<N; i++) { assert( M.mesh.nodes[i].global_number >= 0); assert( M.mesh.nodes[i].global_number < 14065); assert( ! (M.mesh.nodes[i].global_number >= n_start && M.mesh.nodes[i].global_number < n_end) ); ghosts[i] = M.mesh.nodes[i].global_number; }
VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, &ghosts[0], &x);
to:
/*-------------------------------------------------+ | Make necessary PETSc intializations for vetors | +-------------------------------------------------*/ Int nghost = N - n; Indices ghosts; // <---= NEW! for(Int n=0; n<M.mesh.nodes.size(); n++) { assert( M.mesh.nodes[n].global_number >= 0); assert( M.mesh.nodes[n].global_number < 14065); } for(Int i=n; i<N; i++) { assert( M.mesh.nodes[i].global_number >= 0); assert( M.mesh.nodes[i].global_number < 14065); assert( ! (M.mesh.nodes[i].global_number >= n_start && M.mesh.nodes[i].global_number < n_end) ); ghosts.push_back( M.mesh.nodes[i].global_number ); // <---= NEW!
} assert( ghosts.size() == nghost ); // <---= NEW!
VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, &ghosts[0], &x);
I pass the VecCreateGhost phase. "Indices" is an STL container of integers. It seems it works better than classical C array for this case.
However, I still do not see the ghost values, i.e. I get the following error:
[1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: Can only get local values, trying 3529! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Argument out of range! [1]PETSC ERROR: Can only get local values, trying 22! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message ------------------------------------ [2]PETSC ERROR: Argument out of range! [2]PETSC ERROR: Can only get local values, trying 86! [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012
when I am trying to access values in ghost cells. What do I have to use to see them ghosts? I reckon VecGhostGetLocalForm should be used, right?
Kind regards,
Bojan
On 2/23/2012 8:36 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 1:33 PM, Bojan Niceno <[email protected]> wrote:
Dear Matt,
I sent the code as an attached tarball. I sent it with case I run, so is 2 MB big. It is now in the cue for moderator's approval.
No, you HAVE to send it to [email protected], as I said last time, for exactly this reason.
Matt
Thanks.
Kind regards,
Bojan
On 2/23/2012 8:04 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 12:51 PM, Bojan Niceno <[email protected]>wrote:
Dear Matt,
are you sure? It is almost 4000 lines long! Shall I send only the function which bother me?
If the entire code is what you need, shall I make a tarball and attach it?
Send something the builds and runs. Don't care how long it is.
Matt
Kind regards,
Bojan
On 2/23/2012 7:44 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 12:28 PM, Bojan Niceno <[email protected]>wrote:
On 2/23/2012 7:24 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 12:05 PM, Bojan Niceno <[email protected]>wrote:
Dear Matthew,
thank you for your response. When I use VecCreateGhost, I get the following:
It appears that you passed a bad communicator. Did you not initialize a 'comm' variable?
I pass PETSC_COMM_WORLD to VecCreateGhost.
I don't know what you mean by 'comm' variable :-( I called all the routines to initialize PETSc.
Send your code to [email protected].
Matt
Cheers,
Bojan
Matt
[0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] PetscCommDuplicate line 140 src/sys/objects/tagm.c [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 30 src/sys/objects/inherit.c [0]PETSC ERROR: [0] VecCreate line 32 src/vec/vec/interface/veccreate.c [0]PETSC ERROR: [0] VecCreateGhostWithArray line 567 src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: [0] VecCreateGhost line 647 src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./PSI-Flow on a arch-linu named lccfd06 by niceno Thu Feb 23 19:02:45 2012 [0]PETSC ERROR: Libraries linked from /homecfd/niceno/PETSc-3.2-p6/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Fri Feb 10 10:24:13 2012 [0]PETSC ERROR: Configure options [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
I don't understand what could be causing it. I took very good care to match the global numbers of ghost cells when calling VecCreateGhost
Kind regards,
Bojan
On 2/23/2012 5:53 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 10:46 AM, Bojan Niceno <[email protected]>wrote:
Hi all,
I've never used a mailing list before, so I hope this message will reach PETSc users and experts and someone might be willing to help me. I am also novice in PETSc.
I have developed an unstructured finite volume solver on top of PETSc libraries. In sequential, it works like a charm. For the parallel version, I do domain decomposition externally with Metis, and work out local and global numberings, as well as communication patterns between processor. (The latter don't seem to be needed for PETSc, though.) When I run my program in parallel, it also works, but I miss values in vectors' ghost points.
I create vectors with command: VecCreate(PETSC_COMM_WORLD, &x);
Is it possible to get the ghost values if a vector is created like this?
I do not understand this question. By definition, "ghost values" are those not stored in the global vector.
I have tried to use VecCreateGhost, but for some reason which is beyond my comprehension, PETSc goes berserk when it reaches the command: VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, ifrom, &x)
I think you can understand that "berserk" tells me absolutely nothing. Error message? Stack trace? Did you try to run an example which uses VecGhost?
Thanks,
Matt
Can anyone help me? Either how to reach ghost values for vector created by VecCreate, or how to use VecCreateGhost properly?
Kind regards,
Bojan
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
No, I use global. for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); } "n" is defined as the number of cells inside, i.e. without buffers. "unk" is my external array. If I try to access buffer values, I use: for(Int i=0; i<N; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); } But then I end up with tons of warnings, presumably because I am going beyond "n". Vector x was created with VecCreateGhost. On 2/23/2012 10:00 PM, Mohammad Mirzadeh wrote:
Are you using local numbering when accessing the local part of ghost nodes?
On Thu, Feb 23, 2012 at 12:43 PM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
Dear Matt,
I have a new insight, although is not the full resolution. If I change my code in PETScSolver.cpp from:
/*-------------------------------------------------+ | Make necessary PETSc intializations for vetors | +-------------------------------------------------*/ Int nghost = N - n; Int * ghosts = new Int(nghost); for(Int n=0; n<M.mesh.nodes.size(); n++) { assert( M.mesh.nodes[n].global_number >= 0); assert( M.mesh.nodes[n].global_number < 14065); } for(Int i=n; i<N; i++) { assert( M.mesh.nodes[i].global_number >= 0); assert( M.mesh.nodes[i].global_number < 14065); assert( ! (M.mesh.nodes[i].global_number >= n_start && M.mesh.nodes[i].global_number < n_end) ); ghosts[i] = M.mesh.nodes[i].global_number; }
VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, &ghosts[0], &x);
to:
/*-------------------------------------------------+ | Make necessary PETSc intializations for vetors | +-------------------------------------------------*/ Int nghost = N - n; Indices ghosts; // <---= NEW! for(Int n=0; n<M.mesh.nodes.size(); n++) { assert( M.mesh.nodes[n].global_number >= 0); assert( M.mesh.nodes[n].global_number < 14065); } for(Int i=n; i<N; i++) { assert( M.mesh.nodes[i].global_number >= 0); assert( M.mesh.nodes[i].global_number < 14065); assert( ! (M.mesh.nodes[i].global_number >= n_start && M.mesh.nodes[i].global_number < n_end) ); ghosts.push_back( M.mesh.nodes[i].global_number ); // <---= NEW! } assert( ghosts.size() == nghost ); // <---= NEW!
VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, &ghosts[0], &x);
I pass the VecCreateGhost phase. "Indices" is an STL container of integers. It seems it works better than classical C array for this case.
However, I still do not see the ghost values, i.e. I get the following error:
[1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: Can only get local values, trying 3529! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Argument out of range! [1]PETSC ERROR: Can only get local values, trying 22! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message ------------------------------------ [2]PETSC ERROR: Argument out of range! [2]PETSC ERROR: Can only get local values, trying 86! [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012
when I am trying to access values in ghost cells. What do I have to use to see them ghosts? I reckon VecGhostGetLocalForm should be used, right?
Kind regards,
Bojan
On 2/23/2012 8:36 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 1:33 PM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
Dear Matt,
I sent the code as an attached tarball. I sent it with case I run, so is 2 MB big. It is now in the cue for moderator's approval.
No, you HAVE to send it to [email protected] <mailto:[email protected]>, as I said last time, for exactly this reason.
Matt
Thanks.
Kind regards,
Bojan
On 2/23/2012 8:04 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 12:51 PM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
Dear Matt,
are you sure? It is almost 4000 lines long! Shall I send only the function which bother me?
If the entire code is what you need, shall I make a tarball and attach it?
Send something the builds and runs. Don't care how long it is.
Matt
Kind regards,
Bojan
On 2/23/2012 7:44 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 12:28 PM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
On 2/23/2012 7:24 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 12:05 PM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
Dear Matthew,
thank you for your response. When I use VecCreateGhost, I get the following:
It appears that you passed a bad communicator. Did you not initialize a 'comm' variable?
I pass PETSC_COMM_WORLD to VecCreateGhost.
I don't know what you mean by 'comm' variable :-( I called all the routines to initialize PETSc.
Send your code to [email protected] <mailto:[email protected]>.
Matt
Cheers,
Bojan
Matt
[0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] PetscCommDuplicate line 140 src/sys/objects/tagm.c [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 30 src/sys/objects/inherit.c [0]PETSC ERROR: [0] VecCreate line 32 src/vec/vec/interface/veccreate.c [0]PETSC ERROR: [0] VecCreateGhostWithArray line 567 src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: [0] VecCreateGhost line 647 src/vec/vec/impls/mpi/pbvec.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./PSI-Flow on a arch-linu named lccfd06 by niceno Thu Feb 23 19:02:45 2012 [0]PETSC ERROR: Libraries linked from /homecfd/niceno/PETSc-3.2-p6/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Fri Feb 10 10:24:13 2012 [0]PETSC ERROR: Configure options [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
I don't understand what could be causing it. I took very good care to match the global numbers of ghost cells when calling VecCreateGhost
Kind regards,
Bojan
On 2/23/2012 5:53 PM, Matthew Knepley wrote:
On Thu, Feb 23, 2012 at 10:46 AM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
Hi all,
I've never used a mailing list before, so I hope this message will reach PETSc users and experts and someone might be willing to help me. I am also novice in PETSc.
I have developed an unstructured finite volume solver on top of PETSc libraries. In sequential, it works like a charm. For the parallel version, I do domain decomposition externally with Metis, and work out local and global numberings, as well as communication patterns between processor. (The latter don't seem to be needed for PETSc, though.) When I run my program in parallel, it also works, but I miss values in vectors' ghost points.
I create vectors with command: VecCreate(PETSC_COMM_WORLD, &x);
Is it possible to get the ghost values if a vector is created like this?
I do not understand this question. By definition, "ghost values" are those not stored in the global vector.
I have tried to use VecCreateGhost, but for some reason which is beyond my comprehension, PETSc goes berserk when it reaches the command: VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, ifrom, &x)
I think you can understand that "berserk" tells me absolutely nothing. Error message? Stack trace? Did you try to run an example which uses VecGhost?
Thanks,
Matt
Can anyone help me? Either how to reach ghost values for vector created by VecCreate, or how to use VecCreateGhost properly?
Kind regards,
Bojan
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
--
--
On Thu, Feb 23, 2012 at 15:05, Bojan Niceno <[email protected]> wrote:
No, I use global.
The local form is just a local vector. It doesn't even know that a global problem exists. You can't index into it using global indices. (In general, there is no efficient way to look up information in the local vector (includes ghost points) using global indices.)
for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
"n" is defined as the number of cells inside, i.e. without buffers. "unk" is my external array. If I try to access buffer values, I use:
for(Int i=0; i<N; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
But then I end up with tons of warnings, presumably because I am going beyond "n". Vector x was created with VecCreateGhost.
Dear Jed, thanks. Now I have the following: - Array unk, which should hold values inside the partition and in ghost cells. It is big enough to hold both - Vec x, created by command VecCreateGhost, with proper padding for ghost cells - Successful call to linear solver in parallel. But I need to copy ghost values from x to my array unk. How can I do it? Kind regards, Bojan On 2/23/2012 10:10 PM, Jed Brown wrote:
On Thu, Feb 23, 2012 at 15:05, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
No, I use global.
The local form is just a local vector. It doesn't even know that a global problem exists. You can't index into it using global indices. (In general, there is no efficient way to look up information in the local vector (includes ghost points) using global indices.)
for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
"n" is defined as the number of cells inside, i.e. without buffers. "unk" is my external array. If I try to access buffer values, I use:
for(Int i=0; i<N; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
But then I end up with tons of warnings, presumably because I am going beyond "n". Vector x was created with VecCreateGhost.
--
just index x with the local numberings. if you have 'n' local nodes and 'g' ghost nodes in x, ghost nodes indecies run from 'n' to 'n+g-1' On Feb 23, 2012 1:16 PM, "Bojan Niceno" <[email protected]> wrote:
Dear Jed,
thanks.
Now I have the following:
- Array unk, which should hold values inside the partition and in ghost cells. It is big enough to hold both - Vec x, created by command VecCreateGhost, with proper padding for ghost cells - Successful call to linear solver in parallel.
But I need to copy ghost values from x to my array unk.
How can I do it?
Kind regards,
Bojan
On 2/23/2012 10:10 PM, Jed Brown wrote:
On Thu, Feb 23, 2012 at 15:05, Bojan Niceno <[email protected]> wrote:
No, I use global.
The local form is just a local vector. It doesn't even know that a global problem exists. You can't index into it using global indices. (In general, there is no efficient way to look up information in the local vector (includes ghost points) using global indices.)
for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
"n" is defined as the number of cells inside, i.e. without buffers. "unk" is my external array. If I try to access buffer values, I use:
for(Int i=0; i<N; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
But then I end up with tons of warnings, presumably because I am going beyond "n". Vector x was created with VecCreateGhost.
--
Dear Mohammad, it doesn't help me, or I did not understand your explanation. If I do this: /* copy internal values (THIS WORKS, BUT COPIES NO BUFFER VALUES) */ for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); } /* copy ghost values (CREATES MANY WARNINGS */ for(Int i=n; i<N; i++) { VecGetValues(x, 1, &i, &unk[i]); } I get arnings are like this. [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: Can only get local values, trying 3518! [3]PETSC ERROR: --------------------- Error Message ------------------------------------ [3]PETSC ERROR: Argument out of range! [3]PETSC ERROR: Can only get local values, trying 3511! [3]PETSC ERROR: ------------------------------------------------------------------------ What am I doing wrong here? Cheers, Bojan On 2/23/2012 11:23 PM, Mohammad Mirzadeh wrote:
just index x with the local numberings. if you have 'n' local nodes and 'g' ghost nodes in x, ghost nodes indecies run from 'n' to 'n+g-1'
On Feb 23, 2012 1:16 PM, "Bojan Niceno" <[email protected] <mailto:[email protected]>> wrote:
Dear Jed,
thanks.
Now I have the following:
- Array unk, which should hold values inside the partition and in ghost cells. It is big enough to hold both - Vec x, created by command VecCreateGhost, with proper padding for ghost cells - Successful call to linear solver in parallel.
But I need to copy ghost values from x to my array unk.
How can I do it?
Kind regards,
Bojan
On 2/23/2012 10:10 PM, Jed Brown wrote:
On Thu, Feb 23, 2012 at 15:05, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
No, I use global.
The local form is just a local vector. It doesn't even know that a global problem exists. You can't index into it using global indices. (In general, there is no efficient way to look up information in the local vector (includes ghost points) using global indices.)
for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
"n" is defined as the number of cells inside, i.e. without buffers. "unk" is my external array. If I try to access buffer values, I use:
for(Int i=0; i<N; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
But then I end up with tons of warnings, presumably because I am going beyond "n". Vector x was created with VecCreateGhost.
--
--
based on, VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, &ghosts[0], &x); it seems to me that x is your actual ghosted vector. If this is true, then you need to get its "local" form via VecGhostGetLocalForm<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGhostGetLocalForm.html#VecGhostGetLocalForm>(). Once you have done, you should be able to access the ghosted nodes. Are you calling this function anywhere? On Thu, Feb 23, 2012 at 2:32 PM, Bojan Niceno <[email protected]> wrote:
Dear Mohammad,
it doesn't help me, or I did not understand your explanation.
If I do this:
/* copy internal values (THIS WORKS, BUT COPIES NO BUFFER VALUES) */
for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
/* copy ghost values (CREATES MANY WARNINGS */
for(Int i=n; i<N; i++) { VecGetValues(x, 1, &i, &unk[i]); }
I get arnings are like this.
[0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: Can only get local values, trying 3518! [3]PETSC ERROR: --------------------- Error Message ------------------------------------ [3]PETSC ERROR: Argument out of range! [3]PETSC ERROR: Can only get local values, trying 3511! [3]PETSC ERROR: ------------------------------------------------------------------------
What am I doing wrong here?
Cheers,
Bojan
On 2/23/2012 11:23 PM, Mohammad Mirzadeh wrote:
just index x with the local numberings. if you have 'n' local nodes and 'g' ghost nodes in x, ghost nodes indecies run from 'n' to 'n+g-1' On Feb 23, 2012 1:16 PM, "Bojan Niceno" <[email protected]> wrote:
Dear Jed,
thanks.
Now I have the following:
- Array unk, which should hold values inside the partition and in ghost cells. It is big enough to hold both - Vec x, created by command VecCreateGhost, with proper padding for ghost cells - Successful call to linear solver in parallel.
But I need to copy ghost values from x to my array unk.
How can I do it?
Kind regards,
Bojan
On 2/23/2012 10:10 PM, Jed Brown wrote:
On Thu, Feb 23, 2012 at 15:05, Bojan Niceno <[email protected]> wrote:
No, I use global.
The local form is just a local vector. It doesn't even know that a global problem exists. You can't index into it using global indices. (In general, there is no efficient way to look up information in the local vector (includes ghost points) using global indices.)
for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
"n" is defined as the number of cells inside, i.e. without buffers. "unk" is my external array. If I try to access buffer values, I use:
for(Int i=0; i<N; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
But then I end up with tons of warnings, presumably because I am going beyond "n". Vector x was created with VecCreateGhost.
--
--
On 2/23/2012 11:49 PM, Mohammad Mirzadeh wrote:
based on,
VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, &ghosts[0], &x);
it seems to me that x is your actual ghosted vector. If this is true, then you need to get its "local" form via VecGhostGetLocalForm <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGhostGetLocalForm.html#VecGhostGetLocalForm>(). Once you have done, you should be able to access the ghosted nodes. Are you calling this function anywhere?
I tried that before. I did: Vec lx; VecGhostGetLocalForm(x, &lx) then I copied "lx" to my variable, like for(Int i=1; i<N; i++) /* N includes buffer cells */ unk[i] = lx[i] but ghost values were also zero. I am thinking that PETSc somehow clears the ghost values after a call to KSP. Is it the case? Kind regards, Bojan
On Thu, Feb 23, 2012 at 2:32 PM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
Dear Mohammad,
it doesn't help me, or I did not understand your explanation.
If I do this:
/* copy internal values (THIS WORKS, BUT COPIES NO BUFFER VALUES) */
for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
/* copy ghost values (CREATES MANY WARNINGS */
for(Int i=n; i<N; i++) { VecGetValues(x, 1, &i, &unk[i]); }
I get arnings are like this.
[0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: Can only get local values, trying 3518! [3]PETSC ERROR: --------------------- Error Message ------------------------------------ [3]PETSC ERROR: Argument out of range! [3]PETSC ERROR: Can only get local values, trying 3511! [3]PETSC ERROR: ------------------------------------------------------------------------
What am I doing wrong here?
Cheers,
Bojan
On 2/23/2012 11:23 PM, Mohammad Mirzadeh wrote:
just index x with the local numberings. if you have 'n' local nodes and 'g' ghost nodes in x, ghost nodes indecies run from 'n' to 'n+g-1'
On Feb 23, 2012 1:16 PM, "Bojan Niceno" <[email protected] <mailto:[email protected]>> wrote:
Dear Jed,
thanks.
Now I have the following:
- Array unk, which should hold values inside the partition and in ghost cells. It is big enough to hold both - Vec x, created by command VecCreateGhost, with proper padding for ghost cells - Successful call to linear solver in parallel.
But I need to copy ghost values from x to my array unk.
How can I do it?
Kind regards,
Bojan
On 2/23/2012 10:10 PM, Jed Brown wrote:
On Thu, Feb 23, 2012 at 15:05, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
No, I use global.
The local form is just a local vector. It doesn't even know that a global problem exists. You can't index into it using global indices. (In general, there is no efficient way to look up information in the local vector (includes ghost points) using global indices.)
for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
"n" is defined as the number of cells inside, i.e. without buffers. "unk" is my external array. If I try to access buffer values, I use:
for(Int i=0; i<N; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
But then I end up with tons of warnings, presumably because I am going beyond "n". Vector x was created with VecCreateGhost.
--
--
--
You also need calls to VecGhostUpdateBegin()/VecGhostUpdateEnd() functions to update the ghost values if you change them in the global representation. See Petsc Manual 3.2 pp 55-56 Mohammad On Thu, Feb 23, 2012 at 3:18 PM, Bojan Niceno <[email protected]> wrote:
On 2/23/2012 11:49 PM, Mohammad Mirzadeh wrote:
based on,
VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, &ghosts[0], &x);
it seems to me that x is your actual ghosted vector. If this is true, then you need to get its "local" form via VecGhostGetLocalForm<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGhostGetLocalForm.html#VecGhostGetLocalForm>(). Once you have done, you should be able to access the ghosted nodes. Are you calling this function anywhere?
I tried that before. I did:
Vec lx; VecGhostGetLocalForm(x, &lx)
then I copied "lx" to my variable, like
for(Int i=1; i<N; i++) /* N includes buffer cells */ unk[i] = lx[i]
but ghost values were also zero. I am thinking that PETSc somehow clears the ghost values after a call to KSP. Is it the case?
Kind regards,
Bojan
On Thu, Feb 23, 2012 at 2:32 PM, Bojan Niceno <[email protected]> wrote:
Dear Mohammad,
it doesn't help me, or I did not understand your explanation.
If I do this:
/* copy internal values (THIS WORKS, BUT COPIES NO BUFFER VALUES) */
for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
/* copy ghost values (CREATES MANY WARNINGS */
for(Int i=n; i<N; i++) { VecGetValues(x, 1, &i, &unk[i]); }
I get arnings are like this.
[0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: Can only get local values, trying 3518! [3]PETSC ERROR: --------------------- Error Message ------------------------------------ [3]PETSC ERROR: Argument out of range! [3]PETSC ERROR: Can only get local values, trying 3511! [3]PETSC ERROR: ------------------------------------------------------------------------
What am I doing wrong here?
Cheers,
Bojan
On 2/23/2012 11:23 PM, Mohammad Mirzadeh wrote:
just index x with the local numberings. if you have 'n' local nodes and 'g' ghost nodes in x, ghost nodes indecies run from 'n' to 'n+g-1' On Feb 23, 2012 1:16 PM, "Bojan Niceno" <[email protected]> wrote:
Dear Jed,
thanks.
Now I have the following:
- Array unk, which should hold values inside the partition and in ghost cells. It is big enough to hold both - Vec x, created by command VecCreateGhost, with proper padding for ghost cells - Successful call to linear solver in parallel.
But I need to copy ghost values from x to my array unk.
How can I do it?
Kind regards,
Bojan
On 2/23/2012 10:10 PM, Jed Brown wrote:
On Thu, Feb 23, 2012 at 15:05, Bojan Niceno <[email protected]> wrote:
No, I use global.
The local form is just a local vector. It doesn't even know that a global problem exists. You can't index into it using global indices. (In general, there is no efficient way to look up information in the local vector (includes ghost points) using global indices.)
for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
"n" is defined as the number of cells inside, i.e. without buffers. "unk" is my external array. If I try to access buffer values, I use:
for(Int i=0; i<N; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
But then I end up with tons of warnings, presumably because I am going beyond "n". Vector x was created with VecCreateGhost.
--
--
--
A-ha! Will do, thanks :-) On 2/24/2012 12:27 AM, Mohammad Mirzadeh wrote:
You also need calls to VecGhostUpdateBegin()/VecGhostUpdateEnd() functions to update the ghost values if you change them in the global representation. See Petsc Manual 3.2 pp 55-56
Mohammad
On Thu, Feb 23, 2012 at 3:18 PM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
On 2/23/2012 11:49 PM, Mohammad Mirzadeh wrote:
based on,
VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, &ghosts[0], &x);
it seems to me that x is your actual ghosted vector. If this is true, then you need to get its "local" form via VecGhostGetLocalForm <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGhostGetLocalForm.html#VecGhostGetLocalForm>(). Once you have done, you should be able to access the ghosted nodes. Are you calling this function anywhere?
I tried that before. I did:
Vec lx; VecGhostGetLocalForm(x, &lx)
then I copied "lx" to my variable, like
for(Int i=1; i<N; i++) /* N includes buffer cells */ unk[i] = lx[i]
but ghost values were also zero. I am thinking that PETSc somehow clears the ghost values after a call to KSP. Is it the case?
Kind regards,
Bojan
On Thu, Feb 23, 2012 at 2:32 PM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
Dear Mohammad,
it doesn't help me, or I did not understand your explanation.
If I do this:
/* copy internal values (THIS WORKS, BUT COPIES NO BUFFER VALUES) */
for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
/* copy ghost values (CREATES MANY WARNINGS */
for(Int i=n; i<N; i++) { VecGetValues(x, 1, &i, &unk[i]); }
I get arnings are like this.
[0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: Can only get local values, trying 3518! [3]PETSC ERROR: --------------------- Error Message ------------------------------------ [3]PETSC ERROR: Argument out of range! [3]PETSC ERROR: Can only get local values, trying 3511! [3]PETSC ERROR: ------------------------------------------------------------------------
What am I doing wrong here?
Cheers,
Bojan
On 2/23/2012 11:23 PM, Mohammad Mirzadeh wrote:
just index x with the local numberings. if you have 'n' local nodes and 'g' ghost nodes in x, ghost nodes indecies run from 'n' to 'n+g-1'
On Feb 23, 2012 1:16 PM, "Bojan Niceno" <[email protected] <mailto:[email protected]>> wrote:
Dear Jed,
thanks.
Now I have the following:
- Array unk, which should hold values inside the partition and in ghost cells. It is big enough to hold both - Vec x, created by command VecCreateGhost, with proper padding for ghost cells - Successful call to linear solver in parallel.
But I need to copy ghost values from x to my array unk.
How can I do it?
Kind regards,
Bojan
On 2/23/2012 10:10 PM, Jed Brown wrote:
On Thu, Feb 23, 2012 at 15:05, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
No, I use global.
The local form is just a local vector. It doesn't even know that a global problem exists. You can't index into it using global indices. (In general, there is no efficient way to look up information in the local vector (includes ghost points) using global indices.)
for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
"n" is defined as the number of cells inside, i.e. without buffers. "unk" is my external array. If I try to access buffer values, I use:
for(Int i=0; i<N; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
But then I end up with tons of warnings, presumably because I am going beyond "n". Vector x was created with VecCreateGhost.
--
--
--
--
Yeeee-ha! VecGhostUpdateBegin() / VecGhostUpdateEnd() was indeed what I was missing. Thank you Mohammad, thank you all who helped me today! Cheers Bojan On 2/24/2012 12:27 AM, Mohammad Mirzadeh wrote:
You also need calls to VecGhostUpdateBegin()/VecGhostUpdateEnd() functions to update the ghost values if you change them in the global representation. See Petsc Manual 3.2 pp 55-56
Mohammad
On Thu, Feb 23, 2012 at 3:18 PM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
On 2/23/2012 11:49 PM, Mohammad Mirzadeh wrote:
based on,
VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, &ghosts[0], &x);
it seems to me that x is your actual ghosted vector. If this is true, then you need to get its "local" form via VecGhostGetLocalForm <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGhostGetLocalForm.html#VecGhostGetLocalForm>(). Once you have done, you should be able to access the ghosted nodes. Are you calling this function anywhere?
I tried that before. I did:
Vec lx; VecGhostGetLocalForm(x, &lx)
then I copied "lx" to my variable, like
for(Int i=1; i<N; i++) /* N includes buffer cells */ unk[i] = lx[i]
but ghost values were also zero. I am thinking that PETSc somehow clears the ghost values after a call to KSP. Is it the case?
Kind regards,
Bojan
On Thu, Feb 23, 2012 at 2:32 PM, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
Dear Mohammad,
it doesn't help me, or I did not understand your explanation.
If I do this:
/* copy internal values (THIS WORKS, BUT COPIES NO BUFFER VALUES) */
for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
/* copy ghost values (CREATES MANY WARNINGS */
for(Int i=n; i<N; i++) { VecGetValues(x, 1, &i, &unk[i]); }
I get arnings are like this.
[0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: Can only get local values, trying 3518! [3]PETSC ERROR: --------------------- Error Message ------------------------------------ [3]PETSC ERROR: Argument out of range! [3]PETSC ERROR: Can only get local values, trying 3511! [3]PETSC ERROR: ------------------------------------------------------------------------
What am I doing wrong here?
Cheers,
Bojan
On 2/23/2012 11:23 PM, Mohammad Mirzadeh wrote:
just index x with the local numberings. if you have 'n' local nodes and 'g' ghost nodes in x, ghost nodes indecies run from 'n' to 'n+g-1'
On Feb 23, 2012 1:16 PM, "Bojan Niceno" <[email protected] <mailto:[email protected]>> wrote:
Dear Jed,
thanks.
Now I have the following:
- Array unk, which should hold values inside the partition and in ghost cells. It is big enough to hold both - Vec x, created by command VecCreateGhost, with proper padding for ghost cells - Successful call to linear solver in parallel.
But I need to copy ghost values from x to my array unk.
How can I do it?
Kind regards,
Bojan
On 2/23/2012 10:10 PM, Jed Brown wrote:
On Thu, Feb 23, 2012 at 15:05, Bojan Niceno <[email protected] <mailto:[email protected]>> wrote:
No, I use global.
The local form is just a local vector. It doesn't even know that a global problem exists. You can't index into it using global indices. (In general, there is no efficient way to look up information in the local vector (includes ghost points) using global indices.)
for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
"n" is defined as the number of cells inside, i.e. without buffers. "unk" is my external array. If I try to access buffer values, I use:
for(Int i=0; i<N; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
But then I end up with tons of warnings, presumably because I am going beyond "n". Vector x was created with VecCreateGhost.
--
--
--
--
glad to hear that :) On Thu, Feb 23, 2012 at 3:38 PM, Bojan Niceno <[email protected]> wrote:
Yeeee-ha!
VecGhostUpdateBegin() / VecGhostUpdateEnd() was indeed what I was missing.
Thank you Mohammad, thank you all who helped me today!
Cheers
Bojan
On 2/24/2012 12:27 AM, Mohammad Mirzadeh wrote:
You also need calls to VecGhostUpdateBegin()/VecGhostUpdateEnd() functions to update the ghost values if you change them in the global representation. See Petsc Manual 3.2 pp 55-56
Mohammad
On Thu, Feb 23, 2012 at 3:18 PM, Bojan Niceno <[email protected]> wrote:
On 2/23/2012 11:49 PM, Mohammad Mirzadeh wrote:
based on,
VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, &ghosts[0], &x);
it seems to me that x is your actual ghosted vector. If this is true, then you need to get its "local" form via VecGhostGetLocalForm<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGhostGetLocalForm.html#VecGhostGetLocalForm>(). Once you have done, you should be able to access the ghosted nodes. Are you calling this function anywhere?
I tried that before. I did:
Vec lx; VecGhostGetLocalForm(x, &lx)
then I copied "lx" to my variable, like
for(Int i=1; i<N; i++) /* N includes buffer cells */ unk[i] = lx[i]
but ghost values were also zero. I am thinking that PETSc somehow clears the ghost values after a call to KSP. Is it the case?
Kind regards,
Bojan
On Thu, Feb 23, 2012 at 2:32 PM, Bojan Niceno <[email protected]>wrote:
Dear Mohammad,
it doesn't help me, or I did not understand your explanation.
If I do this:
/* copy internal values (THIS WORKS, BUT COPIES NO BUFFER VALUES) */
for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
/* copy ghost values (CREATES MANY WARNINGS */
for(Int i=n; i<N; i++) { VecGetValues(x, 1, &i, &unk[i]); }
I get arnings are like this.
[0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: Can only get local values, trying 3518! [3]PETSC ERROR: --------------------- Error Message ------------------------------------ [3]PETSC ERROR: Argument out of range! [3]PETSC ERROR: Can only get local values, trying 3511! [3]PETSC ERROR: ------------------------------------------------------------------------
What am I doing wrong here?
Cheers,
Bojan
On 2/23/2012 11:23 PM, Mohammad Mirzadeh wrote:
just index x with the local numberings. if you have 'n' local nodes and 'g' ghost nodes in x, ghost nodes indecies run from 'n' to 'n+g-1' On Feb 23, 2012 1:16 PM, "Bojan Niceno" <[email protected]> wrote:
Dear Jed,
thanks.
Now I have the following:
- Array unk, which should hold values inside the partition and in ghost cells. It is big enough to hold both - Vec x, created by command VecCreateGhost, with proper padding for ghost cells - Successful call to linear solver in parallel.
But I need to copy ghost values from x to my array unk.
How can I do it?
Kind regards,
Bojan
On 2/23/2012 10:10 PM, Jed Brown wrote:
On Thu, Feb 23, 2012 at 15:05, Bojan Niceno <[email protected]>wrote:
No, I use global.
The local form is just a local vector. It doesn't even know that a global problem exists. You can't index into it using global indices. (In general, there is no efficient way to look up information in the local vector (includes ghost points) using global indices.)
for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
"n" is defined as the number of cells inside, i.e. without buffers. "unk" is my external array. If I try to access buffer values, I use:
for(Int i=0; i<N; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
But then I end up with tons of warnings, presumably because I am going beyond "n". Vector x was created with VecCreateGhost.
--
--
--
--
I have added the following to the VecGhostGetLocalForm() manual page to make this clearer for everyone: To update the ghost values from the locations on the other processes one must call VecGhostUpdateBegin() and VecGhostUpdateEnd() before accessing the ghost values. Thus normal usage is $ VecGhostUpdateBegin(x,INSERT_VALUES,SCATTER_FORWARD); $ VecGhostUpdateEnd(x,INSERT_VALUES,SCATTER_FORWARD); $ VecGhostGetLocalForm(x,&xlocal); $ VecGetArray(xlocal,&xvalues); $ /* access the non-ghost values in locations xvalues[0:n-1] and ghost values in locations xvalues[n:n+nghost]; */ $ VecRestoreArray(xlocal,&xvalues); $ VecGhostRestoreLocalForm(x,&xlocal); On Feb 23, 2012, at 5:38 PM, Bojan Niceno wrote:
Yeeee-ha!
VecGhostUpdateBegin() / VecGhostUpdateEnd() was indeed what I was missing.
Thank you Mohammad, thank you all who helped me today!
Cheers
Bojan
On 2/24/2012 12:27 AM, Mohammad Mirzadeh wrote:
You also need calls to VecGhostUpdateBegin()/VecGhostUpdateEnd() functions to update the ghost values if you change them in the global representation. See Petsc Manual 3.2 pp 55-56
Mohammad
On Thu, Feb 23, 2012 at 3:18 PM, Bojan Niceno <[email protected]> wrote: On 2/23/2012 11:49 PM, Mohammad Mirzadeh wrote:
based on,
VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, &ghosts[0], &x);
it seems to me that x is your actual ghosted vector. If this is true, then you need to get its "local" form via VecGhostGetLocalForm(). Once you have done, you should be able to access the ghosted nodes. Are you calling this function anywhere?
I tried that before. I did:
Vec lx; VecGhostGetLocalForm(x, &lx)
then I copied "lx" to my variable, like
for(Int i=1; i<N; i++) /* N includes buffer cells */ unk[i] = lx[i]
but ghost values were also zero. I am thinking that PETSc somehow clears the ghost values after a call to KSP. Is it the case?
Kind regards,
Bojan
On Thu, Feb 23, 2012 at 2:32 PM, Bojan Niceno <[email protected]> wrote: Dear Mohammad,
it doesn't help me, or I did not understand your explanation.
If I do this:
/* copy internal values (THIS WORKS, BUT COPIES NO BUFFER VALUES) */
for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
/* copy ghost values (CREATES MANY WARNINGS */
for(Int i=n; i<N; i++) { VecGetValues(x, 1, &i, &unk[i]); }
I get arnings are like this.
[0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: Can only get local values, trying 3518! [3]PETSC ERROR: --------------------- Error Message ------------------------------------ [3]PETSC ERROR: Argument out of range! [3]PETSC ERROR: Can only get local values, trying 3511! [3]PETSC ERROR: ------------------------------------------------------------------------
What am I doing wrong here?
Cheers,
Bojan
On 2/23/2012 11:23 PM, Mohammad Mirzadeh wrote:
just index x with the local numberings. if you have 'n' local nodes and 'g' ghost nodes in x, ghost nodes indecies run from 'n' to 'n+g-1'
On Feb 23, 2012 1:16 PM, "Bojan Niceno" <[email protected]> wrote: Dear Jed,
thanks.
Now I have the following:
- Array unk, which should hold values inside the partition and in ghost cells. It is big enough to hold both - Vec x, created by command VecCreateGhost, with proper padding for ghost cells - Successful call to linear solver in parallel.
But I need to copy ghost values from x to my array unk.
How can I do it?
Kind regards,
Bojan
On 2/23/2012 10:10 PM, Jed Brown wrote:
On Thu, Feb 23, 2012 at 15:05, Bojan Niceno <[email protected]> wrote: No, I use global.
The local form is just a local vector. It doesn't even know that a global problem exists. You can't index into it using global indices. (In general, there is no efficient way to look up information in the local vector (includes ghost points) using global indices.)
for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
"n" is defined as the number of cells inside, i.e. without buffers. "unk" is my external array. If I try to access buffer values, I use:
for(Int i=0; i<N; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
But then I end up with tons of warnings, presumably because I am going beyond "n". Vector x was created with VecCreateGhost.
-- <Mail Attachment.png>
-- <Mail Attachment.png>
-- <Mail Attachment.png>
-- <Signature.png>
Actually now that we are at it, what would you guys recommend as the best way to find local indecies from global ones when accessing the _ghost_ values in the local form? This becomes even more complicated when your global petsc ordering is different from global application ordering. Its quite easy to access the local indecies for the local nodes -- you use ao to convert application global to petsc global and from petsc global, it is easy to get petsc local. However, when you have ghost nodes, there is no such simple relationship between petsc global and the _ghosted_ petsc local (Even though there is a simple one from petsc local to petsc global). The best thing I have came up with so far is to use a sequential second ao to map local indecies of ghost points to petsc global indecies. Is there a better way of doing this? Mohammad On Thu, Feb 23, 2012 at 7:44 PM, Barry Smith <[email protected]> wrote:
I have added the following to the VecGhostGetLocalForm() manual page to make this clearer for everyone:
To update the ghost values from the locations on the other processes one must call VecGhostUpdateBegin() and VecGhostUpdateEnd() before accessing the ghost values. Thus normal usage is $ VecGhostUpdateBegin(x,INSERT_VALUES,SCATTER_FORWARD); $ VecGhostUpdateEnd(x,INSERT_VALUES,SCATTER_FORWARD); $ VecGhostGetLocalForm(x,&xlocal); $ VecGetArray(xlocal,&xvalues); $ /* access the non-ghost values in locations xvalues[0:n-1] and ghost values in locations xvalues[n:n+nghost]; */ $ VecRestoreArray(xlocal,&xvalues); $ VecGhostRestoreLocalForm(x,&xlocal);
On Feb 23, 2012, at 5:38 PM, Bojan Niceno wrote:
Yeeee-ha!
VecGhostUpdateBegin() / VecGhostUpdateEnd() was indeed what I was missing.
Thank you Mohammad, thank you all who helped me today!
Cheers
Bojan
On 2/24/2012 12:27 AM, Mohammad Mirzadeh wrote:
You also need calls to VecGhostUpdateBegin()/VecGhostUpdateEnd() functions to update the ghost values if you change them in the global representation. See Petsc Manual 3.2 pp 55-56
Mohammad
On Thu, Feb 23, 2012 at 3:18 PM, Bojan Niceno <[email protected]> wrote: On 2/23/2012 11:49 PM, Mohammad Mirzadeh wrote:
based on,
VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, &ghosts[0], &x);
it seems to me that x is your actual ghosted vector. If this is true, then you need to get its "local" form via VecGhostGetLocalForm(). Once you have done, you should be able to access the ghosted nodes. Are you calling this function anywhere?
I tried that before. I did:
Vec lx; VecGhostGetLocalForm(x, &lx)
then I copied "lx" to my variable, like
for(Int i=1; i<N; i++) /* N includes buffer cells */ unk[i] = lx[i]
but ghost values were also zero. I am thinking that PETSc somehow clears the ghost values after a call to KSP. Is it the case?
Kind regards,
Bojan
On Thu, Feb 23, 2012 at 2:32 PM, Bojan Niceno <[email protected]>
wrote:
Dear Mohammad,
it doesn't help me, or I did not understand your explanation.
If I do this:
/* copy internal values (THIS WORKS, BUT COPIES NO BUFFER VALUES) */
for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
/* copy ghost values (CREATES MANY WARNINGS */
for(Int i=n; i<N; i++) { VecGetValues(x, 1, &i, &unk[i]); }
I get arnings are like this.
[0]PETSC ERROR: --------------------- Error Message
[0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: Can only get local values, trying 3518! [3]PETSC ERROR: --------------------- Error Message
[3]PETSC ERROR: Argument out of range! [3]PETSC ERROR: Can only get local values, trying 3511! [3]PETSC ERROR:
What am I doing wrong here?
Cheers,
Bojan
On 2/23/2012 11:23 PM, Mohammad Mirzadeh wrote:
just index x with the local numberings. if you have 'n' local nodes
and 'g' ghost nodes in x, ghost nodes indecies run from 'n' to 'n+g-1'
On Feb 23, 2012 1:16 PM, "Bojan Niceno" <[email protected]> wrote: Dear Jed,
thanks.
Now I have the following:
- Array unk, which should hold values inside the partition and in
ghost cells. It is big enough to hold both
- Vec x, created by command VecCreateGhost, with proper padding for ghost cells - Successful call to linear solver in parallel.
But I need to copy ghost values from x to my array unk.
How can I do it?
Kind regards,
Bojan
On 2/23/2012 10:10 PM, Jed Brown wrote:
On Thu, Feb 23, 2012 at 15:05, Bojan Niceno <[email protected]> wrote: No, I use global.
The local form is just a local vector. It doesn't even know that a global problem exists. You can't index into it using global indices. (In general, there is no efficient way to look up information in the local vector (includes ghost points) using global indices.)
for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
"n" is defined as the number of cells inside, i.e. without buffers. "unk" is my external array. If I try to access buffer values, I use:
for(Int i=0; i<N; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
But then I end up with tons of warnings, presumably because I am going beyond "n". Vector x was created with VecCreateGhost.
-- <Mail Attachment.png>
-- <Mail Attachment.png>
-- <Mail Attachment.png>
-- <Signature.png>
I do re-numbering during the domain decomposition stage. I re-number nodes (unknows) in the way I believe PETSc does internaly anyhow. It was a guess-work, I admit, but my communication patterns indeed work. Here is what I do: /*-------------------------------------+ | Assign global numbers to nodes | +-------------------------------------*/ Int global_number = 0; for(Int p=0; p<number_of_partitions; p++) { for(Int n=0; n<size(); n++) { if( nodes[n].partition == p) { nodes[n].global_number = global_number; global_number++; } } } assert( global_number == nodes.size()); -----Original Message----- From: [email protected] on behalf of Mohammad Mirzadeh Sent: Fri 2/24/2012 10:42 AM To: PETSc users list Subject: Re: [petsc-users] Accessing Vector's ghost values Actually now that we are at it, what would you guys recommend as the best way to find local indecies from global ones when accessing the _ghost_ values in the local form? This becomes even more complicated when your global petsc ordering is different from global application ordering. Its quite easy to access the local indecies for the local nodes -- you use ao to convert application global to petsc global and from petsc global, it is easy to get petsc local. However, when you have ghost nodes, there is no such simple relationship between petsc global and the _ghosted_ petsc local (Even though there is a simple one from petsc local to petsc global). The best thing I have came up with so far is to use a sequential second ao to map local indecies of ghost points to petsc global indecies. Is there a better way of doing this? Mohammad On Thu, Feb 23, 2012 at 7:44 PM, Barry Smith <[email protected]> wrote:
I have added the following to the VecGhostGetLocalForm() manual page to make this clearer for everyone:
To update the ghost values from the locations on the other processes one must call VecGhostUpdateBegin() and VecGhostUpdateEnd() before accessing the ghost values. Thus normal usage is $ VecGhostUpdateBegin(x,INSERT_VALUES,SCATTER_FORWARD); $ VecGhostUpdateEnd(x,INSERT_VALUES,SCATTER_FORWARD); $ VecGhostGetLocalForm(x,&xlocal); $ VecGetArray(xlocal,&xvalues); $ /* access the non-ghost values in locations xvalues[0:n-1] and ghost values in locations xvalues[n:n+nghost]; */ $ VecRestoreArray(xlocal,&xvalues); $ VecGhostRestoreLocalForm(x,&xlocal);
On Feb 23, 2012, at 5:38 PM, Bojan Niceno wrote:
Yeeee-ha!
VecGhostUpdateBegin() / VecGhostUpdateEnd() was indeed what I was missing.
Thank you Mohammad, thank you all who helped me today!
Cheers
Bojan
On 2/24/2012 12:27 AM, Mohammad Mirzadeh wrote:
You also need calls to VecGhostUpdateBegin()/VecGhostUpdateEnd() functions to update the ghost values if you change them in the global representation. See Petsc Manual 3.2 pp 55-56
Mohammad
On Thu, Feb 23, 2012 at 3:18 PM, Bojan Niceno <[email protected]> wrote: On 2/23/2012 11:49 PM, Mohammad Mirzadeh wrote:
based on,
VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, &ghosts[0], &x);
it seems to me that x is your actual ghosted vector. If this is true, then you need to get its "local" form via VecGhostGetLocalForm(). Once you have done, you should be able to access the ghosted nodes. Are you calling this function anywhere?
I tried that before. I did:
Vec lx; VecGhostGetLocalForm(x, &lx)
then I copied "lx" to my variable, like
for(Int i=1; i<N; i++) /* N includes buffer cells */ unk[i] = lx[i]
but ghost values were also zero. I am thinking that PETSc somehow clears the ghost values after a call to KSP. Is it the case?
Kind regards,
Bojan
On Thu, Feb 23, 2012 at 2:32 PM, Bojan Niceno <[email protected]>
wrote:
Dear Mohammad,
it doesn't help me, or I did not understand your explanation.
If I do this:
/* copy internal values (THIS WORKS, BUT COPIES NO BUFFER VALUES) */
for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
/* copy ghost values (CREATES MANY WARNINGS */
for(Int i=n; i<N; i++) { VecGetValues(x, 1, &i, &unk[i]); }
I get arnings are like this.
[0]PETSC ERROR: --------------------- Error Message
[0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: Can only get local values, trying 3518! [3]PETSC ERROR: --------------------- Error Message
[3]PETSC ERROR: Argument out of range! [3]PETSC ERROR: Can only get local values, trying 3511! [3]PETSC ERROR:
What am I doing wrong here?
Cheers,
Bojan
On 2/23/2012 11:23 PM, Mohammad Mirzadeh wrote:
just index x with the local numberings. if you have 'n' local nodes
and 'g' ghost nodes in x, ghost nodes indecies run from 'n' to 'n+g-1'
On Feb 23, 2012 1:16 PM, "Bojan Niceno" <[email protected]> wrote: Dear Jed,
thanks.
Now I have the following:
- Array unk, which should hold values inside the partition and in
ghost cells. It is big enough to hold both
- Vec x, created by command VecCreateGhost, with proper padding for ghost cells - Successful call to linear solver in parallel.
But I need to copy ghost values from x to my array unk.
How can I do it?
Kind regards,
Bojan
On 2/23/2012 10:10 PM, Jed Brown wrote:
On Thu, Feb 23, 2012 at 15:05, Bojan Niceno <[email protected]> wrote: No, I use global.
The local form is just a local vector. It doesn't even know that a global problem exists. You can't index into it using global indices. (In general, there is no efficient way to look up information in the local vector (includes ghost points) using global indices.)
for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
"n" is defined as the number of cells inside, i.e. without buffers. "unk" is my external array. If I try to access buffer values, I use:
for(Int i=0; i<N; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
But then I end up with tons of warnings, presumably because I am going beyond "n". Vector x was created with VecCreateGhost.
-- <Mail Attachment.png>
-- <Mail Attachment.png>
-- <Mail Attachment.png>
-- <Signature.png>
Thanks Bojan. I suppose that would work. Problem arises, however, when the actual application numbering matters and you have to preserve it somehow. One example, is where you have a library that is currently sequential and a lot of functionality is directly based on the data structure used for the grid. For example when computing derivatives or performing interpolation and so on. Naturally you would want to reuse as much of the code as possible. So, you would think you can have a function, or a class, that, for example, takes the ghosted vector and the original global numbering and returns the value so that you could still use the old functions, say to compute gradients. Now, when you are constructing the ghosted vector, you know order you are putting the ghost nodes, lets say the first one has global index 42, next has 21, next 56 and so on ... However, when you call the function with the global index 21, you need to know what local ghost index it corresponds to in the local form of the ghosted vector. I can think of the following options to do this: 1- Search the ghost nodes indecies that you created the ghosted vector with: here, search int ghost [] = {42, 21, 56} for 21 and return the index which is 2. This, you can do at O(log n) e.g with binary search 2- Make a vector consisting of _all_ nodes on each processor that returns the ghost index, i.e. have something like ghostIndex[21] = 2, ghostIndex[42] = 1, ghostIndex[56] = 3. Using this you get the index in O(1) but at the cost of extra memory 3- Make an ao that maps {1,2,3} to {42, 21, 56}. This seems to be the best of both worlds So, in short, I'm interested in something that wont alter (at least permanently) the application global numbering so that I can use my old pieces of code and at the same time something that can compute the ghost location fast and efficient from a given global numbering. Mohammad On Fri, Feb 24, 2012 at 1:54 AM, Niceno Bojan <[email protected]> wrote:
I do re-numbering during the domain decomposition stage. I re-number nodes (unknows) in the way I believe PETSc does internaly anyhow. It was a guess-work, I admit, but my communication patterns indeed work. Here is what I do:
/*-------------------------------------+ | Assign global numbers to nodes | +-------------------------------------*/ Int global_number = 0; for(Int p=0; p<number_of_partitions; p++) { for(Int n=0; n<size(); n++) { if( nodes[n].partition == p) { nodes[n].global_number = global_number; global_number++; } } } assert( global_number == nodes.size());
-----Original Message----- From: [email protected] on behalf of Mohammad Mirzadeh Sent: Fri 2/24/2012 10:42 AM To: PETSc users list Subject: Re: [petsc-users] Accessing Vector's ghost values
Actually now that we are at it, what would you guys recommend as the best way to find local indecies from global ones when accessing the _ghost_ values in the local form? This becomes even more complicated when your global petsc ordering is different from global application ordering.
Its quite easy to access the local indecies for the local nodes -- you use ao to convert application global to petsc global and from petsc global, it is easy to get petsc local. However, when you have ghost nodes, there is no such simple relationship between petsc global and the _ghosted_ petsc local (Even though there is a simple one from petsc local to petsc global). The best thing I have came up with so far is to use a sequential second ao to map local indecies of ghost points to petsc global indecies. Is there a better way of doing this?
Mohammad
On Thu, Feb 23, 2012 at 7:44 PM, Barry Smith <[email protected]> wrote:
I have added the following to the VecGhostGetLocalForm() manual page to make this clearer for everyone:
To update the ghost values from the locations on the other processes one must call VecGhostUpdateBegin() and VecGhostUpdateEnd() before accessing the ghost values. Thus normal usage is $ VecGhostUpdateBegin(x,INSERT_VALUES,SCATTER_FORWARD); $ VecGhostUpdateEnd(x,INSERT_VALUES,SCATTER_FORWARD); $ VecGhostGetLocalForm(x,&xlocal); $ VecGetArray(xlocal,&xvalues); $ /* access the non-ghost values in locations xvalues[0:n-1] and ghost values in locations xvalues[n:n+nghost]; */ $ VecRestoreArray(xlocal,&xvalues); $ VecGhostRestoreLocalForm(x,&xlocal);
On Feb 23, 2012, at 5:38 PM, Bojan Niceno wrote:
Yeeee-ha!
VecGhostUpdateBegin() / VecGhostUpdateEnd() was indeed what I was missing.
Thank you Mohammad, thank you all who helped me today!
Cheers
Bojan
On 2/24/2012 12:27 AM, Mohammad Mirzadeh wrote:
You also need calls to VecGhostUpdateBegin()/VecGhostUpdateEnd() functions to update the ghost values if you change them in the global representation. See Petsc Manual 3.2 pp 55-56
Mohammad
On Thu, Feb 23, 2012 at 3:18 PM, Bojan Niceno <[email protected]> wrote: On 2/23/2012 11:49 PM, Mohammad Mirzadeh wrote:
based on,
VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, &ghosts[0], &x);
it seems to me that x is your actual ghosted vector. If this is true, then you need to get its "local" form via VecGhostGetLocalForm(). Once
you
have done, you should be able to access the ghosted nodes. Are you calling this function anywhere?
I tried that before. I did:
Vec lx; VecGhostGetLocalForm(x, &lx)
then I copied "lx" to my variable, like
for(Int i=1; i<N; i++) /* N includes buffer cells */ unk[i] = lx[i]
but ghost values were also zero. I am thinking that PETSc somehow
clears the ghost values after a call to KSP. Is it the case?
Kind regards,
Bojan
On Thu, Feb 23, 2012 at 2:32 PM, Bojan Niceno <[email protected]>
wrote:
Dear Mohammad,
it doesn't help me, or I did not understand your explanation.
If I do this:
/* copy internal values (THIS WORKS, BUT COPIES NO BUFFER VALUES) */
for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
/* copy ghost values (CREATES MANY WARNINGS */
for(Int i=n; i<N; i++) { VecGetValues(x, 1, &i, &unk[i]); }
I get arnings are like this.
[0]PETSC ERROR: --------------------- Error Message
[0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: Can only get local values, trying 3518! [3]PETSC ERROR: --------------------- Error Message
[3]PETSC ERROR: Argument out of range! [3]PETSC ERROR: Can only get local values, trying 3511! [3]PETSC ERROR:
What am I doing wrong here?
Cheers,
Bojan
On 2/23/2012 11:23 PM, Mohammad Mirzadeh wrote:
just index x with the local numberings. if you have 'n' local nodes
and 'g' ghost nodes in x, ghost nodes indecies run from 'n' to 'n+g-1'
On Feb 23, 2012 1:16 PM, "Bojan Niceno" <[email protected]>
wrote:
Dear Jed,
thanks.
Now I have the following:
- Array unk, which should hold values inside the partition and in ghost cells. It is big enough to hold both - Vec x, created by command VecCreateGhost, with proper padding for ghost cells - Successful call to linear solver in parallel.
But I need to copy ghost values from x to my array unk.
How can I do it?
Kind regards,
Bojan
On 2/23/2012 10:10 PM, Jed Brown wrote:
On Thu, Feb 23, 2012 at 15:05, Bojan Niceno <[email protected]> wrote: No, I use global.
The local form is just a local vector. It doesn't even know that a global problem exists. You can't index into it using global indices. (In general, there is no efficient way to look up information in the local vector (includes ghost points) using global indices.)
for(Int i=0; i<n; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
"n" is defined as the number of cells inside, i.e. without buffers. "unk" is my external array. If I try to access buffer values, I use:
for(Int i=0; i<N; i++) { Int gi = mesh.nodes[i].global_number; VecGetValues(x, 1, &gi, &unk[i]); }
But then I end up with tons of warnings, presumably because I am going beyond "n". Vector x was created with VecCreateGhost.
-- <Mail Attachment.png>
-- <Mail Attachment.png>
-- <Mail Attachment.png>
-- <Signature.png>
On Fri, Feb 24, 2012 at 04:43, Mohammad Mirzadeh <[email protected]> wrote:
Thanks Bojan. I suppose that would work. Problem arises, however, when the actual application numbering matters and you have to preserve it somehow. One example, is where you have a library that is currently sequential and a lot of functionality is directly based on the data structure used for the grid. For example when computing derivatives or performing interpolation and so on.
That serial library probably needs the local numbering anyway, if you give it the global indices, it won't be able to access those locations (at least not without a translation step that they did not write). It's better to build the data structure in terms of local indices.
On Fri, Feb 24, 2012 at 03:42, Mohammad Mirzadeh <[email protected]> wrote:
Actually now that we are at it, what would you guys recommend as the best way to find local indecies from global ones when accessing the _ghost_ values in the local form? This becomes even more complicated when your global petsc ordering is different from global application ordering.
Its quite easy to access the local indecies for the local nodes -- you use ao to convert application global to petsc global and from petsc global, it is easy to get petsc local. However, when you have ghost nodes, there is no such simple relationship between petsc global and the _ghosted_ petsc local (Even though there is a simple one from petsc local to petsc global). The best thing I have came up with so far is to use a sequential second ao to map local indecies of ghost points to petsc global indecies. Is there a better way of doing this?
Do the conversion the other way. If your nodes are labeled with global indices, do a traversal (in some ordering, probably owned then ghosted) assigning local indices. As you point out, it is not efficient to convert global indices for ghost points into local indices, so you should build the data structure in terms of local indices and just keep a local-to-global mapping to glue the subdomains together.
On Thu, Feb 23, 2012 at 15:16, Bojan Niceno <[email protected]> wrote:
- Array unk, which should hold values inside the partition and in ghost cells. It is big enough to hold both - Vec x, created by command VecCreateGhost, with proper padding for ghost cells - Successful call to linear solver in parallel.
But I need to copy ghost values from x to my array unk.
I'm not sure what you're asking. You create the ghost array and use VecGhostGetLocalForm()/VecGhostRestoreLocalForm() to access and release access to the local vector (which includes the ghost points). You can into or out of the local form if you want, but usually people set up their indexing to use the ordering in the local form (owned values followed by ghosted values).
2012/2/23 Bojan Niceno <[email protected]>
Hi all,
I've never used a mailing list before, so I hope this message will reach PETSc users and experts and someone might be willing to help me. I am also novice in PETSc.
I have developed an unstructured finite volume solver on top of PETSc libraries. In sequential, it works like a charm. For the parallel version, I do domain decomposition externally with Metis, and work out local and global numberings, as well as communication patterns between processor. (The latter don't seem to be needed for PETSc, though.) When I run my program in parallel, it also works, but I miss values in vectors' ghost points.
I create vectors with command: VecCreate(PETSC_COMM_WORLD, &x);
Is it possible to get the ghost values if a vector is created like this?
I have tried to use VecCreateGhost, but for some reason which is beyond my comprehension, PETSc goes berserk when it reaches the command: VecCreateGhost(PETSC_COMM_**WORLD, n, PETSC_DECIDE, nghost, ifrom, &x)
Can anyone help me? Either how to reach ghost values for vector created by VecCreate, or how to use VecCreateGhost properly?
http://www.mcs.anl.gov/petsc/petsc-current/src/vec/vec/examples/tutorials/ex... could be helpful.
Bojan
Thanks Ju. I studied this case carefully, and it seems clear to me. When I apply the same techniques in my code, I get error messages I sent to my reply to Matthew. Cheers, Bojan On 2/23/2012 6:46 PM, Ju LIU wrote:
2012/2/23 Bojan Niceno <[email protected] <mailto:[email protected]>>
Hi all,
I've never used a mailing list before, so I hope this message will reach PETSc users and experts and someone might be willing to help me. I am also novice in PETSc.
I have developed an unstructured finite volume solver on top of PETSc libraries. In sequential, it works like a charm. For the parallel version, I do domain decomposition externally with Metis, and work out local and global numberings, as well as communication patterns between processor. (The latter don't seem to be needed for PETSc, though.) When I run my program in parallel, it also works, but I miss values in vectors' ghost points.
I create vectors with command: VecCreate(PETSC_COMM_WORLD, &x);
Is it possible to get the ghost values if a vector is created like this?
I have tried to use VecCreateGhost, but for some reason which is beyond my comprehension, PETSc goes berserk when it reaches the command: VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, ifrom, &x)
Can anyone help me? Either how to reach ghost values for vector created by VecCreate, or how to use VecCreateGhost properly?
http://www.mcs.anl.gov/petsc/petsc-current/src/vec/vec/examples/tutorials/ex... could be helpful.
Bojan
--
Did you try running with -on_error_attach_debugger and and using your debugger to try to figure out where your code is segfaulting? Max On Thu, Feb 23, 2012 at 10:07 AM, Bojan Niceno <[email protected]> wrote:
Thanks Ju. I studied this case carefully, and it seems clear to me. When I apply the same techniques in my code, I get error messages I sent to my reply to Matthew.
Cheers,
Bojan
On 2/23/2012 6:46 PM, Ju LIU wrote:
2012/2/23 Bojan Niceno <[email protected]>
Hi all,
I've never used a mailing list before, so I hope this message will reach PETSc users and experts and someone might be willing to help me. I am also novice in PETSc.
I have developed an unstructured finite volume solver on top of PETSc libraries. In sequential, it works like a charm. For the parallel version, I do domain decomposition externally with Metis, and work out local and global numberings, as well as communication patterns between processor. (The latter don't seem to be needed for PETSc, though.) When I run my program in parallel, it also works, but I miss values in vectors' ghost points.
I create vectors with command: VecCreate(PETSC_COMM_WORLD, &x);
Is it possible to get the ghost values if a vector is created like this?
I have tried to use VecCreateGhost, but for some reason which is beyond my comprehension, PETSc goes berserk when it reaches the command: VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, ifrom, &x)
Can anyone help me? Either how to reach ghost values for vector created by VecCreate, or how to use VecCreateGhost properly?
http://www.mcs.anl.gov/petsc/petsc-current/src/vec/vec/examples/tutorials/ex... could be helpful.
Bojan
--
On 2/23/2012 7:22 PM, Max Rudolph wrote:
Did you try running with -on_error_attach_debugger and and using your debugger to try to figure out where your code is segfaulting?
Well, it's a bit embarrassing, but I should admit that I never use debuggers. I use "exit(0)" to find the place where program breaks. It breaks at the very call to VecCreateGhost. Kind regards, Bojan
participants (8)
-
Barry Smith -
Bojan Niceno -
Jed Brown -
Ju LIU -
Matthew Knepley -
Max Rudolph -
Mohammad Mirzadeh -
Niceno Bojan