March 2011 - petsc-dev - mailman3.cels.anl.gov

hg diff question
by Barry Smith 23 Mar '11

23 Mar '11

hg diff src/makefile How come there isn't an option to have it show the most recent set of changes? It seems I have to hunt down some "revision numbers" and type them in the request to get any useful information? How about something like hg diff --last src/makefile Barry

1 0

Re: [petsc-dev] Bug in petsc-dev?
by Thomas Witkowski 23 Mar '11

23 Mar '11

Jed Brown wrote: > On Wed, Mar 23, 2011 at 12:50, Thomas Witkowski > <thomas.witkowski(a)tu-dresden.de > <mailto:[email protected]>> wrote: > > I found the problem: on the parallel system I make use of the > environment variable $DISPLAY is set only on the first node of the > parallel job. On all other nodes this variable is set to an empty > string. This brings trouble in the function PetscSetDisplay(): > > > Thank you for tracking this down. I made this function always > broadcast so that every process is assured the same value for > PetscDisplay. Could you pull from Hg and confirm that this works. Yes, now it works fine! Thanks for the fast fix of the problem! Thomas

1 0

Re: [petsc-dev] Bug in petsc-dev?
by Jed Brown 23 Mar '11

23 Mar '11

On Wed, Mar 23, 2011 at 12:50, Thomas Witkowski < thomas.witkowski(a)tu-dresden.de> wrote: > I found the problem: on the parallel system I make use of the environment > variable $DISPLAY is set only on the first node of the parallel job. On all > other nodes this variable is set to an empty string. This brings trouble in > the function PetscSetDisplay(): Thank you for tracking this down. I made this function always broadcast so that every process is assured the same value for PetscDisplay. Could you pull from Hg and confirm that this works.

1 0

Re: [petsc-dev] Bug in petsc-dev?
by Thomas Witkowski 23 Mar '11

23 Mar '11

I found the problem: on the parallel system I make use of the environment variable $DISPLAY is set only on the first node of the parallel job. On all other nodes this variable is set to an empty string. This brings trouble in the function PetscSetDisplay(): str = getenv("DISPLAY"); if (!str) str = ":0.0"; if (str[0] != ':' || singlehost) { ierr = PetscStrncpy(display,str,sizeof display);CHKERRQ(ierr); } else { if (!rank) { size_t len; ierr = PetscGetHostName(display,sizeof display);CHKERRQ(ierr); ierr = PetscStrlen(display,&len);CHKERRQ(ierr); ierr = PetscStrncat(display,str,sizeof display-len-1);CHKERRQ(ierr); } ierr = MPI_Bcast(display,sizeof display,MPI_CHAR,0,PETSC_COMM_WORLD);CHKERRQ(ierr); } Only those ranks, on which $DISPLAY is set, run into the branch with MPI_Bcast. Therefore the problem with the MPI_Bcast in my code. Thomas Barry Smith wrote: > Is the use of the "current petsc-dev", and "Using PETSc 3.1-p8" both built with the exact same MPI? > > Are you using shared or static libraries for OpenMPI and PETSc? > > Are you using the exact same mpiexec to start up all the cases? > > If you change the order of the four nodes that you run this on does the "oddball" result process rank always refer to the same physical node? That is if the machine that is now used as the fourth node is instead used as the third node does the wrong answer appear on then on the third node or still on the fourth? If you use a different physical machine for the fourth node does the problem persist? > > If you get rid of the rand() call and just set the fileRandomNumber value with say 450385 does it behave the same way? > > The reason I am asking you all these questions is that this is a very strange error that defies easy explanation; since it is just an MPI call the fact that PETSc is used shouldn't matter (yet it does). > > > Barry > > On Mar 22, 2011, at 12:50 PM, Thomas Witkowski wrote: > > >> Zitat von Barry Smith <bsmith(a)mcs.anl.gov>: >> >> >>> On Mar 22, 2011, at 11:08 AM, Thomas Witkowski wrote: >>> >>> >>>> Could some of you test the very small attached example? I make use of the current petsc-dev, OpenMPI 1.4.1 and GCC 4.2.4. In this environment, using 4 nodes, I get the following output, which is wrong: >>>> >>>> [3] BCAST-RESULT: 812855920 >>>> [2] BCAST-RESULT: 450385 >>>> [1] BCAST-RESULT: 450385 >>>> [0] BCAST-RESULT: 450385 >>>> >>>> The problem occurs only when I run the code on different nodes. When I start mpirun on only one node with four threads >>>> >>> You mean 4 MPI processes? >>> >> Yes. >> >> >>> >>>> or I make use of a four core system, everything is fine. valgrind and Allinea DDT, both say that everything is fine. So I'm really not sure where the problem is. Using PETSc 3.1-p8 there is no problem with this example. Would be quite interesting to know if some of you can reproduce this problem or not. Thanks for any try! >>>> >>> Replace the PetscInitialize() and PetscFinalize() with MPI_Init() and MPI_Finalize() and remove the include petsc.h now link under old and new PETSc and run under the different systems. >>> >>> I'm thinking you'll still get the wrong result without the Petsc calls indicating that it is an MPI issue. >>> >> No! When I already did this test. In this case I get the correct results! >> >> Thomas >> >> >> >>> Barry >>> >>> >>>> Thomas >>>> >>>> <test.c> >>>> >>> >>> >> > > > >

1 0

Re: [petsc-dev] ML changing the number of levels
by Barry Smith 22 Mar '11

22 Mar '11

On Mar 22, 2011, at 9:42 PM, Jed Brown wrote: > I was hoping you would just fix it. :) I don't know how to fix it except with a hack. I'll look at as soon as my brain grows back, hopefully in a day or two. Barry > > >> On Mar 23, 2011 3:39 AM, "Barry Smith" <bsmith(a)mcs.anl.gov> wrote: >> >> >> >> On Mar 22, 2011, at 7:34 PM, Jed Brown wrote: >> >> > I just fixed a few things in PCReset_ML, but t... >> >> Right. I forgot about this issue. Maybe put a note about this in the code for the future. >> >> >> Barry >> >

1 0

Re: [petsc-dev] ML changing the number of levels
by Jed Brown 22 Mar '11

22 Mar '11

I was hoping you would just fix it. :) On Mar 23, 2011 3:39 AM, "Barry Smith" <bsmith(a)mcs.anl.gov> wrote: On Mar 22, 2011, at 7:34 PM, Jed Brown wrote: > I just fixed a few things in PCReset_ML, but t... Right. I forgot about this issue. Maybe put a note about this in the code for the future. Barry

1 0

Re: [petsc-dev] ML changing the number of levels
by Barry Smith 22 Mar '11

22 Mar '11

On Mar 22, 2011, at 7:34 PM, Jed Brown wrote: > I just fixed a few things in PCReset_ML, but the current solution is still broken if the number of levels changes. We don't want to always destroy all the levels in PCReset_MG, but ML needs a way to communicate that the number of levels has changed. Right. I forgot about this issue. Maybe put a note about this in the code for the future. Barry

1 0

ML changing the number of levels
by Jed Brown 22 Mar '11

22 Mar '11

I just fixed a few things in PCReset_ML, but the current solution is still broken if the number of levels changes. We don't want to always destroy all the levels in PCReset_MG, but ML needs a way to communicate that the number of levels has changed.

1 0

Re: [petsc-dev] Bug in petsc-dev?
by Barry Smith 22 Mar '11

22 Mar '11

Is the use of the "current petsc-dev", and "Using PETSc 3.1-p8" both built with the exact same MPI? Are you using shared or static libraries for OpenMPI and PETSc? Are you using the exact same mpiexec to start up all the cases? If you change the order of the four nodes that you run this on does the "oddball" result process rank always refer to the same physical node? That is if the machine that is now used as the fourth node is instead used as the third node does the wrong answer appear on then on the third node or still on the fourth? If you use a different physical machine for the fourth node does the problem persist? If you get rid of the rand() call and just set the fileRandomNumber value with say 450385 does it behave the same way? The reason I am asking you all these questions is that this is a very strange error that defies easy explanation; since it is just an MPI call the fact that PETSc is used shouldn't matter (yet it does). Barry On Mar 22, 2011, at 12:50 PM, Thomas Witkowski wrote: > Zitat von Barry Smith <bsmith(a)mcs.anl.gov>: > >> >> On Mar 22, 2011, at 11:08 AM, Thomas Witkowski wrote: >> >>> Could some of you test the very small attached example? I make use of the current petsc-dev, OpenMPI 1.4.1 and GCC 4.2.4. In this environment, using 4 nodes, I get the following output, which is wrong: >>> >>> [3] BCAST-RESULT: 812855920 >>> [2] BCAST-RESULT: 450385 >>> [1] BCAST-RESULT: 450385 >>> [0] BCAST-RESULT: 450385 >>> >>> The problem occurs only when I run the code on different nodes. When I start mpirun on only one node with four threads >> >> You mean 4 MPI processes? > > Yes. > >> >> >>> or I make use of a four core system, everything is fine. valgrind and Allinea DDT, both say that everything is fine. So I'm really not sure where the problem is. Using PETSc 3.1-p8 there is no problem with this example. Would be quite interesting to know if some of you can reproduce this problem or not. Thanks for any try! >> >> Replace the PetscInitialize() and PetscFinalize() with MPI_Init() and MPI_Finalize() and remove the include petsc.h now link under old and new PETSc and run under the different systems. >> >> I'm thinking you'll still get the wrong result without the Petsc calls indicating that it is an MPI issue. > > No! When I already did this test. In this case I get the correct results! > > Thomas > > >> >> Barry >> >>> >>> Thomas >>> >>> <test.c> >> >> >> > >

1 0

Re: [petsc-dev] Bug in petsc-dev?
by Thomas Witkowski 22 Mar '11

22 Mar '11

Zitat von Barry Smith <bsmith(a)mcs.anl.gov>: > > On Mar 22, 2011, at 11:08 AM, Thomas Witkowski wrote: > >> Could some of you test the very small attached example? I make use >> of the current petsc-dev, OpenMPI 1.4.1 and GCC 4.2.4. In this >> environment, using 4 nodes, I get the following output, which is >> wrong: >> >> [3] BCAST-RESULT: 812855920 >> [2] BCAST-RESULT: 450385 >> [1] BCAST-RESULT: 450385 >> [0] BCAST-RESULT: 450385 >> >> The problem occurs only when I run the code on different nodes. >> When I start mpirun on only one node with four threads > > You mean 4 MPI processes? Yes. > > >> or I make use of a four core system, everything is fine. valgrind >> and Allinea DDT, both say that everything is fine. So I'm really >> not sure where the problem is. Using PETSc 3.1-p8 there is no >> problem with this example. Would be quite interesting to know if >> some of you can reproduce this problem or not. Thanks for any try! > > Replace the PetscInitialize() and PetscFinalize() with MPI_Init() > and MPI_Finalize() and remove the include petsc.h now link under > old and new PETSc and run under the different systems. > > I'm thinking you'll still get the wrong result without the Petsc > calls indicating that it is an MPI issue. No! When I already did this test. In this case I get the correct results! Thomas > > Barry > >> >> Thomas >> >> <test.c> > > >

1 0