August 2020 - petsc-dev - mailman3.cels.anl.gov

DMPlex + Cohesive Cells Setup
by Jacob Faibussowitsch 31 Aug '20

31 Aug '20

Hello All, What is the simplest possible configuration required to insert cohesive cells into a plex? Plex/tests/ex5.c would be the example to ape here but there is a lot there that I don’t (think) I need like a DS or any kind of physics. For now, I am essentially looking to: 1. Read the mesh 2. Perform some mesh analysis 3. Slap some cohesive cells along one/multiple areas identified in the previous step (maybe even everywhere?) 4. Save the new mesh This doesn’t even have to parallel. Best regards, Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch) Cell: (312) 694-3391

1 0

Re: [petsc-dev] MacOS firewall annoyance while running tests + solution
by Barry Smith 31 Aug '20

31 Aug '20

I would make your fix a configure option that is off by default. It is silly that Apple makes you use sudo to tell it that a compiled code SHOULD NOT accept outside connections but they have it all bundled together without enough thought for developers. Barry > On Aug 31, 2020, at 4:14 AM, Hapla Vaclav <vaclav.hapla(a)erdw.ethz.ch> wrote: > > >> On 29 Aug 2020, at 02:03, Hapla Vaclav <vaclav.hapla(a)erdw.ethz.ch> wrote: >> >> >> >>> On 28 Aug 2020, at 22:47, Jed Brown <jed(a)jedbrown.org> wrote: >>> >>> "Hapla Vaclav" <vaclav.hapla(a)erdw.ethz.ch> writes: >>> >>>> On MacOS, maybe you also have lots of firewall popups appearing/disappearing when running tests like >>>> Do you want the application "ex29" to accept incoming network connections? >>> >>> Is there a way to express that the application does not need (should not accept) incoming connections? >> >> Oh yes, hadn't thought about before, but it's surely better to _block_ the incoming connections - feels safer for a user at least. >> >> But the way is the same, requiring sudo: >> >> sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add $PETSC_DIR/$PETSC_ARCH/tests/dm/impls/plex/tests/ex9 >> sudo /usr/libexec/ApplicationFirewall/socketfilterfw --block $PETSC_DIR/$PETSC_ARCH/tests/dm/impls/plex/tests/ex9 >> >> Funny enough, these commands don't fail without sudo but they have no effect. >> >>> >>> Normalizing sudo during build/testing seems really bad. >> >> I agree. It shouldn't be a normal part of the makefile. That's why I have been hesitant to create a MR. I think we could >> 1. just add an FAQ entry - but the patch can become out-of-date pretty quickly, and instructions without a patch are gonna be tedious >> 2. activate this part conditionally, requiring a user's action such as passing a very special variable to makefile >> 3. put it into a separate makefile and add a commented out include into gmakefile.test, so that the user has to explicitly uncomment the inclusion - or something like that >> 4. something else? > > Jed, Barry, thoughts here? > > Thanks, > Vaclav > >> >> Vaclav >

1 0

Re: [petsc-dev] MacOS firewall annoyance while running tests + solution
by Hapla Vaclav 31 Aug '20

31 Aug '20

> On 29 Aug 2020, at 02:03, Hapla Vaclav <vaclav.hapla(a)erdw.ethz.ch> wrote: > > > >> On 28 Aug 2020, at 22:47, Jed Brown <jed(a)jedbrown.org> wrote: >> >> "Hapla Vaclav" <vaclav.hapla(a)erdw.ethz.ch> writes: >> >>> On MacOS, maybe you also have lots of firewall popups appearing/disappearing when running tests like >>> Do you want the application "ex29" to accept incoming network connections? >> >> Is there a way to express that the application does not need (should not accept) incoming connections? > > Oh yes, hadn't thought about before, but it's surely better to _block_ the incoming connections - feels safer for a user at least. > > But the way is the same, requiring sudo: > > sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add $PETSC_DIR/$PETSC_ARCH/tests/dm/impls/plex/tests/ex9 > sudo /usr/libexec/ApplicationFirewall/socketfilterfw --block $PETSC_DIR/$PETSC_ARCH/tests/dm/impls/plex/tests/ex9 > > Funny enough, these commands don't fail without sudo but they have no effect. > >> >> Normalizing sudo during build/testing seems really bad. > > I agree. It shouldn't be a normal part of the makefile. That's why I have been hesitant to create a MR. I think we could > 1. just add an FAQ entry - but the patch can become out-of-date pretty quickly, and instructions without a patch are gonna be tedious > 2. activate this part conditionally, requiring a user's action such as passing a very special variable to makefile > 3. put it into a separate makefile and add a commented out include into gmakefile.test, so that the user has to explicitly uncomment the inclusion - or something like that > 4. something else? Jed, Barry, thoughts here? Thanks, Vaclav > > Vaclav

1 0

Re: [petsc-dev] MacOS firewall annoyance while running tests + solution
by Hapla Vaclav 31 Aug '20

31 Aug '20

On 29 Aug 2020, at 05:44, Satish Balay via petsc-dev <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>> wrote: Likely firewall is disabled. I've recently rebuild a bunch of boxes - and don't remember explicity disabling firewall. [well I did something to enable ssh - maybe that was disabling firewall] I see the firewall is disabled on all of them balay@ypro ~ % defaults read /Library/Preferences/com.apple.alf globalstate 0 balay@ypro ~ % Ref: https://raymii.org/s/snippets/OS_X_-_Turn_firewall_on_or_off_from_the_comma… BTW: Perhaps the following setting will prevent popups? [but it might break stuff that need network?] sudo defaults write /Library/Preferences/com.apple.alf globalstate -int 2 2 = on for essential services https://discussions.apple.com/thread/3148672 So I don't think that's an option for normal people - I don't think anybody is keen to limit the network functionality on their laptop (e.g. disable Spotify) just for sake of PETSc testing. So I still think the automated targeted firewall rules I propose are better for most guys. The question is how much automated solution we want to offer. Vaclav Satish On Fri, 28 Aug 2020, Matthew Knepley wrote: Which OS is this? It does not happen on my Catalina. Thanks, Matt On Fri, Aug 28, 2020 at 10:52 PM Barry Smith <bsmith(a)petsc.dev<mailto:[email protected]>> wrote: On Aug 28, 2020, at 3:47 PM, Jed Brown <jed(a)jedbrown.org<mailto:[email protected]>> wrote: "Hapla Vaclav" <vaclav.hapla(a)erdw.ethz.ch<mailto:[email protected]>> writes: On MacOS, maybe you also have lots of firewall popups appearing/disappearing when running tests like Do you want the application "ex29" to accept incoming network connections? Is there a way to express that the application does not need (should not accept) incoming connections? Yes, this also seems to work: sudo $$FW --block $$APP instead of sudo $$FW --unblock $$APP The parallel program still runs correctly to conclusion without the popup. So my conclusion is that at listen() or some later system call it always pops up the window (unless the user as already blocked or unblocked the executable) without regard to whether an outside (from the machine) connection to the process is attempted. The routine has an undocumented option -a <listen or accept> when you run with /usr/libexec/ApplicationFirewall/socketfilterfw -d -a accept it prints ASKWHENACCEPT which seems to indicate it will delay the popup until an accept is called but I can't confirm this because the debugger never stops in accept on one process but the popup still comes up so this argument may be ignored. If I were Junchao I would not do the popup until the code tried to accepted an EXTERNAL connection (the lazy evaluation) but I cannot get it to behave this way. If I disconnect from the network I still get the popups. The pop up is asynchronous also, when the popup is still up the program keeps run (even in parallel) and ends normally. Then the popup disappears. Apple could make this friendly without hurting security but then Apple never cared about external developers for the Mac. Normalizing sudo during build/testing seems really bad.

1 0

Re: [petsc-dev] PETSc issue I cannot post combine WaitForCUDA(); inside PetscLogGpuTimeEnd();
by Barry Smith 29 Aug '20

29 Aug '20

Karl, You are right. If we have some way of knowing the user callback is on the GPU then we will want to wrap the calling back in the GPU timing so the problem should be resolved but we would need to know which callbacks are on the GPU perhaps as our understanding of how through the PETSc API users will use the GPU for their own code this will be clear. Barry > On Aug 29, 2020, at 12:16 AM, Karl Rupp <rupp(a)iue.tuwien.ac.at> wrote: > > >>>>>> Since we cannot post issues (reported here https://forum.gitlab.com/t/creating-new-issue-gives-cannot-create-issue-get…) here is my issue so I don't forget it. >>>>>> I think >>>>>> err = WaitForCUDA();CHKERRCUDA(err); >>>>>> ierr = PetscLogGpuTimeEnd();CHKERRQ(ierr); >>>>>> should be changed to include WaitForCUDA() actually WaitForDevice() inside the PetscLogGpuTimeEnd(). >>>>>> Currently sometimes the WaitForCUDA() is missing in a few places resulting in bad timing. >>>>>> Also some _SeqCUDA() don't have the PetscLogGpuTimeEnd() and need to be fixed. >>>>>> The current model is a maintenance nightmare. >>>>>> Does anyone see a problem with making this change? >>>>> >>>>> I'm fine with this change, as the maintenance benefits outweigh the performance cost for typical use cases. >>>>> >>>>> I propose to also add the WaitForDevice(); at PetscLogGpuTimeBegin(). This will ensure that no previous GPU kernel executions spill over into the timed section. >> Karl, >> When synchronization is turned on the precious GPU kernels should always have their own WaitForDevice(), so are you concerned about buggy code that does not include WaitForDevice? > > I'm primarily thinking of user callback routines here. For example, a FormFunction provided by the user that is running some GPU kernels. We have no guarantee that these user kernels have completed before entering the timed sections inside PETSc, so the logs will be skewed to report an unusually slow kernel in PETSc (the one right after the user form function). Arguably we could add a WaitForDevice() after user callback invocations. > > I didn't think of the WaitForDevice() after each kernel call in PETSc; with that we do get reasonable timings within PETSc (except for the user callbacks mentioned above), so the two-barrier model is not needed. > > Best regards, > Karli > > > >>>> >>>> Might this incur an extra overhead checking the device? Or will it always be true that if there are no outstanding kernels it will not go to the GPU and the check will return immediately? >>> >>> If we want to have a two barrier model, I propose we log the timing for waiting at the first barrier separately. >>>> >>>> Barry >>>> >>>>> >>>>> Best regards, >>>>> Karli >>>

1 0

Re: [petsc-dev] PETSc issue I cannot post combine WaitForCUDA(); inside PetscLogGpuTimeEnd();
by Karl Rupp 29 Aug '20

29 Aug '20

>>>>> ï¿½Since we cannot post issues (reported here >>>>> https://forum.gitlab.com/t/creating-new-issue-gives-cannot-create-issue-get…) >>>>> here is my issue so I don't forget it. >>>>> ï¿½I think >>>>> err ï¿½= WaitForCUDA();CHKERRCUDA(err); >>>>> ierr = PetscLogGpuTimeEnd();CHKERRQ(ierr); >>>>> should be changed to include WaitForCUDA() actually WaitForDevice() >>>>> inside the PetscLogGpuTimeEnd(). >>>>> Currently sometimes the WaitForCUDA() is missing in a few places >>>>> resulting in bad timing. >>>>> Also some _SeqCUDA() don't have the PetscLogGpuTimeEnd() and need >>>>> to be fixed. >>>>> The current model is a maintenance nightmare. >>>>> Does anyone see a problem with making this change? >>>> >>>> I'm fine with this change, as the maintenance benefits outweigh the >>>> performance cost for typical use cases. >>>> >>>> I propose to also add the WaitForDevice(); at >>>> PetscLogGpuTimeBegin(). This will ensure that no previous GPU kernel >>>> executions spill over into the timed section. > > ï¿½ Karl, > > ï¿½ ï¿½When synchronization is turned on the precious GPU kernels should > always have their own WaitForDevice(), so are you concerned about buggy > code that does not includeï¿½WaitForDevice? I'm primarily thinking of user callback routines here. For example, a FormFunction provided by the user that is running some GPU kernels. We have no guarantee that these user kernels have completed before entering the timed sections inside PETSc, so the logs will be skewed to report an unusually slow kernel in PETSc (the one right after the user form function). Arguably we could add a WaitForDevice() after user callback invocations. I didn't think of the WaitForDevice() after each kernel call in PETSc; with that we do get reasonable timings within PETSc (except for the user callbacks mentioned above), so the two-barrier model is not needed. Best regards, Karli > >>> >>> ï¿½Might this incur an extra overhead checking the device? Or will it >>> always be true that if there are no outstanding kernels it will not >>> go to the GPU and the check will return immediately? >> >> If we want to have a two barrier model, I propose we log the timing >> for waiting at the first barrier separately. >>> >>> Barry >>> >>>> >>>> Best regards, >>>> Karli >> >

1 0

Re: [petsc-dev] MacOS firewall annoyance while running tests + solution
by Matthew Knepley 28 Aug '20

28 Aug '20

On Fri, Aug 28, 2020 at 11:44 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > Likely firewall is disabled. > > I've recently rebuild a bunch of boxes - and don't remember explicity > disabling firewall. [well I did something to enable ssh - maybe that was > disabling firewall] > > I see the firewall is disabled on all of them > You are right, as always. Thanks, Matt > balay@ypro ~ % defaults read /Library/Preferences/com.apple.alf > globalstate > 0 > balay@ypro ~ % > > Ref: > https://raymii.org/s/snippets/OS_X_-_Turn_firewall_on_or_off_from_the_comma… > > BTW: Perhaps the following setting will prevent popups? [but it might > break stuff that need network?] > > sudo defaults write /Library/Preferences/com.apple.alf globalstate -int 2 > > Satish > > On Fri, 28 Aug 2020, Matthew Knepley wrote: > > > Which OS is this? It does not happen on my Catalina. > > > > Thanks, > > > > Matt > > > > On Fri, Aug 28, 2020 at 10:52 PM Barry Smith <bsmith(a)petsc.dev> wrote: > > > > > > > > > > > On Aug 28, 2020, at 3:47 PM, Jed Brown <jed(a)jedbrown.org> wrote: > > > > > > "Hapla Vaclav" <vaclav.hapla(a)erdw.ethz.ch> writes: > > > > > > On MacOS, maybe you also have lots of firewall popups > > > appearing/disappearing when running tests like > > > Do you want the application "ex29" to accept incoming network > connections? > > > > > > > > > Is there a way to express that the application does not need (should > not > > > accept) incoming connections? > > > > > > > > > Yes, this also seems to work: > > > > > > sudo $$FW --block $$APP > > > > > > instead of > > > > > > sudo $$FW --unblock $$APP > > > > > > The parallel program still runs correctly to conclusion without the > popup. > > > > > > So my conclusion is that at listen() or some later system call it > always > > > pops up the window (unless the user as already blocked or unblocked the > > > executable) without regard to whether an outside (from the machine) > > > connection to the process is attempted. > > > > > > The routine has an undocumented option -a <listen or accept> when you > run > > > with > > > > > > /usr/libexec/ApplicationFirewall/socketfilterfw -d -a accept > > > > > > it prints ASKWHENACCEPT which seems to indicate it will delay the > popup > > > until an accept is called but I can't confirm this because the debugger > > > never stops in accept on one process but the popup still comes up so > this > > > argument may be ignored. > > > > > > If I were Junchao I would not do the popup until the code tried to > > > accepted an EXTERNAL connection (the lazy evaluation) but I cannot get > it > > > to behave this way. > > > > > > If I disconnect from the network I still get the popups. > > > > > > The pop up is asynchronous also, when the popup is still up the program > > > keeps run (even in parallel) and ends normally. Then the popup > disappears. > > > > > > Apple could make this friendly without hurting security but then Apple > > > never cared about external developers for the Mac. > > > > > > > > > > > > Normalizing sudo during build/testing seems really bad. > > > > > > > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] MacOS firewall annoyance while running tests + solution
by Satish Balay 28 Aug '20

28 Aug '20

Likely firewall is disabled. I've recently rebuild a bunch of boxes - and don't remember explicity disabling firewall. [well I did something to enable ssh - maybe that was disabling firewall] I see the firewall is disabled on all of them balay@ypro ~ % defaults read /Library/Preferences/com.apple.alf globalstate 0 balay@ypro ~ % Ref: https://raymii.org/s/snippets/OS_X_-_Turn_firewall_on_or_off_from_the_comma… BTW: Perhaps the following setting will prevent popups? [but it might break stuff that need network?] sudo defaults write /Library/Preferences/com.apple.alf globalstate -int 2 Satish On Fri, 28 Aug 2020, Matthew Knepley wrote: > Which OS is this? It does not happen on my Catalina. > > Thanks, > > Matt > > On Fri, Aug 28, 2020 at 10:52 PM Barry Smith <bsmith(a)petsc.dev> wrote: > > > > > > > On Aug 28, 2020, at 3:47 PM, Jed Brown <jed(a)jedbrown.org> wrote: > > > > "Hapla Vaclav" <vaclav.hapla(a)erdw.ethz.ch> writes: > > > > On MacOS, maybe you also have lots of firewall popups > > appearing/disappearing when running tests like > > Do you want the application "ex29" to accept incoming network connections? > > > > > > Is there a way to express that the application does not need (should not > > accept) incoming connections? > > > > > > Yes, this also seems to work: > > > > sudo $$FW --block $$APP > > > > instead of > > > > sudo $$FW --unblock $$APP > > > > The parallel program still runs correctly to conclusion without the popup. > > > > So my conclusion is that at listen() or some later system call it always > > pops up the window (unless the user as already blocked or unblocked the > > executable) without regard to whether an outside (from the machine) > > connection to the process is attempted. > > > > The routine has an undocumented option -a <listen or accept> when you run > > with > > > > /usr/libexec/ApplicationFirewall/socketfilterfw -d -a accept > > > > it prints ASKWHENACCEPT which seems to indicate it will delay the popup > > until an accept is called but I can't confirm this because the debugger > > never stops in accept on one process but the popup still comes up so this > > argument may be ignored. > > > > If I were Junchao I would not do the popup until the code tried to > > accepted an EXTERNAL connection (the lazy evaluation) but I cannot get it > > to behave this way. > > > > If I disconnect from the network I still get the popups. > > > > The pop up is asynchronous also, when the popup is still up the program > > keeps run (even in parallel) and ends normally. Then the popup disappears. > > > > Apple could make this friendly without hurting security but then Apple > > never cared about external developers for the Mac. > > > > > > > > Normalizing sudo during build/testing seems really bad. > > > > > > > >

1 0

Re: [petsc-dev] MacOS firewall annoyance while running tests + solution
by Matthew Knepley 28 Aug '20

28 Aug '20

Which OS is this? It does not happen on my Catalina. Thanks, Matt On Fri, Aug 28, 2020 at 10:52 PM Barry Smith <bsmith(a)petsc.dev> wrote: > > > On Aug 28, 2020, at 3:47 PM, Jed Brown <jed(a)jedbrown.org> wrote: > > "Hapla Vaclav" <vaclav.hapla(a)erdw.ethz.ch> writes: > > On MacOS, maybe you also have lots of firewall popups > appearing/disappearing when running tests like > Do you want the application "ex29" to accept incoming network connections? > > > Is there a way to express that the application does not need (should not > accept) incoming connections? > > > Yes, this also seems to work: > > sudo $$FW --block $$APP > > instead of > > sudo $$FW --unblock $$APP > > The parallel program still runs correctly to conclusion without the popup. > > So my conclusion is that at listen() or some later system call it always > pops up the window (unless the user as already blocked or unblocked the > executable) without regard to whether an outside (from the machine) > connection to the process is attempted. > > The routine has an undocumented option -a <listen or accept> when you run > with > > /usr/libexec/ApplicationFirewall/socketfilterfw -d -a accept > > it prints ASKWHENACCEPT which seems to indicate it will delay the popup > until an accept is called but I can't confirm this because the debugger > never stops in accept on one process but the popup still comes up so this > argument may be ignored. > > If I were Junchao I would not do the popup until the code tried to > accepted an EXTERNAL connection (the lazy evaluation) but I cannot get it > to behave this way. > > If I disconnect from the network I still get the popups. > > The pop up is asynchronous also, when the popup is still up the program > keeps run (even in parallel) and ends normally. Then the popup disappears. > > Apple could make this friendly without hurting security but then Apple > never cared about external developers for the Mac. > > > > Normalizing sudo during build/testing seems really bad. > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] MacOS firewall annoyance while running tests + solution
by Barry Smith 28 Aug '20

28 Aug '20

> On Aug 28, 2020, at 3:47 PM, Jed Brown <jed(a)jedbrown.org> wrote: > > "Hapla Vaclav" <vaclav.hapla(a)erdw.ethz.ch> writes: > >> On MacOS, maybe you also have lots of firewall popups appearing/disappearing when running tests like >> Do you want the application "ex29" to accept incoming network connections? > > Is there a way to express that the application does not need (should not accept) incoming connections? Yes, this also seems to work: sudo $$FW --block $$APP instead of sudo $$FW --unblock $$APP The parallel program still runs correctly to conclusion without the popup. So my conclusion is that at listen() or some later system call it always pops up the window (unless the user as already blocked or unblocked the executable) without regard to whether an outside (from the machine) connection to the process is attempted. The routine has an undocumented option -a <listen or accept> when you run with /usr/libexec/ApplicationFirewall/socketfilterfw -d -a accept it prints ASKWHENACCEPT which seems to indicate it will delay the popup until an accept is called but I can't confirm this because the debugger never stops in accept on one process but the popup still comes up so this argument may be ignored. If I were Junchao I would not do the popup until the code tried to accepted an EXTERNAL connection (the lazy evaluation) but I cannot get it to behave this way. If I disconnect from the network I still get the popups. The pop up is asynchronous also, when the popup is still up the program keeps run (even in parallel) and ends normally. Then the popup disappears. Apple could make this friendly without hurting security but then Apple never cared about external developers for the Mac. > > Normalizing sudo during build/testing seems really bad.

1 0