September 2018 - Nek5000-users - mailman3.cels.anl.gov

Re: [Nek5000-users] findpts
by nek5000-users＠lists.mcs.anl.gov 10 Sep '18

10 Sep '18

Dear Nek, When I look at the hemi.usr, the only subroutine it calls for interpolating velocities for list of points as shown below. call interp_nfld(fpts,fwrk,ndim,pts(1),pts(1+n),pts(2*n+1), $ n,iwk,rwk,INTP_NMAX,.true.,intp_h) Could you please explain more? What I want are processor number and ix iy and iz for each point. Thanks so much for your help. Kind regards, Jian

1 1

Re: [Nek5000-users] Proper setup for AMG solver
by nek5000-users＠lists.mcs.anl.gov 10 Sep '18

10 Sep '18

Dear Nek experts, I wanted to add some remarks on my performance tests conducted last week mostly for the turbulent pipe flow at Re_b=5300 - The bad performance of the AMG solver was indeed due to an old version of amg_hypre, as Stefan mentioned already. For Re_b=5300 XXT and AMG showed the same performance. - Additionally, I could increase the performance by about 20% using the following settings: -- lxd=10 (instead of 12, for lx1=8) -- lx2=lx1-0 (PN-PN) -- lower tolerances for p=1e-5 (instead of 1e-8) and 1e-6 for velocity and passive scalars -- turning of projection except for the fields of low Prandtl number - Going from BDF3/EXT3 with a variable DT and targetCFL=0.5 to BDF2/OIFS with variable DT and targetCFL=0.5 resulted in a longer time / timestep (3x) but a also a larger average DT (6x). Assuming the collected statistics require a similar averaging time in wash-outs, BDF2/OIFS is advantageous (even for 10 passive scalars). Before this discussion, I was running at constant DT and collected statistics every 10th step. Now with a larger DT using characteristics, should I collect statistics for each step? - Regarding the filterWeight and filterCutoffRatio, I found out (with Stefan's help) that the divergence error in L2 norm (when using PN-PN) is affected by filterWeight. For the setup at Re_b=5300 a filterWeight of 27 seemed to be OK considering a divergence error in L2 norm of L2=3e-2, whereas a weight of 54 resulted in L2=1e-1. How these settings will affect the statistics, I am testing now. Thank you all for your help and suggestions. Cheers, Steffen

1 1

Re: [Nek5000-users] Proper setup for AMG solver
by nek5000-users＠lists.mcs.anl.gov 10 Sep '18

10 Sep '18

Hello Stefan & Paul, thanks for your suggestions. I use Nek5000 v17 (mver 17.0.4 in makenek). I have taken a closer look at the logfiles as suggested by Paul. It seems like I spend the most time for the scalar fields second most for the velocity fields and pressure solve is the smallest share. The numbers below are in seconds for a typical timestep without calculating statistics or writing out files at Re_b = 5300 (Re_t = u_t D / nu = 360, 768 cores) and Re_b = 37700 (Re_t = 2000, 6144 cores). I use projection for all 10 scalar fields. 360: Scalars done 0.064 Fluid done 0.039 U-PRES gmres 0.021 Step 0.127 2000: Scalars done 0.94 Fluid done 0.51 U-PRES gmres 0.21 Step 1.72 Paul, could you elaborate on why you would not use characteristics when running 10 scalar fields? In my short tests at Re_b=5300, it appears to be more time consuming per step but due to the increase in DT it is worth it to go to characteristics. The time per timestep for targetCFL=2.0 increases by a factor of 3 but DT is increased by a factor of 6. Besides, are such increased timesteps still small enough to capture the temporal evolution of the flow? When I am back on my workstation, I will create a different mesh with lx1=8, lxd=10 and run with the settings Stefan suggested. I expect to get a significant speedup when using the lower tolerances also for velocity and scalars and changing to characteristics. I know about the limit of 350k elements for XXT as I have once commented out the part of the code where this is tested. Since for my setups AMG was always slower than XXT, I am thinking about sticking to XXT. Is there any other reason than your experience with AMG and XXT at large number of elements to enforce AMG? Best Regards, Steffen Message: 5 Date: Sat, 1 Sep 2018 17:10:54 +0200 From: nek5000-users(a)lists.mcs.anl.gov To: nek5000-users(a)lists.mcs.anl.gov <nek5000-users(a)lists.mcs.anl.gov> Subject: Re: [Nek5000-users] Proper setup for AMG solver Message-ID: <mailman.9805.1535814716.86639.nek5000-users(a)lists.mcs.anl.gov> Content-Type: text/plain; charset=utf-8 Try to use lx1=8/lxd=10 with a (potentially) finer mesh BDF2 + OIFS with a targetCFL=3.5 set dt = 0 (this will adjust dt to targetCFL) pressure tol = 1e-5 (residual projection turned on) 1e-6 for velocity and scalars (residual projection turned off) Note, a mesh with more than 350k elements requires AMG. The default parameters are fine. What version of Nek5000 are you using? Cheers, Stefan -----Original message----- > From:[email protected] <nek5000-users(a)lists.mcs.anl.gov> > Sent: Saturday 1st September 2018 16:50 > To: nek5000-users(a)lists.mcs.anl.gov > Subject: [Nek5000-users] Proper setup for AMG solver > > Dear Nek users & experts, > > I am currently running a turbulent pipe flow very similar to the simulations from El Khoury et al 2013. > > Additionally, I solve for 10 thermal fields being treated as passive scalars. > > The Reynolds number is the same as the highest in El Khoury (2013), Re_b = 37700. > > As I am using the relaxation term filtering (RT-Filter), I have a slightly lower resolution of about 250,000 elements at N=11 (5 times less than El Khoury). > > As the simulation is still very heavy, I have been looking into ways for speeding it up. > I found some good suggestions here: > http://nek5000.github.io/NekDoc/faq.html?highlight=amg#computational-speed > > and here (older version?) > http://nek5000.github.io/NekDoc/large_scale.html > > > However, I have some questions regarding these suggestions. > 1) Dealiasing: > Usually I use lxd = 3/2*lx1. Can I lower that or even use lxd=lx1? > > 2) Tolerances: > I have tested to reduce the tolerance for pressure from 1e-8 to 5e-5 for a run at Re_b=5300 without any significant speedup. Would you consider 5e-5 for pressure accurate enough for evaluating statistics like turbulent kinetic energy budgets, Reynolds shear > stress budgets or budget of turbulent heat fluxes? > > 3) Time discretisation: BDF2 and OIFS with Courant=2-5 > If I go from BDF3/EXT3 at C=0.5 to BDF2/OIFS at C=5.0, will I not miss high frequency fluctuations in time, since DT is much larger? > > 4) AMG instead of XXT: > I have tested AMG instead of XXT for both Re_b=5300 and Re_b=37700 without any speedup. Time/timestep is even higher with AMG. My workflow looks like this > 4.1) Set SEMG_AMG in the par file. > 4.2) Run the simulation once to dump amg files. > 4.3) Run amg_hypre (Here I do not know which options to choose, thus I have only uses default settings) > 4.4) Run the simulation. > Maybe I should choose different options for amg_hypre, or should I rather use the amg_matlab2 tools? For the matlab tools I have not found an explanation on how to use them. > > > I am grateful for any advice regarding these aspects. > > Best Regards, > Steffen > > > > _______________________________________________ > Nek5000-users mailing list > Nek5000-users(a)lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users ------------------------------ Message: 6 Date: Sat, 1 Sep 2018 15:15:29 +0000 From: nek5000-users(a)lists.mcs.anl.gov To: "nek5000-users(a)lists.mcs.anl.gov" <nek5000-users(a)lists.mcs.anl.gov> Subject: Re: [Nek5000-users] Proper setup for AMG solver Message-ID: <mailman.9806.1535814932.86639.nek5000-users(a)lists.mcs.anl.gov> Content-Type: text/plain; charset="us-ascii" How much time are you spending in your scalar fields? Do you have projection turned on for all these fields? grep tep logfile will tell you how much time per step grep gmr logfile will tell you how much time in the pressure on each step What's left over is mostly passive scalar, unless you are using characteristics. I would not recommend characteristics when running 10 scalar fields. Paul ________________________________ From: Nek5000-users <nek5000-users-bounces(a)lists.mcs.anl.gov> on behalf of nek5000-users(a)lists.mcs.anl.gov <nek5000-users(a)lists.mcs.anl.gov> Sent: Saturday, September 1, 2018 9:50:38 AM To: nek5000-users(a)lists.mcs.anl.gov Subject: [Nek5000-users] Proper setup for AMG solver Dear Nek users & experts, I am currently running a turbulent pipe flow very similar to the simulations from El Khoury et al 2013. Additionally, I solve for 10 thermal fields being treated as passive scalars. The Reynolds number is the same as the highest in El Khoury (2013), Re_b = 37700. As I am using the relaxation term filtering (RT-Filter), I have a slightly lower resolution of about 250,000 elements at N=11 (5 times less than El Khoury). As the simulation is still very heavy, I have been looking into ways for speeding it up. I found some good suggestions here: http://nek5000.github.io/NekDoc/faq.html?highlight=amg#computational-speed and here (older version?) http://nek5000.github.io/NekDoc/large_scale.html However, I have some questions regarding these suggestions. 1) Dealiasing: Usually I use lxd = 3/2*lx1. Can I lower that or even use lxd=lx1? 2) Tolerances: I have tested to reduce the tolerance for pressure from 1e-8 to 5e-5 for a run at Re_b=5300 without any significant speedup. Would you consider 5e-5 for pressure accurate enough for evaluating statistics like turbulent kinetic energy budgets, Reynolds shear stress budgets or budget of turbulent heat fluxes? 3) Time discretisation: BDF2 and OIFS with Courant=2-5 If I go from BDF3/EXT3 at C=0.5 to BDF2/OIFS at C=5.0, will I not miss high frequency fluctuations in time, since DT is much larger? 4) AMG instead of XXT: I have tested AMG instead of XXT for both Re_b=5300 and Re_b=37700 without any speedup. Time/timestep is even higher with AMG. My workflow looks like this 4.1) Set SEMG_AMG in the par file. 4.2) Run the simulation once to dump amg files. 4.3) Run amg_hypre (Here I do not know which options to choose, thus I have only uses default settings) 4.4) Run the simulation. Maybe I should choose different options for amg_hypre, or should I rather use the amg_matlab2 tools? For the matlab tools I have not found an explanation on how to use them. I am grateful for any advice regarding these aspects. Best Regards, Steffen

1 3

Parameter settings on filterWeight and filterCutoffRatio
by nek5000-users＠lists.mcs.anl.gov 10 Sep '18

10 Sep '18

Hi neks, Recently I am working on an LES simulation in an annular pipe to see how transverse velocity goes with Re increasing. But I am quite wondering how I should set filterWeight and filterCutoffRatio in my example while in the turbChannel demo they are selected to 10 an 0.9 respectively as proposed value. They should be sensitive parameters in LES simulation, don't they? Is there any references concerned about them to help me see them more clearly? If it is convenient for you could you please reply me a url to the reference? Thank you so much for your assistence. best regard --

1 2

findpts
by nek5000-users＠lists.mcs.anl.gov 09 Sep '18

09 Sep '18

Hi nek, I wrote before in terms of findpts subroutine and tried some code. Unfortunately, I failed to get it right. May I attach my code here and please help me to sort it. I want to map the given set of arbitrary points xyz(ldim,n) below to the local coordinates in the mesh (velocity grid). Currently, I stop at how to use findpts results to return ix, iy, iz and iel (element number). Can anyone help me finish it? subroutine interp_p(ielll,ixx,iyy,izz,xyz,n) !find elememt and processor for points xyz include 'SIZE' include 'TOTAL' real xyz(ldim,n), r(n), s(n), s(n) logical ifjac,ifpts integer rcodee(n), ielll(n), nidd(n) integer nfail, inn real dist(lpart) real rst(lpart*ldim) parameter(nmax=lpart,nfldmax=ldim) common /rv_intp/ pts(ldim*nmax) common /iv_intp/ ihandle integer icalld,e save icalld data icalld /0/ nfail = 0 nxyz = nx1*ny1*nz1 ntot = nxyz*nelt if (n.gt.nmax) call exitti ('ABORT: interp_p() n > nmax!$',n) if (nelgt.ne.nelgv) call exitti $ ('ABORT: interp_p() nelgt.ne.nelgv not yet supported!$',nelgv) do i=1,n ! ? not moving -> save? pts(i) = xyz(1,i) pts(i + n) = xyz(2,i) if (if3d) pts(i + n*2) = xyz(3,i) enddo if (icalld.eq.0) then ! interpolation setup !? intpts_done(ih_intp_v)? icalld = 1 tolin = 1.e-8 call intpts_setup(tolin,ihandle) endif nflds = ndim ! number of fields to interpolate ! interpolate ifjac = .true. ! output transpose (of Jacobian) ifpts = .true. ! find points if(nio.eq.0) write(6,*) 'call findpts' call fintpts(ihandle,rcodee,1,ielll,1,rst,ndim,dist,1,pts(1),1,pts(n+1) $ ,1,pts(2*n+1),1,n) do inn = 1, n ! check return code if(rcodee(inn) .eq. 1) then nfail = nfail + 1 if(nfail .le. 5) write(6,'(a,1p4e15.7)') $ ' WARNING: point on boundary or outside the mesh xy[z]d^2: ', $ (pts(inn+k*n),k=0,ndim-1),dist(inn) endif elseif(rcodee(inn) .eq. 2) then nfail = nfail + 1 if(nfail .le. 5) write(6,'(a,1p3e15.7)') $ ' WARNING: point not within mesh xy[z]: !', $ (pts(inn+k*n),k=0,ndim-1) endif enddo Kind regards, Jian

1 1

Cylindrical Channel in Rectangular Prism using .re2
by nek5000-users＠lists.mcs.anl.gov 05 Sep '18

05 Sep '18

Hello Nek developers and users, I am currently developing a case in which a 3D rectangular domain containing a cylindrical channel located at its center. This channel will have a constant temperature BC. Does anyone have any experience with this or a similar setup, i.e. placing a cylinder in the mesh and applying boundary conditions to it? Please direct me to any literature referencing this process or help with some suggestions. All the examples I've found make use of .rea and I am using .re2. Any help is greatly appreciated. I've considered altering the mesh, like in the hillp example, however this leaves the question of BC's. Best Regards, Mario

1 0

Re: [Nek5000-users] Proper setup for AMG solver
by nek5000-users＠lists.mcs.anl.gov 02 Sep '18

02 Sep '18

Do you really need to solve for 10 scalars fields? I would turn off the residual projection for scalars. However it's hard to come up with a general recommondation. That's something you may want to experiment with. Turned it on/off and see what happens. How much faster was the pressure solve with XXT vs AMG? I have never seen a case with more than 10-15%. Based on my experience the crossing point (where AMG beats XXT) is often somewhere between 300-500k elements. The work scales O(N^5/3) vs O(N). Also, the xxt setup timings can be significant for larger element counts. For OIFS I would either use a targetCFL = 1.9 or 3.9 (1 or 2 RK4 substep). Are you using a variable dt? If your time step size is small enough to be accurate depends heavily on what you are interested in. I have done similiar pipe and channel flows simulations using different time step sizes and the results (first and second order statistics) where more or less the same. Again, that's something you need to play with to be sure it's true for your case. Please contact me off-list. I am sure there are ways to speed-up your case ;) Cheers, Stefan -----Original message----- > From:[email protected] <nek5000-users(a)lists.mcs.anl.gov> > Sent: Sunday 2nd September 2018 12:06 > To: nek5000-users(a)lists.mcs.anl.gov > Subject: Re: [Nek5000-users] Proper setup for AMG solver > > Hello Stefan & Paul, > > thanks for your suggestions. I use Nek5000 v17 (mver 17.0.4 in makenek). > > I have taken a closer look at the logfiles as suggested by Paul. It seems like I spend the most time for the scalar fields second most for the velocity fields and pressure solve is the smallest share. The numbers below are in seconds for a typical timestep without calculating statistics or writing out files at Re_b = 5300 (Re_t = u_t D / nu = 360, 768 cores) and Re_b = 37700 (Re_t = 2000, 6144 cores). > I use projection for all 10 scalar fields. > > 360: > Scalars done 0.064 > Fluid done 0.039 > U-PRES gmres 0.021 > Step 0.127 > > 2000: > Scalars done 0.94 > Fluid done 0.51 > U-PRES gmres 0.21 > Step 1.72 > > > Paul, could you elaborate on why you would not use characteristics when running 10 scalar fields? > In my short tests at Re_b=5300, it appears to be more time consuming per step but due to the increase in DT it is worth it to go to characteristics. The time per timestep for targetCFL=2.0 increases by a factor of 3 but DT is increased by a factor of 6. > Besides, are such increased timesteps still small enough to capture the temporal evolution of the flow? > > > When I am back on my workstation, I will create a different mesh with lx1=8, lxd=10 and run with the settings Stefan suggested. I expect to get a significant speedup when using the lower tolerances also for velocity and scalars and changing to characteristics. > > I know about the limit of 350k elements for XXT as I have once commented out the part of the code where this is tested. > Since for my setups AMG was always slower than XXT, I am thinking about sticking to XXT. > Is there any other reason than your experience with AMG and XXT at large number of elements to enforce AMG? > > > Best Regards, > Steffen > > > > Message: 5 > Date: Sat, 1 Sep 2018 17:10:54 +0200 > From: nek5000-users(a)lists.mcs.anl.gov > To: nek5000-users(a)lists.mcs.anl.gov <nek5000-users(a)lists.mcs.anl.gov> > Subject: Re: [Nek5000-users] Proper setup for AMG solver > Message-ID: > <mailman.9805.1535814716.86639.nek5000-users(a)lists.mcs.anl.gov> > Content-Type: text/plain; charset=utf-8 > > Try to use > > lx1=8/lxd=10 with a (potentially) finer mesh > BDF2 + OIFS with a targetCFL=3.5 > set dt = 0 (this will adjust dt to targetCFL) > pressure tol = 1e-5 (residual projection turned on) 1e-6 for velocity and scalars (residual projection turned off) > > Note, a mesh with more than 350k elements requires AMG. The default parameters are fine. > > What version of Nek5000 are you using? > > Cheers, > Stefan > > > -----Original message----- > > From:[email protected] <nek5000-users(a)lists.mcs.anl.gov> > > Sent: Saturday 1st September 2018 16:50 > > To: nek5000-users(a)lists.mcs.anl.gov > > Subject: [Nek5000-users] Proper setup for AMG solver > > > > Dear Nek users & experts, > > > > I am currently running a turbulent pipe flow very similar to the simulations from El Khoury et al 2013. > > > > Additionally, I solve for 10 thermal fields being treated as passive scalars. > > > > The Reynolds number is the same as the highest in El Khoury (2013), Re_b = 37700. > > > > As I am using the relaxation term filtering (RT-Filter), I have a slightly lower resolution of about 250,000 elements at N=11 (5 times less than El Khoury). > > > > As the simulation is still very heavy, I have been looking into ways for speeding it up. > > I found some good suggestions here: > > http://nek5000.github.io/NekDoc/faq.html?highlight=amg#computational-speed > > > > and here (older version?) > > http://nek5000.github.io/NekDoc/large_scale.html > > > > > > However, I have some questions regarding these suggestions. > > 1) Dealiasing: > > Usually I use lxd = 3/2*lx1. Can I lower that or even use lxd=lx1? > > > > 2) Tolerances: > > I have tested to reduce the tolerance for pressure from 1e-8 to 5e-5 for a run at Re_b=5300 without any significant speedup. Would you consider 5e-5 for pressure accurate enough for evaluating statistics like turbulent kinetic energy budgets, Reynolds shear > > stress budgets or budget of turbulent heat fluxes? > > > > 3) Time discretisation: BDF2 and OIFS with Courant=2-5 > > If I go from BDF3/EXT3 at C=0.5 to BDF2/OIFS at C=5.0, will I not miss high frequency fluctuations in time, since DT is much larger? > > > > 4) AMG instead of XXT: > > I have tested AMG instead of XXT for both Re_b=5300 and Re_b=37700 without any speedup. Time/timestep is even higher with AMG. My workflow looks like this > > 4.1) Set SEMG_AMG in the par file. > > 4.2) Run the simulation once to dump amg files. > > 4.3) Run amg_hypre (Here I do not know which options to choose, thus I have only uses default settings) > > 4.4) Run the simulation. > > Maybe I should choose different options for amg_hypre, or should I rather use the amg_matlab2 tools? For the matlab tools I have not found an explanation on how to use them. > > > > > > I am grateful for any advice regarding these aspects. > > > > Best Regards, > > Steffen > > > > > > > > _______________________________________________ > > Nek5000-users mailing list > > Nek5000-users(a)lists.mcs.anl.gov > > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > > > ------------------------------ > > Message: 6 > Date: Sat, 1 Sep 2018 15:15:29 +0000 > From: nek5000-users(a)lists.mcs.anl.gov > To: "nek5000-users(a)lists.mcs.anl.gov" > <nek5000-users(a)lists.mcs.anl.gov> > Subject: Re: [Nek5000-users] Proper setup for AMG solver > Message-ID: > <mailman.9806.1535814932.86639.nek5000-users(a)lists.mcs.anl.gov> > Content-Type: text/plain; charset="us-ascii" > > > How much time are you spending in your scalar fields? > > > Do you have projection turned on for all these fields? > > > grep tep logfile > > > will tell you how much time per step > > > grep gmr logfile will tell you how much time in the pressure on each step > > > What's left over is mostly passive scalar, unless you are using characteristics. > > > I would not recommend characteristics when running 10 scalar fields. > > > Paul > > > ________________________________ > From: Nek5000-users <nek5000-users-bounces(a)lists.mcs.anl.gov> on behalf of nek5000-users(a)lists.mcs.anl.gov <nek5000-users(a)lists.mcs.anl.gov> > Sent: Saturday, September 1, 2018 9:50:38 AM > To: nek5000-users(a)lists.mcs.anl.gov > Subject: [Nek5000-users] Proper setup for AMG solver > > > Dear Nek users & experts, > > I am currently running a turbulent pipe flow very similar to the simulations from El Khoury et al 2013. > Additionally, I solve for 10 thermal fields being treated as passive scalars. > > The Reynolds number is the same as the highest in El Khoury (2013), Re_b = 37700. > As I am using the relaxation term filtering (RT-Filter), I have a slightly lower resolution of about 250,000 elements at N=11 (5 times less than El Khoury). > > As the simulation is still very heavy, I have been looking into ways for speeding it up. > I found some good suggestions here: > http://nek5000.github.io/NekDoc/faq.html?highlight=amg#computational-speed > > and here (older version?) > http://nek5000.github.io/NekDoc/large_scale.html > > > However, I have some questions regarding these suggestions. > 1) Dealiasing: > Usually I use lxd = 3/2*lx1. Can I lower that or even use lxd=lx1? > > 2) Tolerances: > I have tested to reduce the tolerance for pressure from 1e-8 to 5e-5 for a run at Re_b=5300 without any significant speedup. Would you consider 5e-5 for pressure accurate enough for evaluating statistics like turbulent kinetic energy budgets, Reynolds shear stress budgets or budget of turbulent heat fluxes? > > 3) Time discretisation: BDF2 and OIFS with Courant=2-5 > If I go from BDF3/EXT3 at C=0.5 to BDF2/OIFS at C=5.0, will I not miss high frequency fluctuations in time, since DT is much larger? > > 4) AMG instead of XXT: > I have tested AMG instead of XXT for both Re_b=5300 and Re_b=37700 without any speedup. Time/timestep is even higher with AMG. My workflow looks like this > 4.1) Set SEMG_AMG in the par file. > 4.2) Run the simulation once to dump amg files. > 4.3) Run amg_hypre (Here I do not know which options to choose, thus I have only uses default settings) > 4.4) Run the simulation. > Maybe I should choose different options for amg_hypre, or should I rather use the amg_matlab2 tools? For the matlab tools I have not found an explanation on how to use them. > > > I am grateful for any advice regarding these aspects. > > Best Regards, > Steffen > > > >

1 0

re2torea
by nek5000-users＠lists.mcs.anl.gov 01 Sep '18

01 Sep '18

Hi , I am trying to convert re2 to rea file, but the following error appears: name.re2\ byte_read() :: fopen failure2! Does anybody know what the problem is? Thanks, Ali

1 0

Proper setup for AMG solver
by nek5000-users＠lists.mcs.anl.gov 01 Sep '18

01 Sep '18

Dear Nek users & experts, I am currently running a turbulent pipe flow very similar to the simulations from El Khoury et al 2013. Additionally, I solve for 10 thermal fields being treated as passive scalars. The Reynolds number is the same as the highest in El Khoury (2013), Re_b = 37700. As I am using the relaxation term filtering (RT-Filter), I have a slightly lower resolution of about 250,000 elements at N=11 (5 times less than El Khoury). As the simulation is still very heavy, I have been looking into ways for speeding it up. I found some good suggestions here: http://nek5000.github.io/NekDoc/faq.html?highlight=amg#computational-speed and here (older version?) http://nek5000.github.io/NekDoc/large_scale.html However, I have some questions regarding these suggestions. 1) Dealiasing: Usually I use lxd = 3/2*lx1. Can I lower that or even use lxd=lx1? 2) Tolerances: I have tested to reduce the tolerance for pressure from 1e-8 to 5e-5 for a run at Re_b=5300 without any significant speedup. Would you consider 5e-5 for pressure accurate enough for evaluating statistics like turbulent kinetic energy budgets, Reynolds shear stress budgets or budget of turbulent heat fluxes? 3) Time discretisation: BDF2 and OIFS with Courant=2-5 If I go from BDF3/EXT3 at C=0.5 to BDF2/OIFS at C=5.0, will I not miss high frequency fluctuations in time, since DT is much larger? 4) AMG instead of XXT: I have tested AMG instead of XXT for both Re_b=5300 and Re_b=37700 without any speedup. Time/timestep is even higher with AMG. My workflow looks like this 4.1) Set SEMG_AMG in the par file. 4.2) Run the simulation once to dump amg files. 4.3) Run amg_hypre (Here I do not know which options to choose, thus I have only uses default settings) 4.4) Run the simulation. Maybe I should choose different options for amg_hypre, or should I rather use the amg_matlab2 tools? For the matlab tools I have not found an explanation on how to use them. I am grateful for any advice regarding these aspects. Best Regards, Steffen

1 1

Re: [Nek5000-users] Proper setup for AMG solver
by nek5000-users＠lists.mcs.anl.gov 01 Sep '18

01 Sep '18

Try to use lx1=8/lxd=10 with a (potentially) finer mesh BDF2 + OIFS with a targetCFL=3.5 set dt = 0 (this will adjust dt to targetCFL) pressure tol = 1e-5 (residual projection turned on) 1e-6 for velocity and scalars (residual projection turned off) Note, a mesh with more than 350k elements requires AMG. The default parameters are fine. What version of Nek5000 are you using? Cheers, Stefan -----Original message----- > From:[email protected] <nek5000-users(a)lists.mcs.anl.gov> > Sent: Saturday 1st September 2018 16:50 > To: nek5000-users(a)lists.mcs.anl.gov > Subject: [Nek5000-users] Proper setup for AMG solver > > Dear Nek users & experts, > > I am currently running a turbulent pipe flow very similar to the simulations from El Khoury et al 2013. > > Additionally, I solve for 10 thermal fields being treated as passive scalars. > > The Reynolds number is the same as the highest in El Khoury (2013), Re_b = 37700. > > As I am using the relaxation term filtering (RT-Filter), I have a slightly lower resolution of about 250,000 elements at N=11 (5 times less than El Khoury). > > As the simulation is still very heavy, I have been looking into ways for speeding it up. > I found some good suggestions here: > http://nek5000.github.io/NekDoc/faq.html?highlight=amg#computational-speed > > and here (older version?) > http://nek5000.github.io/NekDoc/large_scale.html > > > However, I have some questions regarding these suggestions. > 1) Dealiasing: > Usually I use lxd = 3/2*lx1. Can I lower that or even use lxd=lx1? > > 2) Tolerances: > I have tested to reduce the tolerance for pressure from 1e-8 to 5e-5 for a run at Re_b=5300 without any significant speedup. Would you consider 5e-5 for pressure accurate enough for evaluating statistics like turbulent kinetic energy budgets, Reynolds shear > stress budgets or budget of turbulent heat fluxes? > > 3) Time discretisation: BDF2 and OIFS with Courant=2-5 > If I go from BDF3/EXT3 at C=0.5 to BDF2/OIFS at C=5.0, will I not miss high frequency fluctuations in time, since DT is much larger? > > 4) AMG instead of XXT: > I have tested AMG instead of XXT for both Re_b=5300 and Re_b=37700 without any speedup. Time/timestep is even higher with AMG. My workflow looks like this > 4.1) Set SEMG_AMG in the par file. > 4.2) Run the simulation once to dump amg files. > 4.3) Run amg_hypre (Here I do not know which options to choose, thus I have only uses default settings) > 4.4) Run the simulation. > Maybe I should choose different options for amg_hypre, or should I rather use the amg_matlab2 tools? For the matlab tools I have not found an explanation on how to use them. > > > I am grateful for any advice regarding these aspects. > > Best Regards, > Steffen > > > > _______________________________________________ > Nek5000-users mailing list > Nek5000-users(a)lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users

1 0