December 2016 - Nek5000-users - mailman3.cels.anl.gov

Re: [Nek5000-users] Parallel speedup on supercomputer Tianhe-2
by nek5000-users＠lists.mcs.anl.gov 06 Dec '16

06 Dec '16

Dear Stefan, Thank you for your reply. There are 139056 elements and the polynomial order is 7 (lx1=8). I measure the solver time to compute the speedup. For example, the serial job takes 2564.59s and the same job with 24 tasks takes 302.18s. The speedup is about 8.5. This is on a single Tianhe-2 node. The speedup between nodes is quite good. I also test the code on my 36-core computer (Dual Xeon E5 18-Core). I can only get about 12 times speedup when I use 36 tasks. It is also about 1/3. Best regards, Wei XU From: nek5000-users(a)lists.mcs.anl.gov To: <nek5000-users(a)lists.mcs.anl.gov> Subject: Re: [Nek5000-users] Parallel speedup on supercomputer Tianhe-2 Message-ID: <mailman.7164.1480933423.3602.nek5000-users(a)lists.mcs.anl.gov> Content-Type: text/plain; charset="utf-8" What's your problem size (number of elements and polynomial order)? Let's assume t_MPI << t (this holds if your problem size is reasonably large). Even in this limit you don't get a linear intra-node speedup simply because Nek5000 is not purely compute bound and the cumulative memory bandwidth is saturated with N cores (N < total number of cores). Cheers, Stefan From: <nek5000-users-bounces(a)lists.mcs.anl.gov> on behalf of < nek5000-users(a)lists.mcs.anl.gov> Reply-To: <nek5000-users(a)lists.mcs.anl.gov> Date: Monday, December 5, 2016 at 6:12 AM To: <nek5000-users(a)lists.mcs.anl.gov> Subject: [Nek5000-users] Parallel speedup on supercomputer Tianhe-2 Dear Neks, I'm using Nek5000 to simulate turbulent Rayleigh-Benard convection, which is governed by the coupled Navier-Stokes equations and convective heat equation. I'm running the code on a supercomputer, Tianhe-2, located in Guangzhou, China. Each computer node in Tianhe-2 has 24 cores (2 Xeon E5 12-core CPUs) and 64GB memory. I find the speedup curve is not linear on a single node. For example, a 24-task job is only 8 times faster than the serial one. However, the performance with an increasing number of nodes is quite good. I don't know whether there is any parameter in nek500 that I can change in order to improve the speedup performance of the individual nodes. Thanks in advance! Best regards, Wei XU

1 1

Genbox periodic simple case
by nek5000-users＠lists.mcs.anl.gov 06 Dec '16

06 Dec '16

Hi JP, I had a similar issue with genbox when I wanted to create a simple mesh where periodic BCs are shared between different boxes. I haven't tested Paul's suggestion yet but I wrote a genbox inspired python script that did the job in my case. You can find it here: https://github.com/Steffen1989/code-snippets (genbox.py) The Input is very similar to genbox but you can specify a periodic BC shared between two boxes with P01, a second one with P02, ... and the same with internal BCs, i.e. E01, E02, ... This can create a 2d mesh, which is afterwards extruded with n2to3 in the third direction. Note that it has not been tested thoroughly but you are welcome to give it a try and contact me if you run into troubles. \Steffen

1 0

Maintain flow rate in angled geometry
by nek5000-users＠lists.mcs.anl.gov 05 Dec '16

05 Dec '16

Dear Neks, Recently I have been running test simulations of an angled square duct. A sketch is shown below. The shapes of the inlet and outlet are identical. The actual domain is longer than shown here. ____________________ | \ | _________________ \ \ \ \ \ \ \ \ / \ / Simulations with velocity inlet and outflow conditions worked well. Now I want to set up periodic boundary conditions at the inlet and outlet. I'm wondering how can I maintain a constant flow rate in this simulation? In the normal periodic channel flow simulation, setting parameters p54 and p55 ensures a constant mean velocity. But in my case, the flow changes direction. Does anyone know how to set constant flow rate/mean velocity in this type of simulation? Thank you very much in advance. Kind regards, Tony

1 0

Parallel speedup on supercomputer Tianhe-2
by nek5000-users＠lists.mcs.anl.gov 05 Dec '16

05 Dec '16

Dear Neks, I'm using Nek5000 to simulate turbulent Rayleigh-Benard convection, which is governed by the coupled Navier-Stokes equations and convective heat equation. I'm running the code on a supercomputer, Tianhe-2, located in Guangzhou, China. Each computer node in Tianhe-2 has 24 cores (2 Xeon E5 12-core CPUs) and 64GB memory. I find the speedup curve is not linear on a single node. For example, a 24-task job is only 8 times faster than the serial one. However, the performance with an increasing number of nodes is quite good. I don't know whether there is any parameter in nek500 that I can change in order to improve the speedup performance of the individual nodes. Thanks in advance! Best regards, Wei XU ---------- Wei XU Ph.D. Candidate Nano Science and Technology Program The Hong Kong University of Science and Technology

1 1

Genbox periodic simple case
by nek5000-users＠lists.mcs.anl.gov 04 Dec '16

04 Dec '16

Dear Users, I have used genbox for ages but now I need to split a channel in the x direction and cannot make genmap recognize the periodic boundary conditions. Can a periodic boundary condition be separated by a box? I need periodic boundary conditions in x and z. This is my .box: ------------------------- parameters.rea -3 1 # Left box Box1 -10 -10 -10 0.000 2.000 1.00 0.000 2.000 1.00 0.000 2.000 1.00 P ,E ,W ,W ,P ,P bc's # Right box Box2 -10 -10 -10 2.000 4.000 1.00 0.000 2.000 1.00 0.000 2.000 1.00 E ,P ,W ,W ,P ,P bc's -------------------------- I thought it should be really simple but I cannot get it to work... any help/insight? Regards, JP

1 1

Questions on turbChannel
by nek5000-users＠lists.mcs.anl.gov 02 Dec '16

02 Dec '16

Hello, I have a few questions on the turbChannel example. 1) the ‘eddy viscosity ‘ is computed in the function eddy_visc and called in a loop over the elements of mesh 1. Inside the function eddy_visc there is the following piece of code: if (e.eq.nelv) then ! planar avg and define nu_tau … ntot = nx1*ny1*nz1*nelv do i=1,ntot cdyn = 0 if (den(i,1).gt.0) cdyn = 0.5*num(i,1)/den(i,1) cdyn = 0.16*0.16 ! max(cdyn,0.) ! AS ALTERNATIVE, could clip ediff ediff(i,1) = param(2)+cdyn*dg2(i,1)*snrm(i,1) enddo endif eddy that stores the eddy viscosity is updated when the last element of mesh 1 is called. Is there any reason why it should be done in this manner? 2) I am still confused over the differences between ‘ifsplit’, ‘ifexplvis’ and ‘ifstrs’. From the archive and the documentation I understand the following (please correct me if I am wrong): * ‘ifexplvis’ is set to true when the user specified the viscosity coefficient in uservp. It does not required, however, the stress formation meaning the the diffusive term in the NS equations looks like \mu(x,t) \nabla^2 \vec{u}. * when ‘ifsplit’ is set to true, the viscosity is split in a explicit part and an implicit part. If this correct, how and where are defined the explicit and implicit parts? * ‘ifstrs’: when set to true, ‘ifexplvis’ and ‘if split’ do not have any effect. 3) could someone shed some light on the subroutine ‘set_ds_filt’ in turbChannel.usr, please? It seems to compute some sort of filter that is then used to compute M_{i,j} and L_{i,j} Thanks, Marco

1 2

Re: [Nek5000-users] Nek5000-users Digest, Vol 94, Issue 1
by nek5000-users＠lists.mcs.anl.gov 02 Dec '16

02 Dec '16

Hi Sandeep, Is your self-defined udiiff a non-dimensional one? If not, maybe you can try to set dimensionless properties in uservp subroutine, then the non-positive error may disappear. Cheers, Bolun ---------------------------------------------------------------- Bolun Xu University of Science and Technology of China Department of Modern Mechanics Hefei, Anhui, China -----邮件原件----- 发件人: nek5000-users-bounces(a)lists.mcs.anl.gov [mailto:[email protected]] 代表 nek5000-users-request(a)lists.mcs.anl.gov 发送时间: 2016年12月2日 4:18 收件人: nek5000-users(a)lists.mcs.anl.gov 主题: Nek5000-users Digest, Vol 94, Issue 1 Send Nek5000-users mailing list submissions to nek5000-users(a)lists.mcs.anl.gov To subscribe or unsubscribe via the World Wide Web, visit https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users or, via email, send a message with subject or body 'help' to nek5000-users-request(a)lists.mcs.anl.gov You can reach the person managing the list at nek5000-users-owner(a)lists.mcs.anl.gov When replying, please edit your Subject line so it is more specific than "Re: Contents of Nek5000-users digest..." Today's Topics: 1. uservp MHD error (nek5000-users(a)lists.mcs.anl.gov) ---------------------------------------------------------------------- Message: 1 Date: Thu, 1 Dec 2016 18:31:45 +0100 From: nek5000-users(a)lists.mcs.anl.gov To: nek5000-users(a)lists.mcs.anl.gov Subject: [Nek5000-users] uservp MHD error Message-ID: <mailman.7023.1480623493.3602.nek5000-users(a)lists.mcs.anl.gov> Content-Type: text/plain; charset="utf-8" Hello All, I want to run MHD case with different magnetic diffusivities in different region of the domain. Can we run MHD run with different magnetic diffusivities? Assuming that udiff=magnetic diffusivity (eta), in the uservp file I added the following 1.0000000E+00 p30 > 0 ==> properties set in uservp() subroutine uservp (ix,iy,iz,ieg) include 'SIZE' include 'TOTAL' include 'NEKUSE' if (ifield.eq.1) then ! velocity utrans = param(1) udiff = param(2) elseif (ifield.eq.ifldmhd) then ! B-field utrans = 1.0 R1 = sqrt(x*x+y*y); if(R1 .le. 2.0) then udiff= param(29) else udiff= 1000.0D0*param(29) endif endif return end I get an error done :: set initial conditions ERROR: Non-positive diffusivity ( 0.00 ) specified for field 2, group 0 element ERROR: Non-positive diffusivity ( 0.00 ) specified for field 2, group 0 element Attached are the .usr, .rea and the logfile When I set the p30=0.000 and just utrans=0.0 udiff=0.0 in the uservp, the simulation is running fine. Thank you in advance. Cheers, Sandeep

1 0

uservp MHD error
by nek5000-users＠lists.mcs.anl.gov 01 Dec '16

01 Dec '16

Hello All, I want to run MHD case with different magnetic diffusivities in different region of the domain. Can we run MHD run with different magnetic diffusivities? Assuming that udiff=magnetic diffusivity (eta), in the uservp file I added the following 1.0000000E+00 p30 > 0 ==> properties set in uservp() subroutine uservp (ix,iy,iz,ieg) include 'SIZE' include 'TOTAL' include 'NEKUSE' if (ifield.eq.1) then ! velocity utrans = param(1) udiff = param(2) elseif (ifield.eq.ifldmhd) then ! B-field utrans = 1.0 R1 = sqrt(x*x+y*y); if(R1 .le. 2.0) then udiff= param(29) else udiff= 1000.0D0*param(29) endif endif return end I get an error done :: set initial conditions ERROR: Non-positive diffusivity ( 0.00 ) specified for field 2, group 0 element ERROR: Non-positive diffusivity ( 0.00 ) specified for field 2, group 0 element Attached are the .usr, .rea and the logfile When I set the p30=0.000 and just utrans=0.0 udiff=0.0 in the uservp, the simulation is running fine. Thank you in advance. Cheers, Sandeep

1 1