Smooth Restart feature in Nek
Hi, I found a function in nek which would let you do a restart from 3 previous fld files. This is how I called the function: % ---- in usrchk: prefix = 'rs6' iosave = iostep iosize = 8 nfld = 3 call restart_save(iosave,iosize,nfld) . %---- in rea file : 3 PRESOLVE/RESTART OPTIONS ***** rs6sqr0.f0004 rs6sqr0.f0005 rs6sqr0.f0006 %----- This dumps out six rs6blah0.f***** and is overwritten once every 2*iostep I believe. When I try to restart using this, I still see that at the restart, the pressure values are off. I am guessing that I am not calling it the right way in the rea file. Is there a separate function for loading multiple restart files that I would have to call in usrchk ? Or is this behavior a by-product of SEM ? Thanks for any help. Regards Shriram
Hi Shriram, We're working on this ... should have it set by tomorrow. Paul On Sun, 8 Aug 2010, [email protected] wrote:
Hi,
I found a function in nek which would let you do a restart from 3 previous fld files. This is how I called the function:
% ---- in usrchk:
prefix = 'rs6' iosave = iostep iosize = 8 nfld = 3 call restart_save(iosave,iosize,nfld) . %---- in rea file :
3 PRESOLVE/RESTART OPTIONS ***** rs6sqr0.f0004 rs6sqr0.f0005 rs6sqr0.f0006 %-----
This dumps out six rs6blah0.f***** and is overwritten once every 2*iostep I believe. When I try to restart using this, I still see that at the restart, the pressure values are off. I am guessing that I am not calling it the right way in the rea file. Is there a separate function for loading multiple restart files that I would have to call in usrchk ?
Or is this behavior a by-product of SEM ?
Thanks for any help.
Regards Shriram
Shriram, Attached is a full-restart routine. You need to update the repo to use it. It seems to work ok - hopefully the remarks in the attached .usr file are clear. Paul On Mon, 9 Aug 2010, [email protected] wrote:
Hi Paul,
Thanks a lot. That would be very helpful.
Regards Shriram
Hi Paul, My apologies for the late reply. I just checked by making a trial case and it works. Thanks a lot . Regards Shriram
Hi all. First, i have a comment that may save some time to some users. If i am correct, in Paul's comments rstx.fld03 and rstx.fld04 are the two fields after the x.fld92. Thus if x.fld92 is the field at time step NT=9200 , rstx.fld03 and rstx.fld04 are at NT=9201 and NT=9202. This means that if you want to restart a simulation from NT=10000, you need to run it at least until NT=10002 ( using Torder=3). Is that correct? Second, i have a problem. I think I am able to restart the velocity smoothly, but i may have a problem with the temperature. Attached there are two figures of the perturbation energy (e2=<vx^2+vy^2>) and the temperature root mean square during the restart of an heated turbulent channel. The full restart clearly improves e2, while the temperature has the same behavior of the standard restart. Am I misled by the plots? Do i have to do anything particular for the restart of the scalars? Did anybody else have the same issue? Thank you for any help. francesco On 08/11/2010 03:03 PM, [email protected] wrote:
Shriram,
Attached is a full-restart routine. You need to update the repo to use it. It seems to work ok - hopefully the remarks in the attached .usr file are clear.
Paul
On Mon, 9 Aug 2010, [email protected] wrote:
Hi Paul,
Thanks a lot. That would be very helpful.
Regards Shriram
Is this with CVODE? -Stefan ----- Original Message ----- From: [email protected] To: [email protected] Sent: Fri, 24 Sep 2010 09:53:35 -0600 (GMT-06:00) Subject: Re: [Nek5000-users] Smooth Restart feature in Nek Hi all. First, i have a comment that may save some time to some users. If i am correct, in Paul's comments rstx.fld03 and rstx.fld04 are the two fields after the x.fld92. Thus if x.fld92 is the field at time step NT=9200 , rstx.fld03 and rstx.fld04 are at NT=9201 and NT=9202. This means that if you want to restart a simulation from NT=10000, you need to run it at least until NT=10002 ( using Torder=3). Is that correct? Second, i have a problem. I think I am able to restart the velocity smoothly, but i may have a problem with the temperature. Attached there are two figures of the perturbation energy (e2=<vx^2+vy^2>) and the temperature root mean square during the restart of an heated turbulent channel. The full restart clearly improves e2, while the temperature has the same behavior of the standard restart. Am I misled by the plots? Do i have to do anything particular for the restart of the scalars? Did anybody else have the same issue? Thank you for any help. francesco On 08/11/2010 03:03 PM, [email protected] wrote:
Shriram,
Attached is a full-restart routine. You need to update the repo to use it. It seems to work ok - hopefully the remarks in the attached .usr file are clear.
Paul
On Mon, 9 Aug 2010, [email protected] wrote:
Hi Paul,
Thanks a lot. That would be very helpful.
Regards Shriram
Hi Nek's :-) , Francesco, I also encounter this problem and I was on the edge of writing such an email when the mail-list forwarded yours. I work on stenotic flows (non-linear evolution of infinitesimal perturbations), and because of the long-time dynamics we wanna observe, restarting the simulations from previous time steps is not an option. I set various probes within the flow to plot the velocity. Just like you, I have no problem so far: the velocity restarts smoothly. However, I compute the total energy in usrchk as being the square of the L2-norm and I indeed also get a jump at the restart timestep when plotting this energy. Help (or at least a few hints on how to solve this) would be highly appreciated. Sincerely yours, JC
Hi Stefan, yes, the test was done with CVODE. @ JC , i have the jump in energy only with a standard restart. The energy restarts smoothly ( see e2.eps of the previous email ) if I use the subroutine "my_full_restart_load" that Paul posted here on 8/11/2010. Thanks everybody francesco On 09/28/2010 05:21 PM, [email protected] wrote:
Hi Nek's :-) ,
Francesco, I also encounter this problem and I was on the edge of writing such an email when the mail-list forwarded yours. I work on stenotic flows (non-linear evolution of infinitesimal perturbations), and because of the long-time dynamics we wanna observe, restarting the simulations from previous time steps is not an option.
I set various probes within the flow to plot the velocity. Just like you, I have no problem so far: the velocity restarts smoothly. However, I compute the total energy in usrchk as being the square of the L2-norm and I indeed also get a jump at the restart timestep when plotting this energy.
Help (or at least a few hints on how to solve this) would be highly appreciated. Sincerely yours,
JC _______________________________________________ Nek5000-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
You cannot do an accurate (taking the full BDF history) restart if you use CVODE. Stefan ----- Original Message ----- From: [email protected] To: [email protected] Sent: Tue, 28 Sep 2010 11:19:01 -0600 (GMT-06:00) Subject: Re: [Nek5000-users] Smooth Restart feature in Nek Hi Stefan, yes, the test was done with CVODE. @ JC , i have the jump in energy only with a standard restart. The energy restarts smoothly ( see e2.eps of the previous email ) if I use the subroutine "my_full_restart_load" that Paul posted here on 8/11/2010. Thanks everybody francesco On 09/28/2010 05:21 PM, [email protected] wrote:
Hi Nek's :-) ,
Francesco, I also encounter this problem and I was on the edge of writing such an email when the mail-list forwarded yours. I work on stenotic flows (non-linear evolution of infinitesimal perturbations), and because of the long-time dynamics we wanna observe, restarting the simulations from previous time steps is not an option.
I set various probes within the flow to plot the velocity. Just like you, I have no problem so far: the velocity restarts smoothly. However, I compute the total energy in usrchk as being the square of the L2-norm and I indeed also get a jump at the restart timestep when plotting this energy.
Help (or at least a few hints on how to solve this) would be highly appreciated. Sincerely yours,
JC _______________________________________________ Nek5000-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
_______________________________________________ Nek5000-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
thank you francesco On 09/28/2010 07:43 PM, [email protected] wrote:
You cannot do an accurate (taking the full BDF history) restart if you use CVODE. Stefan
----- Original Message ----- From: [email protected] To: [email protected] Sent: Tue, 28 Sep 2010 11:19:01 -0600 (GMT-06:00) Subject: Re: [Nek5000-users] Smooth Restart feature in Nek
Hi Stefan,
yes, the test was done with CVODE.
@ JC , i have the jump in energy only with a standard restart. The energy restarts smoothly ( see e2.eps of the previous email ) if I use the subroutine "my_full_restart_load" that Paul posted here on 8/11/2010.
Thanks everybody
francesco
On 09/28/2010 05:21 PM, [email protected] wrote:
Hi Nek's :-) ,
Francesco, I also encounter this problem and I was on the edge of writing such an email when the mail-list forwarded yours. I work on stenotic flows (non-linear evolution of infinitesimal perturbations), and because of the long-time dynamics we wanna observe, restarting the simulations from previous time steps is not an option.
I set various probes within the flow to plot the velocity. Just like you, I have no problem so far: the velocity restarts smoothly. However, I compute the total energy in usrchk as being the square of the L2-norm and I indeed also get a jump at the restart timestep when plotting this energy.
Help (or at least a few hints on how to solve this) would be highly appreciated. Sincerely yours,
JC _______________________________________________ Nek5000-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
_______________________________________________ Nek5000-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
_______________________________________________ Nek5000-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
Hi Nek's :-) , I have a problem when using the full_restart routine with the .f???? format: a few iterations after the restart the simulation crashes. Attached are the .usr file I've been using + the logfiles before and after restart. Note however that when using the old .fld format, everything works fine. Best regards, JC /----------------------------------------------------------\\ | _ __ ______ __ __ ______ ____ ____ ____ | | / | / // ____// //_/ / ____/ / __ \\ / __ \\ / __ \\ | | / |/ // __/ / ,< /___ \\ / / / // / / // / / / | | / /| // /___ / /| | ____/ / / /_/ // /_/ // /_/ / | | /_/ |_//_____//_/ |_|/_____/ \\____/ \\____/ \\____/ | | | |----------------------------------------------------------| | | | NEK5000: Open Source Spectal Element Solver | | COPYRIGHT (c) 2008-2010 UCHICAGO ARGONNE, LLC | | Version: 1.0rc1 / SVN r560 | | Web: http://nek5000.mcs.anl.gov | | | \\----------------------------------------------------------/ Number of processors: 8 REAL wdsize : 8 INTEGER wdsize : 4 Beginning session: /home/loiseau/Direct_Numerical_Simulation/Stenosis/stenosis.rea timer accuracy: 9.5367432E-08 sec read .rea file nelgt/nelgv/lelt: 2000 2000 250 lx1 /lx2 /lx3 : 8 6 8 mapping elements to processors 2 250 250 2000 2000 NELV 0 250 250 2000 2000 NELV 4 250 250 2000 2000 NELV 5 250 250 2000 2000 NELV 6 250 250 2000 2000 NELV 1 250 250 2000 2000 NELV 3 250 250 2000 2000 NELV 7 250 250 2000 2000 NELV RANK 0 IEG 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 done :: mapping elements to processors 0 objects found done :: read .rea file 0.76257E-01 sec Reset the target Courant number to .5 setup mesh topology Right-handed check complete for 2000 elements. OK. setvert3d: 8 275309 707309 275309 275309 call usrsetvert done :: usrsetvert gs_setup: 8267 unique labels shared pairwise times (avg, min, max): 6.07878e-05 5.50032e-05 6.66857e-05 crystal router : 0.000134927 0.000133204 0.000136995 all reduce : 0.000445724 0.000443506 0.000448799 used all_to_all method: pairwise setupds time 8.1151E-02 seconds 0 8 275309 2000 8 max multiplicity done :: setup mesh topology call usrdat done :: usrdat generate geomerty data vol_t,vol_v: 157.079659032505 157.079659032505 done :: generate geomerty data call usrdat2 done :: usrdat2 regenerate geomerty data 1 vol_t,vol_v: 38.6317786433067 38.6317786433067 NOTE: All elements deformed , param(59) ^=0 done :: regenerate geomerty data 1 verify mesh topology -0.500000000000000 0.500000000000000 Xrange -0.500000000000000 0.500000000000000 Yrange -5.00000000000000 45.0000000000000 Zrange done :: verify mesh topology 103 Parameters from file:/home/loiseau/Direct_Numerical_Simulation/Stenosis/stenosis.rea 1 1.00000 DENSITY 2 -750.000 VISCOS 7 1.00000 RHOCP 8 1.00000 CONDUCT 11 000500.00 NSTEPS 12 -1.00000E-04 DT 15 0100 IOSTEP 17 1.00000 AXIS 18 -20.0000 GRID 19 -1.00000 INTYPE 20 10.0000 NORDER 21 0.100000E-05 DIVERGENCE 22 0.100000E-06 HELMHOLTZ 24 0.100000E-01 TOLREL 25 0.100000E-01 TOLABS 26 1.00000 COURANT 27 3.00000 TORDER 28 0.000000E+00 49 0.000000E+00 52 1.000000E+00 HISTEP 59 0.000000E+00 65 1.00000 new file each dump (1) 66 4.00000 write form (0), unform(1), universal(2), 67 4.00000 read form (0), unform (1), universal(2), 69 50000.0 frequency of srf dump 93 20.0000 p93: Number of pressure sol'ns saved 94 3.00000 p94: start saving velocity soln's after 95 5.00000 p95: start saving pressure soln's after 99 0.000000E+00 No. of ON 102 1.00000 Dump out divergence at each time step 103 -1.00000 p103: filter strength, < 0 ==> no fil IFTRAN = T IFFLOW = T IFHEAT = F IFSPLIT = F IFLOMACH = F IFUSERVP = F IFUSERMV = F IFSTRS = F IFCHAR = F IFCYCLIC = F IFAXIS = F IFMVBD = F IFMELT = F IFMODEL = F IFKEPS = F IFMOAB = F IFSYNC = T IFVCOR = F IFINTQ = F IFCWUZ = F IFSWALL = F IFGEOM = F IFSURT = F IFWCNO = F IFTMSH for field 1 = F IFADVC for field 1 = T IFNONL for field 1 = F Dealiasing enabled, lxd= 12 Estimated eigenvalues EIGAA = 6.58105221464638 EIGGA = 1071385.12742858 EIGAE = 3.947841760435744E-003 EIGAS = 4.999000199960008E-005 EIGGE = 1071385.12742858 EIGGS = 2.00000000000000 verify mesh topology -0.500000000000000 0.500000000000000 Xrange -0.500000000000000 0.500000000000000 Yrange -5.00000000000000 45.0000000000000 Zrange done :: verify mesh topology E-solver strategy: 1 itr mg_nx: 1 5 7 mg_ny: 1 5 7 mg_nz: 1 5 7 call usrsetvert done :: usrsetvert gs_setup: 203 unique labels shared pairwise times (avg, min, max): 3.44753e-05 3.17097e-05 3.61919e-05 crystal router : 8.609e-05 8.42094e-05 8.82149e-05 all reduce : 9.20773e-05 8.9407e-05 9.36031e-05 used all_to_all method: pairwise setupds time 5.8730E-03 seconds 1 2 2525 2000 setvert3d: 4 42093 58093 42093 42093 call usrsetvert done :: usrsetvert gs_setup: 1579 unique labels shared pairwise times (avg, min, max): 3.82632e-05 3.42846e-05 4.32014e-05 crystal router : 9.83775e-05 9.5892e-05 0.000102186 all reduce : 0.000222489 0.000219297 0.000227308 used all_to_all method: pairwise setupds time 1.7242E-02 seconds 2 4 42093 2000 setvert3d: 6 133021 261021 133021 133021 call usrsetvert done :: usrsetvert gs_setup: 4267 unique labels shared pairwise times (avg, min, max): 4.09126e-05 3.75032e-05 4.33922e-05 crystal router : 0.000106177 0.000103521 0.000110197 all reduce : 0.0003025 0.00030148 0.000305009 used all_to_all method: pairwise setupds time 3.3443E-02 seconds 3 6 133021 2000 setvert3d: 8 275309 707309 275309 275309 call usrsetvert done :: usrsetvert gs_setup: 8267 unique labels shared pairwise times (avg, min, max): 5.32806e-05 4.96149e-05 5.45025e-05 crystal router : 0.00011622 0.000114989 0.000118685 all reduce : 0.000419834 0.000417304 0.000422907 used all_to_all method: pairwise setupds time 6.0193E-02 seconds 4 8 275309 2000 setup h1 coarse grid, nx_crs= 2 call usrsetvert done :: usrsetvert gs_setup: 203 unique labels shared pairwise times (avg, min, max): 3.40521e-05 3.0303e-05 3.64065e-05 crystal router : 9.63628e-05 9.43899e-05 9.77993e-05 all reduce : 9.68575e-05 9.54866e-05 9.86099e-05 used all_to_all method: pairwise gs_setup: 203 unique labels shared pairwise times (avg, min, max): 3.46392e-05 3.16143e-05 3.86e-05 crystal router : 9.02742e-05 8.87156e-05 9.27925e-05 all reduce : 0.000126746 0.000124693 0.000129795 used all_to_all method: pairwise done :: setup h1 coarse grid 0.140328884124756 sec call usrdat3 done :: usrdat3 set initial conditions Checking restart options: my_old_run0.f00004 Reading checkpoint data 0 0 OPEN: my_old_run0.f00004 byte swap: F 6.543210 -2.9312772E+35 0 4.0000E-02 done :: Read checkpoint data avg data-throughput = 375.2MBps io-nodes = 1 xyz min -0.50000 -0.50000 -5.0000 uvwpt min -0.71508 -0.71509 0.0000 -1.1405 0.95865E-20 PS min 0.99000E+22 xyz max 0.50000 0.50000 45.000 uvwpt max 0.71506 0.71508 4.2735 9.0278 0.80000E-19 PS max -0.99000E+22 Restart: recompute geom. factors. regenerate geomerty data 1 vol_t,vol_v: 38.6317789198451 38.6317789198451 NOTE: All elements deformed , param(59) ^=0 done :: regenerate geomerty data 1 done :: set initial conditions call userchk div: davg: 4.8312E-01 2.5000E-01 4.3200E+03 0 4.00000E-02 0.00000E+00 3.66558E-01 8.18424E+00 8.18424E+00 cdiv 0 4320 0.000E+00 8.167E+00 0.000E+00 4.831E-01 8.184E+00 1.344E-01divmnmx done :: userchk gridpoints unique/tot: 707309 1024000 dofs: 667100 432000 Initial time: 0.4000000E-01 Initialization successfully completed 2.4713 sec Starting time loop ... DT/DTCFL/DTFS/DTINIT 0.100E-03 0.000E+00 0.000E+00 0.100E-03 Step 1, t= 4.0100000E-02, DT= 1.0000000E-04, C= 0.016 0.0000E+00 0.0000E+00 Solving for fluid 1.000000000000000E-007 p22 1 1 1 1 Helmholtz VELX F: 8.9001E-02 1.0000E-07 1.3333E-03 1.0000E+04 1 2 Helmholtz VELX F: 7.6040E-04 1.0000E-07 1.3333E-03 1.0000E+04 1 3 Helmholtz VELX F: 1.3868E-05 1.0000E-07 1.3333E-03 1.0000E+04 1 4 Helmholtz VELX F: 2.8342E-07 1.0000E-07 1.3333E-03 1.0000E+04 1 5 Helmholtz VELX F: 7.6577E-09 1.0000E-07 1.3333E-03 1.0000E+04 1 Hmholtz VELX: 4 7.6577E-09 8.9001E-02 1.0000E-07 1.000000000000000E-007 p22 1 1 1 1 Helmholtz VELY F: 8.9001E-02 1.0000E-07 1.3333E-03 1.0000E+04 1 2 Helmholtz VELY F: 7.6041E-04 1.0000E-07 1.3333E-03 1.0000E+04 1 3 Helmholtz VELY F: 1.3868E-05 1.0000E-07 1.3333E-03 1.0000E+04 1 4 Helmholtz VELY F: 2.8341E-07 1.0000E-07 1.3333E-03 1.0000E+04 1 5 Helmholtz VELY F: 7.6574E-09 1.0000E-07 1.3333E-03 1.0000E+04 1 Hmholtz VELY: 4 7.6574E-09 8.9001E-02 1.0000E-07 1.000000000000000E-007 p22 1 1 1 1 Helmholtz VELZ F: 4.7885E-01 1.0000E-07 1.3333E-03 1.0000E+04 1 2 Helmholtz VELZ F: 3.9565E-03 1.0000E-07 1.3333E-03 1.0000E+04 1 3 Helmholtz VELZ F: 6.3684E-05 1.0000E-07 1.3333E-03 1.0000E+04 1 4 Helmholtz VELZ F: 1.8721E-06 1.0000E-07 1.3333E-03 1.0000E+04 1 5 Helmholtz VELZ F: 5.0351E-08 1.0000E-07 1.3333E-03 1.0000E+04 1 Hmholtz VELZ: 4 5.0351E-08 4.7885E-01 1.0000E-07 1 1.00000E-06 1.73934E-05 1.84885E-05 9.40768E-01 1 Divergence 2 1.00000E-06 1.58062E-05 1.84885E-05 8.54918E-01 1 Divergence 3 1.00000E-06 1.06493E-05 1.84885E-05 5.75993E-01 1 Divergence 4 1.00000E-06 5.65715E-06 1.84885E-05 3.05982E-01 1 Divergence 5 1.00000E-06 3.15688E-06 1.84885E-05 1.70748E-01 1 Divergence 6 1.00000E-06 1.47602E-06 1.84885E-05 7.98344E-02 1 Divergence 7 1.00000E-06 8.68565E-07 1.84885E-05 4.69786E-02 1 Divergence 1 U-PRES gmres: 7 8.6856E-07 1.0000E-06 1.8489E-05 2.5989E-01 6.4640E-01 1 DNORM, DIVEX 0.515914336473053 8.685645495340664E-007 1 4.0100E-02 1.2023E+00 Fluid done 1 4.01000E-02 0.00000E+00 3.66558E-01 8.18424E+00 8.18424E+00 cdiv 1 4320 0.000E+00 8.167E+00 0.000E+00 4.831E-01 8.184E+00 1.344E-01divmnmx 1 4.010000000000000E-002 0.707944296161874 energy Reading checkpoint data 0 1 OPEN: rststenosis0.f00001 byte swap: F 6.543210 -2.9312772E+35 1 3.0100E-02 done :: Read checkpoint data avg data-throughput = 404.7MBps io-nodes = 1 schfile: /home/loiseau/Direct_Numerical_Simulation/Stenosis/stenosis.sch Step 2, t= 3.0200000E-02, DT= 1.0000000E-04, C= 0.015 1.9693E+00 1.9693E+00 Solving for fluid 1.000000000000000E-007 p22 2 1 2 Hmholtz VELX: 4 7.2537E-08 4.5509E+00 1.0000E-07 1.000000000000000E-007 p22 2 1 2 Hmholtz VELY: 4 7.2537E-08 4.5509E+00 1.0000E-07 1.000000000000000E-007 p22 2 1 2 Hmholtz VELZ: 5 8.8279E-09 2.5089E+01 1.0000E-07 2 U-PRES gmres: 13 9.4591E-07 1.0000E-06 2.6283E-05 4.8434E-01 1.2558E+00 2 DNORM, DIVEX 0.515914417312535 9.459058029752750E-007 2 3.0200E-02 1.8318E+00 Fluid done 2 3.02000E-02 0.00000E+00 3.66558E-01 8.18424E+00 8.18424E+00 cdiv 2 4320 0.000E+00 8.167E+00 0.000E+00 4.831E-01 8.184E+00 1.344E-01divmnmx 2 3.020000000000000E-002 0.707743814387551 energy Reading checkpoint data 0 2 OPEN: rststenosis0.f00002 byte swap: F 6.543210 -2.9312772E+35 2 3.0200E-02 done :: Read checkpoint data avg data-throughput = 406.1MBps io-nodes = 1 Step 3, t= 3.0300000E-02, DT= 1.0000000E-04, C= 0.015 4.3247E+00 2.3554E+00 Solving for fluid this is mprev: 20 20 alpha1: 3.31851121279889 38.6317789198451 128000 binvm1: 1104069.33194260 0.125000000000000 -7.421624897757315E-011 3 3.3185E+00 3.3185E+00 1.0000E+00 0 alph1x 3 halpha 0 3 Hmholtz VELX: 4 2.3954E-08 3.3185E+00 1.0000E-07 3 3.3185E+00 3.3185E+00 1.0000E+00 0 alph1x 3 halpha 0 3 Hmholtz VELY: 4 2.3954E-08 3.3185E+00 1.0000E-07 3 1.8295E+01 1.8295E+01 1.0000E+00 0 alph1x 3 halpha 0 3 Hmholtz VELZ: 5 2.3873E-09 1.8295E+01 1.0000E-07 3 U-PRES gmres: 56 9.0901E-07 1.0000E-06 3.6217E+02 2.1020E+00 5.5592E+00 3 DNORM, DIVEX 0.515914376683388 9.090138644871518E-007 3 3.0300E-02 6.2240E+00 Fluid done 3 3.03000E-02 0.00000E+00 3.66558E-01 8.18424E+00 8.18424E+00 cdiv 3 4320 0.000E+00 8.167E+00 0.000E+00 4.831E-01 8.184E+00 1.344E-01divmnmx 3 3.030000000000000E-002 3099638.07694348 energy CFL, Ctarg! 150.294455557079 0.500000000000000 Step 4, t= 3.0400000E-02, DT= 1.0000000E-04, C=150.294 1.0927E+01 6.6025E+00 Solving for fluid 4 8.7792E+06 8.7792E+06 1.0000E+00 1 alph1x 4 halpha 1 3.3718E+01 4 Hmholtz VELX: 7 2.3205E-09 8.7792E+06 1.0000E-07 4 1.2428E+07 1.2428E+07 1.0000E+00 1 alph1x 4 halpha 1 8.1972E+02 4 Hmholtz VELY: 7 4.3573E-09 1.2428E+07 1.0000E-07 4 5.9619E+07 5.9619E+07 1.0000E+00 1 alph1x 4 halpha 1 -1.5071E+02 4 Hmholtz VELZ: 7 5.5784E-09 5.9619E+07 1.0000E-07 4 U-PRES gmres: 68 7.5107E-07 1.0000E-06 5.2130E+04 2.5536E+00 6.7898E+00 4 DNORM, DIVEX 0.515914364237373 7.510685083007039E-007 4 3.0400E-02 7.7008E+00 Fluid done 4 3.04000E-02 0.00000E+00 3.66558E-01 8.18424E+00 8.18424E+00 cdiv 4 4320 0.000E+00 8.167E+00 0.000E+00 4.831E-01 8.184E+00 1.344E-01divmnmx 4 3.040000000000000E-002 4949637.82458220 energy CFL, Ctarg! 459.075004239185 0.500000000000000 Step 5, t= 3.0500000E-02, DT= 1.0000000E-04, C=459.075 1.9007E+01 8.0794E+00 Solving for fluid 5 1.0208E+08 9.9350E+07 1.0275E+00 2 alph1x 5 halpha 2 8.0448E+03 -1.0765E+06 5 Hmholtz VELX: 7 4.3937E-08 9.9350E+07 1.0000E-07 5 1.3812E+08 1.2076E+08 1.1437E+00 2 alph1x 5 halpha 2 -9.2642E+02 -3.0772E+06 5 Hmholtz VELY: 7 2.2800E-08 1.2076E+08 1.0000E-07 5 8.9322E+08 8.8686E+08 1.0072E+00 2 alph1x 5 halpha 2 6.6180E+05 -4.8380E+06 5 Hmholtz VELZ: 8 1.8294E-08 8.8686E+08 1.0000E-07 5 U-PRES gmres: 76 8.2494E-07 1.0000E-06 2.9058E+06 2.8501E+00 7.5849E+00 5 DNORM, DIVEX 0.515914370808837 8.249440281207357E-007 5 3.0500E-02 8.5974E+00 Fluid done 5 3.05000E-02 0.00000E+00 3.66558E-01 8.18424E+00 8.18424E+00 cdiv 5 4320 0.000E+00 8.167E+00 0.000E+00 4.831E-01 8.184E+00 1.344E-01divmnmx 5 3.050000000000000E-002 1029761872.67695 energy CFL, Ctarg! 50642.4437126316 0.500000000000000 Step 6, t= 3.0600000E-02, DT= 1.0000000E-04, C=******* 2.7986E+01 8.9789E+00 Solving for fluid 6 9.2689E+11 9.1003E+11 1.0185E+00 3 alph1x 6 halpha 3 5.4776E+06 -2.2091E+08 8.0286E+09 6 Hmholtz VELX: 9 2.5420E-08 9.1003E+11 1.0000E-07 6 1.2533E+12 1.1559E+12 1.0843E+00 3 alph1x 6 halpha 3 1.7943E+08 -6.7335E+07 2.2117E+10 6 Hmholtz VELY: 10 1.7372E-09 1.1559E+12 1.0000E-07 6 1.6358E+13 1.5555E+13 1.0516E+00 3 alph1x 6 halpha 3 3.1476E+10 -9.6210E+09 2.3027E+11 6 Hmholtz VELZ: 11 2.4940E-09 1.5555E+13 1.0000E-07 6 1 alpha: 1.7749E+11 6 1 4.7988E+10 4.1355E+10 1.1604E+00 alph12 6 U-PRES gmres: 108 7.1499E-07 1.0000E-06 4.1355E+10 4.2161E+00 1.1214E+01 6 DNORM, DIVEX 0.515998582206397 7.149864975405526E-007 6 3.0600E-02 1.2572E+01 Fluid done 6 3.06000E-02 0.00000E+00 3.66558E-01 8.18424E+00 8.18424E+00 cdiv 6 4320 0.000E+00 8.167E+00 0.000E+00 4.831E-01 8.184E+00 1.344E-01divmnmx 6 3.060000000000000E-002 2.876071680859043E+017 energy CFL, Ctarg! 2022369530.96645 0.500000000000000 Step 7, t= 3.0700000E-02, DT= 1.0000000E-04, C=******* 4.0940E+01 1.2954E+01 Solving for fluid 7 1.0747E+21 8.2912E+20 1.2963E+00 4 alph1x 7 halpha 4 6.3655E+16 -6.4172E+16 3.6244E+18 3.1175E+19 7 Hmholtz VELX: 14 5.2146E-08 8.2912E+20 1.0000E-07 7 8.1769E+20 6.8887E+20 1.1870E+00 4 alph1x 7 halpha 4 7.2971E+17 -2.6597E+17 7.8029E+18 1.8640E+19 7 Hmholtz VELY: 15 4.9148E-09 6.8887E+20 1.0000E-07 7 1.9107E+22 1.3141E+22 1.4540E+00 4 alph1x 7 halpha 4 2.4419E+19 -7.4102E+18 1.9033E+19 6.3574E+20 7 Hmholtz VELZ: 15 6.8962E-09 1.3141E+22 1.0000E-07 7 2 alpha: 1.3632E+20 3.7288E+20 7 2 4.2549E+19 4.0535E+19 1.0497E+00 alph12 7 U-PRES gmres: 120 2.0045E+00 1.0000E-06 4.0535E+19 4.6681E+00 1.2460E+01 7 DNORM, DIVEX 2493452.02288773 2.00451650526042 7 3.0700E-02 1.4126E+01 Fluid done 7 3.07000E-02 0.00000E+00 3.66558E-01 8.18424E+00 8.18424E+00 cdiv 7 4320 0.000E+00 8.167E+00 0.000E+00 4.831E-01 8.184E+00 1.344E-01divmnmx 7 3.070000000000000E-002 3.997522740961343E+035 energy CFL, Ctarg! 1.889593261729235E+018 0.500000000000000 Step 8, t= 3.0800000E-02, DT= 1.0000000E-04, C=******* 5.5461E+01 1.4521E+01 Solving for fluid 8 1.9579E+39 1.3323E+39 1.4696E+00 5 alph1x 8 halpha 5 -1.4342E+35 -5.9999E+34 -5.4763E+34 1.4859E+37 6.4125E+37 8 Hmholtz VELX: 25 3.2466E-09 1.3323E+39 1.0000E-07 8 1.7206E+39 1.6053E+39 1.0718E+00 5 alph1x 8 halpha 5 3.3515E+35 -2.1596E+35 -2.0268E+36 -1.0105E+37 2.6415E+37 8 Hmholtz VELY: 25 2.7655E-09 1.6053E+39 1.0000E-07 8 3.6055E+40 1.7771E+40 2.0289E+00 5 alph1x 8 halpha 5 2.2298E+37 -1.9645E+37 9.7764E+37 8.1318E+38 1.1842E+39 8 Hmholtz VELZ: 25 8.0500E-08 1.7771E+40 1.0000E-07 8 3 alpha: 2.4511E+38 7.5064E+38 5.9429E+38 8 3 9.1168E+37 4.1623E+37 2.1904E+00 alph12 8 U-PRES gmres: 120 1.7812E+18 1.0000E-06 4.1623E+37 4.6553E+00 1.2474E+01 8 DNORM, DIVEX 3.294681463300302E+024 1.781162806610600E+018 8 3.0800E-02 1.4892E+01 Fluid done 8 3.08000E-02 0.00000E+00 3.66558E-01 8.18424E+00 8.18424E+00 cdiv 8 4320 0.000E+00 8.167E+00 0.000E+00 4.831E-01 8.184E+00 1.344E-01divmnmx 8 3.080000000000000E-002 1.181855307322355E+072 energy CFL, Ctarg! 4.885808314825141E+036 0.500000000000000 9 3.0800E-02 Write checkpoint: 0 9 OPEN: stenosis0.f00001 9 3.0800E-02 done :: Write checkpoint file size = 27. MB avg data-throughput = 304.5MB/s io-nodes = 1 0 Emergency exit: 9 time = 3.080000000000000E-002 Latest solution and data are dumped for post-processing. *** STOP *** 2 Emergency exit: 9 time = 3.080000000000000E-002 Latest solution and data are dumped for post-processing. *** STOP *** 6 Emergency exit: 9 time = 3.080000000000000E-002 Latest solution and data are dumped for post-processing. *** STOP *** 1 Emergency exit: 9 time = 3.080000000000000E-002 Latest solution and data are dumped for post-processing. *** STOP *** 7 Emergency exit: 9 time = 3.080000000000000E-002 Latest solution and data are dumped for post-processing. *** STOP *** 4 Emergency exit: 9 time = 3.080000000000000E-002 Latest solution and data are dumped for post-processing. *** STOP *** 5 Emergency exit: 9 time = 3.080000000000000E-002 Latest solution and data are dumped for post-processing. *** STOP *** 3 Emergency exit: 9 time = 3.080000000000000E-002 Latest solution and data are dumped for post-processing. *** STOP *** runtime statistics: total time 55.4610141323639 inv3 time 597 0.823028087615967 1.483975907205228E-002 invc time 24 1.812195777893066E-002 3.267512875924085E-004 mltd time 1815 9.62307167053223 0.173510560906183 cdtp time 1815 13.3121821880341 0.240027745548667 eslv time 8 57.9872560501099 1.04554986880905 pres time 8 58.3978500366211 1.05295315908303 crsl time 568 0.346214056015015 6.242476114640389E-003 crsl min 0.282711744308472 crsl max 0.635290861129761 crsl avg 0.394825488328934 hmhz time 6 6.41925811767578 0.115743612303149 spro time 9 2.416491508483887E-002 4.357099390062829E-004 usbc time 8 0.140545845031738 2.534137668242235E-003 usbc min 0.139691829681396 usbc max 0.144717931747437 usb avg 0.142532169818878 axhm time 445 4.52438235282898 8.157770685604553E-002 advc time 24 1.24968290328979 2.253263707560932E-002 vdss time 598 3.41643881797791 6.160072749164293E-002 vdss min 2.80426836013794 vdss max 3.43248867988586 vdss avg 2.99572038650513 dsum time 2109 3.53541040420532 6.374586652468472E-002 dsum min 2.98387336730957 dsum max 3.53541040420532 dsum avg 3.32718187570572 dadd time 0 3.09410786628723 5.578888007534082E-002 ddsl time 568 21.7788903713226 0.392688282247145 solv time 568 4.26981401443481 7.698766568249955E-002 prep time 9 0.120651960372925 2.175437327651014E-003 # nid tusbc tdadd tcrsl tvdss tdsum tgop qqq 0 1.4055E-01 3.0941E+00 3.4621E-01 3.4164E+00 3.5354E+00 0.0000E+00 qqq 1 1.4434E-01 2.4822E+00 2.8271E-01 2.8605E+00 3.1515E+00 0.0000E+00 qqq 2 1.4472E-01 2.8438E+00 2.8300E-01 3.4325E+00 3.4284E+00 0.0000E+00 qqq 3 1.4387E-01 2.5183E+00 3.4953E-01 2.8786E+00 3.2094E+00 0.0000E+00 qqq 4 1.4175E-01 2.9689E+00 3.9433E-01 2.8216E+00 3.5147E+00 0.0000E+00 qqq 5 1.4319E-01 2.4075E+00 4.2978E-01 2.8043E+00 2.9839E+00 0.0000E+00 qqq 6 1.4215E-01 2.6402E+00 4.3775E-01 2.8142E+00 3.2847E+00 0.0000E+00 qqq 7 1.3969E-01 3.0568E+00 6.3529E-01 2.9376E+00 3.5095E+00 0.0000E+00 qqq call exitt: dying ... backtrace(): obtained 10 stack frames. ./nek5000 [0x5ca57a] ./nek5000 [0x6c5fe3] ./nek5000 [0x51a868] ./nek5000 [0x50d906] ./nek5000 [0x4106d0] ./nek5000 [0x406111] ./nek5000 [0x405d22] ./nek5000 [0x40419c] /lib64/libc.so.6(__libc_start_main+0xf4) [0x3393c1d974] ./nek5000 [0x4040a9] total elapsed time : 7.33246E+01 sec total solver time incl. I/O : 5.54610E+01 sec time/timestep : 6.16233E+00 sec CPU seconds/timestep/gridpt : 6.96989E-05 sec c----------------------------------------------------------------------- C C USER SPECIFIED ROUTINES: C C - boundary conditions C - initial conditions C - variable properties C - local acceleration for fluid (a) C - forcing function for passive scalar (q) C - general purpose routine for checking errors etc. C c----------------------------------------------------------------------- subroutine uservp (ix,iy,iz,eg) include 'SIZE' include 'TOTAL' include 'NEKUSE' integer e,f,eg c e = gllel(eg) udiff =0. utrans=0. return end c----------------------------------------------------------------------- subroutine userf (ix,iy,iz,eg) include 'SIZE' include 'TOTAL' include 'NEKUSE' integer e,f,eg c e = gllel(eg) c Note: this is an acceleration term, NOT a force! c Thus, ffx will subsequently be multiplied by rho(x,t). ffx = 0.0 ffy = 0.0 ffz = 0.0 return end c----------------------------------------------------------------------- subroutine userq (ix,iy,iz,eg) include 'SIZE' include 'TOTAL' include 'NEKUSE' integer e,f,eg c e = gllel(eg) qvol = 0.0 source = 0.0 return end c----------------------------------------------------------------------- subroutine userchk include 'SIZE' include 'TOTAL' real linf,l2,h1,semi,Energy c call hpts() call fill_div(usrdiv) if (istep.gt.0) then call normvc(h1,semi,l2,linf,vx,vy,vz) Energy = .5*l2*l2 if (nid.eq.0) write(06,*) istep,time,Energy,' energy' endif if (istep.gt.0.and.istep.lt.nbdinp) call my_full_restart_load call my_full_restart_save ! save add'l files for full-restart return end c----------------------------------------------------------------------- subroutine userbc (ix,iy,iz,iside,ieg) include 'SIZE' include 'TOTAL' include 'NEKUSE' rr = x*x + y*y ux = 0.0 uy = 0.0 uz = 2*(1 - 4*rr) ! Hagen-Poiseiulle flow, pipe diamter = 1. temp= 0.0 return end c----------------------------------------------------------------------- subroutine useric (ix,iy,iz,ieg) include 'SIZE' include 'TOTAL' include 'NEKUSE' rr = x*x +y*y ux=0.0 uy=0.0 uz=2*(1 - 4*rr) temp=0 return end c----------------------------------------------------------------------- subroutine usrdat include 'SIZE' include 'TOTAL' c return end c----------------------------------------------------------------------- subroutine usrdat3 include 'SIZE' include 'TOTAL' c return end c----------------------------------------------------------------------- subroutine usrdat2 ! Modify geometry include 'SIZE' include 'TOTAL' logical ifstenosis save ifstenosis c param(66) = 4. c param(67) = 4. x0 = -.5 x1 = .5 call rescale_x (xm1,x0,x1) ! Rescale x & y so pipe diameter is 1. call rescale_x (ym1,x0,x1) ifstenosis = .false. ifstenosis = .true. if (.not.ifstenosis) return one = 1.0 pi = 4.*atan(one) n = nx1*ny1*nz1*nelv z_outflow = 45. z_inflow = -5.0 call rescale_x(zm1,z_inflow,z_outflow) stenosis_length = 1. do i=1,n x = xm1(i,1,1,1) y = ym1(i,1,1,1) z = zm1(i,1,1,1)/stenosis_length arg = pi amp = .25 ! 75% stenosis for 3D case if (abs(z).le.1) then xm1(i,1,1,1) = x*(1.-amp*(1.+cos(arg*z))) ym1(i,1,1,1) = y*(1.-amp*(1.+cos(arg*z))) c write(6,1) i,x,y,z,xm1(i,1,1,1),ym1(i,1,1,1),zm1(i,1,1,1) endif enddo param(59) = 1 ! Force ifdfrm=.true. ( 8/26/03 ) ifxyo = .true. c call outpost(xm1,ym1,zm1,pr,t,' ') c call exitt return end c----------------------------------------------------------------------- subroutine fill_div(div) c c Fill the domain with a nontrivial divergence, where desired. c c include 'SIZE' include 'TOTAL' c integer icalld save icalld data icalld /0/ c common /cvflow_d/ dist(lx2,ly2,lz2,lelt),dmax common /cvflow_l/ ifdivf logical ifdivf c real div(lx2,ly2,lz2,lelv) c ntot2 = nx2*ny2*nz2*nelv call rzero(div,ntot2) if (ifdivf) return c if (icalld.eq.0) then icalld = 1 call set_outflow_dist endif dmax2 = dmax*dmax c call get_div_const(cdiv) c nd = 0 do i=1,ntot2 dd = dist(i,1,1,1)*dist(i,1,1,1) dd = min(dmax2,dd) if (dist(i,1,1,1).ne.0) div(i,1,1,1) = cdiv*(1-dd/dmax2) if (dist(i,1,1,1).ne.0) nd = nd+1 enddo dsmin = glmin(dist,ntot2) dsmax = glmax(dist,ntot2) dvmin = glmin(div,ntot2) dvmax = glmax(div,ntot2) xd = nd xd = glsum(xd,1) nd = xd if(nid.eq.0)write(6,1) istep,nd,dvmin,dvmax,dsmin,dsmax,cdiv,dmax2 1 format(2i9,1p6e11.3,'divmnmx') 2 format(2i9,1p2e11.3,'divdstm') c return end c----------------------------------------------------------------------- subroutine set_outflow_dist c c Compute projected normal distance from outflow c include 'SIZE' include 'TOTAL' c common /cvflow_d/ dist(lx2,ly2,lz2,lelt),dmax c integer e,f c c nxyz2 = nx2*ny2*nz2 ntot2 = nx2*ny2*nz2*nelv ntot1 = nx1*ny1*nz1*nelv c call rzero(dist,ntot2) davg = 0. wavg = 0. c zmax = glmax(zm1,ntot1) c do e=1,nelv if (cbc(6,e,1).eq.'O ') then ! Outflow do k=1,nxyz2 dist(k,1,1,e) = zmax-zm2(k,1,1,e) davg = davg+dist(k,1,1,e) wavg = wavg+1.0 enddo endif enddo dmax = glmax(dist,ntot2) davg = glsum(davg,1) wavg = glsum(wavg,1) if (wavg.gt.0) davg = davg/wavg if (nid.eq.0) write(6,1) dmax,davg,wavg 1 format('div: davg:',1p3e12.4) dmax = 0.5*(dmax+davg) c return end c----------------------------------------------------------------------- subroutine get_div_const(cdiv) c c Get constant multiplier for divergence c include 'SIZE' include 'TOTAL' c common /cvflow_d/ dist(lx2,ly2,lz2,lelt),dmax real flx(0:2) c integer e,f vnmo_desired = 2. cdivt = 1.5*vnmo_desired/dmax cdiv = max(0.,cdivt) ! No contraction! if (nid.eq.0) write(6,1) istep,time,vnmo,dmax,cdiv,cdivt 1 format(i9,1p5e13.5,' cdiv') return end c----------------------------------------------------------------------- subroutine my_full_restart_save c Saves files for next full restart include 'SIZE' include 'TOTAL' include 'RESTART' ! max_rst c This is the full-restart save part: max_rst = 2*(nbdinp-1) ! max # of rst files saved nps1 = 0 if (ifheat) nps1 = 1 + npscal mostep = mod1(istep,iostep) if (istep.gt.iostep.and.mostep.lt.nbdinp) $ call outpost2(vx,vy,vz,pr,t,nps1,'rst') return end c----------------------------------------------------------------------- subroutine my_full_restart_load include 'SIZE' include 'TOTAL' character*80 s80 call blank(s80,80) if (istep.eq.1) s80 ='rststenosis0.f00001' ! This would be the case for Torder=3 if (istep.eq.2) s80 ='rststenosis0.f00002' ! using the first restart pair c if (istep.eq.1) s80 ='rststenosis.fld03' ! This would be the case for Torder=3 c if (istep.eq.2) s80 ='rststenosis.fld04' ! using the second restart pair c if (istep.eq.1) s80 ='rststenosis0.f00001' ! This would be the case for Torder=2, file 1 c if (istep.eq.1) s80 ='rststenosis0.f00002' ! This would be the case for Torder=2, file 2 call bcast(s80,80) call chcopy(initc,s80,80) c time_curr = time nfiles = 1 call restart(nfiles) ! Note -- time is reset. if (nid.ne.0) time=0 time = glmax(time,1) ! Synchronize time across all processors c time = time_curr ! Preserve current simulation time return end c----------------------------------------------------------------------- c automatically added by makenek subroutine usrsetvert(glo_num,nel,nx,ny,nz) ! to modify glo_num integer*8 glo_num(1) return end
JC, I think you have loaded the wrong rst* files. I think your my_old_run0.f00004 is the field at t=4.00000E-02 while 'rststenosis0.f00001' and rststenosis0.f00002 are the fields at 3.0100E-02 and 3.0200E-02. francesco On 09/30/2010 03:37 PM, [email protected] wrote:
Hi Nek's :-) ,
I have a problem when using the full_restart routine with the .f???? format: a few iterations after the restart the simulation crashes. Attached are the .usr file I've been using + the logfiles before and after restart. Note however that when using the old .fld format, everything works fine.
Best regards, JC
_______________________________________________ Nek5000-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
Yes indeed, the rst* files specified in the .usr attached to my previous mail are not the correct ones. However, I have also tried with the correct restart pair and it did not change anything. The simulation still crashes after a few time steps. JC
participants (1)
-
nek5000-users＠lists.mcs.anl.gov