<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>New Operational Short-Range Numerical Weather Prediction System of Khabarovsk Regional Specialized Meteorological Center</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Stanislav Romanskiy</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Eugenia Verbitskaya</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Far Eastern Regional Hydrometeorological Research Institute</institution>
          ,
          <addr-line>Vladivostok</addr-line>
          ,
          <country country="RU">Russia</country>
        </aff>
      </contrib-group>
      <pub-date>
        <year>2019</year>
      </pub-date>
      <fpage>16</fpage>
      <lpage>19</lpage>
      <abstract>
        <p>A description of new operational short-range numerical weather prediction (NWP) system of the Regional Specialized Meteorological Center of Khabarovsk, Russia is presented in this article. Proposed NWP system is based on nonhydrostatic Weather Research and Forecasting model of atmosphere with horizontal grid spacing of 4.5 km and 51 vertical levels. The model is run on cluster CRAY XC-40 four times per day and will replace current short-range forecasting system of 'Khab-15' after an extensive testing. It is shown comparison of forecast skill between the NWP systems of 'Khab-4.5' and 'Khab-15' for winter and spring months of 2019. Samples of product are given. Further development of the 'Khab-4.5' model system is discussed.</p>
      </abstract>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>Introduction</title>
      <p>
        The Advanced Research version of Weather Research and Forecasting model (WRF-ARW) [
        <xref ref-type="bibr" rid="ref6 ref7">6, 7</xref>
        ] is used as a
basic numerical model in both NWP systems of RSMC Khabarovsk. The WRF model is started to develop in 1990s
in the United States of America (USA) by group of various organizations. Today it is used by wide range of
specialists for numerical simulations of atmospheric phenomena in idealized or real cases and it is suitable for
weather forecasting. Source code of WRF-ARW is open. The WRF model is a main part of the High-Resolution
Rapid Refresh model [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ] of National Centers of Environmental Prediction (NCEP, USA) as well.
      </p>
      <p>
        Operational initial and lateral boundary data for WRF-ARW model provides from output of Global Forecast
System (GFS, NCEP) [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ]. GFS data with grid spacing of 0.5° updated every 3 hours during the calculation of the
‘Khab-4.5’ system.
      </p>
      <p>
        In Table 1 technical characteristics of ‘Khab-15’ and ‘Khab-4.5’ NWP systems of RSMC Khabarovsk are
presented. WRF-ARW release was transited from version 3.4.1 to 3.9.1. The last version of WRF is 4 but the WRF
version 3.9 has been intensely tested. WRF-ARW model of version 3.9.1 has new parameterizations of atmospheric
processes and some old schemes were updated to increase the accuracy of a simulation. In NWP system of ‘Khab-4.5’
WRF-ARW model uses Thompson microphysics parameterization and does not apply any convection scheme due to
grid step of 4.5 km. The short- and longwave radiation balance is calculated by rapid radiative transfer model.
MellorYamada-Nakanishi-Niino scheme is planned to use to figure out turbulent kinetic energy budget in planetary
boundary layer (PBL). The Noah land-surface model of four soil levels performs parameterization of surface layer
processes and provides surface sensible and latent heat fluxes and upward short- and longwave radiation fluxes to the
parent atmospheric model. These parameterizations are recommended to use [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ].
      </p>
      <p>
        The NWP system of ‘Khab-4.5’ is specially designed for new high-powered computing platform based on cluster
CRAY XC-40 which is installed in 2018. This cluster includes 60 nodes of two 18-cores Intel Broadwell processors
with Dragonfly topology and has total peak performance of 76 TFlops. Network storage under parallel-distributed file
system of Lustre serves to implement parallel input and output operations to accelerate calculations of the
WRFARW model. Output files of the WRF-ARW model have large size (each of it takes near 8 GB of memory storage).
However, a file has to be written in parallel to cut saving time. The saving time of one model output file was
shortened from 23-25 to 1.8-2.2 s by using parallel netCDF library [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ].
      </p>
      <p>The calculations of the NWP system of ‘Khab-4.5’ will start automatically four times daily (every 6 hours from
standard times of 0000, 0600, 1200 and 1800 UTC at 0430, 1030, 1630 and 2230 UTC respectively). At present the
system starts twice a day in quasi-operational regime (from 0000 and 1200 UTC).</p>
      <p>One run of the NWP system ‘Khab-4.5’ includes three main steps: WRF preprocessing, calculation of the
WRFARW model and post-processing. These steps are joined and maintained by main script of bash and python. The
major functions of the main script are managing the interactions between parts of technology; synchronization and
monitoring a calculation cycle; processing of typical errors. The main script keeps track statuses of running programs,
manages start of applications and proceeds intermediate data between parts of technology. Log messages of the NWP
system are stored to database and an operator in charge may view it on-line.</p>
      <p>The programs of the first step of preprocessing decode initial and boundary data and interpolate it to the grids of
WRF-ARW model. Furthermore, surface observations in code forms of SYNOP and METAR are processed on this
stage. Weather reports are used to determine the accuracy of previous model forecasts and, moreover, it is transferred
to data assimilation system to initialize new run of a numerical model. However, the procedure of data assimilation is
still under development and NWP system of ‘Khab-4.5’ does not implement it at the current state. The WRF-ARW
model is run on the second stage. Programs of post-processing start at the same time as numerical model to handle
new raw forecasting data of parallel netCDF format on the fly. WRF-ARW model uses Message-passing interface
and OpenMP subroutines to conduct calculations in parallel on cluster CRAY XC-40. Post-processing programs are
also written with OpenMP calls and mostly implement procedures of various methods to produce forecasts of clouds,
make special weather prediction for aviation and prepare simulated data for other models of RSMC Khabarovsk (i.e.
sea level model). Weather maps, meteograms (weather forecast figure with charts for a special point of interest), text
messages and other visual products are produced on post-processing stage. These products are published on special
web-sites (khabmeteo.ru, meteo-dv.ru and ferhri.org) and distributed to end users by e-mails, ftp and internal
network.</p>
      <p>
        Programs of WRF preprocessor [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ] includes following items: geogrid.exe to process statical data like orography
and soil types to the horizontal model grids; ungrid.exe to unpack downloaded GFS data of GRIB2 format;
metgrid.exe to horizontally interpolate unpacked GFS data to the WRF-ARW model grids. All these programs run on
10 nodes of CRAY XC-40 cluster. This step is fulfilled in 15-20 minutes. Then, program real.exe is started to
vertically interpolate all prepared data to the WRF-ARW model levels. This operation demands high-performance
computing; it uses 15 nodes of CRAY XC-40 cluster and requires 20 minutes. Following these actions the
WRFARW model is ready to run on 40 nodes of CRAY XC-40 cluster; model works near 90 minutes (depends on
maximum lead time). At the same time bash script of post-processing is started to handle WRF output on fly.
Programs of post-processing run on a front-end server of two 18-cores Intel processors; therefore, it may use up to 36
threads of execution. Post-processing is a complex set of procedures. It includes following main actions: calculation
of methods to diagnose weather parameters and meteorological fields (cloud cover, icing, atmospheric pressure
adjusted to mean sea level, position of zero-isotherm, type of precipitation, TAF etc.); interpolation of WRF data to
standard pressure levels (1000, 850, 700, 500, 300, 150 hPa etc.); conversion of interpolated data to GRIB format;
distribution of products to end users; preparation of meteograms, various weather maps, tables and automated text
messages; publication of forecasts on specialized web-cites (i.e. meteo-dv.ru and ferhri.org); store of output model
data for further estimation of forecasts’ accuracy). For some operations like conversation to GRIB format and
interpolation to pressure levels unified post processor [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ] is used. Post-processing programs work in parallel;
therefore, they will have finished in 20 minutes after last WRF output file would have fulfilled.
      </p>
      <p>In conclusion, one run of NWP system of ‘Khab-4.5’ gets near 2 hours without taking into the account the period
of time to download input GFS forecasts available online on NCEP servers.
2.2</p>
      <p>Output Products of the NWP System‘Khab-4.5’</p>
      <p>
        The NWP system of ‘Khab-4.5’ releases wide range of automated output products. It includes binary messages of
GRIB format, set of weather maps, meteograms for points of interest and group of text messages (special tables,
warning messages, TAF etc.). Weather maps include slides for the layers of free atmosphere, planetary boundary
layer and surface. This information generates in automatic by fortran and python programs. Slides are plotted with
calls of matplotlib library [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ] in parallel (package multiprocessing). Our aim is making specialized products for end
users depending on their demands.
      </p>
      <p>Binary messages of GRIB format are published. Messages include meteorological fields (geopotencial height,
uand v-wind components, temperature, humidity, divergence, vorticity etc.) on surface and standard pressure levels.
These data are converted to lat-lon projection with horizontal grid step of 0.5°.</p>
      <p>Maps of geopotential height and horizontal wind vector (with emphasizing of strong wind) are produced for
standard isobaric surfaces of 100, 200, 300, 500, 700, 850, 925 and 1000 hPa. One type of surface weather map
depicts cloud cover, areas of precipitation and pressure reduced to mean sea level (pmsl) (Fig. 2a). Another surface
weather map shows air temperature on 2 m above ground layer (AGL) and 10-m horizontal wind vector (with
emphasizing of strong wind). Furthermore, maps of total accumulated precipitation are prepared for hydrological
applications.</p>
      <p>In addition ‘Khab-4.5’ produces some special weather maps for aviation sector. These slides depict pmsl by
standard atmosphere, areas of possible icing, heights of zero-isotherm, index of convection, areas of low-level
turbulence etc. In the near future output production will be expanded by forecasts of clear-air turbulence and
mountain waves.</p>
      <p>Meteograms (Fig. 2b) are also produced by NWP system of ‘Khab-4.5’. One figure includes 3 panels. The first
panel shows surface forecast: timelines of 2-m air temperature, 2-m dew point, pmsl and wind shift near surface (or
horizontal wind vector on 10 m AGL). Next panel depicts timelines of 3-h liquid and solid precipitation and
characteristics of clouds: top and bottom heights of clouds, position of zero-isotherm, cloud cover and index of
convection. Last panel illustrates timeline of air temperature, humidity and wind vectors on specific vertical levels.</p>
      <p>Text messages includes fixed-term data like 2-m air temperature, psml (or surface pressure), 10-m wind vector and
pmsl with a step of 1, 3, 6 etc. hours for points of interest.</p>
      <p>
        The products of NWP system of ‘Khab-4.5’ may use as input data for specialized forecasting models. For
example, current sea level model of RSMC Khabarovsk [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ] uses as input 10-m wind and pmsl fields of ‘Khab-15’
NWP system.
      </p>
      <p>The NWP system of ‘Khab-4.5’ works in quasi-operational regime from the beginning of 2019. Examination of
forecasts from both NWP systems of RSMC Khabarovsk allows to obtain objective measurement of models’ skill.
Comparison of surface forecasts between NWP systems of ‘Khab-4.5’ and ‘Khab-15’ was conducted for the period of
time from 14 January to 14 April, 2019 for 130 points of Amur, Sakhalin, Primorskiy and Khabarovskiy regions of
Russia. Surface observations are available 8 times per day. However, numerical forecasts were matched with
observations every 12 hours for lead times of 12, 24, 36, 48, 60 and 72 hours to shorten calculations. Surface
observations have diurnal variations. However, reduction of observation times does not significantly affect result due
to the fact that the time period of 12 hours provides forecasts’ estimations for day as well as night; mentioned above
regions of Russia are located in 3 time zones (from +9 to +11 UTC). Forecasting of surface wind speed and direction
on 10 m above AGL, 2-m air temperature and surface pressure are key parameters for weather guidance.</p>
      <p>The accuracy of forecasts was estimated by calculation of mean errors (ME), absolute errors (ABSE) and
rootmean-squared errors (RMS).</p>
      <p>ME =
ABSE =
RMS =
1 n</p>
      <p> ( f i − oi ),
n i=1
1 n</p>
      <p> | f i − oi |,
n i=1
n
 ( f i − oi ) 2
i=1
n
where n – number of pairs ‘observation – forecast’, f – forecast, o – observation.</p>
      <p>Comparison of errors in surface forecasting of ‘Khab-4.5’ and ‘Khab-15’ NWP systems is shown on Fig. 3.
Forecast in a point of interest corresponds to the data from the nearest grid point. Accuracy of forecasts does not
(1)
(2)
(3)
reduce uniformly with the passing of time and both models show very close estimations. Occasionally night errors is
worse than day. Almost all values have diurnal fluctuations, especially, errors of 2-m temperature forecasts.</p>
      <p>Absolute errors of 10-m wind vector forecasts (Fig. 3a) slightly increase from 2.5 to 3.4 m/s for lead times from 12
to 72 hours. Most rapidly the accuracy of forecasts is getting worse between 12 and 24 hours and from 60 to 72 hours;
most stable estimated values are for lead times from 24 to 60 hours. It is necessary to remember that the estimations
have been obtained for the cold period of a year. Wind is mostly determined by dynamical factors (not temperature)
in this period of year. However, errors show the quality of model’s forecasting which is not affected by error coming
from simulated convective processes. Errors show the simulated surface wind on average is a little bit greater than
observed data.</p>
      <p>Absolute errors of surface pressure forecasts (Fig. 3b) steady increase from 1.2 to 3.3 hPa and mean errors fall
unevenly to -2 hPa for the lead time from 12 to 72 hours. However, ABSE of surface pressure is under the value of
1.8 hPa and mean error is near 0 hPa for the first day of forecasting. Forecasts of surface pressure with these errors
are suitable for aviation (TAF messages are mostly published for the period of time from 6 to 24 hours). Both models
show similar quality of surface pressure forecasting. However, mean errors of ‘Khab-4.5’ surface pressure forecasting
is better than ‘Khab-15’.</p>
      <p>Prediction of 2-m temperature is the most complex task for NWP system especially in cold season. Variation of
surface temperature is defined not only by the state atmosphere but also orography and vicinity of seas. Temperature
inversions often have place on the territory of Russian Far East in cold season. This process is very hard to predict.
Figure 3c shows that the accuracy of 2-m temperature forecasts is better for day period not night. Absolute errors of
2-m forecasts gradually increase from 3.2° to 4.9° for the lead time from 12 to 72 hours.</p>
      <p>Amount of observations for the territory of Russian Far East is not enough for such vast area. Therefore, initial
data also has errors in surface temperature and ‘Khab-4.5’ cannot neglect it well. The accuracy of 2-m temperature
prediction in cold season reveals the necessity of data assimilation system for the NWP system of ‘Khab-4.5’. Mean
errors of 2-m temperature forecasts are not stable; therefore, model output statistics will not have good results for
these forecast data. Amplitude of simulated daily run of surface temperature is insufficient in comparison with the
observed data.</p>
      <p>
        In conclusion, forecasting in cold season is most complex task in comparison with warm period of a year. The
errors in forecasting of ‘Khab-4.5’ NWP system will be decrease in warm period of a year as previously shown for
the ‘Khab-15’ NWP system [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ].
4
      </p>
    </sec>
    <sec id="sec-2">
      <title>Future of ‘Khab-4.5’</title>
      <p>The ‘Khab-4.5’ NWP system will provide improvement to short-range weather forecasting of RSMC Khabarovsk
including forecasts for aviation and prediction of severe weather events. The configuration of coastal line and
mountain regions is much improved in ‘Khab-4.5’ according to transition of horizontal grid step from 15 to 4.5 km.
Increase in the number of vertical levels without doubt improves the representation of a current state of atmosphere in
the model.</p>
      <p>Current experiments show negligible advantages in surface forecast skills as opposite to current ‘Khab-15’ NWP
system. However, this fact means the necessity of fine tuning of new model system. Tests of new microphysics and
PBL parameterizations will be implemented to select more suitable schemes for small-scale processes. Another way
to improve the accuracy of surface forecasts lies in enchantment of procedure to produce weather prediction in a point
of interest. The area of a grid cell with horizontal step of 15 km is 225 km2. However, the area for a grid cell with
horizontal resolution of 4.5 km is only 20.25 km2. Therefore, a forecast from the nearest grid point to a particular spot
may be of insufficiently accurate in comparison with a forecast from the nearest grid point in height or an averaged
value obtained from 9 or 16 points of the encirclement.</p>
      <p>NWP system of ‘Khab-4.5’ has great potential for weather forecasting; it is flexible in configuration and provides
possibility to include new prediction methods to a calculation cycle.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <article-title>Meteorological service for international air navigation</article-title>
          ,
          <source>ICAO</source>
          . Montreal, Canada,
          <volume>187</volume>
          (
          <year>2007</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <surname>Verbitskaya</surname>
            ,
            <given-names>E.M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Romanskij</surname>
            ,
            <given-names>S.O.</given-names>
          </string-name>
          :
          <article-title>Rezul'taty ispytaniy kratkosrochnykh operativnykh prognozov mezomasshtabnoy modeli WRF-ARW "KHab-15" v punktakh Dal'nevostochnogo regiona Rossii, Rezul'taty ispytaniya novykh i usovershenstvovannykh tekhnologij, modelej i metodov gidrometeorologicheskikh prognozov</article-title>
          .
          <volume>43</volume>
          :
          <fpage>32</fpage>
          -
          <lpage>62</lpage>
          (
          <year>2016</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>Benjamin</surname>
            ,
            <given-names>S.G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Weygandt</surname>
            ,
            <given-names>S.S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Brown</surname>
            ,
            <given-names>J.M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hu</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          et. al.:
          <article-title>A North American hourly assimilation and model forecast cycle: the rapid refresh</article-title>
          ,
          <source>Monthly weather review</source>
          <volume>144</volume>
          :
          <fpage>1669</fpage>
          -
          <lpage>1694</lpage>
          (
          <year>2016</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <surname>Akimoto</surname>
          </string-name>
          , H.:
          <article-title>Global air quality and pollution</article-title>
          .
          <source>Science</source>
          ,
          <volume>302</volume>
          ,
          <fpage>1716</fpage>
          -
          <lpage>1719</lpage>
          (
          <year>2003</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          <article-title>5. Manual on the global data-processing and forecasting system, WMO</article-title>
          . Geneva, Switzerland,
          <volume>147</volume>
          (
          <year>2017</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <surname>Wang</surname>
            ,
            <given-names>W.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bruyere</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Duda</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dudhia</surname>
          </string-name>
          , J.:
          <article-title>WRF-ARW version 3 modeling system user's guide, NCAR</article-title>
          . Boulder, USA,
          <volume>434</volume>
          (
          <year>2017</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <surname>Skamarock</surname>
            ,
            <given-names>W. C.</given-names>
          </string-name>
          and
          <article-title>Coauthors: A description of the advanced research WRF version 3</article-title>
          ,
          <string-name>
            <given-names>NCAR</given-names>
            <surname>Tech</surname>
          </string-name>
          .
          <source>Note NCAR/TN-475+STR</source>
          ,
          <volume>113</volume>
          (
          <year>2008</year>
          ), https://opensky.ucar.edu/islandora/object/technotes:500
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          <article-title>8. The GFS atmospheric model</article-title>
          ,
          <source>NCEP Office Note</source>
          <volume>442</volume>
          ,
          <issue>14</issue>
          (
          <year>2003</year>
          ), https://www.emc.ncep.noaa.gov/officenotes/ newernotes/on442.pdf
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <surname>Kui</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          , Wei-keng, L.,
          <string-name>
            <surname>Ross</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Latham</surname>
          </string-name>
          , R.:
          <article-title>Combining I/O operations for multiple array variables in parallel netCDF, Proceedings of the Workshop on interfaces and architectures for scientific data storage, held in conjunction with the IEEE Cluster Conference (</article-title>
          <year>2009</year>
          ), http://cucis.ece.northwestern.edu/publications/ pdf/GaoLia09B.pdf
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10.
          <article-title>UPP users' guide version 4, DTS</article-title>
          . Boulder, USA,
          <volume>24</volume>
          (
          <year>2019</year>
          ), https://dtcenter.org/upp/users/docs/user_guide/ V4/upp_users_guide.pdf
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <surname>Hunter</surname>
            ,
            <given-names>J. D.</given-names>
          </string-name>
          :
          <article-title>Matplotlib: a 2d graphics environment, Computing in science</article-title>
          and engineering.
          <volume>9</volume>
          :
          <fpage>90</fpage>
          -
          <lpage>95</lpage>
          (
          <year>2007</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <surname>Lyubitskiy</surname>
          </string-name>
          ,
          <string-name>
            <surname>Yu</surname>
          </string-name>
          . V.:
          <article-title>Possible improvement of wind characteristics' calculations for the storm surge forecast, Pacific oceanography</article-title>
          .
          <volume>1</volume>
          :
          <fpage>44</fpage>
          -
          <lpage>49</lpage>
          (
          <year>2003</year>
          )
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>