2. Activities during the Project
 2.1
Model Preparation
We planned to use the DREAM-Eta and
WRF-NMM-dust models contributed by Univ.
of Arizona in the
beginning of the project. Because of Dr. Dazhong Yin left at the beginning of
the project, funding resource to Univ.
of Arizona (including for Peja and Nickovic) and most of George Mason Univ.
(Jibo) resources for this project were redirected to finalize/reconfigure the
DREAM-Eta 8 bin (the 8-particle-size-bin version of DREAM-Eta, developed by
Nickovic and Peja), and also to develop a new model based on the NCEP NMM
model. The new model leverages theoretical knowledge from Nickovic and was
developed by Peja in rewriting from scratch line-by-line with Jibo virtually
side-by-side. We refer to the new NMM-based dust model as NMM-dust.
DREAM-Eta works
on-line with the NWS Eta operational forecast model. The Eta model is defined on the semi staggered
Arakawa E grid and uses the technique for preventing grid separation in
combination with split-explicit time differencing. The step-mountain Eta model has shown considerable
skill in forecasting severe storms, but is of a spatial resolution too coarse
for many potential applications. The horizontal grid spacing of DREAM-Eta 8
particle is 1/3 of a degree. With current horizontal resolutions, models used
for numerical weather prediction (NWP) are approaching limits of validity of
the hydrostatic approximation. The Eta model was replaced in NWS operations by
a Non-hydrostatic Mesoscale Model (NMM), which has higher resolution and
greater computational efficiency (Janjic et al., 2001, Janjic 2003), and NMM
was now replaced by WRF-NMM for operation at NCEP.
The NMM-dust model is used to test concepts of
interoperability with DREAM-eta and supercomputing for operation over the U.S.
The NMM-dust is executable in parallel mode on HPC that is based on the NCEP_NMM
weather forecasting model. The NMM-dust model can produce higher resolution
results for weather forecasting and can parallel run on HPC to obtain higher
resolution dust simulation results up to about one KM2.
Model
testing was performed through the following steps:
-
Initial tests were accomplished
by specifying first the elevated and then the surface point source with
arbitrary values of injected dust concentration. Only model dynamics were used
to test model performance, i.e. horizontal and vertical advection and
horizontal and vertical diffusion were used to disperse dust from the point
source.
-
When expected results were
obtained from the previous step, the second phase replaced the point source
with the geographically distributed dust sources, specified according MODIS (MOD12)
land cover data.
-
Final tests included all dust
components (dynamics and source/sink components) and results were compared with
the DREAM-Eta with the same domain/resolution specification; very comparable
features of the dust fields in both models were obtained.
 2.2 High Performance Computing (HPC)
The
project utilized GMU CISC cluster and its environment to provide HPC support.
The HPC has a configuration as illustrated in Figure 1.
The NMM dust model (with 1/9 degree
resolution) was tested by using 1, 4, 8, 16, 32, 64 and 112 CPU cores on the
GMU HPC server (with 28 computing nodes and 224 CPU cores) for performance
comparison. As illustrated in Figure 2, the best performance (cost of 111
seconds with a speedup factor of 9.54) was obtained when 64 CPU cores were used
although the model is in an efficiency of 9.54/64 = 15% in utilizing the CPU
cores.
The NMM dust model with 3km resolution was
tested by using 8, 36, 64, 81, 100, 210 CPU cores on the GMU cluster. Excluding preprocessing and postprocessing
time (Postprocessing is used to quilt the
separate tiles into composite result. It is time consuming, but can only run in
serial computing mode therefore fix in performance), the performance of the
model runs are tested and the best performance was obtained when using 81 CPU
cores are used as illustrated in Figure 3. 
2.3 Model Interoperability
Initially, DREAM-eta 8p model is run at a
coarser resolution and NMM-dust is run with a higher resolution. It would be
ideal if we can run NMM-dust for the entire forecasting region, but the NMM-dust
is very computing intensive and forecast run time is increased by a three to
four power magnitude with resolution increased in 3D space and time. For
example, if we increase the resolution from 1/3 of a degree to 1/9 of a degree
resolution, the computing time will increase by more than
3(latitude)*3(longitude)*3(altitude, because of the fixed layers of altitude,
this factor may not increase so fast) *3(time steps) = 81 times Therefore, it’s
not feasible to run the high resolution NMM model for the entire continent or
the world. Instead, we can use DREAM-eta with coarse resolution when a large
geographic area should be covered. The higher resolution model can be executed for
specific subregions based on initial dust storm identification of the coarse
model. This approach requires the interaction or interoperability between the
two models.
The ideal
approach of model interoperability is to use the coarse model to identify
hotspots of higher predicted dust concentration and run the fine-grain model on
the hotspot areas. This requires access to both models, and entails triggering the
high resolution NMM model based upon the output of the coarse resolution DREAM-eta
model. Therefore, we did a proof-of-concept study here to identify dust storms
on a static basis and simulate the near real-time dust model interactions.
The use case we
identified is to use the coarse model (DREAM-eta 8-bin) to identify hotspots of
higher predicted dust concentration and run the fine-grain model (NMM-dust) on
the hotspot areas. The dust simulation results from coarse resolution (1/3
degree) DREAM-eta dust model can serve as the initial background dust and can provide
more reasonable simulation results. The preprocessing of the NMM-dust model is
designed to transform and ingest the output from DREAM-eta dust model. After ingesting the DREAM-eta output the
NMM-dust can start simulation. Experiments have been performed to show results
of model interoperability. Figure 4
shows the coarse results simulated by DREAM-eta 8p dust model for the southwest
U.S.
region. Arizona and New Mexico sub
domains are selected for high resolution (1/10 degree) NMM-dust model runs as illustrated
in Figures 5 and 7 (white dotted). The
higher resolution results by the NMM-dust model are shown in Figures 6 and
8.
Model
interoperability work for the project has focused on the definition of standard
input and output file formats for both the 8p and NMM-dust models so that the
outputs of one could be used as inputs into the other. Specifically, it was
decided to use the GRIB1 as the file format for the outputs of both models,
allowing standard meteorological data processing tools to access and process
these products. Specifically, both models are able to read GRIB1 files for
model initialization and boundary condition specification, while GRIB1 files
may also be read and converted using the wgrib
utility for use in other systems. Figure 9 illustrates the flow of data through
the various system components enhanced or developed as part of this project.
The figure shows
the GRIB1 products (labeled “Met. / Dust Forecast”) generated by both the PHAiRS
project’s 4-bin model and the Interoperability Test’s 8-bin and NMM models
outputs. These products are then linked to the “Previous Model Run” model input
GRIB1 component that may be used to initialize any of the three models for a
subsequent model run. The use of a common, well supported data format for both
model initialization and output significantly streamlines the process of
developing multi-model workflows where the output of one model is used to
initialize another, either for a model run for a subsequent time step, or for
the execution of a higher resolution model for the same time period over which
a low-resolution model has already been run.
GRIB (GRIdded Binary) is a
mathematically concise data format commonly used in meteorology to store historical
and forecast weather data. It is standardized by the World Meteorological
Organization's Commission for Basic Systems, known under number GRIB FM 92-IX,
described in WMO Manual on Codes No.306 (WMO, 1992). Currently there are two
versions of GRIB, first edition (current sub-version is 2) is used
operationally world-wide by all meteorological centers, for Numerical Weather
Prediction output (NWP). A newer generation was introduced, known as GRIB
second edition, but it is used only by few centers and in many cases not for
operational broadcast. In our project, version 1 is used.
Using GRIB is an efficient vehicle for transmitting large volumes of gridded
data over high-speed telecommunication lines. By packing information into the
GRIB code, messages (or records - the terms are synonymous in this context) can
be made more compact than character oriented data formats, which permits faster
computer-to-computer transmissions. GRIB can equally serve as a data storage
format, generating the same efficiencies relative to information storage and
retrieval devices.
Each GRIB record
intended for either transmission or storage contains a single parameter with
values located at an array of grid points, or represented as a set of spectral
coefficients, for a single level (or layer), encoded as a continuous bit
stream. Logical divisions of the record are designated as "sections",
each of which provides control information and/or data. A GRIB record consists
of six sections, two of which are optional:
(0) Indicator Section
(1) Product Definition Section (PDS)
(2) Grid Description Section (GDS) - optional
(3) Bit Map Section (BMS) - optional
(4) Binary Data Section (BDS)
(5) '7777' (ASCII Characters)
Although the Grid
Description Section is indicated as optional, it is highly desirable that it be
included in all messages. That way there will be no question about just what is
the "correct" geographical grid for a particular field. The output dust
format is documented in annex A.
2.4
System Interoperability
The system interoperability activities for
this project have been related to the development of interoperable interfaces
with external data resources used by the PHAiRS system, and in the development
of enhanced interoperable services for the delivery of products and data to
PHAiRS system users. The interoperability work accomplished for this project
relates to the existing PHAiRS architecture (Figure 10) in two specific areas,
data ingest (labeled 1 in Figure 10) and map image delivery (labeled 2 in
Figure 10).
The open
standards employed in this project closely relate and add to the existing Open
Geospatial Consortium (OGC) standards deployed as part of the PHAiRS project.
Specifically, system support for OGC’s Web Map Service
(WMS,
de la Beaujardiere, 2006)
was
expanded to include time-enabled delivery of DREAM model outputs as a complement
of the existing time-enabled WMS for EPA AirNOW data. This enhancement greatly
streamlined the delivery of model outputs, and led to a re-engineering of some
of the existing web-based interfaces developed for the PHAiRS project. These
time-enabled WMS (such as http://phairs-devel.unm.edu/cgi-bin/dreamwms
and http://129.24.63.59/cgi-bin/mapserv?map=mapmodule_dream8bin_wms.map&) for the DREAM-Eta output are also the foundation
for the direct integration of DREAM-Eta data into the SYRIS syndromic
surveillance system – the public health decision support system that the PHAiRS
project is targeting for enhancement.
The delivery of
these time-enabled WMS products was further enhanced through the development of
automated KML generation scripts that are executable through a basic web form
(Figure 11). These scripts produce a custom KML file that allows for the
visualization of a time series of model outputs in any client that supports the
WMS and timespan components of the
OGC KML standard
(Wilson,
2008)
. This capability was
tested and demonstrated through the Google Earth virtual globe application
(Figure 12). 
In addition to the data delivery services described above, the use of interoperable standards-based interfaces was also increased through this project. In particular, automated data access scripts for the OGC Web Coverage Services (WCS 'Whiteside & Evans, 2006) published by NASA's Land Process Distributed Archive Center (http://lpdaac.usgs.gov/main.asp) were developed for the acquisition of MOD12 land cover data products. These data are used in the initialization of all versions of the DREAM model used in the PHAiRS and PHAiRS interoperability projects (Figure 1),with access to recent land cover data via WCS facilitating execution of the DREAM model with current vegetation data.
Additional work in the development of WCS for the outputs of the DREAM-Eta (4-bin) was performed, with initial progress, but ultimate success limited by the current capabilities of the software solutions currently in use by the PHAiRS project. Specifically, initial services were developed, but ultimate success was limited by the combination of server software (MapServer) and related database application (PostgreSQL/PostGIS) and discovered limitations in the combination of these two technologies to deliver the required time-enabled WCS services.
|
