I am coding a python plugin to QGIS that connects to the UK National Biodiversity Network (NBN) Web Mapping Service (WMS). In general, I don't have any problems doing this with the QgsRasterLayer API when I want to access the WMS as a public user. The problem is when I want to access the WMS as a registered user (to get access to secured data).
The NBN WMS provides a method for doing this by supplying a username and an MD5 hashed password as additional parameters to the web service called username and userkey respectively (https://data.nbn.org.uk/Documentation/Web_Services/Web_Map_Services/).
The parameters work as advertised when requests are tested in a browser. However, they have no effect when used through QGIS and when I used Fiddler to examine the HTTP requests that QGIS is actually sending to the NBN WMS, I find that the username and userkey parameters are no sent by QGIS.
They are non-standard parameters, but the OGC standard says that non-standard parameters are allowed (and don't break the standard). Does anyone know why QGIS strips these non-standard parameters out of the HTTP request or how I can get around it?
I believe that the url you provide to the QgsRasterLayer needs to be url encoded.
Otherwise it seems that the url parameter which you pass to the QgsRasterLayer splits upon the '&' character.
The following snippet demonstrates the creation of a QgsRasterlayer for Cochlearia danica L.
wmsService = 'https://gis.nbn.org.uk/SingleSpecies/NBNSYS0000002854?username=NBN_USER&userkey=PASS_KEY&' url = 'url=' + urllib.quote_plus(wmsService) # encode the url layer = QgsRasterLayer(url, 'NBN WMS layer', 'wms')
I have tested this in QGis 2.4
As a workaround you can include extra parameters into the WMS server base URL. Remember also to check the "Ignore GetMap URI reported in capabilities". If username and userkey are changed you must edit the WMS connection details accordingly. Standard does not say either that WMS clients should keep the vendor parameters. Make a feature request for QGIS and suggest that there should be a box for vendor parameters. Ask to make it large so that also SLD_BODY fits into the box. Check first if such feature request exists already.
A WMS (or Web Map Server) allows for use of data from several different servers, and enables for the creation of a network of Map Servers from which clients can build customized maps. The following document contains information about using MapServer’s WMS connection type to include layers from remote WMS servers in MapServer applications.
MapServer supports the following WMS versions when acting as client: 1.0.0, 1.0.7, 1.1.0, 1.1.1 (see MapServer OGC Specification support for an updated list).
This document assumes that you are already familiar with certain aspects of MapServer:
- MapServer application development and setting up .map files.
- Familiarity with the WMS spec would be an asset. A link to the WMS specification document is included below.
Getting started with GeoMesa using Geodocker
In a previous post, I showed how to use docker to run a single application (GeoServer) in a container and connect to it from your local QGIS install. Today’s post is about running a whole bunch of containers that interact with each other. More specifically, I’m using the images provided by Geodocker. The Geodocker repository provides a setup containing Accumulo, GeoMesa, and GeoServer. If you are not familiar with GeoMesa yet:
GeoMesa is an open-source, distributed, spatio-temporal database built on a number of distributed cloud data storage systems … GeoMesa aims to provide as much of the spatial querying and data manipulation to Accumulo as PostGIS does to Postgres.
The following sections show how to load data into GeoMesa, perform basic queries via command line, and finally publish data to GeoServer. The content is based largely on two GeoMesa tutorials: Geodocker: Bootstrapping GeoMesa Accumulo and Spark on AWS and Map-Reduce Ingest of GDELT, as well as Diethard Steiner’s post on Accumulo basics. The key difference is that this tutorial is written to be run locally (rather than on AWS or similar infrastructure) and that it spells out all user names and passwords preconfigured in Geodocker.
This guide was tested on Ubuntu and assumes that Docker is already installed. If you haven’t yet, you can install Docker as described in Install using the repository.
To get Geodocker set up, we need to get the code from Github and run the docker-compose command:
When docker-compose is finished, use a second console to check the status of all containers:
At the time of writing this post, the Geomesa version installed in this way is 1.3.2:
First we need to get some data. The available tutorials often refer to data published by the GDELT project. Let’s download data for three days, unzip it and copy it to the geodockeraccumulogeomesa_accumulo-master_1 container for further processing:
Loading or importing data is called “ingesting” in Geomesa parlance. Since the format of GDELT data is already predefined (the CSV mapping is defined in geomesa-tools/conf/sfts/gdelt/reference.conf), we can ingest the data:
Once the data is ingested, we can have a look at the the created table by asking GeoMesa to describe the created schema:
In the background, our data is stored in Accumulo tables. For a closer look, open an interactive terminal in the Accumulo master image:
and open the Accumulo shell:
When we store data in GeoMesa, there is not only one table but several. Each table has a specific purpose: storing metadata, records, or indexes. All tables get prefixed with the catalog table name:
By default, GeoMesa creates three indices:
Z2: for queries with a spatial component but no temporal component.
Z3: for queries with both a spatial and temporal component.
Record: for queries by feature ID.
But let’s get back to GeoMesa …
Now we are ready to query the data. Let’s perform a simple attribute query first. Make sure that you are in the interactive terminal in the Accumulo master image:
This query filters for a certain event id:
If the attribute query runs successfully, we can advance to some geo goodness … that’s why we are interested in GeoMesa after all … and perform a spatial query:
Functions that can be used in export command queries/filters are (E)CQL functions from geotools for the most part. More sophisticated queries require SparkSQL.
Publishing GeoMesa tables with GeoServer
To view data in GeoServer, go to http://localhost:9090/geoserver/web. Login with admin:geoserver.
First, we create a new workspace called “geomesa”.
Then, we can create a new store of type Accumulo (GeoMesa) called “gdelt”. Use the following parameters:
Then we can configure a Layer that publishes the content of our new data store. It is good to check the coordinate reference system settings and insert the bounding box information:
To preview the WMS, go to GeoServer’s preview:
Which will look something like this:
GeoMesa data filtered using CQL in GeoServer preview
For more display options, check the official GeoMesa tutorial.
If you check the preview URL more closely, you will notice that it specifies a time window:
This is exactly where QGIS TimeManager could come in: Using TimeManager for WMS-T layers. Interoperatbility for the win!
A PROJECTION must be set in the mapfile for the MAP unless you are sure that all your WMS layers support only a single projection which is the same as the PROJECTION of the map. The MAP PROJECTION can be set using «init=epsg:xxxx» codes or using regular PROJ parameters. Failure to set a MAP PROJECTION may result in blank maps coming from remote WMS servers (because of inconsistent BBOX+SRS combination being used in the WMS connection URL).
Storing Temporary Files¶
Since MapServer version 6.0 when wms_cache_to_disk metadata is turned on (and for MapServer < 6.0), you have to set the IMAGEPATH value in the WEB object of your mapfile to point to a valid and writable directory. MapServer will use this directory to store temporary files downloaded from the remote servers. The temporary files are automatically deleted by MapServer so you won’t notice them.
Example 3. Setting IMAGEPATH Parameter in Mapfile
MS4W Users Should Specify the Following for IMAGEPATH and IMAGEURL:
If you want to keep this temporary file for debugging purposes, you should add the following statement to the LAYER object of your mapfile:
Configuring for HTTPS Connections¶
Follow the Accessing OGC services over HTTPS document to make sure that your local server can connect through HTTPS secured connections.
Adding a WMS Layer¶
WMS layers are accessed via the WMS connection type in the Mapfile . Here is an example of a layer using this connection type:
You can also combine remote WMS layers from a server into one layer, separated by a comma (,) such as:
Layer names must be separated by a comma, without any spaces around the comma
Required Layer Parameters and Metadata¶
Description: This is the remote server’s online resource URL, just the base URL without any of the WMS parameters. The server version, image format, layer name, etc. will be provided via metadata, see below.
Note that if the CONNECTION parameter value is not set the the value of the «wms_onlineresource» metadata will be used. If both CONNECTION and «wms_onlineresource» are set then the «wms_onlineresource» metadata takes precedence.
Description: The image format to use in GetMap requests.
If wms_formatlist is provided then wms_format is optional and MapServer will pick the first supported format in wms_formatlist for use in GetMap requests. If both wms_format and wms_formatlist are provided then wms_format takes precedence. Also note that WMS Servers only advertize supported formats that are part of the GD/GDAL libraries.
Description: Comma-separated list of layers to be fetched from the remote WMS server. This value is used to set the LAYERS and QUERY_LAYERS WMS URL parameters. Note that when specifying multiple layers there must not be any spaces between the comma and the layer name.
Description: The version of the WMS protocol supported by the remote WMS server and that will be used for issuing GetMap requests.
Description: Space-delimited list of EPSG projection codes supported by the remote server. You normally get this from the server’s capabilities output. This value should be upper case (EPSG:4236…..not epsg:4236) to avoid problems with case sensitive platforms. The value is used to set the SRS WMS URL parameter.
Optional Layer Parameters and Metadata¶
Description: If the remote server’s capabilities contains a ScaleHint value for this layer then you might want to set the MINSCALE and MAXSCALE in the LAYER object in the mapfile. This will allow MapServer to request the layer only at scales where it makes sense
Description: It is optional at this point. MapServer will create one internally if needed. Including one may allow MapServer to avoid looking up a definition in the PROJ init files.
Description: msEncrypt-style authorization string. Empty strings are also accepted.
any (the underlying http library picks the best among the opotions supported by the remote server)
anysafe (the underlying http library picks only safe methods among the options supported by the remote server)
Description: The maximum time to wait for a remote WMS layer to load, set in seconds (default is 30 seconds). This metadata can be added at the layer level so that it affects only that layer, or it can be added at the map level (in the web object) so that it affects all of the layers. Note that wms_connectiontimeout at the layer level has priority over the map level.
Description: Set this to «1» to mark this WMS layer as essential. Normally, when a request to a WMS layer fails, this layer will be skipped and the map is rendered without the layer. If the layer is essential and fails, the map will not be rendered and an exception is thrown.
enabling wms_essential means that if there is a problem with the connection, (such as the CONNECTION url cannot be reached, or an incorrect wms_name or wms_format ) an XML error will be returned in the browser. If you are calling MapServer through commandline, your MAP setting for CONFIG «ON_MISSING_DATA» will still be leveraged for how that error is handled locally.
Description: Set the format for exceptions (as of MapServer 4.6). MapServer defaults to application/vnd.ogc.se_inimage (the exception will be in a picture format). You can check the GetCapabilities of the server to see what formats are available for exceptions. The application/vnd.ogc.se_inimage exception format is actually a non-required exception format in the WMS 1.1.1 spec, so there are servers out there which don’t support this format. In that case you would use:
Which would return this xml exception in the MS_ERRORFILE:
Description: Set this to «1» to force this WMS layer to be requested using its own separate GetMap request. By default MapServer will try to merge multiple adjacent WMS layers from the same server into a single multi-layer GetMap request to reduce the load on remote servers and improve response time. This metadata is used to bypass that behavior.
Description: Comma-separated list of image formats supported by the remote WMS server. Note that wms_formatlist is used only if wms_format is not set. If both wms_format and wms_formatlist are provided then wms_format takes precedence.
Description: The bounding box of this layer in geographic coordinates in the format «lon_min lat_min lon_max lat_max». If it is set then MapServer will request the layer only when the map view overlaps that bounding box. You normally get this from the server’s capabilities output.
Description: The authorization type to use for a proxy connection. Supported types include:
any (the underlying http library picks the best among the opotions supported by the remote server)
anysafe (the underlying http library picks only safe methods among the options supported by the remote server)
Description: The hostname of the proxy to use, in «dot-quad» format, with an optional port component (e.g. “192.168.2.10:8080”).
Description: The port to use for a proxy connection.
Description: The type of the proxy connection. Valid values are “http” and “socks5”, which are case sensitive.
Description: msEncrypt-style string for a proxy connection. Empty strings are also accepted.
Description: Can be used to specify an inline SLD document.
Description: Can be used to specify a link to an SLD document.
Description: Name of style to use for the STYLES parameter in GetMap requests for this layer.
Description: URL of a SLD to use in GetMap requests. Replace <stylename> in the metadta name with the name of the style to which the SLD applies.
For more information on SLDs in MapServer see the SLD HowTo document .
Description: Value to use for the TIME parameter in GetMap requests for this layer. Please see the WMS Time HowTo for more information.
Description: Specifies the color to be used as the background of the map. The general format of BGCOLOR is a hexadecimal encoding of an RGB value where two hexadecimal characters are used for each of Red, Green, and Blue color values. The values can range between 00 and FF for each (0 and 255, base 10). The format is 0xRRGGBB either upper or lower case characters are allowed for RR, GG, and BB values. The «0x» prefix shall have a lower case «x».
Description: Specifies whether the map background is to be made transparent or not. TRANSPARENT can take on two values, «TRUE» or «FALSE». If not specified, MapServer sets default to «TRUE».
Description: Set this to «1» to force MapServer to write fetched images to disk. Writing to disk is necessary to take advantage of MapServer’s caching logic to avoid refetching WMS requests made previously. This feature is new to MapServer 6.0 - previously results were always written to disk.
Description: Set this to «0» to indicate that the remote WMS only supports requests for square pixels. In this case MapServer will be careful to only make square pixel requests even if it means oversampling in one dimension compared to the resolution of image data required. This feature is new to MapServer 6.0.
Description: If there is exactly one SRS supported by this layer (as listed in the wms_srs metadata), and if the wms_extent metadata item (or an extent specified via the EXTENT keyword) is set then MapServer will take care to only making requests within this area. This can short circuit requests completely outside the layer, reduce processing for layers that only partially overlap the target map area and avoid poor behaviors with reprojection in some areas. The contents of this metadata item should be of the form «minx miny maxx maxy». This feature is new to MapServer 6.0.
Description: Set this to «1» or «true» to force WMS requests to use strict axis order according to the EPSG code, or «0» or «false» to force WMS requests to always use xy (or lonlat) axis order. This should only be necessary if the axis order interpretation of the server for the chosen EPSG code does not conform to the standard: In the case of WMS 1.0 and 1.1 the assumption is to always use xy (default 0/false), for WMS 1.3.0 the assumption is strict axis order according to the EPSG database (default 1/true).
Note that each of the above metadata can also be referred to as “ows_*” instead of “wms_*”. MapServer tries the “wms_*” metadata first, and if not found it tries the corresponding “ows_*” name. Using this reduces the amount of duplication in mapfiles that support multiple OGC interfaces since «ows_*» metadata can be used almost everywhere for common metadata items shared by multiple OGC interfaces.
Old CONNECTION parameter format from version 3.5 and 3.6 (deprecated)¶
In MapServer version 3.5 and 3.6, the CONNECTION parameter had to include at a minimum the VERSION, LAYERS, FORMAT and TRANSPARENT WMS parameters. This mode of operation is still supported but is deprecated and you are encouraged to use metadata items for those parameters as documented in the previous section above.
Here is an example of a layer definition using this deprecated CONNECTION parameter format:
WMS 1.3.0 Support¶
MapServer 5.4 adds support for WMS 1.3.0. Although the general mechanism in MapServer to support this new specification are the same, there are some notable upgrades.
Major features related to the WMS 1.3.0 support¶
Support WMS 1.3.0 basic operations: GetCapabilities, GetMap and GetFeatureInfo.
Implement the Styled Layer Descriptor profile of the Web Map Service Implementation Specification. This specification extends the WMS 1.3.0 and allows to advertise styling capabilities (Styled Layer Descriptor (SLD) support). It also defines two additional operations, GetLegendGraphic and DescribeLayer.
Implement the Symbology Encoding Implementation Specification, which is the new version of the SLD. Read support was added for Point, Line, Polygon, Raster symbolizers.
Upgrade the generation of SLD to version 1.1.0 (SLD generated through through the GetStyles operation or through MapScript).
Coordinate Systems and Axis Orientation¶
The most notable changes introduced in WMS 1.3.0 are the:
the introduction of new coordinate reference systems
the use of CRS parameter (instead of SRS)
The axis order in previous versions of the WMS specifications was to always use easting (x or lon ) and northing (y or lat). WMS 1.3.0 specifies that, depending on the particular CRS, the x axis may or may not be oriented West-to-East, and the y axis may or may not be oriented South-to-North. The WMS portrayal operation shall account for axis order. This affects some of the EPSG codes that were commonly used such as ESPG:4326. MapServer 5.x makes sure that coordinates passed to the server (as part of the GetMap BBOX parameter) as well as those advertised in the capabilities document reflect the inverse axe orders for EPSG codes between 4000 and 5000.
MapServer 6.0 and up holds a list of EPSG codes with inverted axis order. It is currently based on EPSG database version 7.6. It is also possible to define the axis order at build time for a specific ESPG code (see issue #3582). This allows for example to use the «normal» axis order for some of EPSG codes between 4000 and 5000.
In addition, the WMS 1.3.0 defines a series of new coordinate system. These are the ones that are currently supported in MapServer:
CRS:84 (WGS 84 longitude-latitude)
CRS:83 (NAD83 longitude-latitude)
CRS:27 (NAD27 longitude-latitude)
AUTO2:42001 (WGS 84 / Auto UTM )
AUTO2:42002 (WGS 84 / Auto Tr. Mercator)
AUTO2:42003 (WGS 84 / Auto Orthographic)
AUTO2:42004 (WGS 84 / Auto Equirectangular)
AUTO2:42005 (WGS 84 / Auto Mollweide)
Example of requests¶
Users can use the CRS:84 coordinate system and order the BBOX coordinates as long/lat:
Users can also use the ESPG:4326 coordinates and use the axis ordering of lat/long:
Other notable changes¶
valid values for the EXCEPTIONS parameter in a GetMap request are XML, INIMAGE, BLANK
valid value for the EXCEPTIONS parameter in a GetFeatureInfo request is XML
LayerLimit is introduced, allowing a server to advertise and limit the number of layers a client is allowed to include in a GetMap request
Some Missing features¶
WMS 1.3.0 Post request should be an XML document containing the different operations and parameters.
SLD documents containing elements from the Feature Encoding 1.1 specification could potentially use ESPG projections with some filters. It is not yet clear nor implemented if the axis ordering should be taken into account in these specific cases.
OCG compliance tests¶
As of version 5.4, MapServer passes all the basic and query tests of the OGC CITE test suite for WMS 1.3.0.
Consuming WMS services
To connect to a WMS service, you need to know the URL. WMS services published with ArcGIS Server have this URL format:
Remember that the WMS capability is available for both map services and image services. This is why there are two options for the service type.
For example, if you have a folder Japan containing the map service Tokyo, running on myServer with the default instance arcgis, the URL of your WMS service would look like this:
If you have an image service IdahoImages running on myServer with the instance name of PublicLands, your URL for the WMS service would look like this:
Common WMS clients
A Web browser is the simplest client of a WMS service. WMS requests can be issued through HTTP, and the responses or exceptions are returned through the browser. WMS services support three operations: GetCapabilities, GetMap, and GetFeatureInfo. Through URL parameters, a client can use these operations to obtain metadata, maps, and feature information from the WMS service. These operations and parameters are detailed in the OGC WMS specifications.
ESRI WMS clients include ArcGIS Desktop (ArcCatalog and ArcMap), ArcExplorer, ArcGIS Server Web Application Developer Framework (ADF) for .NET, ArcGIS Server Web ADF for Java, and ESRI GIS Portal Toolkit Map Viewer.
The namespace parameter causes WMS GetCapabilities responses to be filtered to only contain layers in to a particular namespace. The syntax is:
where <namespace> is the namespace prefix.
Using an invalid namespace prefix will not cause an error, but the capabilities document returned will contain no layers, only layer groups.
This affects the capabilities document only, not other requests. Other WMS operations will still process all layers, even when a namespace is specified.
Compilation / Installation¶
The WMS connection type is enabled by the --with-wmsclient configure switch. It requires PROJ, GDAL and libcurl version 7.10.1 or more recent. Windows users who do not want to compile MapServer should use MS4W (which comes ready for WMS/WFS client and server use), or check for the availability of other Windows binaries with WMS support.
For PROJ and GDAL installation, see the MapServer Compilation HowTo ( Compiling on Unix / Compiling on Win32 )
For libcurl, make sure you have version 7.10.1 or more recent installed on your system. You can find out your libcurl version using curl-config --version. (if your system came with an older version of libcurl preinstalled then you MUST uninstall it prior to installing the new version)
Once the required libraries are installed, then configure MapServer using the --with-wmsclient switch (plus all the other switches you used to use) and recompile. This will give you a new set of executables (and possibly php_mapscript.so if you requested it). See the MapServer Compilation HowTo (links above) for installation details.
Check your MapServer executable¶
To check that your mapserv executable includes WMS support, use the "-v" command-line switch and look for "SUPPORTS=WMS_CLIENT".
Example 1. Mapserv Version Info on Unix:
Example 2. Mapserv Version Info on Windows:
WMS servers that support tiling should publish a profile as part of their GetCapabilities advertisement, which specifies all of the information needed to request valid tiles from the tile server. The required information comes in two varieties: (1) WMS request parameters which must be supplied to the server verbatim, and (2) a tile grid definition, which a client can use to calculate the extents of valid tiles.
Verbatim Request Parameters
Each tiling profile needs to establish verbatim values for the following WMS request parameters:
Additionally, each tiled layer is defined in terms of verbatim values for:
Tile Grid Definition
Rather than rely on the much-overloaded concept of "scale," this recommendation instead will describe scales or zoom levels in terms of "resolution," defined in terms of map units (typically degrees or meters) per pixel.
In order to be cacheable, requested tiles must have bounding boxes aligned to a set of consistent grids at successively larger map resolutions in geographic space. Tile grids originate in the lower left corner of the tiled layer's BoundingBox.
A tiling profile must supply a list of supported resolutions, corresponding to the different "zoom levels" it may be used to serve.
The bounding box for a valid tile request must align to the grid given by the layer's tiling profile. In practical terms, this means the bbox coordinates must be equal to the grid origin, plus some non-negative integer multiple of the tile size in pixels, multiplied by one of the supported resolutions listed in the layer's tiling profile.
The following pre-defined global profiles use resolution values that are related by powers of two. Servers may offer tiles at as many resolution levels as are appropriate to the data represented.
All tiling WMS servers should be able to deliver tiles in a generic "unprojected" global profile, with the following tiling parameters:
- Width: 256 px
- Height: 256 px
- Format: image/png
- SRS: EPSG:4326
- BoundingBox: -180 -90, 180 90
- Resolutions: 0.703125, 0.3515625 .
There are two tiles at the highest resolution of the Unprojected Profile.
- Width: 256 px
- Height: 256 px
- Format: image/png
- SRS: OSGEO:41001
- BoundingBox: -20037508.34 -20037508.34 20037508.34 20037508.34
- Resolutions: 156543.03390625 78271.516953125 .
There is one tile at the highest resolution of the basic Mercator profile.
Tiling WMS servers should include the following internal DTD to advertise the presence of tiling metadata, as recommended by the OGC WMS 1.1.1 specification:
A suitable tiling WMS GetCapabilities response might look like this:
This declaration asserts that our hypothetical tiling WMS server supports tiling in the two basic global profiles for the coastline layer defined elsewhere in the capabilities response. Note that the Resolutions list is whitespace separated.
Once the petascope servlet is deployed (TODO see installation guide), the following service endpoints are available:
- rasdaman/ows : serving OGC Web Services (OWS) like WCS, WCPS, WMS and WCS-T:
- rasdaman/rasql : direct RasQL.
For example, assuming that the service’s IP address is 123.456.789.1 and the service port is 8080 , the following request URLs would deliver the Capabilities documents for OGC WMS and WCS, respectively:
The results presented provide an initial evaluation of the potential usefulness and reliability of GloFAS-Seasonal forecasts. For decision-making purposes, it is important to measure the ability of a forecasting system to predict the correct category of an event. As such, an event-based evaluation of the forecasts is used to assess whether the forecasts were able to correctly predict observed high and low river flow events over a 17-year period and whether it is able to do so with good reliability. The initial results are promising, indicating that the forecasts are, on average, potentially useful up to 1–2 months ahead in many rivers worldwide and up to 3–4 months ahead in some locations. The GloFAS-Seasonal forecasts have sharpness, i.e. they are able to predict forecasts with probabilities that differ from climatology, and overall have better reliability than a forecast of climatology, but with a tendency to over-predict at higher probabilities. It is also clear that there is a frequency bias in the reliability results, as often there is a small sample of high-probability forecasts. Typically, the reliability is seen to be better when there is a higher forecast frequency on which to base the results. As would be expected, the potential usefulness and reliability of the forecasts vary by region, season, and forecast lead time.
Considering the evaluation results by season allows for further analysis of the times of year in which the forecasts are potentially useful and/or reliable. For example, in south-east Australia, forecasts are seen to be potentially useful up to 4 months ahead in JJA and SON, but for forecasts produced in DJF the skill only extends to 1 month ahead, and forecasts are less skilful than climatology at several of the stations in MAM. In many rivers across the globe, it is the case that forecasts are potentially useful in some seasons, but not in others, and may be more reliable in certain seasons than others. As such, the maps provided in Figs. S1 and S2 are intended to highlight where and when the forecasts are likely to be useful, information that is key in terms of decision-making.
It is clear that there are regions and seasons in which the forecasts are less skilful than climatology and do not have good reliability, and thus in these rivers it would be more useful to use a long-term average climatology than seasonal hydro-meteorological forecasts of river flow. This lack of skill could be due to several factors, such as certain hydrological regimes that may not be well-represented in the hydrological model or may be difficult to forecast at these lead times (for example, snow-dominated catchments or regions where convective storms produce most of the rainfall in some seasons), poor skill of the meteorological forecast input, poor initial conditions from the ERA5-R reanalysis, extensive management of rivers that cannot be represented by the current model, or the lack of model calibration. While this initial evaluation is designed to provide an overview of whether the forecasts are potentially useful and reliable in predicting high and low flow events, more extensive analysis is required to diagnose the sources of predictability in the forecasts and the potential causes of poor skill. Additionally, it is evident that observations of river flow, particularly covering the reforecast period, are both spatially and temporally limited across large areas of the globe. A more extensive analysis should make use of the globally consistent ERA5-R river flow reanalysis as a benchmark in order to fully assess the forecast skill worldwide, including in regions where no observations are available.
The verification metrics used also require that a high or low flow event is predicted with the correct timing in the same week as that in which it occurred. This is asking a lot of a seasonal forecasting system and for many applications, such as water resources and reservoir management, a forecast of the exact week in which an event is expected at a lead time of several months ahead may not be necessary. That such a system shows real skill despite this being a tough test for the model and is able to successfully predict observed high or low river flow in a specific week, several weeks or months ahead, provides optimism for the future of global-scale seasonal hydro-meteorological forecasting. Further evaluation should aim to assess the skill of the forecasts with a more relaxed constraint on the event timing and also make use of alternative skill measures to cover different aspects of the forecast skill, such as the spread and bias of the forecasts. It will also be important to assess whether the use of weekly averaged river flow is the most appropriate way to display the forecasts. While this is commonly used for applications such as drought early awareness and water resources management, there may be other aspects of decision-making, such as flood forecasting, for which other measures may be more appropriate, for example daily averages or floodiness (Stephens et al., 2015).
Future development of GloFAS-Seasonal will aim to address these evaluation results and improve the skill and reliability of the current forecasts it will also aim to overcome some of the grand challenges in operational hydrological forecasting, such as seamless forecasting and the use of data assimilation. Seamless forecasting will be key in the future development of GloFAS the use of two different meteorological forecast inputs for the medium-range and seasonal versions of the model means that discrepancies can occur between the two timescales, thus producing confusing and inconsistent forecast information for users. Additionally, the use of river flow observations could lead to significant improvements in skill through calibration of the model using historical observations and assimilation of real-time data to adjust the forecasts. This remains a grand challenge due to the lack of openly available river flow data, particularly in real time.
In this paper, the development and implementation of a global-scale operational seasonal hydro-meteorological forecasting system, GloFAS-Seasonal, was presented, and an event-based forecast evaluation was carried out using two different but complementary verification metrics to assess the capability of the forecasts to predict high and low river flow events.
GloFAS-Seasonal provides forecasts of high or low river flow out to 4 months ahead for the global river network through three new forecast product layers via the openly available GloFAS web interface at http://www.globalfloods.eu (last access: 16 August 2018). Initial evaluation results are promising, indicating that in many rivers, forecasts are both potentially useful, i.e. more skilful than a long-term average climatology out to several months ahead in some cases, and overall more reliable than a forecast of climatology. Forecast skill and reliability vary significantly by region and by season.
The initial evaluation, however, also indicates a tendency of the forecasts to over-predict in general, and in some regions forecasts are currently less skilful than climatology future development of the system will aim to improve the forecast skill and reliability with a view to providing potentially useful forecasts across the globe. Development of GloFAS-Seasonal will continue based on results of the forecast evaluation and on feedback from GloFAS partners and users worldwide in order to provide a forecast product that remains state of the art in hydro-meteorological forecasting and caters to the needs of its users. Future versions are likely to address some of the grand challenges in hydro-meteorological forecasting in order to improve forecast skill, such as data assimilation, and will also include more features, such as flexible percentile thresholds and indication of the forecast skill via the interface. A further grand challenge that is important in terms of global-scale hydro-meteorological forecasting, and indeed for the development of GloFAS, is the need for more observed data (Emerton et al., 2016), which is essential not only for providing initial conditions to force the models, but also for evaluation of the forecasts and continuous improvement of forecast accuracy.
While such a forecasting system requires extensive computing resources, the potential for use in decision-making across a range of water-related sectors, and the promising results of the initial evaluation, suggest that it is a worthwhile use of time and resources to develop such global-scale systems. Recent papers have highlighted the fact that seasonal forecasts of precipitation are not necessarily a good indicator of potential floodiness and called for investment in better forecasts of seasonal flood risk (Coughlan De Perez et al., 2017 Stephens et al., 2015). Coughlan de Perez et al. (2017) state that “ultimately, the most informative forecasts of flood hazard at the seasonal scale could be seasonal streamflow forecasts using hydrological models” and that better seasonal forecasts of flood risk could be hugely beneficial for disaster preparedness.
GloFAS-Seasonal represents a first attempt at overcoming the challenges of producing and providing openly available seasonal hydro-meteorological forecast products, which are key for organisations working at the global scale and for regions where no other forecasting system exists. We provide, for the first time, seasonal forecasts of hydrological variables for the global river network by driving a hydrological model with seasonal meteorological forecasts. GloFAS-Seasonal forecasts could be used in addition to other forecast products, such as seasonal rainfall forecasts and short-range forecasts from national hydro-meteorological centres across the globe, to provide useful added information for many water-related applications from water resources management and agriculture to disaster risk reduction.
The ECMWF IFS source code is available subject to a licence agreement, and as such access is available to the ECMWF member-state weather services and other approved partners. The IFS code is also available for educational and academic purposes as part of the OpenIFS project (ECMWF, 2011, 2018a), with full forecast capabilities and including the HTESSEL land surface scheme, but without modules for data assimilation. Similarly, the GloFAS river routing component source code is not openly available however, the “forecast product” code (prior to implementation in ecFlow) that was newly developed for GloFAS-Seasonal and used for a number of tasks such as computing exceedance probabilities and producing the graphics for the interface is provided in the Supplement.
ECMWF's ERA5 reanalysis and SEAS5 reforecasts are available through the Copernicus Climate Data Store (Copernicus, 2018a). The ERA5-R river flow reanalysis and the GloFAS-Seasonal reforecasts (daily data) are currently available from the authors on request and will be made available through ECMWF's data repository in due course. The majority of the observed river flow data were provided by the Global Runoff Data Centre (GRDC BfG, 2017). These data are freely available from https://www.bafg.de/ (last access: 16 August 2018). Additional data were provided by the Russian State Hydrological Institute (SHI, 2018), the European Flood Awareness System (EFAS, 2017), Somalia Water and Land Information Management (SWALIM, 2018), South Africa Department for Water and Sanitation (DWA, 2018), Colombia Institute of Hydrology, Meteorology and Environmental Studies (IDEAM, 2014), Nicaragua Institute of Earth Studies (INETER, 2016), Dominican Republic National Institute of Hydraulic Resources (INDRHI, 2017), Brazil National Centre for Monitoring and Forecasting of Natural Hazards (Cemaden, 2017), Environment Canada Water Office (Environment Canada, 2014), Nepal Department of Hydrology and Meteorology (DHM, 2017), Red Cross Red Crescent Climate Centre (RCCC, 2018), Chile General Water Directorate (DGA, 2018), and the Historical Database on Floods (BDHI, 2018).
The supplement related to this article is available online at: https://doi.org/10.5194/gmd-11-3327-2018-supplement.
FP proposed the development of GloFAS-Seasonal, RE wrote the GloFAS-Seasonal forecast product code, and RE and LA designed the forecast products. EZ built the ecFlow suite and produced ERA5-R and the GloFAS-Seasonal reforecasts, and DM provided technical support for the website and operational implementation. RE evaluated the forecasts and wrote the paper, with the exception of Sect. 2.4, written by DM. All authors were involved in discussions throughout development, and all authors commented on the paper.
The authors declare that they have no conflict of interest.
This work has been funded by the Natural Environment Research Council (NERC) as part of the SCENARIO Doctoral Training Partnership under grant NE/L002566/1. Ervin Zsoter, Davide Muraro, Christel Prudhomme, and Peter Salamon were supported by the Copernicus Emergency Management Service – Early Warning Systems (CEMS-EWS EFAS). Louise Arnal, Hannah L. Cloke, and Florian Pappenberger acknowledge financial support from the Horizon 2020 IMPREX project (grant agreement no. 641811). Elisabeth M. Stephens is thankful for support from NERC and the Department for International Development (grant number NE/P000525/1) under the Science for Humanitarian Emergencies and Resilience (SHEAR) research programme (project FATHUM: Forecasts for AnTicipatory HUManitarian action).
Edited by: Jeffrey Neal
Reviewed by: two anonymous referees
Alfieri, L., Burek, P., Dutra, E., Krzeminski, B., Muraro, D., Thielen, J., and Pappenberger, F.: GloFAS – global ensemble streamflow forecasting and flood early warning, Hydrol. Earth Syst. Sci., 17, 1161–1175, https://doi.org/10.5194/hess-17-1161-2013, 2013.
Arnal, L., Cloke, H. L., Stephens, E., Wetterhall, F., Prudhomme, C., Neumann, J., Krzeminski, B., and Pappenberger, F.: Skilful seasonal forecasts of streamflow over Europe?, Hydrol. Earth Syst. Sci., 22, 2057–2072, https://doi.org/10.5194/hess-22-2057-2018, 2018.
Bahra, A.: Managing work flows with ecFlow, ECMWF Newsl., 129, 30–32 available from: https://www.ecmwf.int/sites/default/files/elibrary/2011/14594-newsletter-no129-autumn-2011.pdf (last access: 18 April 2018), 2011.
Balsamo, G., Pappenberger, F., Dutra, E., Viterbo, P., and van den Hurk, B.: A revised land hydrology in the ECMWF model: a step towards daily water flux prediction in a fully-closed water cycle, Hydrol. Process., 25, 1046–1054, https://doi.org/10.1002/hyp.7808, 2011.
BDHI: Base de Donnees Historiques sur les Inondations, available at: http://bdhi.fr/appli/web/welcome , last access: 23 April 2018.
Bell, V. A., Davies, H. N., Kay, A. L., Brookshaw, A., and Scaife, A. A.: A national-scale seasonal hydrological forecast system: development and evaluation over Britain, Hydrol. Earth Syst. Sci., 21, 4681–4691, https://doi.org/10.5194/hess-21-4681-2017, 2017.
Bennett, J. C., Wang, Q. J., Li, M., Robertson, D. E., and Schepen, A.: Reliable long-range ensemble streamflow forecasts: Combining calibrated climate forecasts with a conceptual runoff model and a staged error model, Water Resour. Res., 52, 8238–8259, https://doi.org/10.1002/2016WR019193, 2016.
Bennett, J. C., Wang, Q. J., Robertson, D. E., Schepen, A., Li, M., and Michael, K.: Assessment of an ensemble seasonal streamflow forecasting system for Australia, Hydrol. Earth Syst. Sci., 21, 6007–6030, https://doi.org/10.5194/hess-21-6007-2017, 2017.
BfG: The GRDC, available at: http://www.bafg.de/GRDC/EN/Home/homepage_node.html (last accessed: 23 April 2018), 2017.
BoM: Seasonal Streamflow Forecasts: Water Information: Bureau of Meteorology, available at: http://www.bom.gov.au/water/ssf/about.shtml , last access: 24 April 2018.
Candogan Yossef, N., van Beek, R., Weerts, A., Winsemius, H., and Bierkens, M. F. P.: Skill of a global forecasting system in seasonal ensemble streamflow prediction, Hydrol. Earth Syst. Sci., 21, 4103–4114, https://doi.org/10.5194/hess-21-4103-2017, 2017.
Cemaden: Cemaden – Centro Nacional de Monitoramento e Alertas de Desastres Naturais, available at: http://www.cemaden.gov.br/ (last access: 23 April 2018), 2017.
Chiew, F. H. S. and McMahon, T. A.: Global ENSO-streamflow teleconnection, streamflow forecasting and interannual variability, Hydrol. Sci. J., 47, 505–522, https://doi.org/10.1080/02626660209492950, 2002.
Chow, V. Te, Maidment, D. R., and Mays, L. W.: Applied hydrology, Tata McGraw-Hill Education, available at: https://books.google.co.uk/books/about/Applied_Hydrology.html?id=RRwidSsBJrEC&redir_esc=y (last access: 17 November 2017), 2010.
Cloke, H., Pappenberger, F., Thielen, J., and Thiemig, V.: Operational European Flood Forecasting, in Environmental Modelling, John Wiley & Sons, Ltd, Chichester, UK, 415–434, 2013.
Copernicus: Copernicus Climate Data Store, available at: https://climate.copernicus.eu/climate-data-store , last access: 23 April 2018a.
Copernicus: SWICCA, Service for Water Indicators in Climate Change Adaptation, available at: http://swicca.climate.copernicus.eu/ , last access: 12 January 2018b.
Coughlan de Perez, E., Stephens, E., Bischiniotis, K., van Aalst, M., van den Hurk, B., Mason, S., Nissan, H., and Pappenberger, F.: Should seasonal rainfall forecasts be used for flood preparedness?, Hydrol. Earth Syst. Sci., 21, 4517–4524, https://doi.org/10.5194/hess-21-4517-2017, 2017.
Crochemore, L., Ramos, M.-H., and Pappenberger, F.: Bias correcting precipitation forecasts to improve the skill of seasonal streamflow forecasts, Hydrol. Earth Syst. Sci., 20, 3601–3618, https://doi.org/10.5194/hess-20-3601-2016, 2016.
Demargne, J., Wu, L., Regonda, S. K., Brown, J. D., Lee, H., He, M., Seo, D.-J., Hartman, R., Herr, H. D., Fresch, M., Schaake, J., Zhu, Y., Demargne, J., Wu, L., Regonda, S. K., Brown, J. D., Lee, H., He, M., Seo, D.-J., Hartman, R., Herr, H. D., Fresch, M., Schaake, J., and Zhu, Y.: The Science of NOAA's Operational Hydrologic Ensemble Forecast Service, . Am. Meteorol. Soc., 95, 79–98, https://doi.org/10.1175/BAMS-D-12-00081.1, 2014.
DGA: Ministerio de Obras Públicas – Dirección de General de Aguas, available at: http://www.dga.cl/Paginas/default.aspx , last access: 23 April 2018.
DHM: Department of Hydrology and Meteorology, available at: http://www.dhm.gov.np/ (last access: 23 April 2018), 2017.
DWA: Department: WAter and Sanitation, available at: http://www.dwa.gov.za/default.aspx , last access: 23 April 2018.
ECMWF: OpenIFS, available at: https://www.ecmwf.int/en/research/projects/openifs (last access: 16 August 2018), 2011.
ECMWF: ecFlow Documentation, available at: https://software.ecmwf.int/wiki/display/ECFLOW/Documentation (last access: 18 April 2018), 2012.
ECMWF: What are the changes from ERA-Interim to ERA5? – Copernicus Knowledge Base – ECMWF Confluence Wiki, available at: https://software.ecmwf.int/wiki/pages/viewpage.action?pageId=74764925 (last access: 24 April 2018), 2017b.
ECMWF: About OpenIFS, available at: https://software.ecmwf.int/wiki/display/OIFS/About+OpenIFS , last access: 26 April 2018a.
EFAS: European Flood Awareness System (EFAS), available at: https://www.efas.eu/ (last access: 23 April 2018), 2017.
Emerton, R., Cloke, H. L., Stephens, E. M., Zsoter, E., Woolnough, S. J., and Pappenberger, F.: Complex picture for likelihood of ENSO-driven flood hazard, Nat. Commun., 8, 14796, https://doi.org/10.1038/ncomms14796, 2017.
Emerton, R. E., Stephens, E. M., Pappenberger, F., Pagano, T. C., Weerts, A. H., Wood, A. W., Salamon, P., Brown, J. D., Hjerdt, N., Donnelly, C., Baugh, C. A., and Cloke, H. L.: Continental and global scale flood forecasting systems, Wiley Interdiscip. Rev. Water, 3, 391–418, https://doi.org/10.1002/wat2.1137, 2016.
Environment Canada: Water Level and Flow – Environment Canada, available at: https://wateroffice.ec.gc.ca/ (last access: 23 April 2018), 2014.
Environmental Systems Research Institute: ArcMap, ArcGIS Desktop, available at: http://desktop.arcgis.com/en/arcmap/ , last access: 26 April 2018.
Fekete, B. M., Vörösmarty, C. J., and Lammers, R. B.: Scaling gridded river networks for macroscale hydrology: Development, analysis, and control of error, Water Resour. Res., 37, 1955–1967, https://doi.org/10.1029/2001WR900024, 2001.
GloFAS: GloFAS Web Map Service Time (WMS-T) User Manual, available at: http://www.globalfloods.eu/static/downloads/GloFAS-WMS-T_usermanual.pdf last access: 26 April 2018a.
IDEAM: IDEAM, available at: http://www.ideam.gov.co/ (last access: 23 April 2018), 2014.
INDRHI: INDRHI – National Institute of Hydraulic Resources, available at: http://indrhi.gob.do/ (last access: 23 April 2018), 2017.
INETER: Ineter, Instituto Nicaragüense de Estudios Territoriales, available at: http://www.ineter.gob.ni/ (last access: 23 April 2018), 2016.
Lehner, B. and Grill, G.: Global river hydrography and network routing: baseline data and new approaches to study the world's large river systems, Hydrol. Process., 27, 2171–2186, https://doi.org/10.1002/hyp.9740, 2013.
Lehner, B., Verdin, K., and Jarvis, A.: New Global Hydrography Derived From Spaceborne Elevation Data, Eos, Trans. Am. Geophys. Union, 89, 93–94, https://doi.org/10.1029/2008EO100001, 2008.
Lorenz, E. N.: The essence of chaos, University of Washington Press, 1993.