Issue with GRIB data from new ECMWF data server?

Post questions and find resources to convert meteorological data into a format HYSPLIT can read.
Andrew.Klekociuk
Posts: 2
Joined: November 13th, 2013, 2:42 am
Registered HYSPLIT User: Yes

Issue with GRIB data from new ECMWF data server?

Post by Andrew.Klekociuk »

Hi. I have been sucessfully using the PC version of HYSPLIT to convert ERA-Interim 1.5 degree gridded data downloaded from http://data-portal.ecmwf.int/data/d/interim_daily/ into the ARL format using the Meteorology>Convert to ARL>ECMWF ERA menu option. That's all nice!

However I am having trouble converting 0.75 degree resolution data dowloaded from the new server at http://apps.ecmwf.int/datasets/data/interim_full_daily/. Now that the higher resolution data is available, I want to see if this provides any advantages over the 1.5 degree data.

GRIB2ARL appears to do the conversion OK in the sense that no error messages are thrown up, but when I use the Check File option (under Meteorology>Display Data) I get error messages like the following:
*ERROR* metset: meteorological data time interval varies
Changed from 360 min to -1080 min
At day/hr 1 0

The listing in Working/checkfile.txt is not what I expect, with incorrect time stamps and fields zeroed from part way through the listing, presumably due to a problem associated with decoding time information from the GRIB files. Trajectory runs using the created ARL data get terminated at the point where the time information in the checkfile.txt listing goes awry. I've been careful to ensure that the 3 files required (invariant, surface and level) have consistent gridding. I've used the ECMWF GRIB-API to examine the files from the new server - they look to be in GRIB1 format, and I can't see an issue with the contents compared with the 1.5 degree data from the old server, except possibly from some extra padding.

I haven't yet used API2ARL to convert the data - I'm wondering if I have to do this (noting there is mention on the board of the ECMWF server moving to GRIB2 format) or if there is some other solution.

Has anyone had a similar issue? Thanks!
Andrew.Klekociuk
Posts: 2
Joined: November 13th, 2013, 2:42 am
Registered HYSPLIT User: Yes

Re: Issue with GRIB data from new ECMWF data server?

Post by Andrew.Klekociuk »

OK - I have made some progress.

The issue with data from the new server occurs when forecast fields are selected for the surface data. I can get the conversion to ARL format to work with data from the new server IF when requesting the data I don't select a forecast field (total precipitation) AND I explicitly select the time step as 0 (setting 'select step' to 'all' isn't appropriate even if you don't specify a forecast field such as precipitation).

So in summary, it appears that if you are interested in wet deposition (as I am), there is an issue for ARL conversion of data from the new server.

Another issue I discovered is that the ECMWF batch server (https://software.ecmwf.int/wiki/display ... s+in+batch) will not concatenate forecast and analysis fields - this is unlike the situation for the web interface where you can combine forecast and analysis fields in the one file.

Andrew
John.Johansson
Posts: 14
Joined: February 24th, 2016, 2:30 pm
Registered HYSPLIT User: Yes

Re: Issue with GRIB data from new ECMWF data server?

Post by John.Johansson »

Hi! I realize this thread has not been active for a while, but since it my problem is very similar to the one described here I'll give it a try.

I am also downloading ERA-Interim grib data from ECMWF and trying to use them in HYSPLIT by means of the included ERA->ARL conversion tool. This seems to work fine as long as I don't include precipitation data. When precipitation is included I get the same warnings as described by Andrew and if I include precipitation in the trajectory output (Advanced->Configuration Setup->Trajectory->Add METEOROLOGY output along trajectory->Precipitation), it invariably becomes 0.0 for all the output points (I've tested several different times and locations where it seemed very likely that there would be at least some precipitation along the trajectory).

As Andrew wrote, this is probably due to the fact that precipitation is a forecast field, while the other surface fields are analysis fields. The options in terms of times and steps available for downloading forecast and analysis field are different in the web interface and its not obvious how they should be chosen to get matching data. I have tried a few different ways that seemed logical to me, but none of them seemed to work. Does anyone have a clue on how to get this working?
alicec
Posts: 240
Joined: February 8th, 2016, 12:56 pm
Registered HYSPLIT User: Yes

Re: Issue with GRIB data from new ECMWF data server?

Post by alicec »

We are working on a converter for the ecmwf interim dataset and will have a solution soon.
There are several reasons why the current generic grib2arl does handle the ecmwf interim data well.

As noted, forecast fields for the interim data set are put in a separate file from analysis fields.
TPPT as well as SHTF, DSWF, LTHF, PBLH, CAPE are all only available as forecast fields (precipitation 228.128, surface sensible heat flux 146.128, surface solar radiation downward 169.128, surface latent heat flux 147.128, boundary layer height 159.128, convective available potential energy 59.128).

This means that the converter program needs to consider an extra input file for 2d forecast fields in addition to the file with 2d analysis fields (which include T02M 167.128, U10M 165.128, V10M 166.128).

Another issue which we are working on resolving is that SHTF, DSWF and LTHF are accumulated in this dataset in the same way as precipitation. So currently, HYSPLIT is not able to use them.

For retrieving the 2d analysis fields I am using a time of 00/06/12/18 and step of 0.

For retrieving 2d forecast fields I use a time of 00/12 and step of 6/12.

Will update with progress.

Thanks,
Alice
Julia
Posts: 2
Joined: May 23rd, 2016, 5:50 am
Registered HYSPLIT User: Yes

Re: Issue with GRIB data from new ECMWF data server?

Post by Julia »

Hello,
I am writing here, because my top belongs to that of Andrew.

For me a similar issue applies:
I am using HYSPLIT as PC version to convert 1-month ERA-Interim 1 degree (default global area) into the ARL format (>Meteorology > convert to ARL > ECMWF ERA). When I check the "chkfile.txt everything seems normal but from a certain date (18th of September) on the following problem occurs:

11626 4 91918 0 099INDX 0 0.0000000E+00 0.0000000E+00
11627 4 91918 0 099SHGT 12 0.1612598E+02 0.2839801E+04
11628 4 91918 0 099U10M 4 0.6299213E-01-0.5167999E+01
11629 4 91918 0 099V10M 4 0.6299213E-01-0.1480408E+01
11630 4 91918 0 099T02M 4 0.6299213E-01 0.2195769E+03
11631 4 91918-1 199HGTS 0 0.0000000E+00 0.0000000E+00
11632 4 91918-1 199TEMP 0 0.0000000E+00 0.0000000E+00
11633 4 91918-1 199UWND 0 0.0000000E+00 0.0000000E+00
11634 4 91918-1 199VWND 0 0.0000000E+00 0.0000000E+00
11635 4 91918-1 199WWND 0 0.0000000E+00 0.0000000E+00
11636 4 91918-1 199RELH 0 0.0000000E+00 0.0000000E+00
11637 4 91918-1 299HGTS 0 0.0000000E+00 0.0000000E+00
11638 4 91918-1 299TEMP 0 0.0000000E+00 0.0000000E+00
11639 4 91918-1 299UWND 0 0.0000000E+00 0.0000000E+00
11640 4 91918-1 299VWND 0 0.0000000E+00 0.0000000E+00

also when I try do the daily and cluster analysis, some files could not be found because the endpoints file are empty.
When I did the same with a subregion and 0.75 deg resolution all worked well.

Thanks for help!

Julia
alicec
Posts: 240
Joined: February 8th, 2016, 12:56 pm
Registered HYSPLIT User: Yes

Re: Issue with GRIB data from new ECMWF data server?

Post by alicec »

Try converting the files in chunks smaller than a month and see if that works.

How large are your input GRIB files?

Also, could you please let me know how you retrieved the GRIB files so I can try to reproduce the problem.
(What pressure levels were you retrieving, what surface fields...)

Cheers,
Alice
Julia
Posts: 2
Joined: May 23rd, 2016, 5:50 am
Registered HYSPLIT User: Yes

Re: Issue with GRIB data from new ECMWF data server?

Post by Julia »

Hi Alice,

Thanks for the fast answer!

> Try converting the files in chunks smaller than a month and see if that works.
I tried it and it worked. So, I think the problem was generated by the large grib file of 3.5GB.

I used following data fields in the area of 90N, -90S, -180W, 180E (global), 1deg, September 2004:

3d-grib: I downloaded all 37 pressure levels (1000-1) because it simplified the download with geopotential, rel. humidity, temperature, u/v wind and vertical velocity in 1 deg.
2d-grib: 2mtemp, 10m-u and v.
1d-grib:
http://apps.ecmwf.int/datasets/data/int ... vtype=sfc/

Cheers,
Julia
alicec
Posts: 240
Joined: February 8th, 2016, 12:56 pm
Registered HYSPLIT User: Yes

Re: Issue with GRIB data from new ECMWF data server?

Post by alicec »

Ok good - I can reproduce this problem if I try to convert a large grib file (> 2GB).
You can make the grib files smaller by using shorter time periods or downloading less pressure levels.
You can concatenate the resulting ARL files if you need to.

Cheers,
Alice
alicec
Posts: 240
Joined: February 8th, 2016, 12:56 pm
Registered HYSPLIT User: Yes

Re: Issue with GRIB data from new ECMWF data server?

Post by alicec »

The ecmwf2arl converter has been updated so that it resolves the following issues:


As noted, forecast fields for the interim data set are put in a separate file from analysis fields.
TPPT as well as SHTF, DSWF, LTHF, PBLH, CAPE are all only available as forecast fields (precipitation 228.128, surface sensible heat flux 146.128, surface solar radiation downward 169.128, surface latent heat flux 147.128, boundary layer height 159.128, convective available potential energy 59.128).


ecmwf2arl will accept an additional grib file with 2d forecast fields so the forecast fields can be used in addition to the analysis fields.


Another issue which we are working on resolving is that SHTF, DSWF and LTHF are accumulated in this dataset in the same way as precipitation. So currently, HYSPLIT is not able to use them.


ecmwf2arl will calculate the average SHTF, DSWF and LTHF from the accumulated values so that HYSPLIT may use them.
mas
Posts: 12
Joined: April 25th, 2017, 12:32 pm
Registered HYSPLIT User: No

Re: Issue with GRIB data from new ECMWF data server?

Post by mas »

Hi!
I had the same problem with the ERA files, and after reading this post I re-installed Hysplit to have the newer version.

Just to be clear: I have still to download the 2D forecast fields (time 00/12, step 6/12) and the 2D analysis fields (time 00/06/12/18, step 0) separately and add them in the converter? When I try this only one file is read, it is not possible to put 2 files in it. Or is it now possible to download all 2 D parameters in only one file?

I am avoiding large files (I am downloading about 10 days) as suggested in other posts. I can extract the profile, the area is the right one, but when running my control file which worked with GDAS data, everything blocks. This i why I looked up this post and then found I had an older Hysplit version and I re-installed everything.

Thanks for any suggestion.
Martina
Post Reply