ERA5 dataset

Post questions and find resources to convert meteorological data into a format HYSPLIT can read.
Post Reply
alicec
Posts: 101
Joined: February 8th, 2016, 12:56 pm
Registered HYSPLIT User: Yes

ERA5 dataset

Post by alicec » November 14th, 2017, 12:48 pm

A conversion utility program is now available to convert ERA5 grib files to ARL format for use by HYSPLIT.
The conversion program is in the trunk/data2arl/era52arl directory and is called era52arl.

Please note the following:
the conversion utility requires the ecmwf eccodes library
the utility currently only handles data on pressure levels (not model levels)

Some python code to download the era5 grib files which utilizes the ecmwfapi can be found at
https://github.com/amcz/hysplit_metdata
Please use or consult to figure out which variables are needed by HYSPLIT.
The era52arl utility will take three input grib files
1. pressure level fields (grib1)
2. analysis surface fields (grib1)
3. analysis forecast fields (grib1)

For more information on the ERA5 dataset please see:
https://software.ecmwf.int/wiki/display ... at+is+ERA5

Note that this is a global dataset with high temporal and spatial resolution (hourly, 0.3 degree).
Consequently, the grib files may be very large.

xinxuAdia
Posts: 6
Joined: November 13th, 2018, 5:44 am
Registered HYSPLIT User: No

Re: ERA5 dataset

Post by xinxuAdia » November 13th, 2018, 6:06 am

Hello, I am using this conversion tool now, but I missed one error:

Allocating agrib 14 SFC.GRIB
Allocating igrib 222 DATA.GRIB
Number of time periods found 1
ECCODES ERROR : get: latitudeOfFirstGridPointInDegrees Key/value not found

It seems that the " latitudeOfFirstGridPointInDegrees " cannot be got but according to the major code,
call grib_get(igrib(i),'latitudeOfFirstGridPointInDegrees', clat2)

It is the value "clat2" which is read at the beginning of the code.
real :: clat2,clon2, aclat2, aclon2

I think basically, it is just the definition of the domain in the GRIB file. Can you provide me with some suggestions about this error?

alicec
Posts: 101
Joined: February 8th, 2016, 12:56 pm
Registered HYSPLIT User: Yes

Re: ERA5 dataset

Post by alicec » November 13th, 2018, 12:06 pm

The first thing to do would be to make sure that latitudeOfFirstGridPointInDegrees is contained in your grib file.

Try using the command line utility grib_dump that comes with eccodes.

https://confluence.ecmwf.int/display/ECC/grib_dump

PATH/grib_dump -D DATA.GRIB

will output all the key names in the grib file and you should make sure that this key name is in there.

Alice

xinxuAdia
Posts: 6
Joined: November 13th, 2018, 5:44 am
Registered HYSPLIT User: No

Re: ERA5 dataset

Post by xinxuAdia » November 13th, 2018, 2:14 pm

Thx, I have checked the DATA.GRIB using this comment. I do not find this key name in the file. This key name is not written into GRIB file.

What can be the reason why this key name is not inside?

Cause the GRIB file is directly downloaded using the python code which I got from the ECMWF website:

#!/usr/bin/env python
from ecmwfapi import ECMWFDataServer
server = ECMWFDataServer()
server.retrieve({
"class": "ea",
"dataset": "era5",
"date": "2018-08-01",
"expver": "1",
"levtype": "sfc",
"param": "3.228/59.128/129.128/134.128/146.128/147.128/159.128/164.128/165.128/166.128/167.128/168.128/169.128/228.128",
"step": "0",
"stream": "oper",
"time": "09:00:00",
"type": "4v",
"target": "output",
})

But I do not define the area and use the default domain (global) when I download the ERA5 data. Is it maybe the reason?

alicec
Posts: 101
Joined: February 8th, 2016, 12:56 pm
Registered HYSPLIT User: Yes

Re: ERA5 dataset

Post by alicec » November 13th, 2018, 4:21 pm

Try using type='an' rather than '4v'

Please also see python code for downloading ERA5 data that can
be used in the converter at
https://github.com/amcz/hysplit_metdata

xinxuAdia
Posts: 6
Joined: November 13th, 2018, 5:44 am
Registered HYSPLIT User: No

Re: ERA5 dataset

Post by xinxuAdia » November 16th, 2018, 3:48 am

Thank you very much! I have downloaded the GRIB file according to the code. But I was confused that I cannot save files in my cloud in which I handed in the task to ECMWF. I checked several address types to figure it out but all of them failed (e.g., [username@***.**.***.***:/data/xinxu/] or without the address of my cloud or default path[./]). Would you give me some help about it? because the right .cfg cannot be saved and I only had one GRIB file in the download system every time.

Then I run the downloaded GRIB file, and one error appears:
>./era52arl -o
Existing decoding configuration: era52arl.cfg
Allocating igrib 1332 DATA.GRIB
Number of time periods found 6

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0 0x7F41BA62A777
#1 0x7F41BA62AD7E
#2 0x7F41B9D62CAF
#3 0x7F41BA93D73F
#4 0x7F41BA944441
#5 0x4060BC in era52arl at era52arl.f:668
Segmentation fault (core dumped)

It broke here at era52arl.f:


call grib_get(agrib(i),'latitudeOfFirstGridPointInDegrees', aclat2) <here Line 668
call grib_get(agrib(i),'longitudeOfFirstGridPointInDegrees',aclon)
call grib_get(agrib(i), 'latitudeOfLastGridPointInDegrees', aclat)
call grib_get(agrib(i), 'longitudeOfLastGridPointInDegrees',aclon2)


And I checked the key name 'latitudeOfFirstGridPointInDegrees' is in the GRIB file. I doubted that it may be caused by the default .cfg file which is not matched with the current GRIB file.

Thank you very much :)

alicec
Posts: 101
Joined: February 8th, 2016, 12:56 pm
Registered HYSPLIT User: Yes

Re: ERA5 dataset

Post by alicec » November 16th, 2018, 10:46 am

I'm not sure I understand the trouble you are having with downloading the files. How are you trying to download them?
Are you using the python code or other way? Can you describe this problem in more detail?

The reason you are getting the error from era52arl is that you need to input at least 2 grib files - one
for the surface fields and one form the pressure level fields. The default name for the pressure level fields
is DATA.GRIB. The default name for the surface fields is SFC.GRIB. You can specify other names for these with the -i and -a options (see below).
The error you are getting is because it is not finding the SFC.GRIB file.
After it says Allocating igrib, you should also get a message that says Allocating agrib.

a default era52arl.cfg will be created if none exists or
alternate name is not specified with the -d option.
This file specifies variables and pressure levels to be written.
-d[decoding configuration file {name | create era52arl.cfg}]
-i[input grib1 file with pressure level fields name {DATA.GRIB}]
-a[input grib1 analysis surface fields name {SFC.GRIB}]
-f[input grib1 forecast surface fields name {SFC2.GRIB}]
-o[output data file name {DATA.ARL}]
-v[verbose mode {False}]

xinxuAdia
Posts: 6
Joined: November 13th, 2018, 5:44 am
Registered HYSPLIT User: No

Re: ERA5 dataset

Post by xinxuAdia » November 16th, 2018, 12:14 pm

Hi Alice

Thank you for your reply! I cannot directly attach the python code, so I shared the edited part in the code here:

##Parse command line arguments and set defaults#####
parser = OptionParser()
parser.add_option("-y", type="int" , dest="year", default=2018,
help="{2018} Year to retrieve. type integer.")
parser.add_option("-m", type="int" , dest="month" , default=8,
help = "{8} Month to retrieve. type integer")
parser.add_option("-d", type="int" , dest="day" , default='1',
help = "Default is to retrieve one day split into four files. ")
parser.add_option("-f", action="store_true" , dest="getfullday" , default=False,
help = "If set then will retrieve one grib file per day. \
default is to split the 3d and 2d analysis files into 6 hour increments.\
The reason for this is that full day grib files for the global dataset will \
be too large for the conversion program. If a smaller area is being extracted then\
you may wish to use this option. The conversion program requires grib files less than\
2 gb in size. \
2d forecast is always retreived for the full day." )
parser.add_option("--dir", type="string" , dest="dir" , default='/data/xinxu/wrf/WPS/ECMWF_Stored',
help = "{./} Directory where files are to be stored. ")
parser.add_option("-o", type="string" , dest="fname" , default='',
help = "Output filename stem. 3D.grib and 2D.grib will be added on. \
The default is to call it DATASET_YYYY.MMM where MMM is the three letter abbreviation for month. \
If a day range is specified then the default will be DATASET_YYYY.MMM.dd-dd.")
parser.add_option("--3d", action="store_false" , dest="retrieve2d" , default='true',
help = "If set then it will only retrieve 3d data.")
parser.add_option("-s", type="string" , dest="stream" , default='oper',
help = "default is oper which retrieves deterministic analyses. \
enda will retrieve ensemble.")
parser.add_option("-e", type="string" , dest="enlist" , default='0:1:2:3:4:5:6:7:8:9',
help = "list of ensemble members to download. Numbers (0 -9) separated by colons. \
default is to download all 10 ensembles 0:1:2:3:4:5:6:7:8:9")
parser.add_option("--noprecip", action="store_false" , dest="get_precip" , default= True,
help = "Default is to retrieve 2D fields in a .2d.grib file which has analysis values for \
and a separate 2D file in a .2df.grib file which has forecast values. \
If the --noprecip field is set then the .2df.grib file will not be retrieved. \
")
parser.add_option("--2d", action="store_false" , dest="retrieve3d" , default='true',
help = "If set then it will only retrieve 2d data. The default is to retrieve both." \
"If --2d and --3d are both set then no data will be retrieved.")
parser.add_option("--check", action="store_false" , dest="run" , default='true',
help = "If set then simply echo command. Do not retrieve data" )
parser.add_option("-g", action="store_true" , dest="grib2arl" , default= False,
help = "If set then will append lines to a shell script for converting the files to arl format.")
parser.add_option("-l", type="int" , dest="toplevel" , default= 1,
help = "Set top pressure level to be retrieved. Default is to retrieve all levels. ")
#currently will only retrieve pressure levels. no converter available for model levels yet.
#parser.add_option("-t", type="string" , dest="leveltype" , default= "pl",
# help = "default is pressure levels (pl) which will retrieve grib1 file \
# Can also choose model levels (ml). There are 137 model levels. This will retrieve grib2 file.")
parser.add_option("--area", type="string" , dest="area" , default= "75/-20/10/60",
help = "choose the area to extract. Format is North/West/South/East \
North/West gives the upper left corner of the bounding box. \
South/East gives the lower right corner of the bounding box. \
Southern latitudes and western longiutdes are given negative numbers.")

I think now my problem is that I cannot save the created files in my set folder. Do you think that my address type is wrong? Because I thought that the default setting is './', but the first time when I used the default setting, it also does not work. Should I write more detailed?

Thank you very much!

Xinxu

Post Reply