ERA5 model-level conversion: issues regarding file size

Post questions and find resources to convert meteorological data into a format HYSPLIT can read.
Post Reply
wkjones
Posts: 3
Joined: September 5th, 2018, 1:02 pm
Registered HYSPLIT User: Yes

ERA5 model-level conversion: issues regarding file size

Post by wkjones » September 5th, 2018, 2:46 pm

Hi all,

I have recently been converting ERA5 data at full resolution on model levels into ARL packed data. However, at 3GB of data per time step, these exceed the maximum file size that hysplit can read. I am guessing that this is a similar problem to that encountered in this thread: viewtopic.php?f=25&t=98&p=276&hilit=data+size+max#p276.
As the file cannot be split up further, and no fields can be removed, can the file reading routines be modified to handle larger files in any way?

Cheers, Will

wkjones
Posts: 3
Joined: September 5th, 2018, 1:02 pm
Registered HYSPLIT User: Yes

Re: ERA5 model-level conversion: issues regarding file size

Post by wkjones » September 6th, 2018, 7:37 pm

I have since found that the problem was caused by compiling the conversion programme without the "-assume byterecl" flag, which meant that the record lengths were being quadrupled. This fixed the problem regarding file reading, and reduced the size per time step to 750MB, which is more manageable for splitting up the files

Cheers, Will

alicec
Posts: 77
Joined: February 8th, 2016, 12:56 pm
Registered HYSPLIT User: Yes

Re: ERA5 model-level conversion: issues regarding file size

Post by alicec » September 10th, 2018, 9:30 am

Thanks for posting the resolution to your problem!

wkjones
Posts: 3
Joined: September 5th, 2018, 1:02 pm
Registered HYSPLIT User: Yes

Re: ERA5 model-level conversion: issues regarding file size

Post by wkjones » September 11th, 2018, 4:30 am

No problem, hopefully this may help someone else in the same circumstance!

I was also able to compile hysplit with 64-bit default integers, using the -i8 option, which has allowed the use of large ARL datasets (~20GB) with no problems so far. This required a few changes to the integer variable declarations in the shpsub.f file in the hysplit libraries to specifiy integer(4), and correspondending changes to the interface to these subroutines in shpbdy.f in order to compile, however apart from this (and a large number of performance warnings) there were no other problems. This may be a topic for another thread though!

Finally, I've been working on a python module that calls the libhysplit packing routines, in order to make it quicker and easier to write conversion programmes. If this would be useful to the larger hysplit community I'd be happy to share it

Cheers, Will

Post Reply