ERA5 model-level conversion: issues regarding file size

Post questions and find resources to convert meteorological data into a format HYSPLIT can read.
Post Reply
wkjones
Posts: 3
Joined: September 5th, 2018, 1:02 pm
Registered HYSPLIT User: Yes

ERA5 model-level conversion: issues regarding file size

Post by wkjones »

Hi all,

I have recently been converting ERA5 data at full resolution on model levels into ARL packed data. However, at 3GB of data per time step, these exceed the maximum file size that hysplit can read. I am guessing that this is a similar problem to that encountered in this thread: viewtopic.php?f=25&t=98&p=276&hilit=data+size+max#p276.
As the file cannot be split up further, and no fields can be removed, can the file reading routines be modified to handle larger files in any way?

Cheers, Will
wkjones
Posts: 3
Joined: September 5th, 2018, 1:02 pm
Registered HYSPLIT User: Yes

Re: ERA5 model-level conversion: issues regarding file size

Post by wkjones »

I have since found that the problem was caused by compiling the conversion programme without the "-assume byterecl" flag, which meant that the record lengths were being quadrupled. This fixed the problem regarding file reading, and reduced the size per time step to 750MB, which is more manageable for splitting up the files

Cheers, Will
alicec
Posts: 418
Joined: February 8th, 2016, 12:56 pm
Registered HYSPLIT User: Yes

Re: ERA5 model-level conversion: issues regarding file size

Post by alicec »

Thanks for posting the resolution to your problem!
wkjones
Posts: 3
Joined: September 5th, 2018, 1:02 pm
Registered HYSPLIT User: Yes

Re: ERA5 model-level conversion: issues regarding file size

Post by wkjones »

No problem, hopefully this may help someone else in the same circumstance!

I was also able to compile hysplit with 64-bit default integers, using the -i8 option, which has allowed the use of large ARL datasets (~20GB) with no problems so far. This required a few changes to the integer variable declarations in the shpsub.f file in the hysplit libraries to specifiy integer(4), and correspondending changes to the interface to these subroutines in shpbdy.f in order to compile, however apart from this (and a large number of performance warnings) there were no other problems. This may be a topic for another thread though!

Finally, I've been working on a python module that calls the libhysplit packing routines, in order to make it quicker and easier to write conversion programmes. If this would be useful to the larger hysplit community I'd be happy to share it

Cheers, Will
Post Reply

Return to “Conversion programs”