"Missing data" file boundary problem with CFS/GFS

[chron]

I've encountered a problem with missing data warnings in converted 0.5deg CFS output. I used grib2arl since the output is on the same grid as some versions of GFS, and modified CFG_GRIB to match the necessary variables. Once I've converted multiple files, I can run both the trajectory and dispersion model on them, and the output appears to be reasonable for both forward and backward cases. However, I've noticed that in some cases I get "missing data" warnings in MESSAGE:

Code: Select all

  NOTICE metpos: (mtime,ftime) -     58056480    58057200
 WARNING metinp: missing data - HGTS
           Time:           10           5          21           0          -1
          Level:            1    Variable:            1
 WARNING metinp: missing data - TEMP
           Time:           10           5          21           0          -1
          Level:            1    Variable:            2
 WARNING metinp: missing data - UWND
           Time:           10           5          21           0          -1
          Level:            1    Variable:            3
 WARNING metinp: missing data - VWND
           Time:           10           5          21           0          -1
          Level:            1    Variable:            4
 WARNING metinp: missing data - WWND
           Time:           10           5          21           0          -1
          Level:            1    Variable:            5
...

This only occurs on the boundary of the files (e.g., if I have pgbh06.gdas.20100516-20100520.bin and pgbh06.gdas.20100521-20100525.bin, the error will only show up when the model passes through 2010-05-21 00Z). Oddly, it still occurs at the original boundary if I concatenate the two files into one:

Code: Select all

cat pgbh06.gdas.20100516-20100520.bin pgbh06.gdas.20100521-20100525.bin > bothfiles.bin

There are two main things that I can't figure out:

1) Why the file transition isn't handled correctly so that hysplit cannot find the data.
2) What does the model do in the absence of these data? I was surprised that it continued with the integration.

Could the problem be related to the fact that the CFS output is a sequence of 6-hour forecasts? Is hysplit expecting a single base time with a sequence of forecasts?

Any help greatly appreciated!

[barbara.stunder]

To help us figure this out please run chk_file on those datasets. Missings will be indicated by a "-1" in the header forecast hour (9th and 10th characters of record). What is missing? Is it all the fields at one time period? All at more than one time period? Or only which fields?

[chron]

According to chk_file, the missing fields are the first two levels of the last time period of each file:

Code: Select all

...
  3706  10 516 0 6 099MSLP   2 0.1574803E-01 0.1013963E+04
  3707  10 516 0-1 199HGTS   0 0.0000000E+00 0.0000000E+00
  3708  10 516 0-1 199TEMP   0 0.0000000E+00 0.0000000E+00
  3709  10 516 0-1 199UWND   0 0.0000000E+00 0.0000000E+00
  3710  10 516 0-1 199VWND   0 0.0000000E+00 0.0000000E+00
  3711  10 516 0-1 199WWND   0 0.0000000E+00 0.0000000E+00
  3712  10 516 0-1 199RELH   0 0.0000000E+00 0.0000000E+00
  3713  10 516 0-1 299HGTS   0 0.0000000E+00 0.0000000E+00
  3714  10 516 0-1 299TEMP   0 0.0000000E+00 0.0000000E+00
  3715  10 516 0-1 299UWND   0 0.0000000E+00 0.0000000E+00
  3716  10 516 0-1 299VWND   0 0.0000000E+00 0.0000000E+00
  3717  10 516 0-1 299WWND   0 0.0000000E+00 0.0000000E+00
  3718  10 516 0-1 299RELH   0 0.0000000E+00 0.0000000E+00
  3719  10 516 0 6 399HGTS   7 0.5039370E+00 0.4557747E+03
  3720  10 516 0 6 399TEMP   4 0.6299213E-01 0.2412000E+03
...

I believe that these are the last records stored (level direction is reversed). These data are definitely present in the original grib file. Looking at the MESSAGE output from grib2arl, I think I may have found the problem -- the files are large and I'm converting many fields, and consequently run into an overrun in the buffer address space:

Code: Select all

-----------------------------------------------------
 Finished time:           10           5          10          18
 INDEX record written after record:         3667
 -------------------------------------------------------
 SUBROUTINE extract: buffer starting location:   2045976894
     0 LISD     0    6    6
     0 CAPE     0    6    6
...
     3 VWND   950    6    6
     2 HGTS   975    6    6
 Finished (kret,file):            1 all_sorted_nodupes.grib
 -----------------------------------------------------
 Finished time:           10           5          11           0
 INDEX record written after record:         3849
 -------------------------------------------------------
 SUBROUTINE extract: buffer starting location:  -2147144818
...

This is followed by lots of repeated " SUBROUTINE extract: buffer starting location: -2147144818" messages. Of course, the max Int32 in fortran is 2147483647, and with the last successful record stored at KBYTE = 2045976894, I conclude that I'm simply running over that register limit. I guess this is easily fixed by either:
[*] reducing the number of timesteps in the file,
[*] recompiling as 64 bit (eek! want to avoid this),
[*] getting rid of some fields (also want to avoid this).

Let me know if you can think of another option, otherwise I think I'll just split the files up. Thanks for your help!

[barbara.stunder]

Agree. Splitting up the files seems reasonable.