Clustering memory limit?

[arl.noaa]

During my research project I am running the Hysplit model to calculate back-trajectories at a site for the period 1998-2011. The final goal is to do a cluster analysis of my back-trajectories and to link the obtained clusters with the concentrations of some atmosheric tracers measured at that site. Since I have data only for some periods, the final dataset consists of a total 9020 trajectories. When I start the cluster calculation on my PC-Windows I get this message: " ERROR - allocation cluster mean array: 40675690 "
I think that it comes from the limited RAM or memory disk or both, but I do not know exactly. Could you help me and tell the characteristics for a PC that I need to run the cluster calculation? Moreover, I cannot find a version for a PC Linux-based.
Thank you very much for your help

[glenn.rolph]

Hello,

Yes, it is a memory issue. We haven't used this cluster program on more than about 1000 trajectories. The limit will depend on your particular PC, but we don't have a simple way to calculate that limit. Try to reduce the number of trajectories and/or number of endpoints along the trajectory that are used in the cluster calculation.

There is a linux version of the HYSPLIT system, however scripts to run the cluster program are not included. This is something we could add in the future. If reducing the memory requirements above is not sufficient, please contact us for the linux version.

[egribbin]

I would definitely agree that it's a memory limit. I've tried clustering a large number of trajectories (about 2000) on a PC [Intel Core 2 Duo E8500 @ 3.16GHz, Windows XP Pro, 4GB RAM] and then the same PC (same processor, Windows 7 Enterprise 64-bit, and 10GB RAM]. With a little experimentation, I found that the clustering on the latter set up tends to "crash" somewhere close to 900 trajectories - your mileage may vary of course.

Are there any hypotheses on whether these limitations can be overcome on a linux system or cluster?

Thanks in advance!

Erik

[Ramirez]

Hello,
I have similar issue. I think your answer can be useful for me.
Thank you