I have been struggling with the following problem.
Ever since having a new operating system installed (Fedora 13, GNOME) I have problems with reading multiple HDF4/HDF5/HDF-EOS/NETCDF files in IDL (either 7.1 or 8.0). After reading a number of files - it depends on the amount of data that is being read - IDL simply stalls without error message and the only way out is to actually kill IDL. I encountered this first when running a script for reading one year of daily MOPITT data, which I have been using for several years but started giving this problem only after the switch to Fedora 13. Given that I do not have an error
message, finding the origin of the problem is difficult. Originally I thought the problem was related to HDF5 or HDF-EOS, but after some testing I realized the same problem occured when reading NETCDF or HDF4 files (problem = stalling of IDL). Worrysome, since I read and process loads of HDF data and IDL is my main software tool.
I have managed to come up with a minimal script that reproduces the problem (opening and closing a small NETCDF file a zillion times; sorry, 10 million times; IDL stalls typically before 1 million times opening and closing). See below. Does not mean that running it on another computer will reproduce the issue, see further comments.
I have also noticed the following:
- the problem occurs when running my minimal code in IDL interactively, but not when compiling it and then running the program on the command line by "idl -rt='program.sav'. However, this does not help with the original problem (i.e. reading multiple MOPITT data files).
- the problem never occurs after opening and closing the files the same number of times. One time it is after X times, the next try is after Y times, although X and Y are pretty much always the same order of magnitude.
- some colleagues reproduce the problem, but others don't. This suggest that the problem is not in the coding or the data files, but rather some memory allocation problem or how environmental variables are set. As far as I know we all use Fedora 13.
- the problem occurs also for Fedora 13 with KDE.
However, I am no particular IDL or UNIX or LINUX guru, so my skills of tracking down this problem are limited.
On the other hand, I am getting quite desparate as I regularly read and process huge amounts of satellite data stored in HDF or similar formats and I have already spent way too much time traying to solve this.
Anyone any suggestions on what could be wrong and how to solve it?
for i=0L,9999999 do begin
ncid = NCDF_OPEN('file.nc')
## file.nc contains just a little bit of data.