PrevPrev Go to previous topic
NextNext Go to next topic
Last Post 30 Mar 2011 07:42 AM by  Jos
Reading multiple HDF/NCDF files in IDL running under Fedora 13
 1 Replies
Sort:
You are not authorized to post a reply.
Author Messages

Jos



New Member


Posts:
New Member


--
30 Mar 2011 07:42 AM
    I have been struggling with the following problem. Ever since having a new operating system installed (Fedora 13, GNOME) I have problems with reading multiple HDF4/HDF5/HDF-EOS/NETCDF files in IDL (either 7.1 or 8.0). After reading a number of files - it depends on the amount of data that is being read - IDL simply stalls without error message and the only way out is to actually kill IDL. I encountered this first when running a script for reading one year of daily MOPITT data, which I have been using for several years but started giving this problem only after the switch to Fedora 13. Given that I do not have an error message, finding the origin of the problem is difficult. Originally I thought the problem was related to HDF5 or HDF-EOS, but after some testing I realized the same problem occured when reading NETCDF or HDF4 files (problem = stalling of IDL). Worrysome, since I read and process loads of HDF data and IDL is my main software tool. I have managed to come up with a minimal script that reproduces the problem (opening and closing a small NETCDF file a zillion times; sorry, 10 million times; IDL stalls typically before 1 million times opening and closing). See below. Does not mean that running it on another computer will reproduce the issue, see further comments. I have also noticed the following: - the problem occurs when running my minimal code in IDL interactively, but not when compiling it and then running the program on the command line by "idl -rt='program.sav'. However, this does not help with the original problem (i.e. reading multiple MOPITT data files). - the problem never occurs after opening and closing the files the same number of times. One time it is after X times, the next try is after Y times, although X and Y are pretty much always the same order of magnitude. - some colleagues reproduce the problem, but others don't. This suggest that the problem is not in the coding or the data files, but rather some memory allocation problem or how environmental variables are set. As far as I know we all use Fedora 13. - the problem occurs also for Fedora 13 with KDE. However, I am no particular IDL or UNIX or LINUX guru, so my skills of tracking down this problem are limited. On the other hand, I am getting quite desparate as I regularly read and process huge amounts of satellite data stored in HDF or similar formats and I have already spent way too much time traying to solve this. Anyone any suggestions on what could be wrong and how to solve it? Cheers, Jos. >>> for i=0L,9999999 do begin print,i ncid = NCDF_OPEN('file.nc') NCDF_CLOSE, ncid endfor;i end ## file.nc contains just a little bit of data.

    Deleted User



    New Member


    Posts:
    New Member


    --
    01 Apr 2011 09:41 AM
    Hi Laatdej, Have you tried getting some error message by running IDL under "strace"?. In other words, try running IDL from the Linux prompt in this way: strace -o idlstrace.txt idl or strace -o idldestrace.txt idlde And then work as usual until you get the crash. Check then the .txt file to see if there is any clue as to what is happening at the moment of the freeze or crash. Cheers, Fernando
    You are not authorized to post a reply.