IDL Keyword Forwarding Perils

You have to be careful when you add keywords to a routine because you want to forward them on to another routine that you invoke. It’s okay when the keywords are always present, but if they are optional output keywords, you may end up copying around data that the caller of the outermost routine never sees. Here is a short example of this:

When I call layer1 with no keywords, we see how both FOO and BAR are not present in layer1, but they are in layer2:

IDL> layer1
Layer1, N_ELEMENTS(foo) = 0
Layer1, N_ELEMENTS(bar) = 0
Layer1, ARG_PRESENT(foo) = 0
Layer1, ARG_PRESENT(bar) = 0
Layer2, N_ELEMENTS(foo) = 0
Layer2, N_ELEMENTS(bar) = 0
Layer2, ARG_PRESENT(foo) = 1
Layer2, ARG_PRESENT(bar) = 1

This is because the interpreter only looks at the way layer1 invokes layer2, and has to assume that layer1 is using FOO and BAR as output keywords. While it is certainly possible that the interpreter could inspect how the foo and bar variables are used, and try to optimize them away when it realizes that these variables aren’t used elsewhere inside layer1 and weren’t passed into it, that would have a major impact on performance. Compiled languages can do this with an optimizing linker, but interpreted languages like IDL can afford to take the time to do that every time they run a line of code.

One way to interpret the values of N_ELEMENTS() and ARG_PRESENT() is shown in this table:

	N_ELEMENTS() EQ 0	N_ELEMENTS() GT 0
ARG_PRESENT() EQ 0	Keyword not used	Iput keyword
ARG_PRESENT() EQ 1	Output keyword	Int/out keyword

If I call layer1 with FOO and BAR set to a literal and an undefined variable, we see different output:

IDL> layer1, FOO=1, BAR=b
Layer1, N_ELEMENTS(foo) = 1
Layer1, N_ELEMENTS(bar) = 0
Layer1, ARG_PRESENT(foo) = 0
Layer1, ARG_PRESENT(bar) = 1
Layer2, N_ELEMENTS(foo) = 1
Layer2, N_ELEMENTS(bar) = 0
Layer2, ARG_PRESENT(foo) = 1
Layer2, ARG_PRESENT(bar) = 1

Here we see how FOO is an input to layer1, and BAR is an output. The more interesting part is in layer2, where FOO in an in/out keyword while BAR is still only an output keyword. The situation changes a little when I call layer1 with a defined variable for one of the keywords:

IDL> b2 = 2
IDL> layer1, FOO=1, BAR=b2
Layer1, N_ELEMENTS(foo) = 1
Layer1, N_ELEMENTS(bar) = 1
Layer1, ARG_PRESENT(foo) = 0
Layer1, ARG_PRESENT(bar) = 1
Layer2, N_ELEMENTS(foo) = 1
Layer2, N_ELEMENTS(bar) = 1
Layer2, ARG_PRESENT(foo) = 1
Layer2, ARG_PRESENT(bar) = 1

This time FOO is an input to layer1 and in/out to layer2, but BAR is an in/out keyword to both layer1 and layer2.

Now you may be wondering what the point of this whole analysis is. It matters when an output keyword takes a lot of time to build or space to store, and is incorrectly identified as being present. Imagine that BAR uses a gigabyte of memory, if the user calls layer1 without BAR, then layer2 will allocate that memory and return it to layer1, but it gets thrown away when layer1 returns to the caller.

How do we defensively implement the code to prevent this waste of time and space? Unfortunately the best solutions I’ve come up with are to add some logic to layer1 to conditionally forward keywords to layer2. It matters what type of keyword we are talking about here, input vs output vs in/out. Pure input keywords can be blindly forwarded, while output and in/out need the extra logic.

The big drawback with this approach, as you can expect, is that it is exponential in the number of keywords and quickly becomes untenable. We need a solution that is O(n) instead of O(2ⁿ), and I’ve come up with two completely different solutions which each have their pros and cons. First the simpler solution, which uses the EXECUTE () function:

This approach builds up a command string to execute, starting with the name of the layer2 routine it invokes. It then conditionally appends the keywords to the command string, using N_ELEMENTS() and/or ARG_PRESENT(), depending on what type of keyword it is. This doesn’t add a lot of new code, but it takes a performance hit because EXECUTE () has to compile the command string and then execute it. One might point out that using CALL_PROCEDURE() is more efficient, but it doesn’t support output and in/out keywords so we have to use EXECUTE().

The other approach avoids the performance hit of EXECUTE(), but it is a bit more involved, requiring the creation of a new wrapper function and the use of SCOPE_VARFETCH(), which some people are leery to use.

This version of layer1 conditionally adds the keywords to a Hash, which is converted to a struct when calling the new wrapper function. By assigning the struct to the _EXTRA keyword during invocation, the keywords are properly mapped. One will note that the variables put into the Hash depend on whether it is an input or output keyword. For input keywords, I just used the variables assigned to those keywords in layer1’s signature, but for output keywords I had to create a local variable and put that in the Hash. The reason for this is that IDL structs cannot contain !NULL or undefined values, so I had to assign them a value. In this case I used 0, but depending on the expectations of layer2 a special “not set” value may need to be used.

The new layer2Wrapper() function is worth examining more closely. It is declared using _REF_EXTRA, so that we get pass by reference semantics instead of pass by value. That _REF_EXTRA bag is simply passed along to layer2, using the _EXTRA keyword like we’re supposed to. After calling layer2, the wrapper then uses SCOPE_VARFETCH() with the /REF_EXTRA keyword to grab each keyword’s value and put it into a Hash that is returned to its caller. It’s this SCOPE_VARFETCH() trick that necessitates the creation of this wrapper routine. Once layer2Wrapper returns that Hash to layer1, we then copy any of the existing values out of the Hash into the appropriate output keyword variables.

Both solutions may seem like a lot of work for what seems like a minor inconvenience, but there are real world cases where we could waste a lot of time and/or memory on keywords that only exist for parts of the callstack and are never specified by the original caller. A prime example of this is in ENVIRaster::GetData(). This method always returns an array of pixel values, but there is the optional PIXEL_STATE keyword that can be used to retrieve a byte array of the same dimensions that indicates which pixels are good and which are bad. Without using a solution like the ones I’ve presented, we would end up with the PIXEL_STATE array being allocated and calculated every time a user calls GetData(), whether they specified that keyword or not.