The ESE API for IDL

With IDL 8.4.1, the ENVI Services Engine (ESE) has a PRO-code interface. This “ESE API” makes it easy for IDL programs to run ESE tasks, monitor them and get results. There are some other goodies in there as well.

First, let’s recall what ESE does. It is an application that provides on-demand IDL, ENVI and ENVI LiDAR processing as a web service  It can be run as an app or a service.  Client applications make requests that look like:

                 http://<host>:<port>/ese/<catalog>/<folders>/<service>/execute?<query-string>

A real world example for an ESE installation running on my CentOS machine, penny64, is:

                http://penny64:8181/ese/services/SyncService/ese_addition/execute?a=1&b=2

 The “ese_addition” task that ships with ESE simply adds two numbers. The result is in JSON form (edited for brevity):

        {

         "jobId": "42”,

         "jobStatus": "esriJobSucceeded",

         "jobStatusURL": "ese/jobs/42/status",

         "jobProgress": 100,

         "jobProgressMessage": "",

         "taskName": "/services/SyncService/ese_addition",

         "jobErrorMessage": "",

         "results": [

           {

             "name": "answer",

             "dataType": "double",

             "parameterType": "required",

             "value": 3

           }

         ],

         "messages": […]

       }

Any program that can make HTTP requests and parse JSON can use ESE to do processing.

Now, with IDL 8.4.1, the ESE API takes the complexity out of IDL programs invoking ESE tasks. The API lets you do the following:

 •             introspect an ESE installation with respect to services and their tasks

•             run tasks (synchronous and asynchronous)

•             monitor and manage job progress

•             retrieve job results

•             upload and download data

Any IDL program can use this API, be they desktop applications or tasks themselves. The API ships with IDL 8.4.1 and does not require a local ENVI nor ESE installation.

The ESE API consists of several classes that generally mirror ESE constructs. For example, ESE has the notion of services, tasks and jobs. So, the API has the ESEService, ESETask and ESEJob classes. There are also classes for ESE servers, folders and even task parameters. All this is in the IDL documentation, so let’s jump straight into an example. Let’s add two numbers:

       oTask = ESE.findTask( 'penny64', 'ese_addition' )

       oJob = oTask.run( a = 1, b = 2 )

       print, oJob.answer

Those three lines fetch the “ese_addition” task, run it and print the answer. The API takes care of all the details: creation of the HTTP request string, issuing the request, parsing the JSON result, error handling, etc. A less obvious duty of the API is to convert IDL variables to and from JSON (the transport mechanism). The API includes facilities for converting custom data types back and forth between IDL variables and JSON.

It should be mentioned that "a" and "b" are input parameters defined on the task and that "answer" is the name of the task's output parameter.

Now, let's try a more complex example that demonstrates task chaining, where the output of one task is used as the input of another.  Call it a workflow.  Let's add three numbers:

        oTask = ESE.findTask( 'penny64', 'ese_addition' )

       oJob1 = oTask.run( a = 1, b = 2 )

       oJob2 = oTask.run( a = oJob1.answer, b = 3 )

       print, oJob2.answer

There are a couple things to note:

 ·         the first job's output parameter, "answer", is just an IDL variable that can be used in the invocation of subsequent tasks (or anything else one wants to do in IDL)

·         one task can be run multiple times; each time, a brand new ESEJob object is returned

 Without going into all the details of the API, here are some of the more interesting features:

 •             run: Execute synchronous and asynchronous task in the style of IDL methods.

•             join: Wait for multiple jobs to complete before continuing.  Good for synchronizing jobs that are running in parallel.

•             monitor: Get notifications whenever a job's status changes.

•             conversion and validation: Program in the style of IDL without thinking too much about data type conversion and validation.

•             upload and download: Copy data between client and server.

 

Although the ESE API provides data "upload" and "download" functionality, the best practice is to keep the data where the processing is and not move it around. So, when the data is large, the mantra is to “bring processing to the data.”

As we’ve seen, the ESE API exposes the power of ESE task processing. Given the atomic nature of tasks, they lend themselves well to both linear chaining and parallel processing. The power of ESE tasks is now closer to your mind's eye.