Data-interchange in Scheme

In my last post I talked about data-interchange using JSON. I remarked that the JSON format maps readily onto a very similar Lisp representation and that the Lisp version had the advantage of being parsed automatically by the Lisp (or Scheme) reader. In this post, I'd like to follow up on that by discussing the paper Experiences with Scheme in an Electro-Optics Laboratory by Richard Cleis and Keith Wilson. I forget when I first read this paper, but since then I've reread it several times and I learn something new each time I do.

The authors work at the Starfire Optical Range, an Air Force Laboratory doing research on controlling the distortion in optical telescopes caused by air turbulence. The laboratory has five telescope systems, which are run by about a dozen computers that provide tracking calculations, motion control, gimbal control, and command and status functions. Starfire uses Scheme in a variety of ways: as an extension language by embedding it in the legacy C application that provides motion control; as a configuration language for the telescopes and experiments; and as a bridge to proprietary hardware controllers by extending Scheme with the vendors' libraries. But what I want to talk about is their use of Scheme for data-interchange. All communications between the telescopes, their subsystems, and the computers are conducted with s-expressions that are evaluated by Scheme (or in one case by a C-based s-expression library).

This is more subtle than merely using s-expressions as a substitute for JSON. Every request for data or command to perform some function is sent as an s-expression. These messages have the form

(list 'callback (function1 parameter1...) ...)

where the optional callback function is defined by the sender and intended to handle the remote system's response. Each of the functions and their parameters (function1, parameter1, etc.) are commands and data to the remote system.

This seems a little strange until you realize that the remote system merely evaluates the message and returns the result using code like the stylized version below—see the paper for the actual, slightly more complex code.

(define handle-request-and-reply
 (lambda (udp-socket)
   (let* ((buffer (make-string max-buffer-size))
          (n (read-udp-socket udp-socket buffer)))
     (write-udp-socket
       udp-socket
       (format #f "~s"
               (eval
                 (read (open-input-string
                         (substring buffer 0 n)))))))))

Now consider what happens when the remote system evaluates the message

(list 'callback (do-something arg1 arg2))

First the arguments to list are evaluated. The callback function is quoted so the remote system merely treats it as a symbol. The second argument is a call to a local (to the remote system) function called do-something. When the argument (do-something arg1 arg2) is evaluated, do-something is called and returns a result, say the-result. Finally, the function list is called with the arguments callback and the-result so the result of the eval is

(callback the-result)

and this is returned to the sender on the same socket that received the original message. When the sender receives this message it is read by a routine similar to the one above that evaluates the message but does not send a response. The result of that evaluation is to call callback with the argument the-result.

This is truly a beautiful thing. The function handle-request-and-reply takes care of all communication with the remote system's clients. Notice that it is completely general and will handle any legal message a client sends. There is no need to write special code for each type of message, merely a function to do whatever the client is requesting. The actual code is wrapped in an error handler so that even malformed messages are caught and handled gracefully with an error message returned to the sender.

This is, I submit, a better solution in many cases than something like JSON or XML. Just a few lines of code completely handles data-interchange, message parsing, and communication, and it shows the power inherent in the humble s-expression.