Data-interchange in Scheme
In my last post I talked about data-interchange using JSON. I remarked that the JSON format maps readily onto a very similar Lisp representation and that the Lisp version had the advantage of being parsed automatically by the Lisp (or Scheme) reader. In this post, I'd like to follow up on that by discussing the paper Experiences with Scheme in an Electro-Optics Laboratory by Richard Cleis and Keith Wilson. I forget when I first read this paper, but since then I've reread it several times and I learn something new each time I do.
The authors work at the Starfire Optical Range, an Air Force Laboratory doing research on controlling the distortion in optical telescopes caused by air turbulence. The laboratory has five telescope systems, which are run by about a dozen computers that provide tracking calculations, motion control, gimbal control, and command and status functions. Starfire uses Scheme in a variety of ways: as an extension language by embedding it in the legacy C application that provides motion control; as a configuration language for the telescopes and experiments; and as a bridge to proprietary hardware controllers by extending Scheme with the vendors' libraries. But what I want to talk about is their use of Scheme for data-interchange. All communications between the telescopes, their subsystems, and the computers are conducted with s-expressions that are evaluated by Scheme (or in one case by a C-based s-expression library).
This is more subtle than merely using s-expressions as a substitute for JSON. Every request for data or command to perform some function is sent as an s-expression. These messages have the form
(list 'callback (function1 parameter1...) ...)
where the optional callback function is defined by the sender and intended to handle the remote system's response. Each of the functions and their parameters (function1, parameter1, etc.) are commands and data to the remote system.
This seems a little strange until you realize that the remote system merely evaluates the message and returns the result using code like the stylized version below—see the paper for the actual, slightly more complex code.
(define handle-request-and-reply (lambda (udp-socket) (let* ((buffer (make-string max-buffer-size)) (n (read-udp-socket udp-socket buffer))) (write-udp-socket udp-socket (format #f "~s" (eval (read (open-input-string (substring buffer 0 n)))))))))
Now consider what happens when the remote system evaluates the message
(list 'callback (do-something arg1 arg2))
First the arguments to
list are evaluated. The
is quoted so the remote system merely treats it as a symbol. The
second argument is a call to a local (to the remote system) function
do-something. When the argument
(do-something arg1 arg2)
do-something is called and returns a result, say
the-result. Finally, the function
list is called with the
the-result so the result of the
and this is returned to the sender on the same socket that received
the original message. When the sender receives this message it is
read by a routine similar to the one above that evaluates the message
but does not send a response. The result of that evaluation is to
callback with the argument
This is truly a beautiful thing. The function
handle-request-and-reply takes care of all communication with the
remote system's clients. Notice that it is completely general and
will handle any legal message a client sends. There is no need to
write special code for each type of message, merely a function to do
whatever the client is requesting. The actual code is wrapped in an
error handler so that even malformed messages are caught and handled
gracefully with an error message returned to the sender.
This is, I submit, a better solution in many cases than something like JSON or XML. Just a few lines of code completely handles data-interchange, message parsing, and communication, and it shows the power inherent in the humble s-expression.