Lisp Is Not An Acceptable Java

Today (April 16) on the Lisp reddit I saw a link to Lisp Is Not An Acceptable Java, a post that repeats every known myth and piece of misinformation about Lisp. Even the title is snort-worthy. After reading the post and shaking my head sadly, I moved on to other things but my mind kept jumping back to the article and I thought that, really, this shouldn't go unanswered. So I got all fired up to give it a good Fisking. But when I read it again, I thought, “this guy is obviously a troll and shouldn't be fed.” Then I noticed that the date on the post was 2011-04-01. OK, an April fools' joke and a good one. The author, Matthias Benkard, is, of course, a Lisper.

What's sad about this is that it works so well. The attitudes and misinformed opinions that the Benkard parodied are common and you do see them expressed all the time. Just last week I blogged about LispWorks' article Common Lisp—Myths and Legends, which was written to rebut the very things that Benkard was satirizing.

I'm not sure what to do about all this. Nobody, it seems, is. All we can do is smack down gently correct those who repeat these misconceptions and hope that the meme of Lisp's irrelevance and manifest shortcomings is not too firmly established to be destroyed.

Common Lisp — Myths and Legends

Via Hacker News, I came across this interesting article about Common Lisp. As its name suggests, it discusses the myths that have grown up around Lisp and talks about Lisp's strengths and weaknesses. The article is by LispWorks so it can hardly be said to be a disinterested analysis but it is, I think, an honest and accurate accounting.

The article starts off by characterizing the types of projects for which Lisp is particularly well suited and follows up with some types of projects where Lisp might not make sense. We all know about Paul Graham's WYSIWYG editor for the Yahoo! Store and about ITA's use of Lisp (although ITA isn't mentioned specifically, the article does list airline scheduling among Lisp uses) but did you know that LispWorks is developing a realtime Lisp for use in AT&T's switched virtual circuit capabilities?

The next section is entitled What Does Lisp Offer Me?. It's a good run down of the standard Lisp capabilities, data types, compilers, and standard library. It also includes common (heh) Lisp add ons that, while not in the Common Lisp standard are usually provided.

The final section is entitled Myths About Lisp. This will be familiar ground to any Lisper but might help dispel some misconceptions held by management or engineers who haven't used Lisp.

The article is fairly short and well worth a read.

Common Lisp Features

Abhishek Reddy has a nice list of Common Lisp features that help make the language so useful and flexible. There's nothing there that will come as news to even a semi-experienced Lisper, but for programmers who've heard about Lisp and are wondering what all the excitement is about, this list is a useful overview.

Compiling DSLs

Recently I had occasion to revisit this post from Peter Seibel's blog about compiling queries in Lisp. It's a cliche in the Lisp world that “if you're using eval, you're doing something wrong.” A more helpful formulation is, “if you're using eval, you should probably rewrite your code to use macros instead.” This is generally good advice—see here for some contrary examples—but it does not tell the whole story; that's where Seibel's post comes in.

Seibel considers the problem of a DSL for performing simple keyword queries on a block of text. He gives the example

(or
 (and "space" "mission")
 (and (or "moon" "lunar") "landing"))

which tests whether the text contains the words space and mission, or moon and landing, or lunar and landing. It's pretty easy to translate the above into a Lisp expression such as

(or
 (and (word-in-string-p "space" text)
      (word-in-string-p "mission" text))
 (and
   (or (word-in-string-p "moon" text)
       (word-in-string-p "lunar" text))
   (word-in-string-p "landing" text)))

where word-in-string-p returns t if the first argument is a substring of the second and nil otherwise.

The next problem is to evaluate this expression. The naive way to do this is to pass it to eval, and you know what? The earth will still be orbiting the sun if you do. But Seibel goes on to show us a better way and to illustrate why the prejudice against eval is usually well-founded.

Instead of invoking eval, we turn the above expression into a function

(lambda (#:G1)
 (or
   (and (word-in-string-p "space" #:G1)
        (word-in-string-p "mission" #:G1))
   (and
     (or (word-in-string-p "moon" #:G1)
         (word-in-string-p "lunar" #:G1))
     (word-in-string-p "landing" #:G1))))

where the #:G1 is a gensym'd symbol. Now an immediate advantage of

this is that we have a function that we can pass to some other function or otherwise use in any of the ways that Lisp's first class functions can be used. This means that it can be called over and over again on a series of text blocks (the original problem is posed in terms of a database query). The key here is that we can take this function and compile it using compile-expression so that (for systems that support native compilation) we have a fast implementation compiled to machine language.

Notice that we're not talking about much more work than using eval. We merely have to wrap the code in the lambda form and then call compile-expression rather than eval. Seibel's got all the code in the post I linked above, so you should go take a look at it.

Lisp Outside the Box

Nick Levine has posted the first batch of sample chapters (Chapters 13–16) from his book-in-progress Lisp Outside the Box, which is being published by O'Reilly. My understanding of the book's purpose is that it's intended to be an introduction to Lisp crafted specifically to show that Lisp is applicable to a wide range of problems and that extensions and external libraries extend its application domain outside of what's provided “inside the box.”

With that understanding in hand and recognizing that we are looking at only a small sliver of the book, largely without context, it is worth while taking a look at the chapters and seeing how well they live up to their stated goals. The weakest chapters, in my opinion, are 13 and 14. These two chapters are devoted to a pretty thorough look at AllegroCache, the proprietary persistent object database that comes with Allegro Common Lisp (ACL) from Franz Inc. The question is: Are these chapters useful to someone learning Lisp? They do, it's true, introduce new Lispers to the notion of persistent objects, but in a way that's not apt to be useful to them. Not useful because someone who is new to Lisp and wants to experiment with it is unlikely to be using ACL. Yes, Franz provides a free (but crippled) version of ACL that includes (an even more crippled) version of AllegroCache, but why would a casual usual bother when there are free, industrial strength implementations such as SBCL and Clozure available? An academic license for ACL Professional starts at $599 and versions that can be used to produce commercial code cost much more. In other words, the most likely user of ACL is a professional Lisp developer who is using a version of ACL provided by his employer, not someone trying to learn about Lisp and its capabilities.

At the end of Chapter 14, the author devotes about a page to Rucksack and Elephant, two open source alternatives to AllegroCache. Rather than devote two chapters out of 32 to a proprietary solution that the intended reader is not apt to be using, it would have made more sense to devote those chapters to Rucksack and Elephant and mention AllegroCache as a commercial (and perhaps better) solution to the same problem.

Chapter 15 deals with concurrency. The author repeats the mistake of the previous two chapters by using ACL as his primary example, but here, at least, the sample code easily translates to other implementations. My main complaint about this chapter is that the author uses the term process to describe what everyone else calls threads. He does this because it's the traditional terminology in the Lisp community, and perhaps he has a point, but I think it's a disservice to the new Lisper because it uses process to mean something different from what it means everywhere else. Indeed, in most contemporary writing about Lisp you see the term thread used when discussing the concept that he calls process. Other than these two nits, the chapter is well written and should be helpful to someone trying to learn to use Lisp in the real world.

The last chapter, on garbage collection and memory management, is the strongest. The main lesson here is to trust the garbage collector. In almost every case the beginning Lisper is likely to encounter, it's better to let Lisp handle memory management rather than trying to hand optimize it by using, for example, the destructive versions of commands. The chapter provides a nice, elementary explanation of how garbage collection works and mentions some things to watch out for, such as creating memory leaks by inadvertently maintaining pointers to old data.

All in all this appears to be an interesting and worthwhile book that can help the new Lisper become familiar with the parts of Lisp that1 aren't mentioned in the Common Lisp Specification. The problem, of course, is that these parts are not standardized and thus are implemented differently in different Common Lisps. By considering several Common Lisps, the author shows the reader the various ways these important but nonstandard parts of Lisp are implemented. I do think, however, that it would improve the book to take the majority of the examples from Lisps that a new user is actually apt to be using.

Yet Another “No Parenthesis Lisp”

I really don't understand this. It's bad enough that noobs keep coming up with stuff like this, do we have to advertise their silliness on the Lisp Aggregators? If you want Python, go use Python already.

JSON

ESR recently put up a paper on his redesign of the request-response protocol for gpsd. The gpsd project is interesting in its own right, but what interests me is his use of JSON to serialize command and response data to the back end of the daemon.

Like many hackers, I'm indifferent to Java. I've never had the urge to learn it and I've been fortunate that my employment has never required its use. I have noticed, however, that Java-related things always seem to start with a capital J, so when I saw references to JSON I always assumed it was yet another Java something or other and passed it by without further investigation. Like most forms of prejudice, of course, this was merely ignorance. While it's true that the J does stand for Java, JSON actually stands for Java Script Object Notation and it's a really neat way of serializing complex data for transfer between applications.

JSON has two types of data structures:

• arrays, in which the values are comma separated and enclosed in square brackets, and
• objects, which are serialized representations of what Lispers call hash tables or alists and Pythonistas call dictionaries. An object is a comma separated list of name/value pairs enclosed in curly braces. The name/value pairs have the form "name" : value.

These structures are fully recursive and each can contain instances of itself or the other. Names are always strings, but object and array values can be strings, numbers, objects, arrays, true, false, or null. The exact syntax, complete with railroad diagrams, is given on the JSON.org site linked above; a formal description is given in RFC 4627. As an example, here is a hypothetical object representing a family:

{"father":{"name":"John Smith", "age":45, "employer":"YoYodyne, inc."},
"mother":{"name":"Wilma Smith", "age":42},
"children":[{"name":"William Smith", "age":15},
         {"name":"Sally Smith","age":17}]}

Notice that white space can be inserted between any pair of tokens to

make the object more human-readable, but that it is not required. Also notice that unused fields in objects (but not arrays) can be omitted as the employer field is in all but the father's object.

What I like about this is that it's fundamentally Lispy, which shouldn't be a surprise given JavaScript's Scheme lineage. For example, the above JSON family object rendered in a typically Lisp way is shown below. Notice how similar it is to the JSON rendering.

'((father . ((name . "John Smith") (age . 45) (employer . "YoYodynce, inc.")))
(mother . ((name . "Wilma Smith") (age . 42)))
(children . #(((name . "William Smith") (age . 15))
            ((name . "Sally Smith") (age . 17)))))

The nice thing about this representation is that the Lisp/Scheme reader will parse it directly. True, the name/value pairs are represented as alists, but for many applications that's the appropriate choice. Common Lisp hash tables don't have an external representation that is directly readable, but we could certainly use alists as such a representation at the cost of spinning down the list and hashing the names.

None of this is to say that we should eschew JSON in favor of s-expressions, especially when doing data-interchange between applications written in non-Lisp languages or even when interchanging data between a Lisp application and, say, a C application. Rather, my point is that Lispers should feel familiar and comfortable with it.

So what's the take away on this? First, that JSON is a simple and ubiquitous (the JSON.org page has links to parsers for dozens of languages, including CL and Scheme) data-interchange format that can make interprocess communication much easier when complex data is involved. Second, I think it nicely illustrates the power of Lisp ideas and data structures, even if JSON itself is neither.