What makes a good extension language?

“Scripting languages are designed for gluing”, wrote John Ousterhout, the creator of Tcl, in 1998.¹ What Ousterhout had in mind was writing an application in a scripting language and having it delegate certain functions to components written in system programming languages. A similar idea inspired the wrapper generator SWIG, developed for scientific computing.² Nowadays, Python is widely used in this way, thanks to its vast ecosystem of modules written in system programming languages.

But there is another way to combine a scripting language with a system programming language: embed an interpreter in an application. In 1994, Paul Dubois asked us to imagine a program “whose user interface is a programming language”.³ As the creators of Lua wrote in 1996:

There is increasing demand for customizable applications. As applications became more complex, customization with simple parameters became impossible: users now want to make configuration decisions at execution time; users also want to write macros and scripts to increase productivity. In response to these needs, there is an important trend nowadays to split complex systems in two parts: kernel and configuration. The kernel implements the basic classes and objects of the system, and is usually written in a compiled, statically typed language, like C or Modula-2. The configuration part, usually written in an interpreted, flexible language, connects these classes and objects to give the final shape to the application.⁴

Several examples come to mind: Basic in Excel (an example of of end-user programming), JavaScript in web browsers and in Visual Studio Code, Emacs Lisp, and Lua in many programs, such as Redis, Neovim, pandoc, and game engines. Embedded languages are often called extension languages, a term that suggests that the host application only delegates a few tasks to scripts, but in some cases, most of the application’s functionality is implemented in the interpreted language (about 1.5 million lines of code in Emacs), and the kernel is much smaller. Andy Wingo, a contributor to the extension language Guile, wrote in 2009:

Emacs, when it was first created in its GNU form in 1984, was a new take on the problem of “how to make a program”. The Emacs thesis is that it is delightful to create composite programs based on an orthogonal kernel written in a low-level language together with a powerful, high-level extension language.

What are the requirements for an embedded language? The authors of the Lua paper list a few, focusing on the design of the language itself. Here I would like to ask: are they the same as the requirements for a glue language? One difference can be found in something outside language design, namely the language’s library ecosystem. Python’s rich ecosystem is important for making it a good glue language, but Lua is widely used as an extension language even though its library ecosystem is much smaller and less active. This is partly because embedded scripts often rely on functionality provided by the host application, and don’t need many libraries. Sometimes, as with web browsers and Emacs, a library ecosystem develops that’s specific to the application.

Wingo explains that the success of Emacs inspired the creation of Guile, a Scheme dialect intended to be a more general-purpose extension language for the GNU project. While Guile has found some niches, it hasn’t been widely adopted. Is that because it’s a Lisp? The authors of the Lua paper wrote: “Lisp cannot be called user-friendly when it comes to customization. Its syntax is rather crude for non-programmers.” The designers of Emacs Lisp disagreed; in their view,

The language should be accessible to a wide audience, so that as many people as possible can adapt Emacs to their own needs, without being dependent on the availability of someone with technical expertise. For example, the Introduction to Programming in Emacs Lisp tutorial targets users with no programming experience.⁵

I’m inclined to think that use of extension languages is driven more by the popularity of the applications that support them than by their syntax. When Microsoft chose Basic as the extension language for Excel in 1993, people who wanted to extend Excel learned Basic because they had to. In 1995, Netscape created JavaScript as the extension language for its web browser, and people who wanted to write scripts that ran in web browsers then learned JavaScript because they had to. As a result, twenty years later, a large number of programmers knew JavaScript. When Microsoft then developed Visual Studio Code and needed an extension language, they could have chosen Basic again, but this time they chose JavaScript, no doubt because many developers already knew it.

Lua doesn’t have such an application to drive its adoption, but it has something else. I suspect that many of the developers who use Lua do so not because they have to, but because they have strict requirements concerning performance and/or executable size, and therefore choose LuaJIT, which is used in “network switches, set-top boxes and other embedded devices”. (If you want those features but you also want to program in Lisp, there’s Fennel, a Lisp that compiles to Lua.)

Extension languages are often designed to be embedded in C programs, and need substantial glue code (“bindings”) to integrate them into programs written in other languages. A lack of convenient bindings therefore becomes an obstacle to using the extension language. For example, there are good libraries for embedding Python and Lua in Rust programs, but nothing comparable for Guile. (Instead, there are several Scheme interpreters implemented in Rust.) Lack of IDE support can be another obstacle.

In short, what makes a good extension language can depend as much on its environment (the host application’s functionality and constraints) and ecosystem (including libraries, JITs, bindings, and IDE support) as on the language itself.

Bibliography #

Beazley, David M., and Peter S. Lomdahl. 1996. “Lightweight Computational Steering of Very Large Scale Molecular Dynamics Simulations.” In Proceedings of the 1996 ACM/IEEE Conference on Supercomputing. IEEE Computer Society. https://doi.org/10.1145/369028.369136.

Dubois, Paul F. 1994. “Making Applications Programmable.” Computers in Physics 8 (1): 70–73. https://doi.org/10.1063/1.4823265.

Ierusalimschy, Roberto, Luiz Henrique de Figueiredo, and Waldemar Celes Filho. 1996. “Lua—An Extensible Extension Language.” Software: Practice and Experience 26 (6): 635–52. https://www.lua.org/spe.html.

Monnier, Stefan, and Michael Sperber. 2020. “Evolution of Emacs Lisp.” Proceedings of the ACM on Programming Languages 4 (HOPL). https://doi.org/10.1145/3386324.

Ousterhout, John K. 1998. “Scripting: Higher Level Programming for the 21st Century.” IEEE Computer 31 (3): 23–30. https://www.tcl-lang.org/doc/scripting.html.