This is a description of the Common Lisp ecosystem, as of January, 2021, from the perspective of a user and contributor.
The purpose of this article is both to give an overview of the ecosystem, and to help drive consolidation in each domain.
Each application domain has recommendations for consolidating that part of the ecosystem, and pointers for interesting future work.
This article is derived from Fernando Borretti’s State of the Common Lisp ecosystem from 2015, hence the introduction that sounded familiar. This new one will be an opportunity to look at what was achieved, or what is still lacking.
Disclaimer: This article is not a list of every project or article of interest that came out in the last years. I wrote an overview of 2018 closer to that goal here. More libraries can be discovered on the Awesome-cl list, on GitHub and on Cliki.
Acknowledgements I would like to thank @borodust, @ambrevar and @digikar for their kind feedback.
Table of Contents
- Application domains
- Languages interop
- Last words
There used to be several options to ease building and distribution of command line programs, but now Roswell has gained most momentum, and that’s a good thing. Roswell is an implementation manager, installer and a script runner, and one of its neat features is support for very easily compiling tiny scripts into executables.
Now, GNU Guix has gained many CL libraries, and becomes a contender to Roswell. Guix can be used as a package manager on top of your Unix distribution. It brings reproducible builds, rollbacks, the ability to install exact versions of any library (including system dependencies), contained environments and user profiles. It makes it easy too to install the latest version of a CL implementation and libraries and, to a certain extent, to share scripts. See the article A Lisp REPL as my main shell for insights.
To parse command line arguments, unix-opts shows decent activity. As a reminder, the CLI arguments are stored portably in
Adams is a new UNIX system administration tool, not unlike Chef or Ansible.
More features to the sripting libraries.
The Lem editor has built a great user interface and REPL on top of ncurses, with the cl-charms library. It would be great to re-use its components, so that Lispers could easily build similar rich terminal-based interfaces.
Mito is an ORM for Common Lisp with migrations, relationships and PostgreSQL support. It is based on cl-dbi (a uniform interface to the various database server-specific libraries such as cl-postgres and cl-mysql) and SxQL (a DSL for building safe, automatically parameterized SQL queries).
It also has a tutorial in the Cookbook: Cookbook/databases.
There are of course more libraries in that field. Some new ones since 2015 are:
cl-yesql (by the author of Serapeum, Spinneret and other great libraries) is based on Clojure’s Yesql.
vivace-graph is a graph database and Prolog implementation, taking design and inspiration from CouchDB, neo4j and AllegroGraph.
Vsevolod Dyomkin, the author of Rutils, the Programming Algorithms book and other libraries, is writing cl-agraph, a minimal client to Franz Inc’s AllegroGraph. AllegroGraph is a “horizontally distributed, multi-model (document and graph), entity-event knowledge graph technology”. It is proprietary and has a free version with a limit of 5 million triples. Surely one of those Lisp hidden gems we should know more about.
A general migration tool was lacking. We now have cl-migratum, a “system which provides facilities for performing database schema migrations, designed to work with various databases”.
And of course, pgloader is still a Common Lisp success story.
Among the emerging ORMs, Mito is the one actively maintained that Lispers seem to have chosen. Good. CLSQL certainly still works, but we don’t hear about it and it looks outdated. So, Mito it is.
Mito has 11 contributors and is actively watched, but it probably should have another(s) core maintainers.
Bindings for the new databases coming out.
In the last year, Manfred Bergmann developed cl-gserver. It is a “message passing” library/framework with actors similar to Erlang or Akka. It is an important achievement.
Its v1 features:
- actors can use a shared pool of message dispatchers which effectively allows to create millions of actors.
- the possibility to create actor hierarchies. An actor can have child actors. An actor now can also “watch” another actor to get notified about its termination.
Many other libraries exist in this area:
- BordeauxThreads - Portable, shared-state concurrency
- the “de-facto” concurrency library.
- lparallel - A library for parallel programming.
- also solid, battle-tested and popular, aka de-facto.
- calispel - CSP-like channels for common lisp. With blocking, optionally buffered channels and a “CSP select” statement. ISC-style.
- “It is complete, flexible and easy to use. I would recommend Calispel over Lparallel and ChanL.” @Ambrevar. discussion
- ChanL - Portable, channel-based concurrency.
- cl-async - A library for general-purpose, non-blocking programming.
- works atop libuv
- Moira - Monitor and restart background threads. In-lisp process supervisor.
- trivial-monitored-thread - a Common Lisp library offering a way of spawning threads and being informed when one any of them crash and die.
- lfarm - distributing work across machines (on top of lparallel and usocket).
- cl-gearman - a library for the Gearman distributed job system.
- Alexander Artemenko used it instead of lfarm for Ultralisp: https://40ants.com/lisp-project-of-the-day/2020/06/0095-cl-gearman.html, because “lfarm is not well suited to environments where worker hosts can go down and return back later”.
- swank-crew - distributed computation framework implemented using Swank Client.
- cl-coroutine - a coroutine library. It uses the CL-CONT continuations library in its implementation.
- CMTX: high performance transactional memory for Common Lisp.
- In our opinion, a library not well known and under-appreciated.
Bordeaux-Threads is the “de-facto” library, but there is some choice paralysis between Lparallel, Calispel, Bordeaux-Threads and SBCL’s contribs. Use the libraries in the wild and write about them.
There exist Common Lisp libraries for all the major file formats:
- XML: Plump (and Lquery), as well as CXML, which can parse large files incrementally.
- JSON: Jonathan, cl-json or more. With utilities:
- YAML: cl-yaml
- CSV: cl-csv
New in 2015, Jonathan is now a good first choice for an easy to use and fast JSON encoder and decoder.
There is not a predominant JSON library. This leads to choice paralysis.
They all represent null values differently. We need a library that “does the right thing”. See maybe the massive web-toolkit for its JSON handling ?
It distinguishes null, false and  from Lisp’s NIL thus supports identical transformation between JSON values. It provides object constructor and accessor to build and access nesting JSON objects.
Give the XPath library some love and documentation.
Still valid from 2015:
A YAML parser so that cl-yaml doesn’t depend on the libyaml library would make distribution far simpler.
A usual complain in Common Lisp land is the lack of a complete, cross-platform GUI solution. Ltk is a very good library, but Tk is limited. Qtools is great, but is only for Qt4.
Matthew Kennedy wrote excellent FFI bindings to the IUP Portable User Interface library: IUP. IUP is cross-platform (Windows, macOS, GNU/Linux, with new Android, iOS, Cocoa and Web Assembly drivers), has many widgets (but less than Qt), has a small API and is actively developed. IUP was created at the PUC university of Rio de Janeiro.
Nicolas Hafner started Alloy, a new user interface protocol and toolkit implementation, which he uses in his Kandria game.
Very recently, David Botton released CLOG, “the Common Lisp Omnificent GUI”:
CLOG uses web technology to produce graphical user interfaces for applications locally or remotely. CLOG can take the place, or work alongside, most cross-platform GUI frameworks and website frameworks. The CLOG package starts up the connectivity to the browser or other websocket client (often a browser embedded in a native template application.)
It is complete enough for most uses.
There are more GUI libraries and frameworks: https://github.com/CodyReichert/awesome-cl#Gui (and more under the works). In particular, LispWorks’ CAPI is still presented as the best in town by the ones who tried it.
Since roughly October, 2020, Nicolas Hafner works full time on Kandria. Supporting his work, through GitHub sponsors or ko-fi would be 1) a great sign of recognition and 2) useful for the ecosystem, especially for Alloy.
I wrote an introduction to these frameworks in the Cookbook: Cookbook/gui. More examples or demo projects would be welcome.
There are two actively maintained diverged forks of the GTK bindings. A reunification effort is required.
Write a desktop application with IUP/your toolkit of choice for everyday use and make it a Common Lisp flagship.
It seems that not much changed since 2015, but libraries are still being developed:
- mgl-mat - a library for working with multi-dimensional arrays which supports efficient interfacing to foreign and CUDA code. BLAS and CUBLAS bindings are available.
Others are less active:
- Antik - a foundation for scientific and engineering computation in Common Lisp. It is designed not only to facilitate numerical computations, but to permit the use of numerical computation libraries and the interchange of data and procedures, whether foreign (non-Lisp) or Lisp libraries.
- more than 2000 commits, last update 2 years ago.
To quote Fernando:
UIOP, ASDF’s portable compatibility layer, contains a large set of tools for portably doing everything from querying the hostname to running external programs to manipulating environment variables.
We should not require cl-fad anymore (but we need Osicat, which unlike UIOP is POSIX friendly).
Built on top of UIOP, Paul M. Rodriguez’s cmd brings in short and handy helpers to run and pipe programs.
Common Lisp’s main web servers are Hunchentoot and Clack. Since 2015, Clack’s documentation state barely improved and is still lacking.
Clack is the equivalent of WSGI/Rack. It has existed since 2009. It is an HTTP server abstraction, that allows the user to write web applications (or, more reasonably, web application frameworks) without depending on a particular server. Some web frameworks are built on top of it, for example Caveman2.
The importance of using Clack cannot be understated: If you build an application directly on, say, Hunchentoot, you’re tied to Hunchentoot, and if a new, faster server – like Woo – comes out, you have to rewrite the entire application to use it. If you write a plugin for Clack – like clack-errors – it is automatically usable by all applications, regardless of framework, that are built on Clack, reducing useless duplication of code.
With Clack, switching from Hunchentoot to Woo, and enjoying the incredible speedup, is a simple matter of installing libev and changing a keyword argument.
This still holds true, but the situation didn’t improve much. In comparison, Hunchentoot is very well documented (and you can read its documentation on a better looking readthedocs here), and it is “fast enough”.
About Hunchentoot: Mariano Montone wrote easy-routes, a little but handy route handling facility on top of Hunchentoot. It brings:
- dispatch by HTTP method,
- arguments extraction from the URL path,
- “decorators” to, for example, quickly add authorization checks,
- integration with the Djula framework to generate URLs from route names.
Several Clack plugins were written, such as a single-sign on middleware.
Write more documentation for Clack. While Lispers know about it, they don’t necessarily adopt it because of the lack of documentation. We can expand this getting started guide.
Build a batteries-included framework.
Many HTML generators and template libraries exist (see the list below). However, some new and good ones appeared lately:
- TEN, by Djula’s maintainer, brings the completness of Djula with the usability of Eco (by Fernando Borretti), aka: you write Django-like HTML templates but you can interleave any Lisp code.
- markup - a JSX-like templating engine, where HTML tags are Common Lisp code. Comes with an Emacs package.
Other HTML generators and templating engines include:
- spinneret - Common Lisp HTML5 generator.
- cl-who - The venerable HTML generator.
- Djula - A port of Django’s template engine to Common Lisp.
- cl-closure-template - Implementation of Google’s Closure templates. [LLGPL].
- clip - An HTML template processor where the templates are written in HTML.
We have nice other building blocks, such as a nice form handling library (cl-forms) and libraries to create Open-API interfaces. An integrated, opinionated all-in-one solution could be a productivity boom.
Djula is easy to work with. It could do with more built-in filters.
As in 2015:
The foundation is finished, now it’s time to write higher-level layers. An extensible administration framework for Clack applications, like Django’s Admin, would be a good example.
- JSCL, a CL-to-JS compiler designed to be self-hosting from day one. JSCL is not complete (yet), it lacks CLOS, format and loop.
Two new are in development:
Help develop one of the existing CL-to-JS implementations. Why not have a look at JSCL’s issues?
Isomorphic web frameworks
Weblocks was getting old and unmaintained but Alexander Artemenko greatly updated and refactored it in his Reblocks branch. He uses it for the Ultralisp website, and more apps. You can reach users and developers on Gitter.
Recently, a very new web framework appeared: ISSR, for Interactive Server-Side rendering. It links a client to the server with a websocket connection and updates the DOM selectively. It is thus not unlike Phoenix’s LiveView or Hotwire.
See this todo-app tutorial.
Reviving Weblocks and releasing CLOG and ISSR are great achievements. However, work is only started to create a community of users around them.
New solutions arose to interoperate with other runtimes.
April brings the APL programming language (a subset thereof) to Common Lisp. Replace hundreds of lines of number-crunching code with a single line of APL.
C, C++, Objective C
It will be a great achievement when CLAW is officially ready to use. This is not yet the case (though the GLM bindings basically do their hello world on Android, which is an achievement per se).
ABCLJ provides a “dead easy Clojure to Common lisp interop”:
instead of rewriting the whole Clojure langugage on CL I’m embedding ABCL in Clojure. Since both are implemented in Java and Clojure has an awesome java interop is easy to have full access on the ABCL Common Lisp environment. This way we have complete support for both Clojure and Common Lisp.
The reason I wanted to see Clojure and Common Lisp working with each other was to use CL programs/libraries on Clojure, especially Maxima and ACL2. Since ABCL already compiles and runs Maxima it should be possible but we are very far from it 🤷.
There are others of attempts to shorten the gap between clojure and common lisp like Cloture and clclojure. Once they are complete Clojure will benefit from native binaries and excelent compilers like SBCL, however they are far from complete.
On the topic, see this talk by Alan Dipert: “Common Lisp for the curious Clojurian”.
“If I had to be stranded with something other than Clojure, I’d be happiest with a good Common Lisp and its source code.” - Rich Hickey, 2011
Common Lisp (CL) and Clojure are both dialects of Lisp. Rich Hickey, the creator of Clojure, learned CL and used it professionally before creating Clojure.
What can Clojure do that CL can’t, and vice versa? Why would anyone use CL today, when both Clojure and ClojureScript exist?
In this talk, I will try to answer these questions and more, from the perspective of a long-time Clojurian with a growing passion for CL.
Calling to Python and hooking into its ecosystem is easier than ever.
Improving CL libraries such as Numcl (a Numpy clone) is what’s required to drive Common Lisp forward.
Bike is a cross-platform .Net Core interface.
All implementations saw new releases, except CLisp, whose development however continues.
Active implementations include: ABCL, CCL, CLASP, ECL, LispWorks, AllegroCL, SBCL. And to a certain extent, GNU CLisp, SICL (which is the newest one) and Corman Lisp (a CL development environment for Windows) (regenerated here).
ABCL jumped to v1.8.0 to support openjdk15.
SBCL still ships monthly releases. It turned 20 and keeps improving (RISC-V port, M1 port, block compilation, more compile-time type checking…). We can read a blog on the party held in Vienna here. Did you know that Doug Katzman of Google fame contributes to SBCL?
Doug Katzman talked about his work at Google getting SBCL to work with Unix better. For those of you who don’t know, he’s done a lot of work on SBCL over the past couple of years, not only adding a lot of new features to the GC and making it play better with applications which have alien parts to them, but also has done a tremendous amount of cleanup on the internals and has helped SBCL become even more Sanely Bootstrappable. That’s a topic for another time, and I hope Doug or Christophe will have the time to write up about the recent improvements to the process, since it really is quite interesting.
Anyway, what Doug talked about was his work on making SBCL more amenable to external debugging tools, such as gdb and external profilers. It seems like they interface with aliens a lot from Lisp at Google, so it’s nice to have backtraces from alien tools understand Lisp. It turns out a lot of prerequisite work was needed to make SBCL play nice like this, including implementing a non-moving GC runtime, so that Lisp objects and especially Lisp code (which are normally dynamic space objects and move around just like everything else) can’t evade the aliens and will always have known locations.
Here too, great progress has been made. While a usual complain of non-Lispers was the lack of editor support besides Emacs (and Vim), we now nearly reach choice paralysis:
- Portacle is the easiest way to get started with Emacs. It is portable and multi-platform, ready-to-use in three clicks. It ships Emacs, SBCL, Slime, Quicklisp and git.
- SLIMA is the Atom extension. It is nearly as good as Slime for Emacs.
- VSCode has two extensions: commonlisp-vscode, using the Language Server Protocol, and Alive, more recent, using a Lisp backend (Swank) as traditional extensions.
- Sublime Text got a good extension: Slyblime is an implementation of SLY and it uses the same backend (SLYNK). It ships advanced features including a debugger with stack frame inspection.
- Lem is an editor written in Common Lisp. It allows to start developing in CL at once, and it supports other languages.
- we have a Jupyter kernel for CL.
- the Dandelion Eclipse plugin was re-discovered. While it isn’t as feature-rich as others (no interactive debugger for example), it has its users. It specifically targets beginners.
Last but not least, if you want to play in your iPhone or iPad, the CodePlayground app got Lisp support via CCL.
Life continues to improve for the developper. We will cite some new tools:
- cl-flamegraph is a wrapper around SBCL’s statistical profiler to generate FlameGraph charts from Common Lisp programs.
- tracer is a tracing profiler for Common Lisp, with output suitable for display in Chrome’s/Chromium’s Tracing Viewer.
- GTFL is a graphical terminal for Lisp, meant for Lisp programmers who want to debug or visualize their own algorithms. It is a graphical trace in the browser.
- Lisp REPL core dumper is a portable wrapper to generate Lisp cores on demand to start a REPL blazingly fast. It can preload provided systems to help build a collection of specialized Lisp cores.
- if you are used to working in different environments that require their own set of libraries, this core dumper (optionally along with SLY’s mrepl) can make switching from one another easier and faster.
Quicklisp is the de-facto package manager. However, we now have:
- Ultralisp, a Quicklisp distribution that builds every 5 minutes. We can add our project in two clicks.
- CLPM, a new package manager that is compatible with Quicklisp, that allows to pin exact versions of dependencies, that is usable from the command line and that supports HTTPS.
Not forgetting Qlot, to install Quicklisp libraries relative to a directory.
Last but not least, as said earlier, many CL libraries were packaged for Guix (most notably by Pierre Neidhart of Nyxt).
Ultralisp solves the 1-month release schedule of Quicklisp (which is a feature, but not to everyone’s taste) and makes it straightforward and quick to publish a library. CLPM by tackling a different approach solves other Quicklisp limitations. Both are great achievements.
Ultralisp also has a search box that searches a symbol on all its registered libraries. Very useful.
Alexander is working on allowing every Ultralisp user to create his own Quicklisp dist in a few clicks.
Same as 2015, ASDF is the de-facto build system.
Every project has an .asd file, called a system definition file, which defines project metadata (author, maintainer, homepage, etc.) and the components.
This, to me, is one of the major selling points of Common Lisp. With languages like Python, every file imports whatever it needs, and your project becomes a massive graph of interdependent files. In ASDF, you basically list the files in your project in the order in which they are defined. Or, you can specify the dependencies between the files, and let ASDF figure out a linear ordering. The point is that dependencies are explicit, and clearly spelled out.
There’s not much to say here, except that Common Lisp has a pretty great type system that is not exploited nearly enough.
And to our greatest pleasure, SBCL’s type system continues to improve. For example, SBCL 1.5.9 now gives type warnings when a slot declared type doesn’t match its
initform. It continued to improve on SBCL 2.0 and onwards.
Moreover, the Coalton library is bringing a dialect of ML on top of CL, in order to write statically typed programs similar in spirit to Standard ML, OCaml, and Haskell.
Help develop Coalton.
Fernando cited FiveAM and recommended it along with the much newer Prove. Prove has a couple issues and is now deprecated by its author, and its younger brother Rove is not in par yet.
So, use FiveAM.
Moreover, Common Lisp has good support for the CI/CD services out there.
CL Foundation’s Docker images have integrated best practices over the years and are recommended: https://common-lisp.net/project/cl-docker-images/
CI-Utils regroups utilities for various platforms (Travis, Circle, Gitlab, Github, Appveyor, Bitbucket, Azure) and test frameworks.
For Travis CI, you can also see cl-travis (for ABCL, Allegro CL, SBCL, CMUCL, CCL and ECL).
You will find an example for Gitlab CI on the Cookbook.
Rove or Parachute would be great alternatives if developed a bit further.
Integration with the CI services’ advanced features such as Gitlab’s auto DevOps.
Common Lisp is very well documented through its standard, the Common Lisp Hyper Spec and many books. However, we felt it was lacking good on-line material. Good news is, the situation improved tremendously in the last three or four years.
New common-lisp.net website
https://common-lisp.net was written anew. It looked dated. This is now fixed. Well done!
The Common Lisp Cookbook on GitHub got revived by many new contributors, included myself. It got many new content and a new UI. It is also now available in ePub and PDF, for free or as a “pay what you want” option.
Write content on the Cookbook. Don’t write tutorials on your blog. Everyone can help, even new Lispers (and in fact: mostly new Lispers can write content best suited to the Cookbook’s target audience).
Make it look world-class with a real and modern theme.
Help revive the minispec ?
The awesome-cl list saw continuous updates and is now a great solution to have an overview of the ecosystem and choose a library.
One of its goals is to break choice paralysis by recommending libraries, with its “+1” marks.
Help furnish and curate it.
I agree with /u/defunkydrummer here:
Note that many, many things that people wish to see, are already available, so perhaps we, as a community, are not fully communicating the state of our ecosystem even to our insiders (!)
Several popular libraries have been ported to readthedocs, so the reading experience is more pleasant: https://common-lisp-libraries.readthedocs.io/.
Michal “phoe” Herda organized many online Lisp meetings, and we can find the videos on Youtube: https://www.youtube.com/c/OnlineLispMeetings/videos
Alexander Artemenko started lisp project of the day, a blog to review a library every day for a month, and he is now at post #219. Lately he reviewed many documentation builders for CL.
On a sadder note, Quickdocs closed :(
We got 3 new books on Common Lisp in 2020:
- Programming Algorithms, originally published by Vsevolod Dyomkin on his website, then self-published in paperback and then published by Apress.
- the Common Lisp Condition System, by Michal “phoe” Herda, was also published by himself and then by Apress.
- The Cookbook that was made available in ePub and PDF :)
- the book Calendrical calculations, 4th edition, by Edward M. Reingold, Nachum Dershowitz, Cambridge Press. It provides Lisp sources.
- Building Problem Solvers, by Kenneth Forbus and Johan de Kleer, MIT Press, was made available.
We now have a curated list of companies using CL: awesome-cl-companies. Before that list, the situation was embarassing:
Everyone says “Nobody uses Lisp” and Lispers say “Yes they do, there’s ITA, and, um, Autocad, and, uh, oh yeah, Paul Graham wrote Viaweb in Lisp!” Not very helpful for either side. It’s about time there was a better resource.
And see also lisp-lang.org’s success stories.
Some additions of this year include GraphMetrix (automation of document extraction and publishing for construction, property and logistics), Doremir Music Research AB (developing ScoreCloud, a music notation software: you sing, it writes the score), Keepit (a cloud-to-cloud backup service provider), Mind AI (an artificial intelligence engine and ecosystem), Virtual Insurance Products Ltd (insurance MGA with a bespoke business to business web platform) or again the Mimix Company (creators of MSL and Nebula, new tools for working with facts and documents).
We are able to compare the number of downloads of the 100 most popular Quicklisp libraries between 2015 and 2020:
We can observe a 3x growth in five years. Of course, these figures need to be taken with a grain of salt, what they really represent is subject to interpretation. What is the role of Continuous Integration here?
Many things are happening in the CL universe. Stay tuned!
The article source.