Caution [from 2017: what a road since then]: this is a draft. I take notes and write more in other resources (the Cookbook, my blog).

update, January 2024: see my new video on Youtube, a demo on how to assemble Hunchentoot, easy-routes, Djula templates (with ready-to-use Bulma themes), with error handling and common traps: https://www.youtube.com/watch?v=h_noB1sI_e8

update, June 2022: see my web project skeleton, it illustrates and fixes common issues: https://github.com/vindarel/cl-cookieweb

update july, 5th 2019: I put this content into the Cookbook: https://lispcookbook.github.io/cl-cookbook/web.html, fixing a long-standing request.

new post: why and how to live-reload one’s running web application: https://lisp-journey.gitlab.io/blog/i-realized-that-to-live-reload-my-web-app-is-easy-and-convenient/

new project skeleton: lisp-web-template-productlist: Hunchentoot + easy-routes + Djula templates + Bulma CSS + a Makefile to build the project

See also the Awesome CL list.

Information is at the moment scarce and spread appart, Lisp web frameworks and libraries evolve and take different approaches.

I’d like to know what’s possible, what’s lacking, see how to quickstart everything, see code snippets and, most of all, see how to do things that I couldn’t do before such as hot reloading, building self-contained executables, shipping a multiplatform web app.

Prior notice:

Some people sell ten pages long ebooks or publish their tutorial on Gitbook to have a purchase option. I prefer to enhance the collaborative Cookbook (I am by far the main contributor). You can tip me on Kofi if you like: https://ko-fi.com/vindarel Thanks !

Table of Contents

Web application environments

Clack, Lack

Clack is to Lisp what WSGI is to Python. However it is mostly undocumented and not as battle-proofed as Hunchentoot.

Web frameworks

Hunchentoot

The de-facto web server, with the best documentation (cough looking old cough), the most websites on production. Lower level than a web framework (defining routes seems weird at first). I think worth knowing.

Its terminology is different from what we are used to (“routes” are not called routes but we create handlers), part I don’t know why and part because the Lisp image-based development allows for more, and thus needs more terminology. For example, we can run two applications on different URLs on the same image.

https://edicl.github.io/hunchentoot/

edit: here’s a modern looking page: https://digikar99.github.io/common-lisp.readthedocs/hunchentoot/

Caveman

A popular web framework, or so it seems by the github stars, written by a super-productive lisper, with nice documentation for basic stuff but lacking for the rest, based on Clack (webserver interface, think Python’s WSGI), uses Hunchentoot by default.

I feel like basic functions are too cumbersome (accessing url parameters).

https://github.com/fukamachi/caveman

Snooze

By the maintainer of Sly, Emacs’ Yasnippet,…

Defining routes is like defining functions. Built-in features that are available as extensions in Clack-based frameworks (setting to get a stacktrace on the browser, to fire up the debugger or to return a 404,…). Definitely worth exploring.

https://github.com/joaotavora/snooze

Radiance

Radiance, with extensive tutorial and existing apps.

It doesn’t look like a web framework to me. It has ready-to-use components:

  • admin page (but what does it do?)
  • auth system
  • user: provide user accounts and permissions
  • image hosting
  • there is an email marketing system in development…

cl-rest-server

cl-rest-server

a library for writing REST Web APIs in Common Lisp.

Features: validation via schemas, Swagger support, authentication, logging, caching, permission checking…

It seems complete, it is maintained, the author seems to be doing web development in CL for a living. Note to self: I want to interview him.

Wookie

https://github.com/orthecreedence/wookie

An asynchronous web server, by an impressive lisper, who built many async libraries. Used for the Turtl api backend. Dealing with async brings its own set of problems (how will be debugging ?).

Nice api to build routes, good documentation: http://wookie.lyonbros.com/

Weblocks (solving the Javascript problem)

Weblocks allows to create dynamic pages without a line of JavaScript, all in Lisp. It was started years ago and it saw a large update and refactoring lately.

It isn’t the easy path to web development in CL but there’s great potential IMO.

It doesn’t do double data binding as in modern JS frameworks. But new projects are being developed…

See our presentation below.

http://40ants.com/weblocks/quickstart.html

Tasks

Accessing url parameters

It is easy and well explained with Hunchentoot or easy-routes in the Cookbook.

Lucerne has a nice with-params macro that makes accessing post or url query parameters a breeze:

@route app (:post "/tweet")
(defview tweet ()
  (if (lucerne-auth:logged-in-p)
      (let ((user (current-user)))
        (with-params (tweet)
          (utweet.models:tweet user tweet))
        (redirect "/"))
      (render-template (+index+)
                       :error "You are not logged in.")))

Snooze’s way is simple and lispy: we define routes like methods and parameters as keys:

(defroute lispdoc (:get :text/* name &key (package :cl) (doctype 'function))
   ...

matches /lispdoc, /lispdoc/foo and /lispdoc/foo?package=arg.

On the contrary, I find Caveman’s and Ningle’s ways cumbersome.

Ningle:

(setf (ningle:route *app* "/hello/:name")
      #'(lambda (params)
          (format nil "Hello, ~A" (cdr (assoc "name" params :test #'string=)))))

The above controller will be invoked when you access to “/hello/Eitaro” or “/hello/Tomohiro”, and then (cdr (assoc “name” params :test #‘string=)) will be “Eitaro” and “Tomohiro”.

and it doesn’t say about query parameters. I had to ask:

(assoc "the-query-param" (clack.request:query-parameter lucerne:*request*) :test 'string=)

Caveman:

Parameter keys contain square brackets (”[” & “]”) will be parsed as structured parameters. You can access the parsed parameters as _parsed in routers.

(defroute "/edit" (&key _parsed)
  (format nil "~S" (cdr (assoc "person" _parsed :test #'string=))))
;=> "((\"name\" . \"Eitaro\") (\"email\" . \"e.arrows@gmail.com\") (\"birth\" . ((\"year\" . 2000) (\"month\" . 1) (\"day\" . 1))))"

Session an cookies

Data storage

SQL

Mito works for MySQL, Postgres and SQLite3 on SBCL and CCL.

https://lispcookbook.github.io/cl-cookbook/databases.html

We can define models with a regular class which has a mito:dao-table-class :metaclass:

(defclass user ()
  ((name :col-type (:varchar 64)
         :initarg :name
         :accessor user-name)
   (email :col-type (:varchar 128)
          :initarg :email
          :accessor user-email))
  (:metaclass mito:dao-table-class)
  (:unique-keys email))

We create the table with ensure-table-exists:

(ensure-table-exists 'user)
;-> ;; CREATE TABLE IF NOT EXISTS "user" (
;       "id" BIGSERIAL NOT NULL PRIMARY KEY,
;       "name" VARCHAR(64) NOT NULL,
;       "email" VARCHAR(128),
;       "created_at" TIMESTAMP,
;       "updated_at" TIMESTAMP
;   ) () [0 rows] | MITO.DAO:ENSURE-TABLE-EXISTS

Persistent datastores

Migrations

Mito has migrations support and DB schema versioning for MySQL, Postgres and SQLite3, on SBCL and CCL. Once we have changed our model definition, we have commands to see the generated SQL and to apply the migration.

We inspect the SQL: (suppose we just added the email field into the user class above)

(mito:migration-expressions 'user)
;=> (#<SXQL-STATEMENT: ALTER TABLE user ALTER COLUMN email TYPE character varying(128), ALTER COLUMN email SET NOT NULL>
;    #<SXQL-STATEMENT: CREATE UNIQUE INDEX unique_user_email ON user (email)>)

and we can apply the migration:

(mito:migrate-table 'user)
;-> ;; ALTER TABLE "user" ALTER COLUMN "email" TYPE character varying(128), ALTER COLUMN "email" SET NOT NULL () [0 rows] | MITO.MIGRATION.TABLE:MIGRATE-TABLE
;   ;; CREATE UNIQUE INDEX "unique_user_email" ON "user" ("email") () [0 rows] | MITO.MIGRATION.TABLE:MIGRATE-TABLE
;-> (#<SXQL-STATEMENT: ALTER TABLE user ALTER COLUMN email TYPE character varying(128), ALTER COLUMN email SET NOT NULL>
;    #<SXQL-STATEMENT: CREATE UNIQUE INDEX unique_user_email ON user (email)>)

Crane advertises automatic migrations, i.e. it would run them after a C-c C-c. Unfortunately Crane has some issues, it doesn’t work with sqlite yet and the author is busy elsewhere. It didn’t work for me at first try.

Let’s hope the author comes back to work on this in a near future.

Forms

There are a few libraries, see the awesome-cl list. At least one is well active.

Debugging

On an error we are usually dropped into the interactive debugger by default.

Snooze gives options:

  • use the debugger,
  • print the stacktrace in the browser (like clack-errors below, but built-in),
  • display a custom 404.

clack-errors. Like a Flask or Django stacktrace in the browser. For Caveman, Ningle and family.

By default, when Clack throws an exception when rendering a page, the server waits for the response until it times out while the exception waits in the REPL. This isn’t very useful. So now there’s this.

It prints the stacktrace along with some request details on the browser. Can return a custom error page in production.

clack-pretend

Are you tired of jumping to your web browser every time you need to test your work in Clack? Clack-pretend will capture and replay calls to your clack middleware stack. When developing a web application with clack, you will often find it inconvenient to run your code from the lisp REPL because it expects a clack environment, including perhaps, cookies or a logged-in user. With clack-pretend, you can run prior web requests from your REPL, moving development back where it belongs.

Testing

Testing with a local DB: example of a testing macro.

We would use envy to switch configurations.

Misc

Oauth, Job queues, etc

Templating engines

HTML-based

Djula: as Django templates. Good documentation. Comes by default in Lucerne and Caveman.

We also use a dot to access attributes of dict-like variables (plists, alists, hash-tables, arrays and CLOS objects), such a feature being backed by the access library.

We wanted once to use structs and didn’t find how to it directly in Djula, so we resorted in a quick helper function to transform the struct in an alist.

Defining custom template filters is straightforward in Djula, really a breeze compared to Django.

Eco - a mix of html with lisp expressions.

Truncated example:

<body>
      <% if posts %>
        <h1>Recent Posts</h1>
        <ul id="post-list">
          <% loop for (title . snippet) in posts %>
            <li><%= title %> - <%= snippet %></li>
          <% end %>
        </ul>
        ...

Lisp-based

I prefer the semantics of Spinneret over cl-who. It also has more features (like embeddable markdown, warns on malformed html, and more).

Javascript

Parenscript

Parenscript is a translator from an extended subset of Common Lisp to JavaScript. Parenscript code can run almost identically on both the browser (as JavaScript) and server (as Common Lisp). Parenscript code is treated the same way as Common Lisp code, making the full power of Lisp macros available for JavaScript. This provides a web development environment that is unmatched in its ability to reduce code duplication and provide advanced meta-programming facilities to web developers.

https://common-lisp.net/project/parenscript/

JSCL

A Lisp-to-Javascript compiler bootstrapped from Common Lisp and executed from the browser.

https://github.com/jscl-project/jscl

https://t-cool.github.io/jscl-playground/

Ajax

Is it possible to write Ajax-based pages only in CL?

The case Webblocks - Reblocks, 2017

Weblocks is an “isomorphic” web frameworks that solves the “Javascript problem”. It allows to write the backend and an interactive client interface in Lisp, without a line of Javascript, in our usual Lisp development environment.

The framework evolves around widgets, that are updated server-side and are automatically redisplayed with transparent ajax calls on the client.

It is being massively refactored, simplified, rewritten and documented since 2017. See the new quickstart:

http://40ants.com/weblocks/quickstart.html

Writing a dynamic todo-app resolves in:

  • defining a widget class for a task:
(defwidget task ()
        ((title
          :initarg :title
          :accessor title)
         (done
          :initarg :done
          :initform nil
          :accessor done)))
  • doing the same for a list of tasks:
(defwidget task-list ()
        ((tasks
          :initarg :tasks
          :accessor tasks)))
  • saying how to render these widgets in html by extending the render method:
(defmethod render ((task task))
        "Render a task."
        (with-html
              (:span (if (done task)
                         (with-html
                               (:s (title task)))
                       (title task)))))

(defmethod render ((widget task-list))
        "Render a list of tasks."
        (with-html
              (:h1 "Tasks")
              (:ul
                (loop for task in (tasks widget) do
                      (:li (render task))))))
  • telling how to initialize the Weblocks app:
(defmethod weblocks/session:init ((app tasks))
         (declare (ignorable app))
         (let ((tasks (make-task-list "Make my first Weblocks app"
                                      "Deploy it somewhere"
                                      "Have a profit")))
           (make-instance 'task-list :tasks tasks)))
  • and then writing functions to interact with the widgets, for example adding a task:
(defmethod add-task ((task-list task-list) title)
        (push (make-task title)
              (tasks task-list))
        (update task-list))

Adding an html form and calling the new add-task function:

(defmethod render ((task-list task-list))
        (with-html
          (:h1 "Tasks")
          (loop for task in (tasks task-list) do
            (render task))
          (with-html-form (:POST (lambda (&key title &allow-other-keys)
                                         (add-task task-list title)))
            (:input :type "text"
                    :name "title"
                    :placeholder "Task's title")
            (:input :type "submit"
                    :value "Add"))))

Shipping

We can build our web app from sources, no worries, that works.

Building

We can build an executable also for web apps. That simplifies a deployment process drastically.

We can even get a Lisp REPL and interact with the running web app, in which we can even install new Quicklisp dependencies. That’s quite incredible, and it’s very useful, if not to hot-reload a web app (which I do anyways but that might be risky), at least to reload a user’s configuration file.

This is the general way:

(sb-ext:save-lisp-and-die #p"name-of-executable" :toplevel #'main :executable t)

But this is an SBCL-specific command, so we can be generic and use asdf:make, with a couple lines inside our system .asd declaration. See the Cookbook: https://lispcookbook.github.io/cl-cookbook/scripting.html#with-asdf

Now if you run your binary, your app will start up all fine, but it will quit instantly. We need to put the web server thread on the foreground:

(defun main ()
    ;; with bordeaux-threads. Also sb-ext: join-thread, thread-name, list-all-threads.
    (bt:join-thread (find-if (lambda (th)
                                (search "hunchentoot" (bt:thread-name th)))
                              (bt:all-threads))))

When I run it, Hunchentoot stays listening at the foreground:

$ ./my-webapp
Hunchentoot server is started.
Listening on localhost:9003.

I can use a tmux session (tmux, then C-b d to detach it) or better yet, start the app with Systemd, see below.

But we need something more for real apps, we need to ship foreign libraries. Deploy to the rescue. Edit your .asd file slightly to have:

:defsystem-depends-on (:deploy)  ;; (ql:quickload "deploy") before
:build-operation "deploy-op"     ;; instead of "program-op" as above
:build-pathname "my-webapp"  ;; doesn't change
:entry-point "my-webapp:main"  ;; doesn't change

Build the app again with asdf:make, and see how Deploy discovers and ships in the required foreign libraries: libssl.so, libosicat.so, libmagic.so…

It puts the final binary and the .so libraries in a bin/ directory. This is what you’ll have to ship.

I can now build my web app, send it to my VPS and see it live \o/

One more thing. We don’t want to ship libssl and libcrypto, so we ask Deploy to not ship them. We prefer to rely on the target OS.

;; .asd
;; Deploy may not find libcrypto on your system.
;; But anyways, we won't ship it to rely instead
;; on its presence on the target OS.
(require :cl+ssl)  ; sometimes necessary.
#+linux (deploy:define-library cl+ssl::libssl :dont-deploy T)
#+linux (deploy:define-library cl+ssl::libcrypto :dont-deploy T)

There’s also a hack if you have an issue with ASDF trying to update itself… see the skeleton: https://github.com/vindarel/cl-cookieweb

To be exhaustive, here’s how to catch a user’s C-c and stop our app correctly. By default, you would get a full backtrace. Yuk.

(defun main ()
  (start-app :port 9003)
  ;; with bordeaux-threads
  (handler-case (bt:join-thread (find-if (lambda (th)
                                             (search "hunchentoot" (bt:thread-name th)))
                                         (bt:all-threads)))
    (#+sbcl sb-sys:interactive-interrupt
      #+ccl  ccl:interrupt-signal-condition
      #+clisp system::simple-interrupt-condition
      #+ecl ext:interactive-interrupt
      #+allegro excl:interrupt-signal
      () (progn
           (format *error-output* "Aborting.~&")
           (clack:stop *server*)
           (uiop:quit 1)) ;; portable exit, included in ASDF, already loaded.
    ;; for others, unhandled errors (we might want to do the same).
    (error (c) (format t "Woops, an unknown error occured:~&~a~&" c)))))

To see:

Multiplatform delivery with Electron (Ceramic)

Ceramic makes all the work for us.

It is as simple as this:

;; Load Ceramic and our app
(ql:quickload '(:ceramic :our-app))

;; Ensure Ceramic is set up
(ceramic:setup)
(ceramic:interactive)

;; Start our app (here based on the Lucerne framework)
(lucerne:start our-app.views:app :port 8000)

;; Open a browser window to it
(defvar window (ceramic:make-window :url "http://localhost:8000/"))

;; start Ceramic
(ceramic:show-window window)

and we can ship this on Linux, Mac and Windows.

More:

Ceramic applications are compiled down to native code, ensuring both performance and enabling you to deliver closed-source, commercial applications.

(so no need to minify our JS)

with one more line:

(ceramic.bundler:bundle :ceramic-hello-world
                                 :bundle-pathname #p"/home/me/app.tar")
Copying resources...
Compiling app...
Compressing...
Done!
#P"/home/me/app.tar"

This last line was buggy for us.

Deployment

When you build a self-contained binary (see above, “Shipping”), deployment gets easy.

Manually

sbcl --load <my-app> --eval '(start-my-app)'

For example, a run Makefile target:

run:
	sbcl --load my-app.asd \
	     --eval '(ql:quickload :my-app)' \
	     --eval '(my-app:start-app)'  ;; given this function starts clack or hunchentoot.

this keeps sbcl in the foreground. We can use tmux to put it in background, or use Systemd.

Then, we need of a task supervisor, that will restart our app on failures, start it after a reboot, handle logging. See the section below and example projects.

with Clack

$ clackup app.lisp
Hunchentoot server is started.
Listening on localhost:5000.

with Docker

So we have various implementations ready to use: sbcl, ecl, ccl… with Quicklisp well configured.

https://lispcookbook.github.io/cl-cookbook/testing.html#gitlab-ci

On Heroku

See heroku-buildpack-common-lisp and the Awesome CL#deploy section.

Systemd: daemonizing, restarting in case of crashes, handling logs

Generally, this depends on your system. But most GNU/Linux distros now come with Systemd. Write a service file like this:

$ /etc/systemd/system/my-app.service
[Unit]
Description=stupid simple example

[Service]
WorkingDirectory=/path/to/your/app
ExecStart=/usr/bin/sbcl --load run.lisp  # your command, full path
Type=simple
Restart=always
RestartSec=10

One gotcha is that your app must be run on the foreground. See the threads snippet above in Shipping, and add it when running the app from sources too. Otherwise, you’ll see a “compilation unit aborted, caught 1 fatal error condition” error. That’s simply your Lisp quitting too early.

Run this command to start the service:

sudo systemctl start my-app.service

to check its status:

systemctl status my-app.service

Systemd handles logging for you. We write to stdout or stderr, it writes logs:

journalctl -u my-app.service

use -f -n 30 to see live updates of logs.

This tells Systemd to handle crashes and to restart the app:

Restart=always

and it can start the app after a reboot:

[Install]
WantedBy=basic.target

to enable it:

sudo systemctl enable my-app.service

Debugging SBCL error: ensure_space: failed to allocate n bytes

If you get this error with SBCL on your server:

mmap: wanted 1040384 bytes at 0x20000000, actually mapped at 0x715fa2145000
ensure_space: failed to allocate 1040384 bytes at 0x20000000
(hint: Try "ulimit -a"; maybe you should increase memory limits.)

then disable ASLR:

sudo bash -c "echo 0 > /proc/sys/kernel/randomize_va_space"

Connecting to a remote Swank server

Little example here: http://cvberry.com/tech_writings/howtos/remotely_modifying_a_running_program_using_swank.html.

It defines a simple function that prints forever:

;; a little common lisp swank demo
;; while this program is running, you can connect to it from another terminal or machine
;; and change the definition of doprint to print something else out!
;; (ql:quickload '(:swank :bordeaux-threads))

(require :swank)
(require :bordeaux-threads)

(defparameter *counter* 0)

(defun dostuff ()
  (format t "hello world ~a!~%" *counter*))

(defun runner ()
  (bt:make-thread (lambda ()
                    (swank:create-server :port 4006)))
  (format t "we are past go!~%")
  (loop while t do
       (sleep 5)
       (dostuff)
       (incf *counter*)))

(runner)

On our server, we run it with

sbcl --load demo.lisp

we do port forwarding on our development machine:

ssh -L4006:127.0.0.1:4006 username@example.com

this will securely forward port 4006 on the server at example.com to our local computer’s port 4006 (swanks accepts connections from localhost).

We connect to the running swank with M-x slime-connect, typing in port 4006.

We can write new code:

(defun dostuff ()
  (format t "goodbye world ~a!~%" *counter*))
(setf *counter* 0)

and eval it as usual with M-x slime-eval-region for instance. The output should change.

There are more pointers on CV Berry’s page.

Hot reload

When we run the app as a script we get a Lisp REPL, so we can hot-reload the running web app. Here we demonstrate a recipe to update it remotely.

Example taken from Quickutil.

It has a Makefile target:

hot_deploy:
	$(call $(LISP), \
		(ql:quickload :quickutil-server) (ql:quickload :swank-client), \
		(swank-client:with-slime-connection (conn "localhost" $(SWANK_PORT)) \
			(swank-client:slime-eval (quote (handler-bind ((error (function continue))) \
				(ql:quickload :quickutil-utilities) (ql:quickload :quickutil-server) \
				(funcall (symbol-function (intern "STOP" :quickutil-server))) \
				(funcall (symbol-function (intern "START" :quickutil-server)) $(start_args)))) conn)) \
		$($(LISP)-quit))

It has to be run on the server (a simple fabfile command can call this through ssh). Beforehand, a fab update has run git pull on the server, so new code is present but not running. It connects to the local swank server, loads the new code, stops and starts the app in a row.

Demo: fetching the GitHub REST API with Common Lisp

In this video I show you how to create a new project and we explore the Lisp ecosystem to make API calls, notably how to fire HTTP requests and parse JSON. We develop in Emacs and SLIME and in the end we get a Lisp library AND an application for the command line. Hope you enjoy the ride.