We just published a long overdue page on the Cookbook: web development in Common Lisp. We have an ambivalent feeling about it since it isn’t really a recipe as in the other pages. Yet it is valuable content that required a certain amount of digging and tryouts. Indeed, it took us about two years to discover and advertise many projects, to learn, try and put a tutorial together. We also wrote a commercial application. During that time, we were taking notes on our web-dev/ page.

We present Hunchentoot, Clack (briefly), we have an overview of other web frameworks, of templating libraries, we introduce Weblocks, we give recipes for common tasks (such as checking if a user is logged in, encrypting passwords), and we speak about deployment.

Some topics still need to be adressed, so check for updates on the Cookbook !

Prior notice:

Some people sell ten pages long ebooks or publish their tutorial on Gitbook to have a purchase option. I prefer to enhance the collaborative Cookbook (I am by far the main contributor). You can tip me on liberapay if you like: https://liberapay.com/vindarel/. Thanks !

For web development as for any other task, one can leverage Common Lisp’s advantages: the unmatched REPL and exception handling system, performance, the ability to build a self-contained executable, stability, good threads story, strong typing, etc. We can, say, define a new route and try it right away, there is no need to restart any running server. We can change and compile one function at a time (the usual C-c C-c in Slime) and try it. The feedback is immediate. We can choose the degree of interactivity: the web server can catch exceptions and fire the interactive debugger, or print lisp backtraces on the browser, or display a 404 error page and print logs on standard output. The ability to build self-contained executables eases deployment tremendously (compared to, for example, npm-based apps), in that we just copy the executable to a server and run it.

We’ll present here some established web frameworks and other common libraries to help you getting started in developing a web application. We do not aim to be exhaustive nor to replace the upstream documentation. Your feedback and contributions are appreciated.

Table of Contents

Overview

Hunchentoot and Clack are two projects that you’ll often hear about.

Hunchentoot is

a web server and at the same time a toolkit for building dynamic websites. As a stand-alone web server, Hunchentoot is capable of HTTP/1.1 chunking (both directions), persistent connections (keep-alive), and SSL. It provides facilities like automatic session handling (with and without cookies), logging, customizable error handling, and easy access to GET and POST parameters sent by the client.

It is a software written by Edi Weitz (“Common Lisp Recipes”, cl-ppcre and much more), it’s used and proven solid. One can achieve a lot with it, but sometimes with more friction than with a traditional web framework. For example, dispatching a route by the HTTP method is a bit convoluted, one must write a function for the :uri parameter that does the check, when it is a built-in keyword in other frameworks like Caveman.

Clack is

a web application environment for Common Lisp inspired by Python’s WSGI and Ruby’s Rack.

Also written by a prolific lisper (E. Fukamachi), it actually uses Hunchentoot by default as the server, but thanks to its pluggable architecture one can use another web server, like the asynchronous Woo, built on the libev event loop, maybe “the fastest web server written in any programming language”.

We’ll cite also Wookie, an asynchronous HTTP server, and its companion library cl-async, for general purpose, non-blocking programming in Common Lisp, built on libuv, the backend library in Node.js.

Clack being more recent and less documented, and Hunchentoot a de-facto standard, we’ll concentrate on the latter for this recipe. Your contributions are of course welcome.

Web frameworks build upon web servers and can provide facilities for common activities in web development, like a templating system, access to a database, session management, or facilities to build a REST api.

Some web frameworks include:

  • Caveman, by E. Fukamachi. It provides, out of the box, database management, a templating engine (Djula), a project skeleton generator, a routing system à la Flask or Sinatra, deployment options (mod_lisp or FastCGI), support for Roswell on the command line, etc.
  • Radiance, by Shinmera (Qtools, Portacle, lquery, …), is a web application environment, more general than usual web frameworks. It lets us write and tie websites and applications together, easing their deployment as a whole. It has thorough documentation, a tutorial, modules, pre-written applications such as an image board or a blogging platform, and more. For example websites, see https://shinmera.com/, reader.tymoon.eu and events.tymoon.eu.
  • Snooze, by João Távora (Sly, Emacs’ Yasnippet, Eglot, …), is “an URL router designed around REST web services”. It is different because in Snooze, routes are just functions and HTTP conditions are just Lisp conditions.
  • cl-rest-server is a library for writing REST web APIs. It features validation with schemas, annotations for logging, caching, permissions or authentication, documentation via OpenAPI (Swagger), etc.
  • last but not least, Weblocks is a venerable Common Lisp web framework that permits to write ajax-based dynamic web applications without writing any JavaScript, nor writing some lisp that would transpile to JavaScript. It is seeing an extensive rewrite and update since 2017. We present it in more details below.

For a full list of libraries for the web, please see the awesome-cl list #network-and-internet and Cliki. If you are looking for a featureful static site generator, see Coleslaw.

Installation

Let’s install the libraries we’ll use:

(ql:quickload '("hunchentoot" "caveman" "spinneret" "djula"))

To try Weblocks, please see its documentation. The Weblocks in Quicklisp is not yet, as of writing, the one we are interested in.

We’ll start by serving local files and we’ll run more than one local server in the running image.

Simple webserver

Serve local files

Hunchentoot

Create and start a webserver like this:

(defvar *acceptor* (make-instance 'hunchentoot:easy-acceptor :port 4242))
(hunchentoot:start *acceptor*)

We create an instance of easy-acceptor on port 4242 and we start it. We can now access http://127.0.0.1:4242/. You should get a welcome screen with a link to the documentation and logs to the console.

By default, Hunchentoot serves the files from the www/ directory in its source tree. Thus, if you go to the source of easy-acceptor (M-. in Slime), which is probably ~/quicklisp/dists/quicklisp/software/hunchentoot-v1.2.38/, you’ll find the root/ directory. It contains:

  • an errors/ directory, with the error templates 404.html and 500.html,
  • an img/ directory,
  • an index.html file.

To serve another directory, we give the option document-root to easy-acceptor. We can also set the slot with its accessor:

(setf (hunchentoot:acceptor-document-root *acceptor*) #p"path/to/www")

Let’s create our index.html first. Put this in a new www/index.html at the current directory (of the lisp repl):

<html>
  <head>
    <title>Hello!</title>
  </head>
  <body>
    <h1>Hello local server!</h1>
    <p>
    We just served our own files.
    </p>
  </body>
</html>

Let’s start a new acceptor on a new port:

(defvar *my-acceptor* (make-instance 'hunchentoot:easy-acceptor :port 4444
                                   :document-root #p"www/"))
(hunchentoot:start *my-acceptor*)

go to p://127.0.0.1:4444/ and see the difference.

Note that we just created another acceptor on a different port on the same lisp image. This is already pretty cool.

Access your server from the internet

Hunchentoot

With Hunchentoot we have nothing to do, we can see the server from the internet right away.

If you evaluate this on your VPS:

(hunchentoot:start (make-instance 'hunchentoot:easy-acceptor :port 4242))

You can see it right away on your server’s IP.

Stop it with (hunchentoot:stop *).

Routing

Simple routes

Hunchentoot

To bind an existing function to a route, we create a “prefix dispatch” that we push onto the *dispatch-table* list:

(defun hello ()
   (format nil "Hello, it works!"))

(push
  (hunchentoot:create-prefix-dispatcher "/hello.html" #'hello)
  hunchentoot:*dispatch-table*)

To create a route with a regexp, we use create-regex-dispatcher, where the url-as-regexp can be a string, an s-expression or a cl-ppcre scanner.

If you didn’t yet, create an acceptor and start the server:

(defvar *server* (make-instance 'hunchentoot:easy-acceptor :port 4242))
(hunchentoot:start *server*)

and access it on [http://localhost:4242/hello.html]http://localhost:4242/hello.html).

We can see logs on the REPL:

127.0.0.1 - [2018-10-27 23:50:09] "get / http/1.1" 200 393 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0"
127.0.0.1 - [2018-10-27 23:50:10] "get /img/made-with-lisp-logo.jpg http/1.1" 200 12583 "http://localhost:4242/" "Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0"
127.0.0.1 - [2018-10-27 23:50:10] "get /favicon.ico http/1.1" 200 1406 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0"
127.0.0.1 - [2018-10-27 23:50:19] "get /hello.html http/1.1" 200 20 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0"

define-easy-handler allows to create a function and to bind it to an uri at once.

Its form follows

define-easy-handler (function-name :uri <uri> …) (lambda list parameters)

where <uri> can be a string or a function.

Example:

(hunchentoot:define-easy-handler (say-yo :uri "/yo") (name)
  (setf (hunchentoot:content-type*) "text/plain")
  (format nil "Hey~@[ ~A~]!" name))

Visit it at p://localhost:4242/yo and add parameters on the url: http://localhost:4242/yo?name=Alice.

Just a thought… we didn’t explicitely ask Hunchentoot to add this route to our first acceptor of the port 4242. Let’s try another acceptor (see previous section), on port 4444: http://localhost:4444/yo?name=Bob It works too ! In fact, define-easy-handler accepts an acceptor-names parameter:

acceptor-names (which is evaluated) can be a list of symbols which means that the handler will only be returned by DISPATCH-EASY-HANDLERS in acceptors which have one of these names (see ACCEPTOR-NAME). acceptor-names can also be the symbol T which means that the handler will be returned by DISPATCH-EASY-HANDLERS in every acceptor.

So, define-easy-handler has the following signature:

define-easy-handler (function-name &key uri acceptor-names default-request-type) (lambda list parameters)

It also has a default-parameter-type which we’ll use in a minute to get url parameters.

There are also keys to know for the lambda list. Please see the documentation.

Caveman

Caveman provides two ways to define a route: the defroute macro and the @route pythonic annotation:

(defroute "/welcome" (&key (|name| "Guest"))
  (format nil "Welcome, ~A" |name|))

@route GET "/welcome"
(lambda (&key (|name| "Guest"))
  (format nil "Welcome, ~A" |name|))

A route with an url parameter (note :name in the url):

(defroute "/hello/:name" (&key name)
  (format nil "Hello, ~A" name))

It is also possible to define “wildcards” parameters. It works with the splat key:

(defroute "/say/*/to/*" (&key splat)
  ; matches /say/hello/to/world
  (format nil "~A" splat))
;=> (hello world)

We must enable regexps with :regexp t:

(defroute ("/hello/([\\w]+)" :regexp t) (&key captures)
  (format nil "Hello, ~A!" (first captures)))

Accessing GET and POST parameters

Hunchentoot

First of all, note that we can access query parameters anytime with

(hunchentoot:parameter "my-param")

It acts on the default *request* object which is passed to all handlers.

There is also get-paramater and post-parameter.

Earlier we saw some key parameters to define-easy-handler. We now introduce default-parameter-type.

We defined the following handler:

(hunchentoot:define-easy-handler (say-yo :uri "/yo") (name)
  (setf (hunchentoot:content-type*) "text/plain")
  (format nil "Hey~@[ ~A~]!" name))

The variable name is a string by default. Let’s check it out:

(hunchentoot:define-easy-handler (say-yo :uri "/yo") (name)
  (setf (hunchentoot:content-type*) "text/plain")
  (format nil "Hey~@[ ~A~] you are of type ~a" name (type-of name)))

Going to http://localhost:4242/yo?name=Alice returns

Hey Alice you are of type (SIMPLE-ARRAY CHARACTER (5))

To automatically bind it to another type, we use default-parameter-type. It can be one of those simple types:

  • 'string (default),
  • 'integer,
  • 'character (accepting strings of length 1 only, otherwise it is nil)
  • or 'boolean

or a compound list:

  • '(:list <type>)
  • '(:array <type>)
  • '(:hash-table <type>)

where <type> is a simple type.

Error handling

In all frameworks, we can choose the level of interactivity. The web framework can return a 404 page and print output on the repl, it can catch errors and invoke the interactive lisp debugger, or it can show the lisp backtrace on the html page.

Hunchentoot

The global variables to set are *catch-errors-p*, *show-lisp-errors-p* and *show-lisp-backtraces-p*.

Hunchentoot also defines condition classes.

See the documentation: https://edicl.github.io/hunchentoot/#conditions.

Clack

Clack users might make a good use of plugins, like the clack-errors middleware: https://github.com/CodyReichert/awesome-cl#clack-plugins.

Weblocks - solving the “JavaScript problem”©

Weblocks is a widgets-based and server-based framework with a built-in ajax update mechanism. It allows to write dynamic web applications without the need to write JavaScript or to write lisp code that would transpile to JavaScript.

Weblocks is an old framework developed by Slava Akhmechet, Stephen Compall and Leslie Polzer. After nine calm years, it is seeing a very active update, refactoring and rewrite effort by Alexander Artemenko.

It was initially based on continuations (they were removed to date) and thus a lispy cousin of Smalltalk’s Seaside. We can also relate it to Haskell’s Haste, OCaml’s Eliom, Elixir’s Phoenix LiveView and others.

The Ultralisp website is an example Weblocks website in production known in the CL community.

Weblock’s unit of work is the widget. They look like a class definition:

(defwidget task ()
        ((title
          :initarg :title
          :accessor title)
         (done
          :initarg :done
          :initform nil
          :accessor done)))

Then all we have to do is to define the render method for this widget:

(defmethod render ((task task))
        "Render a task."
        (with-html
              (:span (if (done task)
                         (with-html
                               (:s (title task)))
                       (title task)))))

It uses the Spinneret template engine by default, but we can bind any other one of our choice.

To trigger an ajax event, we write lambdas in full Common Lisp:

...
(with-html
          (:p (:input :type "checkbox"
            :checked (done task)
            :onclick (make-js-action
                      (lambda (&key &allow-other-keys)
                        (toggle task))))
...

The function make-js-action creates a simple javascript function that calls the lisp one on the server, and automatically refreshes the HTML of the widgets that need it. In our example, it re-renders one task only.

Is it appealing ? Carry on this quickstart guide here: http://40ants.com/weblocks/quickstart.html.

Templates

Djula - HTML markup

Djula is a port of Python’s Django template engine to Common Lisp. It has excellent documentation.

Caveman uses it by default, but otherwise it is not difficult to setup. We must declare where our templates are with something like

(djula:add-template-directory (asdf:system-relative-pathname "webapp" "templates/"))

A Djula template looks like this, no surprises (forgive the antislash in \%, this is a Jekyll limitation):

{\% extends "base.html" \%}
{\% block title %}Memberlist{\% endblock \%}
{\% block content \%}
  <ul>
  {\% for user in users \%}
    <li><a href="{{ user.url }}">{{ user.username }}</a></li>
  {\% endfor \%}
  </ul>
{\% endblock \%}

Djula compiles the templates before rendering them.

It is, along with its companion access library, one of the most downloaded libraries of Quicklisp.

Spinneret - lispy templates

Spinneret is a “lispy” HTML5 generator. It looks like this:

(with-page (:title "Home page")
     (:header
      (:h1 "Home page"))
     (:section
      ("~A, here is *your* shopping list: " *user-name*)
      (:ol (dolist (item *shopping-list*)
             (:li (1+ (random 10)) item))))
     (:footer ("Last login: ~A" *last-login*)))

The author finds it is easier to compose the HTML in separate functions and macros than with the more famous cl-who. But it has more features under it sleeves:

  • it warns on invalid tags and attributes
  • it can automatically number headers, given their depth
  • it pretty prints html per default, with control over line breaks
  • it understands embedded markdown
  • it can tell where in the document a generator function is (see get-html-tag)

Connecting to a database

Please see the databases section. The Mito ORM supports SQLite3, PostgreSQL, MySQL, it has migrations and db schema versioning, etc.

In Caveman, a database connection is alive during the Lisp session and is reused in each HTTP requests.

Checking a user is logged-in

A framework will provide a way to work with sessions. We’ll create a little macro to wrap our routes to check if the user is logged in.

In Caveman, *session* is a hash table that represents the session’s data. Here are our login and logout functions:

(defun login (user)
  "Log the user into the session"
  (setf (gethash :user *session*) user))

(defun logout ()
  "Log the user out of the session."
  (setf (gethash :user *session*) nil))

We define a simple predicate:

(defun logged-in-p ()
  (gethash :user cm:*session*))

and we define our with-logged-in macro:

(defmacro with-logged-in (&body body)
  `(if (logged-in-p)
       (progn ,@body)
       (render #p"login.html"
               '(:message "Please log-in to access this page."))))

If the user isn’t logged in, there will nothing in the session store, and we render the login page. When all is well, we execute the macro’s body. We use it like this:

(defroute "/account/logout" ()
  "Show the log-out page, only if the user is logged in."
  (with-logged-in
    (logout)
    (render #p"logout.html")))

(defroute ("/account/review" :method :get) ()
  (with-logged-in
    (render #p"review.html"
            (list :review (get-review (gethash :user *session*))))))

and so on.

Encrypting passwords

In this recipe we use the de-facto standard Ironclad cryptographic toolkit and the Babel charset encoding/decoding library.

This snippet creates the password hash that should be stored in your database. Note that Ironclad expects a byte-vector, not a string.

(defun password-hash (password)
  (ironclad:pbkdf2-hash-password-to-combined-string
   (babel:string-to-octets password)))

pbkdf2 is defined in RFC2898. It uses a pseudorandom function to derive a secure encryption key based on the password.

The following function checks if a user is active and verifies the entered password. It returns the user-id if active and verified and nil in all other cases even if an error occurs. Adapt it to your application.

(defun check-user-password (user password)
  (handler-case
      (let* ((data (my-get-user-data user))
             (hash (my-get-user-hash data))
             (active (my-get-user-active data)))
        (when (and active (ironclad:pbkdf2-check-password (babel:string-to-octets password)
                                                          hash))
          (my-get-user-id data)))
    (condition () nil)))

And the following is an example on how to set the password on the database. Note that we use (password-hash password) to save the password. The rest is specific to the web framework and to the DB library.

(defun set-password (user password)
  (with-connection (db)
    (execute
     (make-statement :update :web_user
                     (set= :hash (password-hash password))
                     (make-clause :where
                                  (make-op := (if (integerp user)
                                                  :id_user
                                                  :email)
                                           user))))))

Credit: /u/arvid on /r/learnlisp.

Building

Building a self-contained executable

As for all Common Lisp applications, we can bundle our web app in one single executable, including the assets. It makes deployment very easy: copy it to your server and run it.

$ ./my-web-app
Hunchentoot server is started.
Listening on localhost:9003.

See this recipe on scripting#for-web-apps.

Continuous delivery with Travis CI or Gitlab CI

Please see the section on testing#continuous-integration.

Multiplatform delivery with Electron

Ceramic makes all the work for us.

It is as simple as this:

;; Load Ceramic and our app
(ql:quickload '(:ceramic :our-app))

;; Ensure Ceramic is set up
(ceramic:setup)
(ceramic:interactive)

;; Start our app (here based on the Lucerne framework)
(lucerne:start our-app.views:app :port 8000)

;; Open a browser window to it
(defvar window (ceramic:make-window :url "http://localhost:8000/"))

;; start Ceramic
(ceramic:show-window window)

and we can ship this on Linux, Mac and Windows.

There is more:

Ceramic applications are compiled down to native code, ensuring both performance and enabling you to deliver closed-source, commercial applications.

Thus, no need to minify our JS.

Deployment

Deploying manually

We can start our executable in a shell and send it to the bakcground (C-z bg), or run it inside a tmux session. These are not the best but hey, it works©.

Daemonizing, restarting in case of crashes, handling logs with Systemd

This is actually a system-specific task. See how to do that on your system.

Most GNU/Linux distros now come with Systemd, so here’s a little example.

Deploying an app with Systemd is as simple as writing a configuration file:

$ emacs -nw /etc/systemd/system/my-app.service
[Unit]
Description=stupid simple example

[Service]
WorkingDirectory=/path/to/your/app
ExecStart=/usr/local/bin/sthg sthg
Type=simple
Restart=always
RestartSec=10

Then we have a command to start it:

sudo systemctl start my-app.service

a command to check its status:

systemctl status my-app.service

and Systemd can handle logging (we write to stdout or stderr, it writes logs):

journalctl -f -u my-app.service

and it handles crashes and restarts the app:

Restart=always

and it can start the app after a reboot:

[Install]
WantedBy=basic.target

to enable it:

sudo systemctl enable my-app.service

With Docker

There are several Docker images for Common Lisp. For example:

  • 40ants/base-lisp-image is based on Ubuntu LTS and includes SBCL, CCL, Quicklisp, Qlot and Roswell.
  • container-lisp/s2i-lisp is CentOs based and contains the source for building a Quicklisp based Common Lisp application as a reproducible docker image using OpenShift’s source-to-image.

With Guix

GNU Guix is a transactional package manager, that can be installed on top of an existing OS, and a whole distro that supports declarative system configuration. It allows to ship self-contained tarballs, which also contain system dependencies. For an example, see the Next browser.

Deploying on Heroku and other services

See heroku-buildpack-common-lisp and the Awesome CL#deploy section for interface libraries for Kubernetes, OpenShift, AWS, etc.

Monitoring

See Prometheus.cl for a Grafana dashboard for SBCL and Hunchentoot metrics (memory, threads, requests per second,…).

Connecting to a remote Lisp image

This this section: debugging#remote-debugging.

Hot reload

This is an example from Quickutil. It is actually an automated version of the precedent section.

It has a Makefile target:

hot_deploy:
	$(call $(LISP), \
		(ql:quickload :quickutil-server) (ql:quickload :swank-client), \
		(swank-client:with-slime-connection (conn "localhost" $(SWANK_PORT)) \
			(swank-client:slime-eval (quote (handler-bind ((error (function continue))) \
				(ql:quickload :quickutil-utilities) (ql:quickload :quickutil-server) \
				(funcall (symbol-function (intern "STOP" :quickutil-server))) \
				(funcall (symbol-function (intern "START" :quickutil-server)) $(start_args)))) conn)) \
		$($(LISP)-quit))

It has to be run on the server (a simple fabfile command can call this through ssh). Beforehand, a fab update has run git pull on the server, so new code is present but not running. It connects to the local swank server, loads the new code, stops and starts the app in a row.

Credits