Caution [from 2017: what a road since then]: this is a draft. I take notes and write more in other resources (the Cookbook, my blog).
update, June 2022: see my web project skeleton, it illustrates and fixes common issues: https://github.com/vindarel/cl-cookieweb
update july, 5th 2019: I put this content into the Cookbook: https://lispcookbook.github.io/cl-cookbook/web.html, fixing a long-standing request.
new post: why and how to live-reload one’s running web application: https://lisp-journey.gitlab.io/blog/i-realized-that-to-live-reload-my-web-app-is-easy-and-convenient/
new project skeleton: lisp-web-template-productlist: Hunchentoot + easy-routes + Djula templates + Bulma CSS + a Makefile to build the project
See also the Awesome CL list.
Information is at the moment scarce and spread appart, Lisp web frameworks and libraries evolve and take different approaches.
I’d like to know what’s possible, what’s lacking, see how to quickstart everything, see code snippets and, most of all, see how to do things that I couldn’t do before such as hot reloading, building self-contained executables, shipping a multiplatform web app.
Some people sell ten pages long ebooks or publish their tutorial on Gitbook to have a purchase option. I prefer to enhance the collaborative Cookbook (I am by far the main contributor). You can tip me on Kofi if you like: https://ko-fi.com/vindarel Thanks !
Table of Contents
- Web application environments
- Web frameworks
- Templating engines
Web application environments
Clack is to Lisp what WSGI is to Python. However it is mostly undocumented and not as battle-proofed as Hunchentoot.
The de-facto web server, with the best documentation (cough looking old cough), the most websites on production. Lower level than a web framework (defining routes seems weird at first). I think worth knowing.
Its terminology is different from what we are used to (“routes” are not called routes but we create handlers), part I don’t know why and part because the Lisp image-based development allows for more, and thus needs more terminology. For example, we can run two applications on different URLs on the same image.
edit: here’s a modern looking page: https://digikar99.github.io/common-lisp.readthedocs/hunchentoot/
A popular web framework, or so it seems by the github stars, written by a super-productive lisper, with nice documentation for basic stuff but lacking for the rest, based on Clack (webserver interface, think Python’s WSGI), uses Hunchentoot by default.
I feel like basic functions are too cumbersome (accessing url parameters).
By the maintainer of Sly, Emacs’ Yasnippet,…
Defining routes is like defining functions. Built-in features that are available as extensions in Clack-based frameworks (setting to get a stacktrace on the browser, to fire up the debugger or to return a 404,…). Definitely worth exploring.
Radiance, with extensive tutorial and existing apps.
It doesn’t look like a web framework to me. It has ready-to-use components:
- admin page (but what does it do?)
- auth system
- user: provide user accounts and permissions
- image hosting
- there is an email marketing system in development…
a library for writing REST Web APIs in Common Lisp.
Features: validation via schemas, Swagger support, authentication, logging, caching, permission checking…
It seems complete, it is maintained, the author seems to be doing web development in CL for a living. Note to self: I want to interview him.
An asynchronous web server, by an impressive lisper, who built many async libraries. Used for the Turtl api backend. Dealing with async brings its own set of problems (how will be debugging ?).
Nice api to build routes, good documentation: http://wookie.lyonbros.com/
It isn’t the easy path to web development in CL but there’s great potential IMO.
It doesn’t do double data binding as in modern JS frameworks. But new projects are being developed…
See our presentation below.
Accessing url parameters
It is easy and well explained with Hunchentoot or
easy-routes in the Cookbook.
Lucerne has a nice
with-params macro that makes accessing post or url query parameters a breeze:
@route app (:post "/tweet") (defview tweet () (if (lucerne-auth:logged-in-p) (let ((user (current-user))) (with-params (tweet) (utweet.models:tweet user tweet)) (redirect "/")) (render-template (+index+) :error "You are not logged in.")))
Snooze’s way is simple and lispy: we define routes like methods and parameters as keys:
(defroute lispdoc (:get :text/* name &key (package :cl) (doctype 'function)) ...
On the contrary, I find Caveman’s and Ningle’s ways cumbersome.
(setf (ningle:route *app* "/hello/:name") #'(lambda (params) (format nil "Hello, ~A" (cdr (assoc "name" params :test #'string=)))))
The above controller will be invoked when you access to “/hello/Eitaro” or “/hello/Tomohiro”, and then (cdr (assoc “name” params :test #‘string=)) will be “Eitaro” and “Tomohiro”.
and it doesn’t say about query parameters. I had to ask:
(assoc "the-query-param" (clack.request:query-parameter lucerne:*request*) :test 'string=)
Parameter keys contain square brackets (”[” & “]”) will be parsed as structured parameters. You can access the parsed parameters as _parsed in routers.
(defroute "/edit" (&key _parsed) (format nil "~S" (cdr (assoc "person" _parsed :test #'string=)))) ;=> "((\"name\" . \"Eitaro\") (\"email\" . \"email@example.com\") (\"birth\" . ((\"year\" . 2000) (\"month\" . 1) (\"day\" . 1))))"
Session an cookies
Mito works for MySQL, Postgres and SQLite3 on SBCL and CCL.
We can define models with a regular class which has a
(defclass user () ((name :col-type (:varchar 64) :initarg :name :accessor user-name) (email :col-type (:varchar 128) :initarg :email :accessor user-email)) (:metaclass mito:dao-table-class) (:unique-keys email))
We create the table with
(ensure-table-exists 'user) ;-> ;; CREATE TABLE IF NOT EXISTS "user" ( ; "id" BIGSERIAL NOT NULL PRIMARY KEY, ; "name" VARCHAR(64) NOT NULL, ; "email" VARCHAR(128), ; "created_at" TIMESTAMP, ; "updated_at" TIMESTAMP ; ) () [0 rows] | MITO.DAO:ENSURE-TABLE-EXISTS
Mito has migrations support and DB schema versioning for MySQL, Postgres and SQLite3, on SBCL and CCL. Once we have changed our model definition, we have commands to see the generated SQL and to apply the migration.
We inspect the SQL: (suppose we just added the email field into the
user class above)
(mito:migration-expressions 'user) ;=> (#<SXQL-STATEMENT: ALTER TABLE user ALTER COLUMN email TYPE character varying(128), ALTER COLUMN email SET NOT NULL> ; #<SXQL-STATEMENT: CREATE UNIQUE INDEX unique_user_email ON user (email)>)
and we can apply the migration:
(mito:migrate-table 'user) ;-> ;; ALTER TABLE "user" ALTER COLUMN "email" TYPE character varying(128), ALTER COLUMN "email" SET NOT NULL () [0 rows] | MITO.MIGRATION.TABLE:MIGRATE-TABLE ; ;; CREATE UNIQUE INDEX "unique_user_email" ON "user" ("email") () [0 rows] | MITO.MIGRATION.TABLE:MIGRATE-TABLE ;-> (#<SXQL-STATEMENT: ALTER TABLE user ALTER COLUMN email TYPE character varying(128), ALTER COLUMN email SET NOT NULL> ; #<SXQL-STATEMENT: CREATE UNIQUE INDEX unique_user_email ON user (email)>)
Crane advertises automatic
migrations, i.e. it would run them after a
C-c C-c. Unfortunately Crane has
some issues, it doesn’t work with sqlite yet and the author is busy
elsewhere. It didn’t work for me at first try.
Let’s hope the author comes back to work on this in a near future.
There are a few libraries, see the awesome-cl list. At least one is well active.
On an error we are usually dropped into the interactive debugger by default.
Snooze gives options:
- use the debugger,
- print the stacktrace in the browser (like clack-errors below, but built-in),
- display a custom 404.
clack-errors. Like a Flask or Django stacktrace in the browser. For Caveman, Ningle and family.
By default, when Clack throws an exception when rendering a page, the server waits for the response until it times out while the exception waits in the REPL. This isn’t very useful. So now there’s this.
It prints the stacktrace along with some request details on the browser. Can return a custom error page in production.
Are you tired of jumping to your web browser every time you need to test your work in Clack? Clack-pretend will capture and replay calls to your clack middleware stack. When developing a web application with clack, you will often find it inconvenient to run your code from the lisp REPL because it expects a clack environment, including perhaps, cookies or a logged-in user. With clack-pretend, you can run prior web requests from your REPL, moving development back where it belongs.
Testing with a local DB: example of a testing macro.
We would use envy to switch configurations.
Oauth, Job queues, etc
Djula: as Django templates. Good documentation. Comes by default in Lucerne and Caveman.
We also use a dot to access attributes of dict-like variables (plists, alists, hash-tables, arrays and CLOS objects), such a feature being backed by the access library.
We wanted once to use structs and didn’t find how to it directly in Djula, so we resorted in a quick helper function to transform the struct in an alist.
Defining custom template filters is straightforward in Djula, really a breeze compared to Django.
Eco - a mix of html with lisp expressions.
<body> <% if posts %> <h1>Recent Posts</h1> <ul id="post-list"> <% loop for (title . snippet) in posts %> <li><%= title %> - <%= snippet %></li> <% end %> </ul> ...
I prefer the semantics of Spinneret over cl-who. It also has more features (like embeddable markdown, warns on malformed html, and more).
Is it possible to write Ajax-based pages only in CL?
The case Webblocks - Reblocks, 2017
The framework evolves around widgets, that are updated server-side and are automatically redisplayed with transparent ajax calls on the client.
It is being massively refactored, simplified, rewritten and documented since 2017. See the new quickstart:
Writing a dynamic todo-app resolves in:
- defining a widget class for a task:
(defwidget task () ((title :initarg :title :accessor title) (done :initarg :done :initform nil :accessor done)))
- doing the same for a list of tasks:
(defwidget task-list () ((tasks :initarg :tasks :accessor tasks)))
- saying how to render these widgets in html by extending the
(defmethod render ((task task)) "Render a task." (with-html (:span (if (done task) (with-html (:s (title task))) (title task))))) (defmethod render ((widget task-list)) "Render a list of tasks." (with-html (:h1 "Tasks") (:ul (loop for task in (tasks widget) do (:li (render task))))))
- telling how to initialize the Weblocks app:
(defmethod weblocks/session:init ((app tasks)) (declare (ignorable app)) (let ((tasks (make-task-list "Make my first Weblocks app" "Deploy it somewhere" "Have a profit"))) (make-instance 'task-list :tasks tasks)))
- and then writing functions to interact with the widgets, for example adding a task:
(defmethod add-task ((task-list task-list) title) (push (make-task title) (tasks task-list)) (update task-list))
Adding an html form and calling the new
(defmethod render ((task-list task-list)) (with-html (:h1 "Tasks") (loop for task in (tasks task-list) do (render task)) (with-html-form (:POST (lambda (&key title &allow-other-keys) (add-task task-list title))) (:input :type "text" :name "title" :placeholder "Task's title") (:input :type "submit" :value "Add"))))
We can build our web app from sources, no worries, that works.
We can build an executable also for web apps. That simplifies a deployment process drastically.
We can even get a Lisp REPL and interact with the running web app, in which we can even install new Quicklisp dependencies. That’s quite incredible, and it’s very useful, if not to hot-reload a web app (which I do anyways but that might be risky), at least to reload a user’s configuration file.
This is the general way:
(sb-ext:save-lisp-and-die #p"name-of-executable" :toplevel #'main :executable t)
But this is an SBCL-specific command, so we can be generic and use
asdf:make, with a couple lines inside our system .asd
declaration. See the Cookbook:
Now if you run your binary, your app will start up all fine, but it will quit instantly. We need to put the web server thread on the foreground:
(defun main () ;; with bordeaux-threads. Also sb-ext: join-thread, thread-name, list-all-threads. (bt:join-thread (find-if (lambda (th) (search "hunchentoot" (bt:thread-name th))) (bt:all-threads))))
When I run it, Hunchentoot stays listening at the foreground:
$ ./my-webapp Hunchentoot server is started. Listening on localhost:9003.
I can use a
tmux session (
C-b d to detach it) or better yet, start the app with Systemd, see below.
But we need something more for real apps, we need to ship foreign libraries. Deploy to the rescue. Edit your .asd file slightly to have:
:defsystem-depends-on (:deploy) ;; (ql:quickload "deploy") before :build-operation "deploy-op" ;; instead of "program-op" as above :build-pathname "my-webapp" ;; doesn't change :entry-point "my-webapp:main" ;; doesn't change
Build the app again with
asdf:make, and see how Deploy discovers and
ships in the required foreign libraries: libssl.so, libosicat.so, libmagic.so…
It puts the final binary and the .so libraries in a
directory. This is what you’ll have to ship.
I can now build my web app, send it to my VPS and see it live \o/
One more thing. We don’t want to ship
libcrypto, so we
ask Deploy to not ship them. We prefer to rely on the target OS.
;; .asd ;; Deploy may not find libcrypto on your system. ;; But anyways, we won't ship it to rely instead ;; on its presence on the target OS. (require :cl+ssl) ; sometimes necessary. #+linux (deploy:define-library cl+ssl::libssl :dont-deploy T) #+linux (deploy:define-library cl+ssl::libcrypto :dont-deploy T)
There’s also a hack if you have an issue with ASDF trying to update itself… see the skeleton: https://github.com/vindarel/cl-cookieweb
To be exhaustive, here’s how to catch a user’s
C-c and stop our app
correctly. By default, you would get a full backtrace. Yuk.
(defun main () (start-app :port 9003) ;; with bordeaux-threads (handler-case (bt:join-thread (find-if (lambda (th) (search "hunchentoot" (bt:thread-name th))) (bt:all-threads))) (#+sbcl sb-sys:interactive-interrupt #+ccl ccl:interrupt-signal-condition #+clisp system::simple-interrupt-condition #+ecl ext:interactive-interrupt #+allegro excl:interrupt-signal () (progn (format *error-output* "Aborting.~&") (clack:stop *server*) (uiop:quit 1)) ;; portable exit, included in ASDF, already loaded. ;; for others, unhandled errors (we might want to do the same). (error (c) (format t "Woops, an unknown error occured:~&~a~&" c)))))
- a Debian package for every Quicklisp system: http://margaine.com/2015/12/22/quicklisp-packagecloud-debian-packages.html.
Multiplatform delivery with Electron (Ceramic)
Ceramic makes all the work for us.
It is as simple as this:
;; Load Ceramic and our app (ql:quickload '(:ceramic :our-app)) ;; Ensure Ceramic is set up (ceramic:setup) (ceramic:interactive) ;; Start our app (here based on the Lucerne framework) (lucerne:start our-app.views:app :port 8000) ;; Open a browser window to it (defvar window (ceramic:make-window :url "http://localhost:8000/")) ;; start Ceramic (ceramic:show-window window)
and we can ship this on Linux, Mac and Windows.
Ceramic applications are compiled down to native code, ensuring both performance and enabling you to deliver closed-source, commercial applications.
(so no need to minify our JS)
with one more line:
(ceramic.bundler:bundle :ceramic-hello-world :bundle-pathname #p"/home/me/app.tar") Copying resources... Compiling app... Compressing... Done! #P"/home/me/app.tar"
This last line was buggy for us.
When you build a self-contained binary (see above, “Shipping”), deployment gets easy.
sbcl --load <my-app> --eval '(start-my-app)'
For example, a
run Makefile target:
run: sbcl --load my-app.asd \ --eval '(ql:quickload :my-app)' \ --eval '(my-app:start-app)' ;; given this function starts clack or hunchentoot.
this keeps sbcl in the foreground. We can use
tmux to put it in background, or use Systemd.
Then, we need of a task supervisor, that will restart our app on failures, start it after a reboot, handle logging. See the section below and example projects.
$ clackup app.lisp Hunchentoot server is started. Listening on localhost:5000.
So we have various implementations ready to use: sbcl, ecl, ccl… with Quicklisp well configured.
See heroku-buildpack-common-lisp and the Awesome CL#deploy section.
Systemd: daemonizing, restarting in case of crashes, handling logs
Generally, this depends on your system. But most GNU/Linux distros now come with Systemd. Write a service file like this:
$ /etc/systemd/system/my-app.service [Unit] Description=stupid simple example [Service] WorkingDirectory=/path/to/your/app ExecStart=/usr/bin/sbcl --load run.lisp # your command, full path Type=simple Restart=always RestartSec=10
One gotcha is that your app must be run on the foreground. See the threads snippet above in Shipping, and add it when running the app from sources too. Otherwise, you’ll see a “compilation unit aborted, caught 1 fatal error condition” error. That’s simply your Lisp quitting too early.
Run this command to start the service:
sudo systemctl start my-app.service
to check its status:
systemctl status my-app.service
Systemd handles logging for you. We write to stdout or stderr, it writes logs:
journalctl -u my-app.service
-f -n 30 to see live updates of logs.
This tells Systemd to handle crashes and to restart the app:
and it can start the app after a reboot:
to enable it:
sudo systemctl enable my-app.service
Debugging SBCL error: ensure_space: failed to allocate n bytes
If you get this error with SBCL on your server:
mmap: wanted 1040384 bytes at 0x20000000, actually mapped at 0x715fa2145000 ensure_space: failed to allocate 1040384 bytes at 0x20000000 (hint: Try "ulimit -a"; maybe you should increase memory limits.)
then disable ASLR:
sudo bash -c "echo 0 > /proc/sys/kernel/randomize_va_space"
Connecting to a remote Swank server
Little example here: http://cvberry.com/tech_writings/howtos/remotely_modifying_a_running_program_using_swank.html.
It defines a simple function that prints forever:
;; a little common lisp swank demo ;; while this program is running, you can connect to it from another terminal or machine ;; and change the definition of doprint to print something else out! ;; (ql:quickload '(:swank :bordeaux-threads)) (require :swank) (require :bordeaux-threads) (defparameter *counter* 0) (defun dostuff () (format t "hello world ~a!~%" *counter*)) (defun runner () (bt:make-thread (lambda () (swank:create-server :port 4006))) (format t "we are past go!~%") (loop while t do (sleep 5) (dostuff) (incf *counter*))) (runner)
On our server, we run it with
sbcl --load demo.lisp
we do port forwarding on our development machine:
ssh -L4006:127.0.0.1:4006 firstname.lastname@example.org
this will securely forward port 4006 on the server at example.com to our local computer’s port 4006 (swanks accepts connections from localhost).
We connect to the running swank with
M-x slime-connect, typing in
We can write new code:
(defun dostuff () (format t "goodbye world ~a!~%" *counter*)) (setf *counter* 0)
and eval it as usual with
M-x slime-eval-region for instance. The output should change.
There are more pointers on CV Berry’s page.
When we run the app as a script we get a Lisp REPL, so we can hot-reload the running web app. Here we demonstrate a recipe to update it remotely.
Example taken from Quickutil.
It has a Makefile target:
hot_deploy: $(call $(LISP), \ (ql:quickload :quickutil-server) (ql:quickload :swank-client), \ (swank-client:with-slime-connection (conn "localhost" $(SWANK_PORT)) \ (swank-client:slime-eval (quote (handler-bind ((error (function continue))) \ (ql:quickload :quickutil-utilities) (ql:quickload :quickutil-server) \ (funcall (symbol-function (intern "STOP" :quickutil-server))) \ (funcall (symbol-function (intern "START" :quickutil-server)) $(start_args)))) conn)) \ $($(LISP)-quit))
It has to be run on the server (a simple fabfile command can call this
through ssh). Beforehand, a
fab update has run
git pull on the
server, so new code is present but not running. It connects to the
local swank server, loads the new code, stops and starts the app in a
Demo: fetching the GitHub REST API with Common Lisp
In this video I show you how to create a new project and we explore the Lisp ecosystem to make API calls, notably how to fire HTTP requests and parse JSON. We develop in Emacs and SLIME and in the end we get a Lisp library AND an application for the command line. Hope you enjoy the ride.