¡ THIS IS A DRAFT !

First, see the Awesome CL list.

See also the Cookbook’s issue.

Information is at the moment scarce and spread appart, Lisp web frameworks and libraries evolve and take different approaches.

I’d like to know what’s possible, what’s lacking, see how to quickstart everything, see code snippets and, most of all, see how to do things that I couldn’t do before such as hot reloading, building self-contained executables, shipping a multiplatform web app.

Table of Contents

Web application environments

Clack, Lack

Web frameworks

Hunchentoot

https://edicl.github.io/hunchentoot/

Caveman

https://github.com/fukamachi/caveman

Lucerne

https://github.com/eudoxia0/lucerne

(staling as of writing)

Snooze

https://github.com/joaotavora/snooze

Radiance

Radiance, with extensive tutorial and existing apps.

cl-rest-server

cl-rest-server

a library for writing REST Web APIs in Common Lisp.

Features: validation via schemas, Swagger support, authentication, logging, caching, permission checking…

Seems complete. Didn’t try.

Weblocks (solving the Javascript problem)

See our presentation below.

http://40ants.com/weblocks/quickstart.html

Tasks

Accessing url parameters

Lucerne has a nice with-params macro that makes accessing post or url query parameters a breeze:

@route app (:post "/tweet")
(defview tweet ()
  (if (lucerne-auth:logged-in-p)
      (let ((user (current-user)))
        (with-params (tweet)
          (utweet.models:tweet user tweet))
        (redirect "/"))
      (render-template (+index+)
                       :error "You are not logged in.")))

Snooze’s way is simple and lispy: we define routes like methods and parameters as keys:

(defroute lispdoc (:get :text/* name &key (package :cl) (doctype 'function))
   ...

matches /lispdoc, /lispdoc/foo and /lispdoc/foo?package=arg.


On the contrary, I find Caveman’s and Ningle’s ways cumbersome.

Ningle:

(setf (ningle:route *app* "/hello/:name")
      #'(lambda (params)
          (format nil "Hello, ~A" (cdr (assoc "name" params :test #'string=)))))

The above controller will be invoked when you access to “/hello/Eitaro” or “/hello/Tomohiro”, and then (cdr (assoc “name” params :test #‘string=)) will be “Eitaro” and “Tomohiro”.

and it doesn’t say about query parameters. I had to ask:

(assoc "the-query-param" (clack.request:query-parameter lucerne:*request*) :test 'string=)

Caveman:

Parameter keys contain square brackets (”[” & “]”) will be parsed as structured parameters. You can access the parsed parameters as _parsed in routers.

(defroute "/edit" (&key _parsed)
  (format nil "~S" (cdr (assoc "person" _parsed :test #'string=))))
;=> "((\"name\" . \"Eitaro\") (\"email\" . \"e.arrows@gmail.com\") (\"birth\" . ((\"year\" . 2000) (\"month\" . 1) (\"day\" . 1))))"

Session an cookies

Data storage

SQL

Mito works for MySQL, Postgres and SQLite3 on SBCL and CCL.

https://lispcookbook.github.io/cl-cookbook/databases.html

We can define models with a regular class which has a mito:dao-table-class :metaclass:

(defclass user ()
  ((name :col-type (:varchar 64)
         :initarg :name
         :accessor user-name)
   (email :col-type (:varchar 128)
          :initarg :email
          :accessor user-email))
  (:metaclass mito:dao-table-class)
  (:unique-keys email))

We create the table with ensure-table-exists:

(ensure-table-exists 'user)
;-> ;; CREATE TABLE IF NOT EXISTS "user" (
;       "id" BIGSERIAL NOT NULL PRIMARY KEY,
;       "name" VARCHAR(64) NOT NULL,
;       "email" VARCHAR(128),
;       "created_at" TIMESTAMP,
;       "updated_at" TIMESTAMP
;   ) () [0 rows] | MITO.DAO:ENSURE-TABLE-EXISTS

Persistent datastores

Migrations

Mito has migrations support and DB schema versioning for MySQL, Postgres and SQLite3, on SBCL and CCL. Once we have changed our model definition, we have commands to see the generated SQL and to apply the migration.

We inspect the SQL: (suppose we just added the email field into the user class above)

(mito:migration-expressions 'user)
;=> (#<SXQL-STATEMENT: ALTER TABLE user ALTER COLUMN email TYPE character varying(128), ALTER COLUMN email SET NOT NULL>
;    #<SXQL-STATEMENT: CREATE UNIQUE INDEX unique_user_email ON user (email)>)

and we can apply the migration:

(mito:migrate-table 'user)
;-> ;; ALTER TABLE "user" ALTER COLUMN "email" TYPE character varying(128), ALTER COLUMN "email" SET NOT NULL () [0 rows] | MITO.MIGRATION.TABLE:MIGRATE-TABLE
;   ;; CREATE UNIQUE INDEX "unique_user_email" ON "user" ("email") () [0 rows] | MITO.MIGRATION.TABLE:MIGRATE-TABLE
;-> (#<SXQL-STATEMENT: ALTER TABLE user ALTER COLUMN email TYPE character varying(128), ALTER COLUMN email SET NOT NULL>
;    #<SXQL-STATEMENT: CREATE UNIQUE INDEX unique_user_email ON user (email)>)

Crane advertises automatic migrations, i.e. it would run them after a C-c C-c. Unfortunately Crane has some issues, it doesn’t work with sqlite yet and the author is busy elsewhere. It didn’t work for me at first try.

Let’s hope the author comes back to work on this in a near future.

Forms

Form validation

Debugging

On an error we are usually dropped into the interactive debugger by default.

Snooze gives options:

  • use the debugger,
  • print the stacktrace in the browser (like clack-errors below, but built-in),
  • display a custom 404.

clack-errors. Like a Flask or Django stacktrace in the browser. For Caveman, Ningle and family.

By default, when Clack throws an exception when rendering a page, the server waits for the response until it times out while the exception waits in the REPL. This isn’t very useful. So now there’s this.

It prints the stacktrace along with some request details on the browser. Can return a custom error page in production.

clack-pretend

Are you tired of jumping to your web browser every time you need to test your work in Clack? Clack-pretend will capture and replay calls to your clack middleware stack. When developing a web application with clack, you will often find it inconvenient to run your code from the lisp REPL because it expects a clack environment, including perhaps, cookies or a logged-in user. With clack-pretend, you can run prior web requests from your REPL, moving development back where it belongs.

Testing

Testing with a local DB: example of a testing macro.

We would use envy to switch configurations.

Misc

Oauth, Job queues, etc

Templating engines

HTML-based

Djula: as Django templates. Good documentation. Comes by default in Lucerne and Caveman.

We also use a dot to access attributes of dict-like variables (plists, alists, hash-tables, arrays and CLOS objects), such a feature being backed by the access library.

We wanted once to use structs and didn’t find how to it directly in Djula, so we resorted in a quick helper function to transform the struct in an alist.

Eco - a mix of html with lisp expressions.

Truncated example:

<body>
      <% if posts %>
        <h1>Recent Posts</h1>
        <ul id="post-list">
          <% loop for (title . snippet) in posts %>
            <li><%= title %> - <%= snippet %></li>
          <% end %>
        </ul>
        ...

Lisp-based

I prefer the semantics of Spinneret over cl-who. It also has more features (like embeddable markdown, warns on malformed html, and more).

Javascript

Parenscript

Parenscript is a translator from an extended subset of Common Lisp to JavaScript. Parenscript code can run almost identically on both the browser (as JavaScript) and server (as Common Lisp). Parenscript code is treated the same way as Common Lisp code, making the full power of Lisp macros available for JavaScript. This provides a web development environment that is unmatched in its ability to reduce code duplication and provide advanced meta-programming facilities to web developers.

https://common-lisp.net/project/parenscript/

JSCL

A Lisp-to-Javascript compiler bootstrapped from Common Lisp and executed from the browser.

https://github.com/jscl-project/jscl

https://t-cool.github.io/jscl-playground/

Ajax

The case Webblocks - Reblocks, 2017

Weblocks is an “isomorphic” web frameworks that solves the “Javascript problem”. It allows to write the backend and an interactive client interface in Lisp, without a line of Javascript, in our usual Lisp development environment.

The framework evolves around widgets, that are updated server-side and are automatically redisplayed with transparent ajax calls on the client.

It is being massively refactored, simplified, rewritten and documented since 2017. See the new quickstart:

http://40ants.com/weblocks/quickstart.html

Writing a dynamic todo-app resolves in:

  • defining a widget class for a task:
(defwidget task ()
        ((title
          :initarg :title
          :accessor title)
         (done
          :initarg :done
          :initform nil
          :accessor done)))
  • doing the same for a list of tasks:
(defwidget task-list ()
        ((tasks
          :initarg :tasks
          :accessor tasks)))
  • saying how to render these widgets in html by extending the render method:
(defmethod render ((task task))
        "Render a task."
        (with-html
              (:span (if (done task)
                         (with-html
                               (:s (title task)))
                       (title task)))))

(defmethod render ((widget task-list))
        "Render a list of tasks."
        (with-html
              (:h1 "Tasks")
              (:ul
                (loop for task in (tasks widget) do
                      (:li (render task))))))
  • telling how to initialize the Weblocks app:
(defmethod weblocks/session:init ((app tasks))
         (declare (ignorable app))
         (let ((tasks (make-task-list "Make my first Weblocks app"
                                      "Deploy it somewhere"
                                      "Have a profit")))
           (make-instance 'task-list :tasks tasks)))
  • and then writing functions to interact with the widgets, for example adding a task:
(defmethod add-task ((task-list task-list) title)
        (push (make-task title)
              (tasks task-list))
        (update task-list))

Adding an html form and calling the new add-task function:

(defmethod render ((task-list task-list))
        (with-html
          (:h1 "Tasks")
          (loop for task in (tasks task-list) do
            (render task))
          (with-html-form (:POST (lambda (&key title &allow-other-keys)
                                         (add-task task-list title)))
            (:input :type "text"
                    :name "title"
                    :placeholder "Task's title")
            (:input :type "submit"
                    :value "Add"))))

Shipping

Building

We can build an executable also for web apps. That makes for a simple deployment process.

This is the general way:

(sb-ext:save-lisp-and-die #p"name-of-executable" :toplevel #'main :executable t)

we need a step more for web apps:

(defun main ()
    ;; with bordeaux-threads. Also sb-ext: join-thread, thread-name, list-all-threads.
    (bt:join-thread (find-if (lambda (th)
                                (search "hunchentoot" (bt:thread-name th)))
                              (bt:all-threads))))

I can now build my web app, send it to my VPS and see it live.

When I run it, Hunchentoot stays listening at the foreground:

$ ./my-webapp
Hunchentoot server is started.
Listening on localhost:9003.

I need to put it in the background (C-z bg), or use a tmux session (tmux, then C-b d to detach it).

To be complete, you’ll notice that we can not C-c our running app, we get trapped into the debugger (which responds only to C-z and kill). As with any command line, we have to catch the corresponding signal. We also stop our app. See our cl-torrents tutorial on how to build command-line applications.

(defun main ()
  (start-app :port 9003)
  ;; with bordeaux-threads
  (handler-case (bt:join-thread (find-if (lambda (th)
                                             (search "hunchentoot" (bt:thread-name th)))
                                         (bt:all-threads)))
    (#+sbcl sb-sys:interactive-interrupt
      #+ccl  ccl:interrupt-signal-condition
      #+clisp system::simple-interrupt-condition
      #+ecl ext:interactive-interrupt
      #+allegro excl:interrupt-signal
      () (progn
           (format *error-output* "Aborting.~&")
           (clack:stop *server*)
           (uiop:quit 1)) ;; portable exit, included in ASDF, already loaded.
    ;; for others, unhandled errors (we might want to do the same).
    (error (c) (format t "Woops, an unknown error occured:~&~a~&" c)))))

See also how to daemonize an application (below in Deployment).

To see:

Multiplatform delivery with Electron (Ceramic)

Ceramic makes all the work for us.

It is as simple as this:

;; Load Ceramic and our app
(ql:quickload '(:ceramic :our-app))

;; Ensure Ceramic is set up
(ceramic:setup)
(ceramic:interactive)

;; Start our app (here based on the Lucerne framework)
(lucerne:start our-app.views:app :port 8000)

;; Open a browser window to it
(defvar window (ceramic:make-window :url "http://localhost:8000/"))

;; start Ceramic
(ceramic:show-window window)

and we can ship this on Linux, Mac and Windows.

More:

Ceramic applications are compiled down to native code, ensuring both performance and enabling you to deliver closed-source, commercial applications.

(so no need to minify our JS)

with one more line:

(ceramic.bundler:bundle :ceramic-hello-world
                                 :bundle-pathname #p"/home/me/app.tar")
Copying resources...
Compiling app...
Compressing...
Done!
#P"/home/me/app.tar"

This last line was buggy for us.

Deployment

Radiance’s tutorial talks about deployment. https://github.com/Shirakumo/radiance-tutorial/blob/master/Part%207.md

Running the app on a web server

Manually

sbcl --load <my-app> --eval (start-my-app)

For example, a run Makefile target:

run:
	sbcl --load my-app.asd \
	     --eval '(ql:quickload :my-app)' \
	     --eval '(my-app:start-app)'  ;; given this function starts clack or hunchentoot.

this keeps sbcl in the foreground. Can use tmux or just C-z bg to put it in background.

Then, we need of a task supervisor, that will restart our app on failures, start it after a reboot, handle logging. See the section below and example projects (such as Quickutil).

with Clack

$ clackup app.lisp
Hunchentoot server is started.
Listening on localhost:5000.

with Docker

So we have various implementations ready to use: sbcl, ecl, ccl… with Quicklisp well configured.

https://lispcookbook.github.io/cl-cookbook/testing.html#gitlab-ci

On Heroku

See heroku-buildpack-common-lisp and the Awesome CL#deploy section.

Daemonizing, restarting in case of crashes, handling logs

See how to do that on your system.

Most GNU/Linux distros now come with Systemd.

Examples search result:

It is as simple as writing a configuration file:

$ /etc/systemd/system/my-app.service
[Unit]
Description=stupid simple example

[Service]
WorkingDirectory=/path/to/your/app
ExecStart=/usr/local/bin/sthg sthg
Type=simple
Restart=always
RestartSec=10

running a command to start it:

sudo systemctl start my-app.service

a command to check its status:

systemctl status my-app.service

and Systemd can handle logging (we write to stdout or stderr, it writes logs):

journalctl -f -u my-app.service

and it handles crashes and restarts the app:

Restart=always

and it can start the app after a reboot:

[Install]
WantedBy=basic.target

to enable it:

sudo systemctl enable my-app.service

Debugging SBCL error: ensure_space: failed to allocate n bytes

If you get this error with SBCL on your server:

mmap: wanted 1040384 bytes at 0x20000000, actually mapped at 0x715fa2145000
ensure_space: failed to allocate 1040384 bytes at 0x20000000
(hint: Try "ulimit -a"; maybe you should increase memory limits.)

then disable ASLR:

sudo bash -c "echo 0 > /proc/sys/kernel/randomize_va_space"

Connecting to a remote Swank server

Little example here: http://cvberry.com/tech_writings/howtos/remotely_modifying_a_running_program_using_swank.html.

It defines a simple function that prints forever:

;; a little common lisp swank demo
;; while this program is running, you can connect to it from another terminal or machine
;; and change the definition of doprint to print something else out!
;; (ql:quickload :swank)
;; (ql:quickload :bordeaux-threads)

(require :swank)
(require :bordeaux-threads)

(defparameter *counter* 0)

(defun dostuff ()
  (format t "hello world ~a!~%" *counter*))

(defun runner ()
  (bt:make-thread (lambda ()
                    (swank:create-server :port 4006)))
  (format t "we are past go!~%")
  (loop while t do
       (sleep 5)
       (dostuff)
       (incf *counter*)
       ))

(runner)

On our server, we run it with

sbcl --load demo.lisp

we do port forwarding on our development machine:

ssh -L4006:127.0.0.1:4006 username@example.com

this will securely forward port 4006 on the server at example.com to our local computer’s port 4006 (swanks accepts connections from localhost).

We connect to the running swank with M-x slime-connect, typing in port 4006.

We can write new code:

(defun dostuff ()
  (format t "goodbye world ~a!~%" *counter*))
(setf *counter* 0)

and eval it as usual with M-x slime-eval-region for instance. The output should change.

There are more pointers on CV Berry’s page.

Hot reload

Example with Quickutil.

It has a Makefile target:

hot_deploy:
	$(call $(LISP), \
		(ql:quickload :quickutil-server) (ql:quickload :swank-client), \
		(swank-client:with-slime-connection (conn "localhost" $(SWANK_PORT)) \
			(swank-client:slime-eval (quote (handler-bind ((error (function continue))) \
				(ql:quickload :quickutil-utilities) (ql:quickload :quickutil-server) \
				(funcall (symbol-function (intern "STOP" :quickutil-server))) \
				(funcall (symbol-function (intern "START" :quickutil-server)) $(start_args)))) conn)) \
		$($(LISP)-quit))

It has to be run on the server (a simple fabfile command can call this through ssh). Beforehand, a fab update has run git pull on the server, so new code is present but not running. It connects to the local swank server, loads the new code, stops and starts the app in a row.

Appendice I: Example websites built with Lisp:

  • Quickdocs-server - Caveman, Djula templates, Datafly and Sxql, Envy configuration switcher, Qlot, a simple fabfile for deployment.
  • Quickutil - ningle, closure-template, jquery and pjax, hot deploy with connection to a swank server, a fabfile, nginx, supervisor, watchdog autoreload.

Appendice II: Example software

See also Potato, Turtl and others in the Software section.