In our previous entry, we saw how to deploy our web application with Systemd, either from sources or with a binary. Now we’ll speak more about this building process to produce one binary that contains everything for our web app. We’ll tackle 3 issues:

  • ship foreign libraries alongside your binary, such as libreadline.so or libsqlite3.so,
  • include your Djula templates into your binary,
  • serve static files from your binary, without reading the filesystem,
  • and we’ll see my Gitlab CI recipe.

This allows us to create a binary that is really easy to deploy, to ship to users or to embed in an external process such as an Electron window (more on that later). Coming from Python and JS, what a dream!

Now, I want to thank the people that helped me figure these issues out and who wrote, fixed and extended these libraries: special shout-out to @mmontone for writing a Djula patch so quickly, @shinmera for Deploy and answering my many questions on Discord, @zulu.inoe for finding Hunchentoot answers, and everybody else on Discord for their help (@gavinok, @fstamour et all, sorry if I forgot) and all who dare asking questions to let everybody learn!

Table of Contents - Ship foreign libraries: the need of Deploy - Configuring Deploy: ignore libssl, verbosity - Remember: your program runs on another user’s machine. - Telling ASDF to calm down - Embed HTML Djula templates in your binary - Serve static assets - Gitlab CI - Closing remarks

Ship foreign libraries: the need of Deploy

Deploy is the way to go. If you used asdf:make in your .asd system definition to create a binary already, you just need to change two things:

;; my-project.asd
:defsystem-depends-on (:deploy)  ;; so you need to quickload deploy sometime before.
:build-operation "deploy-op"  ;; instead of program-op for asdf:make

and those two lines stay the same:

:build-pathname "my-application-name"
:entry-point "my-package:my-start-function"

Here’s my Makefile target, where I quickload Deploy before loading my app and calling asdf:make:

LISP ?= sbcl

build:
	$(LISP)	--non-interactive \
		--eval '(ql:quickload "deploy")' \
		--load openbookstore.asd \
		--eval '(ql:quickload :openbookstore)' \
		--eval '(asdf:make :openbookstore)'

This creates a bin/ directory with our binary and the foreign libraries:

  -rwxr-xr-x  1 vindarel vindarel 130545752 Nov 25 18:48 openbookstore
  -rw-rw-r--  1 vindarel vindarel    294632 Aug  3 13:06 libreadline.so.7.0
  -rw-rw-r--  1 vindarel vindarel    319528 Aug 23 18:01 libreadline.so.8.0
  -rw-rw-r--  1 vindarel vindarel   1212216 Aug 24 16:42 libsqlite3.so.0.8.6
  -rw-rw-r--  1 vindarel vindarel    116960 Aug  3 13:06 libz.so.1.2.11

We need to deploy this directory.

When we start the binary, Deploy tells us what it is doing:

$ ./bin/openbookstore --datasource argentina lisp
 ==> Performing warm boot.
   -> Runtime directory is /home/vindarel/projets/openbookstore/openbookstore/bin/
   -> Resource directory is /home/vindarel/projets/openbookstore/openbookstore/bin/
 ==> Running boot hooks.
 ==> Reloading foreign libraries.
   -> Loading foreign library #<LIBRARY READLINE>.
   -> Loading foreign library #<LIBRARY SQLITE3-LIB>.
   -> Loading foreign library #<LIBRARY LIBSSL>.
   -> Loading foreign library #<LIBRARY LIBCRYPTO>.
 ==> Launching application.
OpenBookStore version 0.2-d2ac5f2
[ā€¦]
==> Epilogue.
==> Running quit hooks.

We can configure Deploy.

Configuring Deploy: ignore libssl, verbosity

You can silence the Deploy statuses by pushing :deploy-console into the *features* list, before calling asdf:make. Add this to the Makefile:

--eval '(push :deploy-console *features*)'

Now all seems well, you rsync your app to your server, run it andā€¦ you get a libssl error:

=> Deploying files to /home/vindarel/projets/myapp/commandes-collectivites/bin/
Unhandled SIMPLE-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING
                                    {10007285B3}>:
  #<LIBRARY LIBCRYPTO> does not have a known shared library file path.

Nicolas (@shinmera) explained that we typically want to import libssl or libcrypto from the target system, that “deploying these libraries without them blowing up on Linux is hard”. To do this, we ask Deploy to not handle them. In the .asd:

#+linux (deploy:define-library cl+ssl::libssl :dont-deploy T)
#+linux (deploy:define-library cl+ssl::libcrypto :dont-deploy T)

As a consequence, you now need to quickload or require :cl+ssl before loading the .asd file, because of the cl+ssl::libssl/libcrypto symbols at the top level.

Nicolas built all this for his needs when working on his Trial game engine and on his Kandria game (soon on Steam!), check them out!

Remember: your program runs on another user’s machine.

By this I mean that if you took the habit to use functions that locate a project’s source directory (asdf:system-source-directory, asdf:system-relative-pathname, for example when asking Hunchentoot to serve static assets, more on that below), then you need to re-write them, because your binary runs on another machine and it doesn’t run from sources, so ASDF, Quicklisp and friends are not installed, and your project(s) don’t have source directories, they are embedded in the binary.

Use a deploy:deployed-p runtime check if needed.

Telling ASDF to calm down

Now, we are very happy and confident, what could possibly go wrong? We run our app once again on our naked VPS:

$ ./bin/myapp
 ==> Performing warm boot.
   -> Runtime directory is /home/debian/websites/app/myapp/bin/
   -> Resource directory is /home/debian/websites/app/myapp/bin/
 ==> Running boot hooks.
 ==> Reloading foreign libraries.
   -> Loading foreign library #<LIBRARY LIBSSL>.
   -> Loading foreign library #<LIBRARY LIBRT>.
   -> Loading foreign library #<LIBRARY LIBOSICAT>.
   -> Loading foreign library #<LIBRARY LIBMAGIC>.
 ==> Launching application.
WARNING:
   You are using ASDF version 3.3.4.15 from
   #P"/home/vindarel/common-lisp/asdf/asdf.asd" and have an older version of ASDF
   (and older than 2.27 at that) registered at
   #P"/home/vindarel/common-lisp/asdf/asdf.asd".

  [ long message ellided ]

;
; compilation unit aborted
;   caught 1 fatal ERROR condition
An error occured:
 Error while trying to load definition for system asdf from pathname
 /home/vindarel/common-lisp/asdf/asdf.asd:
    Couldn't load #P"/home/vindarel/common-lisp/asdf/asdf.asd": file does not
    exist. ==> Epilogue.
 ==> Running quit hooks.

Now ASDF wants to do what, update itself? Whatever it tries to do, it crashes. Yes, this happens on the target host, when we run the binary. Damn!

The solution is easy, but it had to be documentend or google-ableā€¦ Add this in your .asd to tell ASDF to not try to upgrade itself:

(deploy:define-hook (:deploy asdf) (directory)
  ;; Thanks again to Shinmera.
  (declare (ignorable directory))
  #+asdf (asdf:clear-source-registry)
  #+asdf (defun asdf:upgrade-asdf () nil))

By the way, if you want a one-liner to upgrade ASDF to 3.3.5 so that you can use package-local nicknames, check this lisp tip

Embed HTML Djula templates in your binary

Our binary now runs fine on our server: super great. But our app has another issue.

I like very much Djula templates, maintained by @mmontone. It is a traditional, no-surprises HTML templating system, very similar to Django templates. It is easy to setup, it is very easy to create custom filters, it has good error messages, both in the browser window and on the Lisp REPL. It’s one of the most downloaded Quicklisp libraries. Like Django templates, its philosophy is that it doesn’t allow a lot of computations in the template. It encourages to prepare your data on the back-end, so it is straightforward to process them in the templates. Sometimes it is limiting, so for more flexibility I’d look at Ten. It isn’t as much used and tested though (and I didn’t try it myself). If you want lispy templates, look at Spinneret. You can say goodby to copy-pasting nice-looking HTML examples, though.

However, by using Spinneret you would have not faced the following issue:

Djula reads templates from your file system.

and when your application runs on someone else’s machine, this is undefined behaviour.

Until now, you had to deploy your web app from sources or, at least, you had to send the HTML files to the server. This was the case until I talked about this issue to Mariano. He sent a patch the day after.

Now, we can choose: by default, Djula reads the HTML files from disk: very well. But now, when we build our binary, we can ask Djula to build the templates in memory, so they are saved into the Lisp binary.

Normally, you only need to tell Djula where to find templates (“add a template directory”), then to compile them into a variable:

;; normal, file-system case.
(djula:add-template-directory (asdf:system-relative-pathname "webapp" "templates/"))
(defparameter +base.html+ (djula:compile-template* "base.html"))

;; and then, we render the template with (djula:render-template* nil +base.html+ ā€¦)

This uses a filesystem-template-store. In addition, it recompiles templates on change. This can be turned off as we’ll see.

For our binary, we need to set Djula’s *current-store* to a memory-template-store AND we need to turn off the djula:*recompile-templates-on-change* setting. Then, we need to compile all the templates of our application, and save our binary.

I actually do all this at the top-level of my web.lisp file. By default I load the app for development, and if we find a custom “feature”, that is added by the “build” target of the Makefile, we compile templates in memory.

So, in order:

  1. in the “build” target of my Makefile, I push a new setting in the *features* list:

    --eval '(push :djula-binary *features*)'

  2. in my web.lisp, I check for this setting (with #+djula-binary) and I create either a filetemplate store or a memory store. This is written at the top-level so it will be executed when we load the file. We can probably come up with better ergonomics.

This will be executed when I quickload my app in the build target of the Makefile, following the one above.

    --eval '(ql:quickload :openbookstore)'

(setf djula:*current-store*
      (let ((search-path (list (asdf:system-relative-pathname "openbookstore"
                                                              "src/web/templates/"))))
        #-djula-binary
        (progn
          (uiop:format! t "~&Openbookstore: compiling templates in file system store.~&")
          ;; By default, use a file-system store and reload templates during development.
          (setf djula:*recompile-templates-on-change* t)
          (make-instance 'djula:filesystem-template-store
		         :search-path search-path))

        ;; But, if this setting is set to NIL at load time, from the Makefile,
        ;; we are building a standalone binary: use an in-memory template store.
        ;;
        ;; We also need to NOT re-compile templates on change.
        #+djula-binary
        (progn
          (uiop:format! t "~&Openbookstore: compiling templates in MEMORY store.~&")
          (setf djula:*recompile-templates-on-change* nil)
          (make-instance 'djula:memory-template-store :search-path search-path))))
  1. compile all the templates. If you used a web framework (or started to develop yours), you might have used a shortcut: calling a render function which takes the name of a template as a string for argument. I’m thinking about Caveman:
@route GET "/"
(defun index ()
  (render #P"index.tmpl"))

This strings denotes the name of the template. For a standalone binary, we need to compile the template before. That’s why Djula shows how to define and compile our templates:

(defparameter +base.html+ (djula:compile-template* "base.html"))

You need this line for every template of your application:

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; Compile and load templates (as usual).
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defparameter +base.html+ (djula:compile-template* "base.html"))
(defparameter +dashboard.html+ (djula:compile-template* "dashboard.html"))
(defparameter +search.html+ (djula:compile-template* "search.html"))
(defparameter +stock.html+ (djula:compile-template* "stock.html"))
(defparameter +card-page.html+ (djula:compile-template* "card-page.html"))
(defparameter +card-stock.html+ (djula:compile-template* "card-stock.html"))
(defparameter +card-create.html+ (djula:compile-template* "card-create.html"))
(defparameter +card-update.html+ (djula:compile-template* "card-edit.html"))
;; and so on.
  1. let asdf:make (and Deploy) save your binary. To try it, rename your templates/ directory to something else and run your app.

Addendum.

An .asd system definition can reference static files, so they are part of the build process and included into the delivered application. That’s how you can ship a README file:

  :components ((:static-file "README.md")
               ...)

I did the same for my templates. To be honest, I don’t recall if there is a solid reason, since they are compiled and saved into the image anyhow with the compilation step above. I show this anyways, it looks like a good practice to me:

            (:module "src/web/templates"
                        :components
                        ;; Order is important.
                        ((:static-file "login.html")
                         (:static-file "404.html")
                         (:static-file "base.html")
                         (:static-file "dashboard.html")
                         (:static-file "history.html")
                         (:static-file "search.html")
                         (:static-file "sell.html")
                         ...))

now we need to programmatically get the list of files from this src/web/templates module and to compile everything:

(let ((paths (djula:list-asdf-system-templates "bookshops" "src/web/templates")))
  (loop for path in paths
     do (uiop:format! t "~&Compiling template file: ~aā€¦~&" path)
       (djula:compile-template* path))
  (values t :all-done))

This snippet and the general instructions are documented: https://mmontone.github.io/djula/djula/Deployment.html#Deployment

Feel free to show how you do it.

Bonus: here’s the list-asdf-system-templates function. We use asdf functions to get a system name, its components, their namesā€¦

(defun list-asdf-system-templates (asdf-system component)
  "List djula templates in ASDF-SYSTEM at COMPONENT.
  A list of template PATHNAMEs is returned."
  (let* ((sys (asdf:find-system asdf-system))
         (children (asdf:component-children sys))
         (module (or (find component children :key #'asdf:component-name :test #'equal)
                     (error "Component ~S not found in system named ~S.~&Available components are: ~S" component asdf-system (mapcar #'asdf:component-name children))))
         (alltemplates (remove-if-not (lambda (x) (typep x 'asdf:static-file))
                                      (asdf:module-components module))))
    (mapcar (lambda (it) (asdf:component-pathname it))
            alltemplates)))

Serve static assets

At that point in time, I figured out static assets would need to be worked on too. Hopefully, the people on Discord helped me and it was quickly solved.

This is how I served static assets with Hunchentoot. We use a “folder dispatcher and handler”:

https://common-lisp-libraries.readthedocs.io/hunchentoot/#create-folder-dispatcher-and-handler

(defun serve-static-assets ()
  "Serve static assets under the /src/static/ directory when called with the /static/ URL root."
  (push (hunchentoot:create-folder-dispatcher-and-handler
         "/static/" (merge-pathnames *default-static-directory*
                                     (asdf:system-source-directory :openbookstore) ;; => NOT src/
                                     ))
        hunchentoot:*dispatch-table*))

But when your app is on another machineā€¦ hence the need to ship the static assets into the standalone binary, and to ask Hunchentoot to serve them.

What we do is pretty obvious: save our static files into a data structure, so this one is saved in the image, but we use a couple Lisp tricks so I comment the code below.

You’ll see that this time I hardcoded the file names and I didn’t declare them on the .asd fileā€¦ clearly there is room for improvement, be my guest.

You can find the file I use for my application here.

;;; pre-web.lisp
;;; Parameters and functions required before loading web.lisp
;;;
;;; We read the content of our static files and put them into variables, so that they can be saved in the Lisp image.
;;; We define %serve-static-file to simply return their content (as string),
;;; and because we use with the #. reader macro, we need to put these functions in another file than web.lisp.

;;; Where my static files are:
(defparameter *default-static-directory* "src/static/"
  "The directory where to serve static assets from (STRING). If it starts with a slash, it is an absolute directory. Otherwise, it will be a subdirectory of where the system :abstock is installed.
  Static assets are reachable under the /static/ prefix.")

;;; We simply use a hash-table that maps a file name to its content, a a string.
;;; I love Serapeum's dict which is a readable hash-table, that's what I use:
(defparameter *static-files-content* (dict)
  "Content of our JS and CSS files.
  Hash-table with file name => content (string).")

;;; I read all my static files and I save them into the hash-table:
(defun %read-static-files-in-memory ()
  "Save the JS and CSS files in a variable in memory, so they can be saved at compile time."
  (loop for file in (list "openbookstore.js"
                          "card-page.js")
     with static-directory = (merge-pathnames *default-static-directory*
                                              (asdf:system-source-directory :bookshops))
     for content = (uiop:read-file-string (merge-pathnames file static-directory))
     do (setf (gethash file *static-files-content*) content)
     finally (return *static-files-content*)))

;; AT COMPILE TIME, read the content of our static files.
(%read-static-files-in-memory)

(defun %serve-static-file (path)
  "Return the content as a string of this static file.
  For standalone binaries delivery."
  ;; "alert('yes, compiled in pre-web.lisp');"  ;; JS snippet to check if this dispatcher works.
  (gethash path *static-files-content*))  ;; this would not work without the #. reader macro.

It is inside “web.lisp” that I set other rules for Hunchentoot. If it recognizes my static files, we simply return their content, as a string.

I don’t know if this works well with very big or with numerous files. But I-want-a-standalone-binary! For serious needs, I’d serve the static files with a proper serverā€¦ I guess.

We use the #. reader macro to get our files’ content at compile time, this is why we needed to define our helper functions in another file, that is loaded before this one.

;;; web.lisp
(defun serve-static-assets-for-release ()
  "In a binary release, Hunchentoot can not serve files under the file system: we are on another machine and the files are not there.
  Hence we need to get the content of our static files into memory and give them to Hunchentoot."
  (push
   (hunchentoot:create-regex-dispatcher "/static/openbookstore\.js"
                                        (lambda ()
                                          ;; Returning the result of the function calls silently fails. We need to return a string.
                                          ;; Here's the string, read at compile time.
                                          #.(%serve-static-file "openbookstore.js")))
   hunchentoot:*dispatch-table*)

  (push
   (hunchentoot:create-regex-dispatcher "/static/card-page\.js"
                                        (lambda ()
                                          #.(%serve-static-file "card-page.js")))
   hunchentoot:*dispatch-table*))

Finally, it is inside my start-app function that I decide how to serve my static assets:

  (hunchentoot:start *server*)
  (if (deploy:deployed-p)
      ;; Binary release: don't serve files by reading them from disk.
      (serve-static-assets-for-release)
      ;; Normal setup, running from sources: serve static files as usual.
      (serve-static-assets))
  (uiop:format! t "~&Application started on port ~a.~&" port)

Find my web.lisp file here.

Gitlab CI

I build my binary on Gitlab.

image: clfoundation/sbcl

# uncomment to run the jobs in parallel. They are now run one after the other.
# stages:
  # - test
  # - build

# We need to install some system dependencies,
# to clone libraries not in Quicklisp,
# and to update ASDF to >= 3.3.5 in order to use local-package-nicknames.
before_script:
  - apt-get update -qy
  - apt-get install -y git-core sqlite3 tar
  # The image doesn't have Quicklisp installed by default.
  - QUICKLISP_ADD_TO_INIT_FILE=true /usr/local/bin/install-quicklisp
  # clone libraries not in Quicklisp or if we need the latest version.
  - make install
  # Upgrade ASDF (UIOP) to 3.3.5 because we want package-local-nicknames.
  - mkdir -p ~/common-lisp/asdf/
  - ( cd ~/common-lisp/ && wget https://asdf.common-lisp.dev/archives/asdf-3.3.5.tar.gz  && tar -xvf asdf-3.3.5.tar.gz && mv asdf-3.3.5 asdf )
  - echo "Content of ~/common-lisp/asdf/:" && ls ~/common-lisp/asdf/

qa:
  allow_failure: true
  # stage: test
  script:
    # QA tools:
    # install Comby:
    - apt-get install -y sudo
    # - bash <(curl -sL get.comby.dev)
    - bash <(curl -sL https://raw.githubusercontent.com/vindarel/comby/set-release-1.0.0/scripts/install.sh)
    # install Colisper for simple lisp checks:
    - git clone https://github.com/vindarel/colisper ~/colisper
    - chmod +x ~/colisper/colisper.sh
    - cd src/ && ~/colisper/colisper.sh

build:
  # stage: build
  script:
    - make build
  artifacts:
    name: "openbookstore"
    # Publish the bin/ directory (todo: rename, include versionā€¦)
    paths:
      - bin/

Closing remarks

I am so excited by the possibilities this brings.

I knew it was possible to do this in CL but I admit I thought it would be simplerā€¦ it turned out it is not a very crowded path. Now the steps are documented and google-able, here and everywhere else I could leave a comment, but it will be nice to come up with shorter and friendlier ready-to-use utilities. In a new web framework? And again, please share how you do all of this in the comments.

Having this standalone binary dramatically simplifies my deployment process. With a small web app, running from sources was easy (once you set up Quicklisp, and ASDF, andā€¦). But with a growing application, that uses my local forks or code not yet pushed to GitHub, deployment was becoming tedious, and it is now greatly simplified. rsync, systemctl restart and done.

Its only limitation is that you need the same libc version on the target OS as on your local machine. So, back in august I could build on my machine and send the result to my VPS, but I upgraded my Debian-ish system, and left the server with its (very) old Ubuntu version, so I can’t run the binary from my machine there any moreā€¦ I must resort to a CI pipeline that uses a matrix of Ubuntu versions, or build with Docker or a virtual machine. Or run from sourcesā€¦ Maybe soon I will build (truly) static executables: they are coming to SBCL.

I repeat again the good side: my friends (on Debian so far) can download the app, run bin/openbookstore and it works \o/

One last thing I’d like to do is to be able to double-click an executable to start the app, and to have one single file (and not an archive that extracts as a directory, although it is not too bad!). This looks possible with Makeself.

If you want to try the standalone binary on your GNU/Linux system (does it actually work on other distros?), download the artifacts of the latest passing build on the pipelines page, or grab it with this direct link. Un-zip, run bin/bookshops and go to localhost address shown in the output (and also create an admin user as shown in the readme). You can leave me a comment here, on Gitter, on Discord or with a good ol’ email.

Stay tuned, OpenBookStore is still a work in progress but it will be a Common Lisp application flagship ;)