Lisp journey

🖌️ Lisp screenshots: today's Common Lisp applications in action

Wed, 11 Feb 2026 23:35:40 +0100

I released a hopefully inspiring gallery:

lisp-screenshots.org

We divide the showcase under the categories Music, Games, Graphics and CAD, Science and industry, Web applications, Editors and Utilities.

Of course:

“Please don’t assume Lisp is only useful for…

thank you ;)

For more example of companies using CL in production, see this list (contributions welcome, of course).

Don’t hesitate to share a screenshot of your app! It can be closed source and yourself as the sole user, as long as it as some sort of a GUI, and you use it. Historical success stories are for another collection.

The criteria are:

built in Common Lisp
with some sort of a graphical interface
targeted at end users
a clear reference, anywhere on the web, by email to me or simply as a comment here, that it is built in CL.

Details:

it can be web applications whose server side is CL, even if the JS/HTML is classical web tech.
no CLI interfaces. A readline app is OK but hey, we can do better.
it can be closed-source or open-source, commercial, research or a personal software
regarding “end users”: I don’t see how to include a tool like CEPL, but I did include a screen of LispWorks.
bonus point if it is developed in a company (we want it on https://github.com/azzamsa/awesome-lisp-companies/), be it a commercial product or an internal tool.

You can reach us on GitHub discussions, by email at (reverse "gro.zliam@stohsneercs+leradniv") and in the comments.

Best,

Practice for Advent Of Code in Common Lisp

Sun, 30 Nov 2025 19:12:07 +0100

Advent Of Code 2025 starts in a few hours. Time to practice your Lisp-fu to solve it with the greatest language of all times this year!

Most of the times, puzzles start with a string input we have to parse to a meaningful data structure, after which we can start working on the algorithm. For example, parse this:

(defparameter *input* "3   4
4   3
2   5
1   3
3   9
3   3")

into a list of list of integers, or this:

(defparameter *input* "....#.....
.........#
..........
..#.......
.......#..
..........
.#..^.....
........#.
#.........
......#...")

into a grid, a map. But how do you represent it, how to do it efficiently, what are the traps to avoid, are there some nice tricks to know? We’ll try together.

You’ll find those 3 exercises of increasing order also in the GitHub repository of my course (see my previous post on the new data structures chapter).

I give you fully-annotated puzzles and code layout. You’ll have to carefully read the instructions, think about how you would solve it yourself, read my proposals, and fill-in the blanks -or do it all by yourself. Then, you’ll have to check your solution with your own puzzle input you have to grab from AOC’s website!

Table of Contents

Prerequisites

You must know the basics, but not so much. And if you are an experienced Lisp developer, you can still find new constraints for this year: solve it with loop, without loop, with a purely-functional data structure library such as FSet, use Coalton, create animations, use the object system, etc.

If you are starting out, you must know at least:

the basic data structures (lists and their limitations, arrays and vectors, hash-tables, sets…)
iteration (iterating over a list, arrays and hash-table keys)
functions

no need of macros, CLOS or thorough error handling (it’s not about production-grade puzzles :p ).

Exercise 1 - lists of lists

This exercise comes from Advent Of Code 2024, day 01: https://adventofcode.com/2024/day/1

Read the puzzle there! Try with your own input data!

Here are the shortened instructions.

;;;
;;; ********************************************************************
;;; WARN: this exercise migth be hard if you don't know about functions.
;;; ********************************************************************
;;;
;;; you can come back to it later.
;;; But, you can have a look, explore and get something out of it.

In this exercise, we use:

;;; SORT
;;; ABS
;;; FIRST, SECOND
;;; EQUAL
;;; LOOP, MAPCAR, REDUCE to iterate and act on lists.
;;; REMOVE-IF
;;; PARSE-INTEGER
;;; UIOP (built-in) and a couple string-related functions
;;;
;;; and also:
;;; feature flags
;;; ERROR
;;;
;;; we don't rely on https://github.com/vindarel/cl-str/
;;; (nor on cl-ppcre https://common-lisp-libraries.readthedocs.io/cl-ppcre/)
;;; but it would make our life easier.
;;;

OK, so this is your puzzle input, a string representing two colums of integers.

(defparameter *input* "3   4
4   3
2   5
1   3
3   9
3   3")

We’ll need to parse this string into two lists of integers.

If you want to do it yourself, take the time you need! If you’re new to Lisp iteration and data structures, I give you a possible solution.

;;;
;;;
;;;
;;;
;;;
;;;
;;;
;;;
;;; [hiding in case you want to do it…]
;;;
;;;
;;;
;;;
;;;
;;;
;;;
;;;
;;;
;;;
;;;
;;;
;;;
;;;
;;;
;;;
;;;
;;;
;;;
;;;
;;;
;;;
;;;
;;;
;;;

(defun split-lines (s)
  "Split the string S by newlines.
  Return: a list of strings."
  ;; If you already quickloaded the STR library, see:
  ;; (str:lines s)
  ;;
  ;; UIOP comes with ASDF which comes with your implementation.
  ;; https://asdf.common-lisp.dev/uiop.html
  ;;
  ;; #\ is a built-in reader-macro to write a character by name.
  (uiop:split-string s :separator '(#\Newline)))

Compile the function and try it on the REPL, or with a quick test expression below a “feature flag”.

We get a result like '("3 4" "4 3" "2 5" "1 3" "3 9" "3 3"), that is a list of strings with numbers inside.

#+lets-try-it-out
;; This is a feature-flag that looks into this keyword in the top-level *features* list.
;; The expression below should be highlihgted in grey
;; because :lets-try-it-out doesn't exist in your *features* list.
;;
;; You can compile this with C-c C-c
;; Nothing should happen.
(assert (equal '("3   4" "4   3" "2   5" "1   3" "3   9" "3   3")
               (split-lines *input*)))
;;                                   ^^ you can put the cursor here and eval the expression with C-x C-e, or send it to the REPL with C-c C-j.

We now have to extract the integers inside each string.

To do this I’ll use a utility function.

;; We could inline it.
;; But, measure before trying any speed improvement.
(defun blank-string-p (s)
  "S is a blank string (no content)."
  ;; the -p is for "predicate" (returns nil or t (or a truthy value)), it's a convention.
  ;;
  ;; We already have str:blankp in STR,
  ;; and we wouldn't need this function if we used str:words.
  (equal "" s))  ;; better: pair with string-trim.

#+(or)
(blank-string-p nil)
#++
(blank-string-p 42)
#+(or)
(blank-string-p "")

And another one, to split by spaces:

(defun split-words (s)
  "Split the string S by spaces and only return non-blank results.

  Example:

    (split-words \"3    4\")
    => (\"3\" \"4\")
  "
  ;; If you quickloaded the STR library, see:
  ;; (str:words s)
  ;; which actually uses cl-ppcre under the hood to split by the \\s+ regexp,
  ;; and ignore consecutive whitespaces like this.
  ;;
  (let ((strings (uiop:split-string s :separator '(#\Space))))
    (remove-if #'blank-string-p strings)))

#+lets-try-it-out
;; test this however you like.
(split-words "3       4")

I said we wouldn’t use a third-party library for this first puzzle. But using cl-ppcre would be so much easier:

(ppcre:all-matches-as-strings "\\d+" "3  6")
;; => ("3" "6")

With our building blocks, this is how I would parse our input string into a list of list of integers.

We loop on input lines and use the built-in function parse-integer.

(defun parse-input (input)
  "Parse the multi-line INPUT into a list of two lists of integers."
  ;; loop! I like loop.
  ;; We see everything about loop in the iteration chapter.
  ;;
  ;; Here, we see one way to iterate over lists:
  ;; loop for … in …
  ;;
  ;; Oh, you can rewrite it in a more functional style if you want.
  (loop :for line :in (split-lines input)
        :for words := (split-words line)
        :collect (parse-integer (first words)) :into col1
        :collect (parse-integer (second words)) :into col2
        :finally (return (list col1 col2))))

#+lets-try-it-out
(parse-input *input*)
;; ((3 4 2 1 3 3) (4 3 5 3 9 3))

The puzzle continues.

“Maybe the lists are only off by a small amount! To find out, pair up the numbers and measure how far apart they are. Pair up the smallest number in the left list with the smallest number in the right list, then the second-smallest left number with the second-smallest right number, and so on.”

=> we need to SORT the columns by ascending order.;;;

“Within each pair, figure out how far apart the two numbers are;”

=> we need to compute their relative, absolute distance.

“you’ll need to add up all of those distances.”

=> we need to sum each relative distance.

“For example, if you pair up a 3 from the left list with a 7 from the right list, the distance apart is 4; if you pair up a 9 with a 3, the distance apart is 6.”

Our input data’s sum of the distances is 11.

We must sort our lists of numbers. Here’s a placeholder function:

(defun sort-columns (list-of-lists)
  "Accept a list of two lists.
  Sort each list in ascending order.
  Return a list of two lists, each sorted."
  ;; no mystery, use the SORT function.
  (error "not implemented"))

;; Use this to check your SORT-COLUMNS function.
;; You can write this in a proper test function if you want.
#+lets-try-it-out
(assert (equal (sort-columns (parse-input *input*))
               '((1 2 3 3 3 4) (3 3 3 4 5 9))))

Compute the absolute distance.

;; utility function.
(defun distance (a b)
  "The distance between a and b.
  Doesn't matter if a < b or b < a."
  ;;
  ;; hint: (abs -1) is 1
  ;;
  (error "not implemented")
  )

(defun distances (list-of-lists)
  "From a list of two lists, compute the absolute distance between each point.
  Return a list of integers."
  (error "not implemented")
  ;; hint:
  ;; (mapcar #'TODO (first list-of-lists) (second list-of-lists))
  ;;
  ;; mapcar is a functional-y way to iterate over lists.
  )


(defun sum-distances (list-of-integers)
  "Add the numbers in this list together."
  (error "not implemented")
  ;; Hint:
  ;; try apply, funcall, mapcar, reduce.
  ;; (TODO #'+ list-of-integers)
  ;; or loop … sum !
  )

Verify.

(defun solve (&optional (input *input*))
  ;; let it flow:
  (sum-distances (distances (sort-columns (parse-input input)))))

#+lets-try-it-out
(assert (equal 11 (solve)))

All good? There’s more if you want.

;;;
;;; Next:
;;; - do it with your own input data!
;;; - do the same with the STR library and/or CL-PPCRE.
;;; - write a top-level instructions that calls our "main" function so that you can call this file as a script from the command line, with sbcl --load AOC-2024-day01.lisp
;;;

Exercise 2 - prepare to parse a grid as a hash-table

This exercise is a short and easy, to prepare you for a harder puzzle. This is not an AOC puzzle itself.

Follow the instructions. We are only warming up.

;; Do this with only CL built-ins,
;; or with the dict notation from Serapeum,
;; or with something else,
;; or all three one after the other.

We will build up a grid stored in a hash-table to represent a map like this:

"....#...##....#"

where the # character represents an obstacle.

In our case the grid is in 1D, it is often rather 2D.

This grid/map is the base of many AOC puzzles.

Take a second: shall we represent a 2D grid as a list of lists, or something else, (it depends on the input size) and how would you do in both cases?

Your turn:

;;
;; 1. Define a function MAKE-GRID that returns an empty grid (hash-table).
;;
(defun make-grid ()
  ;; todo
  )


;;
;; Define a top-level parameter to represent a grid that defaults to an empty grid.
;;

;; def… *grid* …

;;
;; 2. Create a function named CELL that returns a hash-table with those keys:
;; :char -> holds the character of the grid at this coordinate.
;; :visited or :visited-p or even :visited? -> stores a boolean,
;;  to tell us if this cell was already visited (by a person walking in the map). It defaults
;;  to NIL, we don't use this yet.
;;

(defun cell (char &key visited)
  ;; todo
  )

;;
;; 3. Write a function to tell us if a cell is an obstacle,
;;    denoted by the #\# character
;;
(defun is-block (cell)
  "This cell is a block, an obstacle. Return: boolean."
  ;; todo
  ;; get the :char key,
  ;; check it equals the #\# char.
  ;; Accept a cell as NIL.
  )

We built utility functions we’ll likely re-use on a more complex puzzle.

Let’s continue with parsing the input to represent a grid.

If you are a Lisp beginner or only saw the data structures chapter in my course, I give you the layout of the parse-input function with a loop and you only have to fill-in one blank.

In any case, try yourself. Refer to the Cookbook for loop examples.

;;
;; 4. Fill the grid (with devel data).
;;
;; Iterate on a given string (the puzzle input),
;; create the grid,
;; keep track of the X coordinate,
;; for each character in the input create a cell,
;; associate the coordinate to this cell in the grid.
;;

(defparameter *input* ".....#..#.##...#........##...")

(defun parse-grid (input)
  "Parse a string of input, fill a new grid with a coordinate number -> a cell (hash-table).
  Return: our new grid."
  (loop :for char :across input
        :with grid := (make-grid)
        :for x :from 0
        :for cell := (cell char)
        :do
           ;; associate our grid at the X coordinate
           ;; with our new cell.
           ;; (setf … )
        :finally (return grid)))

;; try it:
#++
(parse-grid *input*)

That’s only a simple example of the map mechanism that comes regurlarly in AOC.

Here’s the 3rd exercise that uses all of this.

Harder puzzle - hash-tables, grid, coordinates

This exercise comes from Advent Of Code 2024, day 06. https://adventofcode.com/2024/day/6 It’s an opportunity to use hash-tables.

Read the puzzle there! Try with your own input data!

Here are the shortened instructions.

The solutions are in another file, on my GitHub repository.

;;;
;;; ********************************************************************
;;; WARN: this exercise migth be hard if you don't know about functions.
;;; ********************************************************************
;;;
;;; you can come back to it later.
;;; But, you can have a look, explore and get something out of it.

In this exercise, we use:

;;;
;;; parameters
;;; functions
;;; recursivity
;;; &aux in a lambda list
;;; CASE
;;; return-from
;;; &key arguments
;;; complex numbers
;;; hash-tables
;;; the DICT notation (though optional)
;;; LOOPing on a list and on strings
;;; equality for characters

For this puzzle, we make our life easier and we’ use the DICT notation.

(import 'serapeum:dict)

If you know how to create a package, go for it.

Please, quickload the STR library for this puzzle.

#++
(ql:quickload "str")
;; Otherwise, see this as another exercise to rewrite the functions we use.

This is your puzzle input:

;;; a string representing a grid, a map.
(defparameter *input* "....#.....
.........#
..........
..#.......
.......#..
..........
.#..^.....
........#.
#.........
......#...")

;; the # represents an obstacle,
;; the ^ represents a guard that walks to the top of the grid.

When the guard encounters an obstacle, it turns 90 degrees right, and keeps walking.

Our task is to count the number of distinct positions the guard will visit on the grid before eventually leaving the area.

We will have to: - parse the grid into a data structure - preferably, an efficient data structures to hold coordinates. Indeed, AOC real inputs are large. - for each cell, note if it’s an obstacle, if that’s where the guard is, if the cell was already visited, - count the number of visited cells.

;; We'll represent a cell "object" by a hash-table.
;; With Serapeum's dict:
(defun cell (char &key guard visited)
  (dict :char char
        :guard guard
        :visited visited))

;; Our grid is a dict too.
;; We create a top-level variable, mainly for devel purposes.
(defvar *grid* (dict)
  "A hash-table to represent our grid. Associates a coordinate (complex number which represents the X and Y axis in the same number) to a cell (another hash-table).")
;; You could use a DEFPARAMETER, like I did initially. But then, a C-c C-k (recompile current file) will erase its current value, and you might want or not want this.

For each coordinate, we associate a cell.

What is a coordinate? We use a trick we saw in other people’s AOC solution, to use a complex number. Indeed, with its real and imaginary parts, it can represent both the X axis and the Y axis at the same time in the same number.

#|
;; Practice complex numbers:

(complex 1)
;; => 1
(complex 1 1)
;; => represented #C(1 1)

;; Get the imaginary part (let's say, the Y axis):
(imagpart #C(1 1))

;; the real part (X axis):
(realpart #C(1 1))

|#

Look, we are tempted to go full object-oriented and represent a “coordinate” object, a “cell” object and whatnot, but it’s OK we can solve the puzzle with usual data structures.

;; Let's remember where our guard is.
(defvar *guard* nil
  "The guard coordinate. Mainly for devel purposes (IIRC).")

Task 1: parse the grid string.

We must parse the string to a hash-table of coordinates -> cells.

I’ll write the main loop for you. If you feel ready, take a go at it.

(defun parse-grid (input)
  "Parse INPUT (string) to a hash-table of coordinates -> cells."
  ;; We start by iterating on each line.
  (loop :for line :in (str:lines input)
        ;; start another variable that tracks our loop iteration.
        ;; It it incremented by 1 at each loop by default.
        :for y :from 0  ;; up and down on the map, imagpart of our coordinate number.
        ;; The loop syntax with … = … creates a variable at the first iteration,
        ;; not at every iteration.
        :with grid = (dict)

        ;; Now iterate on each line's character.
        ;; A string is an array of characters,
        ;; so we use ACROSS to iterate on it. We use IN to iterate on lists.
        ;;
        ;; The Iterate library has the generic :in-sequence clause if that's your thing (with a speed penalty).
        :do (loop :for char :across line
                 :for x :from 0   ;; left to right on the map, realpart of our coordinate.
                 :for key := (complex x y)
                  ;; Create a new cell at each character.
                  :for cell := (cell char)
                  ;; Is this cell the guard at the start position?
                 :when (equal char #\^)
                   :do (progn
                         ;; Here, use SETF on GETHASH
                         ;; to set the :guard keyword of the cell to True.

                         (print "we saw the guard")
                         ;; (setf (gethash … …) …)

                         ;; For devel purposes, we will also keep track of
                         ;; where our guard is with a top-level parameter.
                         (setf *guard* key)
                         )
                  :do
                     ;; Normal case:
                     ;; use SETF on GETHAH
                     ;; to associate this KEY to this CELL in our GRID.
                     (format t "todo: save the cell ~S in the grid" cell)
                  )
        :finally (return grid))
  )

;; devel: test and bind a top-level param for ease of debugging/instropection/poking around.
#++
(setf *grid* (parse-grid *input*))

Task 2: walk our guard, record visited cells.

We have to move our guard on the grid, until it exits it.

I’ll give you a couple utility functions.

(defun is-block (cell)
  "Is this cell an obstacle?"
  ;; accept a NIL, we'll stop the walk in the next iteration.
  (when cell
    (equal TODO #\#)))

;; We choose the write the 4 possible directions as :up :down :right :left.
;; See also:
;; exhaustiveness checking at compile-time:
;; https://dev.to/vindarel/compile-time-exhaustiveness-checking-in-common-lisp-with-serapeum-5c5i

(defun next-x (position direction)
  "From a position (complex number) and a direction, compute the next X."
  (case direction
    (:up (realpart position))
    (:down (realpart position))
    (:right (1+ (realpart position)))
    (:left (1- (realpart position)))))

(defun next-y (position direction)
  "From a position (complex number) and a direction, compute the next Y."
  (case direction
    (:up (1- (imagpart position)))
    (:down (1+ (imagpart position)))
    (:right (imagpart position))
    (:left (imagpart position))))

This is the “big” function that moves the guard, records were it went, makes it rotate if it is against a block, and iterates, until the guard goes out of the map.

Read the puzzle instructions carefuly and write the “TODO” placeholders.

(defun walk (&key (grid *grid*) (input *input*)
               (position *guard*)
               (cell (gethash *guard* *grid*))  ;; todo: *grid* is used here. Fix it so we don't use a top-level variable, but only the grid given as a key argument.
               (direction :up)
               (count 0)
               ;; &aux notation: it saves a nested of LET bindings.
               ;; It's old style.
               ;; Those are not arguments to the function we pass around,
               ;; they are bindings inside the function body.
             &aux next-cell
               next-position
               obstacle-coming)
  "Recursively move the guard and annotate cells of our grid,
  count the number of visited cells."

  ;; At each iteration, we study a new cell we take on our grid.
  ;; If we move the guard to a coordinate that doesn't exist in our grid,
  ;; we stop here.
  (unless cell
    (return-from walk count))

  ;; Look in the same direction first and see what we have.
  (setf next-position
        (complex (next-x position direction) (next-y position direction)))

  (setf next-cell (gethash next-position grid))

  ;; obstacle?
  (setf obstacle-coming (is-block next-cell))

  ;; then change direction.
  (when obstacle-coming
    (setf direction
          (case direction
            (:up :right)
            (:down :left)
            (:right :down)
            (:left :up))))

  ;; Count unique visited cells.
  ;; TODO
  (unless (print "if this CELL is visited...")
      (incf count)
      ;; TODO set this cell as visited.
      (print "set this CELL to visited")
    )

  ;; get our next position now.
  (setf next-position
        (complex (next-x position direction) (next-y position direction)))

  ;; This next cell may or may not be in our grid (NIL).
  (setf next-cell (gethash next-position grid))

  (walk :grid grid :input input
        :cell next-cell
        :position next-position
        :direction direction
        :count count))

and that’s how we solve the puzzle:

(defun part-1 (input)
  (walk :grid (parse-grid input)))

#++
(part-1 *input*)
;; 41
;; The right answer for this input.
;; In AOC, you have a bigger, custom puzzle input. This can lead to surprises.

Closing words

Look at other people’s solutions too. For example, ak-coram’s for our last exercise (using FSet). See how Screamer is used for day 06 by bo-tato (reddit). atgreen (ocicl, cl-tuition, cffi…) solution with a grid as a hash-table with complex numbers. lispm’s day 04 solution. Can you read all solutions?

On other days, I used:

alexandria’s map-permutations for day 08 when you want… permutations. It doesn’t “cons” (what does that mean you ask? You didn’t follow my course ;) ). Read here: https://dev.to/vindarel/advent-of-code-alexandrias-map-permutations-was-perfect-for-day-08-common-lisp-tip-16il.
the library fare-memoization, to help in a recursive solution.
to write math, use cmu-infix. When you spot 2 equations with 2 unknows, think “Cramer system”. This came up last year, so maybe not this year.
with very large numbers: use double floats, as in 1.24d0
least common multiple? lcm is a built-in.
str:match can be a thing to parse strings.
if you got CIEL (CIEL Is an Extended Lisp), you have Alexandria, cl-str, Serapeum:dict and more libraries baked-in. It’s also an easy way to run Lisp scripts (with these dependencies) from the shell.

See you and happy lisping!

Your best resources:

🎥 ⭐ Learn Common Lisp data structures: 9 videos, 90 minutes of video tutorials to write efficient Lisp

Tue, 25 Nov 2025 19:11:24 +0100

It is with great pleasure and satisfaction that I published new videos about Common Lisp data structures on my course.

The content is divided into 9 videos, for a total of 90 minutes, plus exercises, and comprehensive lisp snippets for each video so you can practice right away.

The total learning material on my course now accounts for 8.40 hours, in 10 chapters and 61 videos, plus extras. You get to learn all the essentials to be an efficient (Common Lisp) developer: CLOS made easy, macros, error and condition handling, iteration, all about functions, working with projects, etc. All the videos have english subtitles.

learn more about this course (and why it’s a good idea to support me ;) ) on the GitHub page: Common Lisp course in videos: learn Lisp in videos (github),
and register here: 🎥 Common Lisp programming course on Udemy
- if you are a student, drop me an email for a free link.
You can refer to the course with this link (with my referral).

Table of Contents

What is this course anyways?

Hey, first look at what others say about it!

[My employees] said you do a better job of teaching than Peter Seibel.

ebbzry, CEO of VedaInc, August 2025 on Discord. O_o

🔥 :D

I have done some preliminary Common Lisp exploration prior to this course but had a lot of questions regarding practical use and development workflows. This course was amazing for this! I learned a lot of useful techniques for actually writing the code in Emacs, as well as conversational explanations of concepts that had previously confused me in text-heavy resources. Please keep up the good work and continue with this line of topics, it is well worth the price!

@Preston, October of 2024 <3

Now another feedback is that also according to learners, the areas I could improve are: give more practice activities, make the videos more engaging.

I worked on both. With the experience and my efforts, my flow should be more engaging. My videos always have on-screen annotations about what I’m doing or have complementary information. They are edited to be dynamic.

You have 9 freely-available videos in the course so you can judge by yourself (before leaving an angry comment ;) ). Also be aware that the course is not for total beginners in a “lisp” language. We see the basics (evaluation model, syntax…), but quickly. Then we dive in “the Common Lisp way”.

I also created more practice activities. For this chapter on data structures, each video comes with its usual set of extensive lisp snippets to practice (for example, I give you a lisp file with all sequence functions, showing their common use and some gotchas), plus 3 exercises, heavily annotated. Given the time of the year we are on, I prepare you for Advent Of Code :) I drive you into how you can put your knowledge in use to solve its puzzles. If you have access to the course and you are somewhat advanced, look at the new exercise of section 6.

Enough talk, what will you learn?

Course outcome

The goals were:

give you an overview of the available data structures in Common Lisp (lists and the cons cell, arrays, hash-tables, with a mention of trees an sets)
teach you how things work, don’t read everything for you. I show you the usual sequence functions, but I don’t spend an hour listing all of them. Instead I give you pointers to a reference and a lisp file with all of them.
give pointers on where is Common Lisp different and where is Common Lisp similar to any other language. For example, we discuss the time complexity of list operations vs. arrays.
teach common errors, such as using '(1 2 3) with a quote instead of the list constructor function, and how this can lead to subtle bugs.
make your life easier: working with bare-bones hash-tables is too awkward for my taste, and was specially annoying as a beginner. I give you workarounds, in pure CL and with third-party libraries.
- 🆓 this video is free for everybody, hell yes, this was really annoying to me.
present the ecosystem and discuss style: for example I point you to purely-functional data-structures libraries, we see how to deal with functions being destructive or not destructive and how to organize your functions accordingly.

So, suppose you followed this chapter, the one about functions, and a couple videos on iteration: you are ready to write efficient solutions to Advent Of Code.

Chapter content

3.1 Intro [🆓 FREE FOR ALL]

Common Lisp has more than lists: hash-tables (aka dictionaries), arrays, as well as sets and tree operations. Linked lists are made of “CONS” cells. You should adopt a functional style in your own functions, and avoid the built-ins that mutate data. We see how, and I give you more pointers for modern Common Lisp.

3.2 Lists: create lists, plists, alists

What we see: how to create lists (proper lists, plists and alists). A first warning about the ‘(1 2 3) notation with a quote.

PRACTICE: list creation

3.3 Lists (2): lists manipulation

Lists, continued. What we see: how to access elements: FIRST, REST, LAST, NTH…

PRACTICE: accessing sequences elements
PRACTICE: sequences manipulation functions
EXERCISE: parse, sort, compute. AOC day 01. (hard for total beginners. Have a look, take at least something out of it)

3.4 Equality - working with strings gotcha

What we see: explanation of the different equality functions and why knowing this is necessary when working with strings. EQ, EQL, EQUAL, EQUALP (and STRING= et all) explained. Which is too low-level, which you’ll use most often.

PRACTICE: using equality predicates.

3.5 Vectors and arrays

What we see: vectors (one-dimensional arrays), multi-dimensional arrays, VECTOR-PUSH[-EXTEND], the fill-pointer, adjustable arrays, AREF, VECTOR-POP, COERCE, iteration across arrays (LOOP, MAP).

EXERCISE: compare lists and vectors access time.

3.6 The CONS cell

A “CONS cell” is the building block of Common Lisp’s (linked) lists. What do “cons”, “car” and “cdr” even mean?

3.7 The :test and :keys arguments

All CL built-in functions accept a :TEST and :KEY argument. They are great. What we see: when and how to use them, when working with strings and with compound objects (lists of lists, lists of structs, etc).

3.8 Hash-tables and fixing their two ergonomic flaws [🆓 FREE FOR ALL]

Hash-tables (dictionaries, hash maps etc) are efficient key-value stores. However, as a newcomer, I had them in gripe. They were not easy enough to work with. I show you everything that’s needed to work with hash-tables, and my best solution for better ergonomics.

PRACTICE: the video snippet to create hash-tables, access and set content, use Alexandria, Serapeum’s dict notation, iterate on keys and values, serialize a HT to a file and read its content back.

3.9 Using QUOTE to create lists is NOT THE SAME as using the LIST function. Gotchas and solution.

Thinking that ‘(1 2 3) is the same as (list 1 2 3) is a rookie mistake and can lead to subtle bugs. Demo, explanations and simple rule to follow.

At last, EXERCISE of section 6: real Advent Of Code puzzle.

;;;
;;; In this exercise, we use:
;;;
;;; top-level variables
;;; functions
;;; recursivity
;;; &aux in a lambda list
;;; CASE
;;; return-from
;;; &key arguments
;;; complex numbers
;;; hash-tables
;;; the DICT notation (optional)
;;; LOOPing on a list and on strings
;;; equality
;;; characters literal notation

(defparameter *input* "....#.....
.........#
..........
..#.......
.......#..
..........
.#..^.....
........#.
#.........
......#...")

Closing words

Thanks for your support, thanks to everybody who took the course or who shared it, and for your encouragements.

If you wonder why I create a paid course and you regret it isn’t totally free (my past me would def wonder), see some details on the previous announce. The short answer is: I also contribute free resources.

Keep lisping and see you around: improving the Cookbook or Lem, on the Fediverse, reddit and Discord…

What should be next: how the Cookbook PDF quality was greatly improved thanks to Typst. Stay tuned.

Oh, a last shameless plug: since Ari asked me at the beginning of the year, I now do 1-1 Lisp coaching sessions. We settled on 40 USD an hour. Drop me an email! (concatenate 'string "vindarel" "@" "mailz" "." "org").

🎥 Common Lisp course in videos

🕊

Lisp tutorial: variables. defparameter vs defvar, let/let* and lexical scope, unbound variables, style guidelines

Tue, 16 Sep 2025 23:47:24 +0200

Lisp newcomers, I still care about you ;) A section on variables was missing on the Cookbook, here it is.

As usual, this is best read on the Common Lisp Cookbook. This is where it will get updates and fixes.

The Cookbook has many contributors. You can contribute too. I myself mostly contributed (out of frustration) as I was discovering Common Lisp, the language and the ecosystem. It’s been years now, but I still take care of it because I like it, and thanks to your tips. As I don’t have a salary nor a million-dollar company, I do appreciate them. I’m on github sponsors too. Thank you!

Also, I can now generate a good-quality PDF thanks to Typst and Pandoc. Stay tuned.

So, you are writing your first Common Lisp program (again, welcome!) and you want to declare variables. What are your options?

When in doubt, use defparameter for top-level parameters.

Use let or let* for lexical scope:

(let* ((a 2)
       (square (* a a)))
   (format t "the square of ~a is ~a" a square))

Use setf to change them.

`defparameter`: top-level variables

Use defparameter to declare top-level variables, like this:

(defparameter *name* "me")

(defun hello (&optional name)
  "Say hello."
  (format t "Hello ~a!" (or name *name*)))

defparameter accepts an optional third argument: the variable’s docstring:

(defparameter *name* "me"
   "Default name to say hello to.")

The inline docstrings are an important part of the Common Lisp interactive experience. You will encounter them during your coding sessions (and we lispers usually keep our Lisp running for a long time). In Emacs and Slime, you can ask for a symbol’s docstring with C-c C-d d (Alt-x slime-describe-symbol). You can also ask for a docstring programmatically:

(documentation '*name* 'variable)

We ask the documentation of the *name* symbol, not what it holds, hence the quote in '*name* (which is short for (quote *name*). Another “doc-type” is 'function. See: in Common Lisp, variables and functions live in different “namespaces”, and it shows here.

We’ll mention the defparameter form with no value below.

redefining a defparameter

A Common Lisp coding session is usually long-lasting and very interactive. We leave a Lisp running and we interact with it while we work. This is done with Emacs and Slime, Vim, Atom and SLIMA, VSCode and Alive, Lem… and more editors, or from the terminal.

That means that you can do this:

1- write a first defparameter

(defparameter *name* "me")

either write this in the REPL, either write this in a .lisp file and compile+load it with a shortcut (C-c C-c (Alt-x slime-compile-defun) in Slime on this expression, or C-c C-k (Alt-x slime-compile-and-load-file) to compile and load everything you have in the current buffer). If you work from a simple terminal REPL, you can (load …) a .lisp file.

Now the *name* variable exists in the running image.

2- edit the defparameter line:

(defparameter *name* "you")

and load the changes the same way: either with the REPL, or with a C-c C-c. Now, the *name* variable has a new value, “you”.

A defvar wouldn’t be redefined.

`defvar`: no redefinition

defvar defines top-level variables and protects them from redefinition.

When you re-load a defvar, it doesn’t erase the current value, you must use setf for this.

(defvar *names-cache* (list)
  "Store a list of names we said \"hello\" to.")

(defun hello (&optional (name *name*))
   (pushnew name *names-cache* :test #'string-equal)
   (format t "hello ~a!" name))

Let’s see it in use:

CL-USER> (hello)
hello you!
NIL
CL-USER> *names-cache*
("you")
CL-USER> (hello "lisper")
hello lisper!
NIL
CL-USER> *names-cache*
("lisper" "you")

What happens to *names-cache* if you redefine the defvar line (with C-c C-c, or C-c C-k, or on the REPL…)?

It doesn’t change and that is a good thing.

Indeed, this variable isn’t a user-visible parameter, it doesn’t have an immediate use, but it is important for the program correctness, or strength, etc. Imagine it holds the cache of your webserver: you don’t want to erase it when you load new code. During development, we hit a lot C-c C-k to reload the current file, we can as well reload our running app in production, but there are certain things we want untouched. If it is a database connection, you don’t want to set it back to nil, and connect again, everytime you compile your code.

You must use setf to change a defvar’s variable value.

The “earmuff” convention

See how we wrote *name* in-between “*earmuffs*”. That is an important convention, that helps you not override top-level variables in lexical scopes.

(defparameter name "lisper")

;; later…
(let ((name "something else"))
   ;;  ^^^ overrides the top-level name. This will cause bugs.
   …)

This becomes a feature only when using earmuffs:

(defparameter *db-name* "db.db")

(defun connect (&optional (db-name *db-name*))
  (sqlite:connect db-name))

(let ((*db-name* "another.db"))
  (connect))
  ;;^^^^  its db-name optional parameter, which defaults to *db-name*, now sees "another.db".

By the way, for such a use-case, you will often find with-… macros that abstract the let binding.

(with-db "another.db"
  (connect))

By the way again, an earmuff is a thing that covers the ears (but only the ears) in winter. You might have seen it in movies more than in reality. The lasting word is: take care of yourself, stay warm and use earmuffs.

Global variables are created in the “dynamic scope”

Our top-level parameters and variables are created in the so-called dynamic scope. They can be accessed from anywhere else: from function definitions (as we did), in let bindings, etc.

In Lisp, we also say these are dynamic variables or special.

It could also be possible to create one from anywhere by proclaiming it “special”. It really isn’t the thing you do everydays but, you know, in Lisp everything’s possible ;)

A dynamic variable can be referenced outside the dynamic extent of a form that binds it. Such a variable is sometimes called a “global variable” but is still in all respects just a dynamic variable whose binding happens to exist in the global environment rather than in some dynamic environment. [Hyper Spec]

`setf`: change values

Any variable can be changed with setf:

(setf *name* "Alice")
;; => "Alice"

It returns the new value.

Actually, setf accepts pairs of value, variable:

(setf *name* "Bob"
      *db-name* "app.db")
;; => "app.db"

It returned the last value.

What happens if you setf a variable that wasn’t declared yet? It generally works but you have a warning:

;; in SBCL 2.5.8
CL-USER> (setf *foo* "foo")
; in: SETF *FOO*
;     (SETF CL-USER::*FOO* "foo")
;
; caught WARNING:
;   undefined variable: CL-USER::*FOO*
;
; compilation unit finished
;   Undefined variable:
;     *FOO*
;   caught 1 WARNING condition
"foo"

We see the returned “foo”, so it worked. Please declare variables with defparameter or defvar first.

Let’s read the full setf docstring because it’s interesting:

Takes pairs of arguments like SETQ. The first is a place and the second
is the value that is supposed to go into that place. Returns the last
value. The place argument may be any of the access forms for which SETF
knows a corresponding setting form.

Note that setq is another macro, but now seldom used, because setf works on more “places”. You can setf functions and many things.

`let`, `let*`: create lexical scopes

let lets you define variables in a limited scope, or override top-level variables temporarily.

Below, our two variables only exist in-between the parenthesis of the let:

(let* ((a 2)
       (square (* a a)))
   (format t "the square of ~a is ~a" a square))
   ;; so far so good

(format t "the value of a is: ~a" a)
;; => ERROR: the variable A is unbound

“unbound” means the variable is bound to nothing, not even to NIL. Its symbol may exist, but it isn’t associated to anything.

Just after the scope formed by the let, the variables a and square don’t exist anymore.

When the Lisp reader reads the format expression, it reads a a symbol, which now exists in the global environment, but it isn’t bound.

Food for thought: the fact to write a variable name and have the Lisp reader read it creates the symbol, but doesn’t bind it to anything.

Our two variables can be accessed by any form inside the let binding. If we create a second let, its environment inherits the previous one (we see variables declared above, fortunately!).

(defparameter *name* "test")

(defun log (square)
  (format t "name is ~s and square is ~a" *name* square))

(let* ((a 2)
       (square (* a a)))
  ;; inside first environment
  (let ((*name* "inside let"))
    ;; inside second environment,
    ;; we access the dynamic scope.
    (log square)))
;;  => name is "inside let" and square is 4
;;  => NIL

(print *name*)
;; => "test"
;;    ^^^^ outside the let, back to the dynamic scope's value.

We could also define a function inside a let, so that this function definition “sees” a binding from a surrounding let at compile time. This is a closure and it’s for the chapter on functions.

A “lexical scope” is simply

a scope that is limited to a spatial or textual region within the establishing form. “The names of parameters to a function normally are lexically scoped.” [Hyper Spec]

In other words, the scope of a variable is determined by its position in the source code. It’s today’s best practice. It’s the least surprising way of doing: you can see the scope by looking at the source code.

`let` vs `let*`

By the way, what is the syntax of let and what is the difference with let*?

let* lets you declare variables that depend on each other.

let’s basic use is to declare a list of variables with no initial values. They are initialized to nil:

(let (variable1 variable2 variable3) ;; variables are initialized to nil by default.
  ;; use them here
  …)

;; Example:
(let (a b square)
  (setf a 2)
  (setf square (* a a))
  (list a b square))
;; => (2 NIL 4)

;; exactly the same:
(let (a
      b
      square)
  …)

You can give default values by using “pairs” of elements, as in (a 2):

(let ((a 2)     ;; <-- initial value
       square)  ;; <-- no "pair" but still one element: defaults to NIL.
  (setf square (* a a))
  (list a square))

Yes, there are two (( in a row! This is the syntax of Common Lisp. You don’t need to count them. What appears after a let is variable definitions. Usually, one per line.

The let’s logic is in the body, with a meaningful indentation. You can read Lisp code based on indentation. If the project you are looking at doesn’t respect that, it is a low quality project.

Observe that we kept square to nil. We want it to be the square of a, so can we do this?

(let ((a 2)
      (square (* a a))) ;; WARN:
  …)

You can’t do that here, this is the limitation of let. You need let*.

You could write two lets:

(let ((a 2))
  (let ((square (* a a)))
    (list a square)))
;; => (2 4)

This is equivalent to let*:

(let* ((a 2)
       (square (* a a)))
  …)

let is to declare variables that don’t depend on each other, let* is to declare variables which are read one after the other and where one can depend on a previous one.

This is not valid:

(let* ((square (* a a))  ;; WARN!
       (a 2))
   (list a square))
;; => debugger:
;; The variable A is unbound.

The error message is clear. At the time of reading (square (* a a)), a is unknown.

setf inside let

Let’s make it even clearer: you can setf any value that is shadowed in a let binding, once outside the let, the variables are back to the value of the current environment.

We know this:

(defparameter *name* "test")

(let ((*name* "inside let"))
  (format t "*name* inside let: ~s" *name*))
;; => *name* inside let: "inside let"

(format t "*name* outside let: ~s" *name*)
;; => *name* outside let: "test"

we setf a dynamic parameter that was shadowed by a let binding:

(defparameter *name* "test")

(defun change-name ()
   ;; bad style though,
   ;; try to not mutate variables inside your functions,
   ;; but take arguments and return fresh data structures.
   (setf *name* "set!"))
   ;;    ^^^^^ from the dynamic environment, or from a let lexical scope.

(let ((*name* "inside let"))
  (change-name)
  (format t "*name* inside let: ~s" *name*))
;; => *name* inside let: "set!"

(format t "*name* outside let: ~s" *name*)
;; => *name* outside let: "test"

When you don’t use defined variables

Read your compiler’s warnings :)

Below, it tells us that b is defined but never used. SBCL is pretty good at giving us useful warnings at compile time (every time you hit C-c C-c (compile and load the expression at point), C-c C-k (the whole file) or use load).

(let (a b square)
  (list a square))
;; =>
; caught STYLE-WARNING:
;   The variable B is defined but never used.

This example works in the REPL because SBCL’s REPL always compiles expressions.

This may vary with your implementation.

It’s great to catch typos!

(let* ((a 2)
       (square (* a a)))
  (list a squale))
  ;;         ^^^ typo

If you compile this in a .lisp file (or in a Alt-x slime-scratch lisp buffer), you will have two warnings, and your editor will underline each in two different colors:

first, “square” is defined but never used
second, “squale” is an undefined variable.

If you run the snippet in the REPL, you will get the two warnings but, because the snippet is run, you will see the interactive debugger with the error “The variable SQUALE is unbound”.

Unbound variables

“unbound” variables were not bound to anything, not even nil. Their symbol might exist, but they have no associated value.

You can create such variables like this:

(defparameter *connection*)

This defparameter form is correct. You didn’t give any default value: the parameter is unbound.

You can check if a variable (or a function) is bound with boundp (or fboundp). The p is for “predicate”.

You can make a variable (or function) unbound with makunbound (or fmakunbound).

Global variables are thread safe

Don’t be afraid of accessing and set-ing global bindings in threads. Each thread will have its own copy of the variable. Consequently, you can bind them to other values with let bindings, etc. That’s good.

It’s only if you want one single source of truth that you’ll have to share the variable between threads and where the danger lies. You can use a lock (very easy), but that’s all another topic.

Addendum: `defconstant`

defconstant is here to say something is a constant and is not supposed to change, but in practice defconstant is annoying. Use defparameter, and add a convention with a new style of earmuffs:

(defparameter +pi+ pi
  "Just to show that pi exists but has no earmuffs. Now it does. You shouldn't change a variable with +-style earmuffs, it's a constant.")

defconstant is annoying because, at least on SBCL, it can’t be redefined without asking for validation through the interactive debugger, which we may often do during development, and its default test is eql, so give it a string and it will always think that the constant was redefined. Look (evaluate each line one by one in order):

(defconstant +best-lisper+ :me)
;; so far so good.

(defconstant +best-lisper+ :me)
;; so far so good: we didn't redefine anything.

(defconstant +best-lisper+ :you)
;; => the constant is being redefined, we get the interactive debugger (SBCL):

The constant +BEST-LISPER+ is being redefined (from :ME to :YOU)
   [Condition of type SB-EXT:DEFCONSTANT-UNEQL]
See also:
  Common Lisp Hyperspec, DEFCONSTANT [:macro]
  SBCL Manual, Idiosyncrasies [:node]

Restarts:
 0: [CONTINUE] Go ahead and change the value.
 1: [ABORT] Keep the old value.
 2: [RETRY] Retry SLIME REPL evaluation request.
 3: [*ABORT] Return to SLIME's top level.
 4: [ABORT] abort thread (#<THREAD tid=573581 "repl-thread" RUNNING {120633D123}>)

;; => presse 0 (zero) or click on the "Continue" restart to accept changing the value.

With constants as strings:

(defconstant +best-name+ "me")
;; so far so good, we create a new constant.

(defconstant +best-name+ "me")
;; => interactive debugger!!

The constant +BEST-NAME+ is being redefined (from "me" to "me")
…

As you will see in the equality chapter, two strings are not equal by eql that is a low-level equality operator (think pointers), they are equal (or string-equal).

This is defconstant documentation:

Define a global constant, saying that the value is constant and may be compiled into code. If the variable already has a value, and this is not EQL to the new value, the code is not portable (undefined behavior). The third argument is an optional documentation string for the variable.

The eql thing is in the spec, what an implementation should do when redefining a constant is not defined, so it may vary with your implementation.

We invite you to look at:

Alexandria’s define-constant, which has a :test keyword (but still errors out on redefinition).
Serapeum’s defconst
cl:defparameter ;)

Guidelines and best practices

A few style guidelines:

create all your top-level parameters at the top of a file
define first parameters then variables
use docstrings
read your compiler’s warnings
it’s better for your functions to accept arguments, rather than to rely on top-level parameters
your functions shouldn’t mutate (modify) a top-level binding. You should create a new data structure instead, and use your function’s return value as the parameter to another function, and have data flow from one function to another.
parameters are best for: a webserver port, a default value… and other user-facing parameters.
variables are best for long-living and internal variables: caches, DB connections…
you can forget about defconstant
when in doubt, use a defparameter
the pattern where a function parameter is by default a global variable is typical and idiomatic:

;; from the STR library.
(defvar *whitespaces* (list #\Backspace #\Tab #\Linefeed #\Newline #\Vt #\Page
                            #\Return #\Space #\Rubout
                            ;; edited for brevity
                            ))

(defun trim-left (s &key (char-bag *whitespaces*))
  "Removes all characters in `char-bag` (default: whitespaces) at the beginning of `s`."
  (when s
   (string-left-trim char-bag s)))

the default value can also be a function call:

;; from the Lem editor
(defun buffer-modified-p (&optional (buffer (current-buffer)))
  "Return T if 'buffer' has been modified, NIL otherwise."
  (/= 0 (buffer-%modified-p buffer)))

these let bindings over global variables are idiomatic too: (let ((*name* "other")) …).

Lisp error handling (advanced): how handler-bind doesn't unwind the stack

Tue, 08 Jul 2025 11:45:58 +0200

I updated the Condition Handling page on the Common Lisp Cookbook to show what it means that handler-bind “doesn’t unwind the stack”, along with a couple real-world use cases.

This time I must thank Ari. I originally wrote the page (with contributions from jsjolen, rprimus and phoe), starting from Timmy Jose “z0ltan”’s article (linked in the introduction) and the books I had at my disposal. These don’t show handler-bind’s power and use-cases like this, focusing on restarts (which this page certainly could explain better, it shows they aren’t part of my daily toolbelt). So I learned about the real goodness by chance while reading @shinmera’s code and, shame on me, I didn’t update the Cookbook. My video course has been well and complete for years though ;) I needed fresh eyes and that happened with Ari. He asked if I offered 1-1 video lisp mentoring. I didn’t, but now I do!. So, as we talked about a million things (lisp and software development in general) and eventually worked on condition handling, I realized this page was lacking, and I took an extra 1h 40min to update it, “backporting” content from my video course.

These days my attention is more turned towards my tutorials for web development in Common Lisp, my CL projects and libraries, and less to the Cookbook alone like before, but for this edit to the Cookbook 2 things were new and important:

I had fresh eyes again, thanks to the lisp mentoring
I had an extra motivation to do the edit in the context of the 1-1 session: I felt it’s part of the job. By the way, we settled on 40USD an hour and I have a couple more slots available in the week ;) Please contact me by email (@ vindarel (. mailz org)) Thanks! [/end of self plug].

So what is this all about? handler-bind, unlike handler-case, doesn’t unwind the stack: it shows us the full backtrace and gives us absolute control over conditions and restarts.

It’s particularly necessary if you want to print a meaningful backtrace. We’ll give another development tip, where you can decide to either print a backtrace (production mode) or accept to be dropped into the debugger (invoke-debugger).

Feel free to leave feedback, in the comments or in the Cookbook issues or PR.

Table of Contents

Absolute control over conditions and restarts: `handler-bind`

handler-bind is what to use when we need absolute control over what happens when a condition is signaled. It doesn’t unwind the stack, which we illustrate in the next section. It allows us to use the debugger and restarts, either interactively or programmatically.

Its general form is:

(handler-bind ((a-condition #'function-to-handle-it)
               (another-one #'another-function))
    (code that can...)
    (...error out…)
    (... with an implicit PROGN))

For example:

(defun handler-bind-example ()
  (handler-bind
        ((error (lambda (c)
                  (format t "we handle this condition: ~a" c)
                  ;; Try without this return-from: the error bubbles up
                  ;; up to the interactive debugger.
                  (return-from handler-bind-example))))
      (format t "starting example…~&")
      (error "oh no")))

You’ll notice that its syntax is “in reverse” compared to handler-case: we have the bindings first, the forms (in an implicit progn) next.

If the handler returns normally (it declines to handle the condition), the condition continues to bubble up, searching for another handler, and it will find the interactive debugger.

This is another difference from handler-case: if our handler function didn’t explicitely return from its calling function with return-from handler-bind-example, the error would continue to bubble up, and we would get the interactive debugger.

This behaviour is particularly useful when your program signaled a simple condition. A simple condition isn’t an error (see our “conditions hierarchy” below) so it won’t trigger the debugger. You can do something to handle the condition (it’s a signal for something occuring in your application), and let the program continue.

If some library doesn’t handle all conditions and lets some bubble out to us, we can see the restarts (established by restart-case) anywhere deep in the stack, including restarts established by other libraries that this library called.

handler-bind doesn’t unwind the stack

With handler-bind, we can see the full stack trace, with every frame that was called. Once we use handler-case, we “forget” many steps of our program’s execution until the condition is handled: the call stack is unwound (or “untangled”, “shortened”). handler-bind does not rewind the stack. Let’s illustrate this.

For the sake of our demonstration, we will use the library trivial-backtrace, which you can install with Quicklisp:

(ql:quickload "trivial-backtrace")

It is a wrapper around the implementations’ primitives such as sb-debug:print-backtrace.

Consider the following code: our main function calls a chain of functions which ultimately fail by signaling an error. We handle the error in the main function with hander-case and print the backtrace.

(defun f0 ()
  (error "oh no"))

(defun f1 ()
  (f0))

(defun f2 ()
  (f1))

(defun main ()
  (handler-case (f2)
    (error (c)
      (format t "in main, we handle: ~a" c)
      (trivial-backtrace:print-backtrace c))))

This is the backtrace (only the first frames):

CL-REPL> (main)
in main, we handle: oh no
Date/time: 2025-07-04-11:25!
An unhandled error condition has been signalled: oh no

Backtrace for: #<SB-THREAD:THREAD "repl-thread" RUNNING {1008695453}>
0: […]
1: (TRIVIAL-BACKTRACE:PRINT-BACKTRACE … )
2: (MAIN)
[…]

So far so good. It is trivial-backtrace that prints the “Date/time” and the message “An unhandled error condition…”.

Now compare the stacktrace when we use handler-bind:

(defun main-no-stack-unwinding ()
  (handler-bind
      ((error (lambda (c)
                (format t "in main, we handle: ~a" c)
                (trivial-backtrace:print-backtrace c)
                (return-from main-no-stack-unwinding))))
    (f2)))

CL-REPL> (main-no-stack-unwinding)
in main, we handle: oh no
Date/time: 2025-07-04-11:32!
An unhandled error condition has been signalled: oh no

Backtrace for: #<SB-THREAD:THREAD "repl-thread" RUNNING {1008695453}>
0: …
1: (TRIVIAL-BACKTRACE:PRINT-BACKTRACE …)
2: …
3: …
4: (ERROR "oh no")
5: (F0)
6: (F1)
7: (MAIN-NO-STACK-UNWINDING)

That’s right: you can see all the call stack: from the main function to the error through f1 and f0. These two intermediate functions were not present in the backtrace when we used handler-case because, as the error was signaled and bubbled up in the call stack, the stack was unwound, and we lost information.

When to use which?

handler-case is enough when you expect a situation to fail. For example, in the context of an HTTP request, it is a common to anticipate a 400-ish error:

;; using the dexador library.
(handler-case (dex:get "http://bad-url.lisp")
  (dex:http-request-failed (e)
    ;; For 4xx or 5xx HTTP errors: it's OK, this can happen.
    (format *error-output* "The server returned ~D" (dex:response-status e))))

In other exceptional situations, we’ll surely want handler-bind. For example, when we want to handle what went wrong and we want to print a backtrace, or if we want to invoke the debugger manually (see below) and see exactly what happened.

Invoking the debugger manually

Suppose you handle a condition with handler-bind, and your condition object is bound to the c variable (as in our examples above). Suppose a parameter of yours, say *devel-mode*, tells you are not in production. It may be handy to fire the debugger on the given condition. Use:

(invoke-debugger c)

In production, you can print the backtrace instead and have an error reporting tool like Sentry notify you.

Closing words

This is yet another CL feature I wish I had known earlier and learned by chance. I hope you learned a thing or two!

Hacker News now runs on top of Common Lisp

Mon, 26 May 2025 17:46:52 +0200

Hacker News was written in the Arc lisp dialect, a dialect created by Paul Graham. Arc was implemented on top of Racket, but that has now changed. HN runs on top of SBCL since (at least) September of 2024.

But why? For performance reasons.

I recently noticed that Hacker News no longer uses paging for long threads. In the past, when a discussion grew large, we had to click “More” to load the next page of comments, and dang would occasionally post helpful tips to remind us about this feature. Was there an announcement regarding this change? Has anyone else still seen paging recently? I’d love to know more details—especially the technical aspects or considerations that went into the decision.

Answer:

It’s because Clarc is finally out.

dang, Sept. 2024

[Clarc] is much faster and also will easily let HN run on multiple cores. It’s been in the works for years, mainly because I rarely find time to work on it, but it’s all pretty close to done.

dang, 2022

How it’s done:

there’s now an Arc-to-JS called Lilt, and an Arc-to-Common Lisp called Clarc. In order to make those easier to develop, we reworked the lower depths of the existing Arc implementation to build Arc up in stages. The bottom one is called arc0, then arc1 is written in arc0, and arc2 in arc1. The one at the top (arc2, I think) is full Arc. This isn’t novel, but it makes reimplementation easier since you pack as much as possible in the later stages, and only arc0 needs to be written in the underlying system (Racket, JS, or CL).

dang, 2019

But Clarc’s code isn’t released, although it could be done:

open-sourcing the Arc implementation (i.e. Clarc) would be much easier [than the HN site]. The way to do it would be to port the original Arc release (http://arclanguage.org/) to Clarc. It includes a sample application which is an early version of HN, scrubbed of anything HN- or YC-specific.

dang, day of this post

Releasing the new HN code base however wouldn’t work:

Much of the HN codebase consists of anti-abuse measures that would stop working if people knew about them. Unfortunately. separating out the secret parts would by now be a lot of work. The time to do it will be if and when we eventually release the alternative Arc implementations we’ve been working on.

https://news.ycombinator.com/item?id=21546438

Congrats for the successful “splash-free” transition though.

edit 10PM UTC+2: added a quote to clarify about open-sourcing the HN code (“wouldn’t work”) VS Clarc itself (“much easier”).
edit May, 27th: reworded “since a few months” to mention a date.

These years in Common Lisp: 2023-2024 in review

Mon, 17 Feb 2025 19:06:46 +0200

This is a personal pick of the most interesting projects, tools, libraries and articles that popped-up in Common Lisp land in the last two years.

Newcomers might not realize how the Common Lisp ecosystem, though stable in many ways, actually evolves, sharpens, tries new solutions, proposes new tools, ships new libraries, revives projects. And everyone might enjoy a refresher.

Here’s my previous overview for 2022.

The same warnings hold: I picked the most important links, in my view, but this list is by no means a compilation of all new CL projects or articles published on the topic. Look for yourself on Reddit, Quicklisp releases, GitHub, and use your favourite search engine.

There are too many great news and achievements to pick 3. I love what’s happening around SBCL (and ECL, and Clozure’s revival), I love everything that got included into Lem and the work on all other editors, I love the webviews and I love the scripting tools that are emerging. What are your top picks?

OK, there’s a news I want to put at the forefront: HackerNews now runs on top of SBCL ;)

If you are discovering the ecosystem, my recommendaton is to not miss these two resources:

Awesome-cl - a curated list of libraries (there might be more than you think)
- if you are looking for a list of recommended libraries on each topic, look here.
the CL Cookbook

Now let’s dive in and thanks to everyone involved.

Table of Contents

Community

We could start with some reddit stats: 2025 - a New Year for an old programming language! (numbers are up).

The ELS team kept organizing the conference. We have a date and place for 2025: European Lisp Symposium 2025 in Zürich, May ¹⁹⁄₂₀

We saw new and regular Lisp Ireland meetups.

Here’s one of their videos: Lisp Ireland, February 2024 Meetup - Lisp & Hardware Verification with ACL2

@djha-skin ran a survey, which is not an established practice in the community, and analysed the results: Common Lisp Community Survey 2024 Results .

@shinmera (Yukari), the author of many useful libraries and an active member of the ELS, and even the host of the next one, opened a Patreon. “If you’d like to help me continue my full-time open source Lisp work, please consider supporting me.”. Sponsoring Yukari is money well spent. She is on GH sponsors and ko-fi too.

The community is on reddit, Discord, Mastodon, LinkedIn… and also on XMPP.

Documentation

The CL Cookbook is a collaborative resource with new contributors each year: new Cookbook EPUB and PDF release: 2025-01.

We got a great contribution: Cookbook: Building Dynamic Libraries with SBCL-Librarian · by em7

PAIP is a classic, now available on the web: Peter Norvig: Paradigms of Artificial Intelligence Programming, Case Studies in Common Lisp (web version).

New resource: Web Apps in Lisp: Know-how: I wanted a resource specialized for web development in Common Lisp. I mean to continuously extend it from now on.

I’ll include a couple general videos in this section. More videos and more documentation improvements are to be found in their respective sections.

FreeCodeCamp released an extensive Common Lisp course on Youtube: Lisp Programming Language – Full Course for Beginners - freeCodeCamp.org - Youtube.

David Botton of CLOG fame released more beginner material, among which Common Lisp - The Tutorial - Fast, Fun and Practical (with CLOG).

I carry on the work on my Common Lisp course in videos, on the Udemy platform. Lately, I worked on a CLOS tutorial: I published 9 videos (1h 22min) on my course. You’ll know enough to read the sources of Hunchentoot or the Kandria game 🎥 comments. The course is comprised of more than 7 hours of short videos, with a code first approach, divided in 9 chapters. We see some basics but we quickly dive into more advanced Common Lisp topics. You can learn more about it here on GitHub. Students can send me an email for a free link.

Here’s the feedback of redditors:

I can vouch for the Udemy course. From the very first lesson, just firing up the REPL and Emacs/SLIME I was taught something new. It’s a great course.

fuzzmonkey35, January 2025 (reddit)

It is an amazing tutorial. What is really strange is I thought CLOS was complicated. I guess it can be but Vincent is amazing at explaining everything and demystifying it.

intergallactic_llama, January 2025 (reddit)

;)

Implementations

Great times for Common Lisp implementations.

SBCL

SBCL ships monthly releases. You really should look at and appreciate all the activity and the continous improvements.

One noticeable addition: its new garbage collector. SBCL: merge of the mark-region GC.

More improvements include:

“the mark-region parallel garbage collector can be enabled on arm64. (Thanks to Hayley Patton)”,
new contrib module sb-perf, “a performance-analysing tool for Linux. (thanks to Luke Gorrie and Philipp Marek)”
support for cross-compiling the system to Android has been added (thanks to Gleefre)
“support for memory allocation arenas is now available on the arm64 platform.”
haiku support
- Porting Common Lisp to Haiku OS
sb-simd improvements

More good stuff with SBCL:

There are open bounties to improve SBCL:

$2000 USD bounty to see by-value struct passing implemented in SBCL’s native FFI.
- it may be more than $2000 USD now.
- You wouldn’t start from zero, there is existing work. See the thread.
another $700+ bounty to add coroutines in SBCL
- same link. No official bounty page yet, I may work on it.

ABCL

New release: ABCL 1.9.1 “never use a dot oh”: CFFI compatibilities, Java virtual threads, ASDF 3.3.6, fixed loading of Fricas0 and Maxima…

New release ABCL 1.9.2.

New tool: Announcing the First Release of abcl-memory-compiler - Now Available!

CCL

Clozure was a bit active, but rather dormant.

Great news: Clozure is back

Clozure CL 1.13 released.

Allegro

Allegro Common Lisp 11.0 from Franz Inc.

LispWorks

I didn’t spot a patch release (they had a major release in 2022), so let’s link to a discussion: is LispWorks worth it? you might learn some things about LW’s feature set.

ECL

Embeddable, targetting WASM… is it the future?

CLASP

CLASP targets C++ on LLVM.

Release: Clasp v2.5.0

They realeased Clasp v2.7.0 in January, 2025.

For context:

Christian Schafmeister talk - brief update about his “molecular lego” supported by his Lisp compiler
there’s less funding than in the 80s for Common Lisp, but still funding: “CLASP was supported by The Defense Threat Reduction Agency, The National Institutes of Health, The National Science Foundation”.

SICL - the new, portable and modular implementation

The SICL implementation is very active.

SICL is a new implementation of Common Lisp. It is intentionally divided into many implementation-independent modules that are written in a totally or near-totally portable way, so as to allow other implementations to incorporate these modules from SICL, rather than having to maintain their own, perhaps implementation-specific versions.

SICL’s components are used, for example, in the CLASP implementation.

Related, the second-climacs editor shows good activity too. Watch this demo, showing on-the-fly code parsing and feedback, it looks pretty cool. Here too, each achievement is extracted into its own component. So the second-climacs’ codebase actualy shrinks with time.

If you want to hack on CL, those are good places.

New implementations

A Common Lisp implementation in development in C89 (no compiler so far)
- called ALisp, “breakpoints and stepping work quite well”
gsou/LCL: Lua Common Lisp. An implementation of Common Lisp targeting Lua.

Historical: Medley Interlisp

We can run the Medley Interlisp Lisp machine in a browser O_o The work achieved by this group is phenomenal, look:

I suggest to follow @interlisp@fosstodon.org on Mastodon.

Companies and jobs

Yes, some companies still choose Common Lisp today, and some hire with a public job posting.

It’s of course the visible top of the iceberg. If you dream of a Lisp job, I suggest to be active and make yourself visible, you might be contacted by someone without a proper job announce. This could be for an open-source project with funding (happened to me), for a university, etc.

We knew these companies since awesome-lisp-companies -it’s only a list of companies we know about, nothing offical. Additions welcome.

Discussions on the topic:

Anyone using Common Lisp for freelance work? where we learn about cool websites made in CL and cool experience reports.
Running my 4th Common Lisp script in production© - you can do it too aka “you don’t need crazily difficult needs to make yourself a favour and use CL instead of Python in your projects”

Projects

Editors

Please check out the Cookbook: editors for a list of good editors for Common Lisp. You migth be surprised.

Let’s highlight a new editor in town: Neomacs: Structural Lisp IDE/computing environment . Mariano integrated it in his moldable web desktop: Integrating Neomacs into my CLOG-powered desktop.

About Emacs

About VSCode

Experience Report using VS Code + Alive to Write Common Lisp

About Lem and Rooms pair programming environment

Lem 2.0.0 released
- released in May 2023, this version added the SDL2 frontend, adding mouse support, graphic capabilities, and Windows support.
- it brought the possibility to draw images and shapes at any location on a buffer or window.
- addition of many base16 color themes (180), by @lukpank.
Lem 2.1.0 released, with many new contributors. Lem 2.0 definitely caught the eyes of many developers IMO.
- this is when Lem got its website: https://lem-project.github.io/
- @sasanidas worked on supporting other implementations: “ECL and CCL should work fairly well”, “ABCL and Clasp are still work in progress, working but with minor bugs.”.
- I added project-aware commands, find-file-recursively
- @cxxxr added (among everything else) great Lisp mode additions (just look at the release notes and the screenshots)
- added a sidebar / filer
- and much more. Just look at the release.
then came out Lem 2.2.0
- the release notes are less organized ;)
- added libvterm integration
- this is when I added the interactive git mode.

Unfortunately these latest releases do not ship a readily usable executable. But the installation recipes have been greatly simplified and use Qlot instead of Roswell. There’s a one-liner shell command to install Lem on Unixes.

Lem’s creator cxxxr is now on GitHub sponsors.

He is also working on Rooms, aka Lem on the cloud: it’s a Lem-based “pair programming environment where you can share your coding sessions”. Only the client is open-source, so far at least.

Demo: https://www.youtube.com/watch?v=IMN7feOQOak

Those are the Lem related articles that popped up:

About LispWorks

About the Jetbrains plugin

SLT Jetbrains plugin v0.2.0 and 0.2.1 - first version of inspector, macro-expand, basic completion

About Jupyter

Jupyter and Common Lisp: A Powerful Combination for Data Science

Other tools

CL-REPL now supports multiline editing
- it also comes as a ready-to-use binary

Coalton

Coalton is

the implementation of a static type system beyond Haskell 95. Full multiparameter type classes, functional dependencies, some persistent data structures, type-oriented optimization (including specialization and monomorphization). All integrated and native to CL without external tools.

And used in production for years in the quantum industry. See quilc.

I found Coalton-related projects:

E. Fukamachi added Coalton support for Lem: https://lem-project.github.io/modes/coalton-lang/. This adds completion, syntax highlighting, interactive compilation and more inside “coalton-toplevel” forms.

Package managers

Quicklisp had a one year hiatus, because it relies on one man. It finally got an update after 1 year: Quicklisp libraries were updated 2024-10-12. Despite a call for collaboration, we don’t really know how we can help.

But Quicklisp isn’t the only library manager anymore.

Also:

Announcing deptree | list and archive dependency snapshots of (ASDF-defined) projects

Gamedev

The Kandria game was released: https://kandria.com/

If you are into game dev, this is a paper you cannot miss: Kandria: experience report, presented at the ELS 2023.

Great articles:

and more:

Multiplayer game with Common Lisp + SDL2 on WebAssembly (short demo video)

I almost forgot the Lisp Game Jams and the new cool little games. For example: Nano Towers

a simple tower defense game written in Common Lisp with the EON framework based on Raylib, submitted for the Spring Lisp Game Jam 2024.

Links to the jams:

GUI

Many solutions exist. Disclaimer: the perfect GUI library doesn’t exist. Please see the Cookbook/gui and awesome-cl. Also don’t miss the web views available today.

releases:

As always, we might not highlight the work achieved on existing libraries that didn’t get a proper announce. There are more GUI libraries for CL.

demos:

“cl-gtk4 combines strength of GTK and Common Lisp interactive development.” (short screencast)

Web

CLOG appeared in 2022 and is kicking. Its API has been stable for 4 years.

You know Hacker News, the website, right? Hacker News now runs on top of SBCL

HN runs on top of Arc, the language. Arc was implemented on top of Racket (-> MzScheme). A new, faster / more efficient, implementation of Arc in SBCL was in the works by a Hacker News site maintainer for some time: called Clarc. Its source code has not been published. Since [late september, 2024], the official Hacker News site runs using Clarc and SBCL.

Here’s (again) my new resource for web development in Common Lisp: Web Apps in Lisp: Know-how.

Now the links:

CLOG CLOG 2.0 - Now with a complete Common Lisp IDE and GUI Builder (with or w/o emacs)
CLOG OS shell

CLOG CLOG Builder Video Manual Video 5 - Using Projects & Plugins
CLOG debug tools https://github.com/rabbibotton/clog/discussions/361
CLOG got emacs-like tabs

Projects built with CLOG:

mold desktop - A programmable desktop.
CLOG moldable inspector, “A moldable Common Lisp object inspector based on CLOG”

Weblocks (continued in the Reblocks project):

A WASM Common Lisp REPL

Articles:

videos:

How to create new CL library or web-application using the Mystic project generator

libraries:

The web views I mentioned: Electron is a thing, but today we have bindings to webview.h and webUI:

Three web views for Common Lisp: build cross platform GUIs with Electron, WebUI or CLOG Frame

Nyxt 4.0 pre-realease - now on Electron

Nyxt 4.0-pre-release-1 was published in late 2024.

They are publishing a Flatpak featuring the legacy WebKitGTK port and a new Electron one.

Electron has better performance and opens the door for macOS and Windows support.

More libraries

Data structures:

Language extensions, core libraries:

Iteration:

Developer tools:

BRAKE, an extended breakpoint facility for Common Lisp

Threads, actors:

Documentation builders:

Add Documentation, Please… with Github Flavoured Markdown · supports cross references and table of contents.

Databases:

relational database and first order logic:

AP5 - an extension to commonlisp which allows users to “program” in a model of first order logic or a relational database (1990, last update 2024)

Numerical and scientific:

Plotting:

GURAFU: a simple (just usable) plot program for Common Lisp
plotly-user: Use plotly in your browser to explore data from a Common Lisp REPL
- “a week-end hack and an excuse to learn CLOG”

Bindings and interfaces:

Serialization:

Date and time:

Utilities:

Bindings and interfaces to other software:

Networking:

aether · Distributed system emulation in Common Lisp ELS talk

Scripting

(I love what’s being done here)

ruricolist/kiln: Infrastructure for scripting in Common Lisp to make Lisp scripting efficient and ergonomic.
CIEL Is an Extended Lisp, Hacker News
unix-in-lisp - Mount Unix system into Common Lisp image.

Software releases

Videos

Demos:

Web:

More from the ELS (see their Youtube channel):

Learning:

Aaaand that’s it for the tour of the last couple years. Tell me if I missed something. I’ll keep updating this post for a few days.

Happy lisping and show us what you build!

New resource specialized on web development in Common Lisp

Wed, 15 Jan 2025 10:39:53 +0100

I just released a new documentation website specialized on web development in Common Lisp:

🚀 https://web-apps-in-lisp.github.io/

I’d be embarrassed to tell how long it took me to grasp all the building blocks and to assemble a resource that makes sense. I hope it serves you well, now don’t hesitate to share what you are building, it creates emulation!

In the first tutorial we build a simple app that shows a web form that searches and displays a list of products.

We see many necessary building blocks to write web apps in Lisp:

how to start a server
how to create routes
how to define and use path and URL parameters
how to define HTML templates
how to run and build the app, from our editor and from the terminal.

In doing so, we’ll experience the interactive nature of Common Lisp.

In the user log-in section, we build a form that checks a user name and a password:

We also introduce databases, and more topics.

The sources are here: https://github.com/web-apps-in-lisp/web-apps-in-lisp.github.io/ and the GitHub Discussions are open.

Towards a Django-like database admin dashboard for Common Lisp

Mon, 30 Dec 2024 17:23:14 +0100

This is an ongoing work I wanted to share sometimes this year. It isn’t ready, it isn’t released, but if you are a motivated lisper I can add you to the GitHub repository and you can try the demo.

We all want more tools for easier and faster web development in CL I guess right? An automatic database admin dashboard is an important component for me, both for personal use and development purposes, but also for client-facing apps, at least at the beginning of a project.

What I started is based on the Mito ORM. You define your DB tables as usual, and then this package comes into play. Let’s call it mito-admin.

What’s more or less working so far is:

choose which tables to display (let’s say we have books, that can be in one shelf, and have many tags)
you get a welcome page. The tables are listed on the left side of the admin.
click on a table and see a list of records (with pagination, which module is already published as cosmo-pagination. See also lisp-pagination)
CRUD actions on records:
- create
- update
- delete
with CSRF protection
with form validation
with automatic handling of relations and custom HTML widgets
- a one-to-many gives you a select field
- (there’s a lot to do here)
a search bar
built-in login, user auth and rights
all this being customizable with CLOS fields and methods
light and dark themes thanks to Bulma CSS
for SQLite, Postgres, MySQL.

Table of Contents

There’s a demo in the project. Here’s how it works.

Models

Here are 3 Mito models: a book, a shelf, tags.

This is regular Mito and regular classes definitions (note the :metaclass option).

We are excluding the print-object methods for brevity.

(in-package :mito-admin-demo)

(defparameter *db-name* "db.db"
  "SQLite DB name.")

(defvar *db* nil
  "DB connection object.")

(defclass shelf ()
  ((name
    :initarg :shelf
    :accessor name
    :initform ""
    :col-type (or :null (:varchar 128))))
  (:metaclass mito:dao-table-class)
  (:documentation "Shelf: where is the book located.")
  (:unique-keys name))

(defclass tag ()
  ((name
    :initarg :name
    :accessor name
    :initform nil
    :col-type (or :null (:varchar 128))))
  (:metaclass mito:dao-table-class)
  (:documentation "A book can have many tags (categories).")
  (:unique-keys name))

;; necessary intermediate table for m2m.
(defclass book-tags ()
  ((book :references book)
   (tag :col-type tag))
  (:metaclass mito:dao-table-class))


(defclass book ()
  ((title
    :accessor title
    :initarg :title
    :initform nil
    :type string
    :col-type (:varchar 128))

   (title-ascii
     :accessor title-ascii
     :initform nil
     :col-type (:varchar 128))
     :documentation "Same title, only ascii. Processed after the book creation. Used for search and URI slugs.")

   (shelf
    :accessor shelf
    :initform nil
    :col-type (or :null shelf)
    :documentation "A card has only one shelf.")

   (tags
    ;; This column is even optional, we can do everything with the intermediate table
    ;; and a method that collects the tag for a book.
    ;; In fact, this slot won't be populated by Mito, we'll see "slot unbound".
    :accessor tags
    :initform nil
    :col-type (or :null book-tags)
    :documentation "A book can have many tags, aka categories.")

   (cover-url
    :accessor cover-url
    :initarg :cover-url
    :initform nil
    :type (or string null)
    :col-type (or (:varchar 1024) :null))

   (review
    :accessor review
    :initarg :review
    :initform nil
    :type (or string null)
    :col-type (or :text :null)
    :documentation "Let's write reviews about our favourite books."))

  (:metaclass mito:dao-table-class)
  (:documentation   "Book class, simplified. After modification of the DB schema, run (migrate-all)."))

To create the database, you’ll need a couple more Mito invocations. See its README or the Cookbook.

Select tables to show

We need to select the tables we’ll make available in the admin.

Override the mito-admin::tables method:

(defmethod mito-admin::tables ()
  '(
    book
    shelf
    tag
    ))

I deleted some inline comments, but there’s something to keep in mind. We need a list of all classes for Mito, so that it runs the migrations, and another one of a subset of classes for the admin. There’s only one list for now but that’s easy to fix.

Start the admin

Call mito-admin:connect, and bootstrap users and their base roles:

    ;; users and roles:
    (mito-admin-auth/v1::ensure-users-and-roles)

    ;; base roles.
    (mito-admin-auth/v1::bootstrap-base-roles)

Configure the admin

The first thing to do is to register an app name:

(mito-admin::register-app :cosmo-admin-demo)

which is here our lisp’s package. Working in a Lisp image allows to work on multiple (web) apps at the same time (at least with Hunchentoot). So, in a web library you’ll need a layer of indirection if you want to support this use case. I don’t for now, but registering the app’s *package* is necessary internally to resolve table names to fully-qualified symbols. As a user, you shouldn’t need to know all that, but as a tester you might.

You can override the render-slot method to change the default representation of a record’s slot. Here we turn a book’s cover-url from a string to an HTML anchor. This can be automated and abstracted. Another TODO.

(defmethod mito-admin::render-slot ((obj book) (slot (eql 'cover-url)))
  "A book cover URL must be a <a> tag with an href."
  (if (str:non-blank-string-p (mito-admin::slot-value? obj slot))
      (format nil "<a href=\"~a\"> ~a </a>" val val)
      ""))

Form handling

This is the most important, and unfinished, part of the admin.

We could maybe use cl-forms but I didn’t find that it maps well to this admin’s ABI. You might find it useful though, as it’s feature complete. It even has: client-side validation, sub-forms, Spinneret and Djula-based renderers, etc. Look at its demo.

We need to:

be able to include and exclude fields from the HTML forms
be able to choose a different HTML widget for a given field
validate forms
- with custom validation logic
- and nicely render errors
- or create or update the records.

We currently need to create forms explicitely:

(mito-admin:define-forms '(shelf tag))

or also

;; We can define forms manually.
;; (we can override default slots, but we can also override them by redefining the accessor methods)
(defclass book-form (mito-admin::form)
  ())

A form class has several slots we may redefine and use later, such as a list of validators, or the fields to use for the search, fields to exclude, etc.

We can exclude fields:

(defmethod mito-admin::exclude-fields (book-form)
  "Return a list of field names (symbols) to exclude from the creation form."
  '(title-ascii
    ;; TODO: handle relations
    ;; we need to exclude shelf, or we'll get an error on mito:insert-dao if the field is NIL.
    ;; shelf
    shelf-id
    tags-id
    ))

As you see, there are more TODOs here. title-ascii is a private field that we don’t need to expose. That’s fine. shelf-id and all are Mito’s references to the other tables. We need to recognize them and exclude them.

To define an HTML widget, override mito-admin::field-input. Our demo uses a built-in template to render a select field from a list of options.

TODOs: there’s a lot to do here. Our admin app still requires too much configuration, we want it to be more automatic. Recognize the input types better, ship an async select2 input for many-to-many relations.

Form validation

To validate a record, override the validators method. It returns a hash-table that, for all the table’s fields, associates a list of validators.

(defmethod mito-admin::validators ((obj (eql 'book)))
  "To validate a book:

  - its title should not be equal to \"test\".

  Beware that we override the method MITO-ADMIN::VALIDATORS."
  (serapeum:dict 'title (clavier:~= "test"
                                    "this title is too common, please change it!")))

We use the clavier library for this. Here’s a short blog post that shows a quick usage and our :allow-blank passthrough and validate-all function (which were not accepted upstream btw (shit it was nearly a year ago)).

Our form validation mechanism shows a global error message, and a specific one under each input field.

You didn’t write a single HTML line for this o/

One-to-many relations

Those are correctly handled by Mito and it’s easy to have a custom widget in the admin.

Many-to-many relations

See here: https://github.com/fukamachi/mito/discussions/161

At present, Mito doesn’t do much for m2m relations, but we can simply write a short method that will select all related objects of a record.

Here’s for a book’s tags:

(defmethod tags (obj)
  ;; null case
  nil)

(defmethod tags ((obj book))
  (let ((book-tags
            (mito:select-dao 'book-tags
                (sxql:where (:= :book-id
                                (mito:object-id obj))))))
    (when book-tags
     (mapcar #'tag book-tags))))

How to use it:

(tags (mito:find-dao 'book))
;; => (#<TAG tag1> #<TAG tag2>)

(tags (mito:find-dao 'book :id 2))
;; NIL

Simple enough, but that’s another TODO: automate the creation of such methods.

Closing words

This project has too many moving parts to my taste but I’ll get there.

I hate when people don’t release their useful library because “the code is meh” and yes, I’m doing that to you.

However I extracted parts from OpenBookStore (WIP) (notably the login, users auth and rights which was contributed by gnuxie), I published the little pagination module, contributed to a couple libraries and wrote blog posts. It’s already that for you.

You are at the very least helping as duck-brainstorming, so thanks. If you’d like to try the demo and look at horrible code, it should be doable.

I wonder under which license I’ll publish that.

There might be other ways for an admin panel and please try and cook us a plug-and-play solution. The proprietary Airtable with the new cl-airtable library or the open-source NocoDB (they give you a spreadsheet-like web UI + an API), or the lightweight Pocketbase, Mathesar or Supabase for Postgres (would you give this to your non-tech-savvy clients?), or NodeJS-based admin panels, etc, etc, etc. But a pure Common Lisp one? We’ll talk more about it in less than ten years, fingers crossed.

You can support E. Fukamachi for his work on Mito, cl-dbi, SxQL and all his other useful libraries.

Appendix: TODOs

View:

- [X] list of tables
- [-] list of records for each table
  - [X] see records
  - [X] pagination
  - [X] add "create" button
  - [ ] choose fields to display in search result
  - [ ] order records by field
- [X] a specific record
  - [X] view related column
  - [X] view some fields, ignore some fields

Search:

- [X] lax search on given fields
- [ ] more criteria

Create:

- [-] create a record
  - [X] ignore some fields
  - [X] choose related column
    - [X] select input for a to-1 relationship
    - [-] have a usable input widget for many-to-many relationships
      - [X] define a many-to-many in the demo: a book can have many tags.
      - [ ] define a default input widget (probably select2)
      - Mito doesn't help much with many-to-manys though.
  - [-] form handling
    - [X] form validation
      - [X] basics
    - [X] CSRF protection
      - in the create and edit forms.
  - [ ] form handling (cont)
    - [ ] subforms (create a new shelf in the card form)
    - [ ] client-side validation

Update:

- [X] update an existing record
  - [X] with same mechansim as create

Demo:

- [X] demo project with a couple tables and fields
  - [X] decouple the POC from openbookstore. Shit that wasn't that
    easy. <2024-03-21>
- [ ] display the app name instead of mito-admin (in title, header, footer).

Login:

- [-] admin user and rights
  - [X] base mechanism: users, roles, DB migrations (imported from
    OpenBookStore, again. Contributions by gnuxie). <2024-07-31>
  - [X] templates
  - [ ] add access rights to all the admin dashboard routes
  - [ ] add user logout dropdown in app

Actions:

- [ ] in the list of records, have actions: export to CSV, etc.


More:

- reference documentation website and tutorial
- unit tests
- include static assets (Bulma…) to work offline

i18n

- translations. See cl-gettext in OpenBookStore.

style:

- [X] dark mode (thanks Bulma 1.0)

CLOS tutorial: I published 9 videos (1h 22min) on my course. You'll know enough to read the sources of Hunchentoot or the Kandria game 🎥 ⭐

Fri, 20 Dec 2024 18:01:14 +0100

This is a follow-up from yesterday’s post on reddit and an announce I wanted to make since this summer: I created 9 videos on CLOS, for a total of 1 hour and 22 minutes, in which you learn what I detail below. You can watch the course and subscribe here (Christmas coupon) and learn more on GitHub. The whole course is made of 51 videos divided in 9 chapters, for a total of 7 hours and 12 minutes. It is rated 4.71 / 5 as of date (thank you!!).

Yesterday was a great day because I received nice feedback:

It is an amazing tutorial. What is really strange is I thought CLOS was complicated. I guess it can be but [Vincent] is amazing at explaining everything and demystifying it.

/u/intergalactic_llama

🔥 I appreciate any (constructive ;) ) feedback and positive ones a lot.

Oh hey you made that tutorial. I started it but then got distracted by other stuff, been meaning to restart it and make my way through the whole thing. Really liked what I went through (I was on video 12 about redefining functions locally etc).

/u/runevault

Look, other recent feedback on my course:

I have done some preliminary Common Lisp exploration prior to this course but had a lot of questions regarding practical use and development workflows. This course was amazing for this! I learned a lot of useful techniques for actually writing the code in Emacs, as well as conversational explanations of concepts that had previously confused me in text-heavy resources. Please keep up the good work and continue with this line of topics, it is well worth the price!

Preston, October 2024

The instructor shows lots of tricks.

Tom, November 2024

Excellent selection of content. The delivery is not always obvious just for watching, but when I do the examples, it’s absolutely clear that what I need to be learning has been presented.

Steven, November 2024 <3

Table of Contents

Chapter content

1. `defclass`, `make-instance`, `slots`… aka CLOS crash course, part 1. This one is free to watch 🆓

We see in more details: defclass, make-instance, attributes (aka slots), slot options (initarg, initform, reader, writer, accessor, documentation), slot-value, generic functions, defmethod, dispatching on built-in types, how objects are lazily updated, Slime inspector actions, manipulating Slime presentations, unbound slots and slot-boundp, Slime shortcuts to create objects…

We see a LOT already in this video, in an efficient way (way more efficient than when I learned anyways), so if you’re on a budget you can start with it (it’s free to watch) and complement with the Cookbook, and the other free books. Also if you are a student shoot me an email (and avoid the reddit chat, I don’t see the notifications, sorry about that).

1b. Quizz: CLOS crash test

There is a small quizz. Keep in mind that the Udemy plateform doesn’t support any Lisp language so I can’t put any live coding exercises, but we can read code.

2. Inheritance, multimethods, around, before and after methods… aka CLOS crash course, part 2

what we see more precisely: inheritance, multimethods, :around, :before and :after methods (think signals and overwriting default methods in other languages, that allow to control what happens when a method is called, if it is called at all), their order of execution, a Slime shortcut to export all symbols of a class at once…

3. Pretty printing

We see how to change the default printed representation of objects.

What we see: print-object, with print-unreadable-object, the object type, the object identity, classic gotchas.

You know, normally an object is printed un-readable as

#<ROBOT {1005CEBD03}>

(guess what AOC day I am at)

and we can use the print-object method to print it however we like, such as

#<ROBOT x: 47 y: 14 {1005CEBD03}>

4. `defclass` review

We give another pass, slower, to defclass, slot options, make-instance, and to the fact that accessors are generic functions.

You can skip this one if the crash course was crystal clear.

5. Custom constructors and custom logic.

What we see: writing our own “make-person” terse constructor. Adding some logic before the object creation, doing side-effects after the object creation: towards initialize-instance.

6. `initialize-instance`: control if and how any objects are created

What we see: defining a :before and an :after method of initialize-instance for our person class, in order to do the same logic than with our custom constructor, but with a built-in CL Object System mechanism. Note that using INITIALIZE-INSTANCE isn’t a must, only a “can”, that you can use for your own classes, or to control the creation of objects from other systems.

7. Multiple inheritance

What we see: how to inherit from multiple parent classes and who takes precedence, when the parents define the same slot with each a default value. Quick illustration. We use what is known as a mixin class to add functionality to our class.

8. `defgeneric` vs `defmethod`: when to use which, which is better?

What we see: the use of defgeneric and defmethod, either separately, either together. defgeneric has a couple advantages in regards to documentation and keeping your code in sync with your image.

9. Class allocation

What we see: the default :allocation :instance VS :allocation :class. How to automatically count how many objects of a class are created.

8b. Quizz: reading code from real-world projects.

Outcome of the chapter

There was a lot of choices to make and advanced topics to ignore for this first chapter on CLOS. What drove my choices was looking at real-world code out there. As a result, by the end of this chapter, you will know enough to read real-world Common Lisp projects such as the Hunchentoot web server or the Kandria game. Bravo!

Closing words

First of all, thank you for your encouragements, and to everyone who took the course or who shared it!

Today I’d like to answer to my past me, a newcomer to Lisp on a budget: why create a paying course? First of all, I still contribute to the Cookbook, a collaborative resource. It’s not “free or paid” resources, it’s both. Then, preparing and recording structured videos takes so much time that I wouldn’t do this continuous effort if I hadn’t the ambition to make a non-ridiculous hourly rate on them one day. Disclaimer: it isn’t the case yet. Maybe next year, depending on how many videos I release ;) I can pay my rent with them once every few months though, that’s cool. Rest assured I’m not a millionaire. I’m on my own projects and I don’t have a fixed (nor big) income. So your contribution or sponsorship counts, if only for the good vibes that push me to spend more and more time on my growing list of projects.

You can sponsor other lispers too.

Thank you and happy lisping.

Three web views for Common Lisp: build cross platform GUIs with Electron, WebUI or CLOG Frame

Tue, 10 Dec 2024 10:59:06 +0100

You dream to build a cross-platform GUI in Common Lisp? It’s now easy with web views.

Honestly GUIs are a difficult topic. Add in “cross platform” and you can spend your life trying out different solutions and hesitating between the best one for Common Lisp. It’s doable: Tk, Gtk3 and Gtk4, Qt4 and Qt5, CAPI (LispWorks), IUP, Nuklear, Cocoa, McCLIM, Garnet, Alloy, Java Swing… what can of worms do you want to open?

The situation improved in the last years thanks to lispers writing new bindings. So it’s possible you find one that works for your needs. That’s great, but now: you have to learn the GUI framework :p

If like me you already know the web, are developing a web app, and would like to ship a desktop application, web views are making it easy. I know the following ones, listed from least favourite to most favourite.

Table of Contents

Electron

Electron is heavy, but really cross-platform, and it has many tools around it. It allows to build releases for the three major OS from your development machine, its ecosystem has tools to handle updates, etc.

Advise: study it before discarding it.

Ceramic (old but works)

Ceramic is a set of utilities around Electron to help you build an Electron app: download the npm packages, open a browser window, etc.

Here’s its getting started snippet:

;; Start the underlying Electron process
(ceramic:start)
;; ^^^^^ this here downloads ±200MB of node packages under the hood.

;; Create a browser window
(defvar window (ceramic:make-window :url "https://www.google.com/"
                                    :width 800
                                    :height 600))

;; Show it
(ceramic:show window)

When you run (ceramic:bundle :ceramic-hello-world) you get a .tar file with your application, which you can distribute. Awesome!

But what if you don’t want to redirect to google.com but open your own app? You just build your web app in CL, run the webserver (Hunchentoot, Clack…) on a given port, and you’ll open localhost:[PORT] in Ceramic/Electron. That’s it.

Ceramic wasn’t updated in five years as of date and it downloads an outdated version of Electron by default (see (defparameter *electron-version* "5.0.2")), but you can change the version yourself.

The new Neomacs project, a structural editor and web browser, is a great modern example on how to use Ceramic. Give it a look and give it a try!

What Ceramic actually does is abstracted away in the CL functions, so I think it isn’t the best to start with. We can do without it to understand the full process, here’s how.

Ceramic API reference: http://ceramic.github.io/docs/api-reference.html

Electron from scratch

Here’s our web app embedded in Electron:

Our steps are the following:

follow the Electron installation instructions,
build a binary of your Lisp web app, including assets and HTML templates, if any.
- see this post: https://lisp-journey.gitlab.io/blog/lisp-for-the-web-build-standalone-binaries-foreign-libraries-templates-static-assets/ (the process will be a tad simpler without Djula templates)
bundle this binary into the final Electron build.
and that’s it.

You can also run the Lisp web app from sources, of course, without building a binary, but then you’ll have to ship all the lisp sources.

main.js

The most important file to an Electron app is the main.js. The one we show below does the following:

it starts Electron
it starts our web application on the side, as a child process, from a binary name, and a port.
it shows our child process’ stdout and stderr
it opens a browser window to show our app, running on localhost.
it handles the close event.

Here’s our version.

console.log(`Hello from Electron 👋`)

const { app, BrowserWindow } = require('electron')

const { spawn } = require('child_process');

// FIXME Suppose we have our app binary at the current directory.

// FIXME This is our hard-coded binary name.
var binaryPaths = [
    "./openbookstore",
];

// FIXME Define any arg required for the binary.
// This is very specific to the one I built for the example.
var binaryArgs = ["--web"];

const binaryapp = null;

const runLocalApp = () => {
    "Run our binary app locally."
    console.log("running our app locally…");
    const binaryapp = spawn(binaryPaths[0], binaryArgs);
    return binaryapp;
}

// Start an Electron window.

const createWindow = () => {
  const win = new BrowserWindow({
    width: 800,
    height: 600,
  })

  // Open localhost on the app's port.
  // TODO: we should read the port from an environment variable or a config file.
  // FIXME hard-coded PORT number.
  win.loadURL('http://localhost:4242/')
}

// Run our app.
let child = runLocalApp();

// We want to see stdout and stderr of the child process
// (to see our Lisp app output).
child.stdout.on('data', (data) => {
  console.log(`stdout:\n${data}`);
});

child.stderr.on('data', (data) => {
  console.error(`stderr: ${data}`);
});

child.on('error', (error) => {
  console.error(`error: ${error.message}`);
});

// Handle Electron close events.
child.on('close', (code) => {
  console.log(`openbookstore process exited with code ${code}`);
});

// Open it in Electron.
app.whenReady().then(() => {
    createWindow();

    // Open a window if none are open (macOS)
    if (process.platform == 'darwin') {
        app.on('activate', () => {
            if (BrowserWindow.getAllWindows().length === 0) createWindow()
        })
    }
})


// On Linux and Windows, quit the app main all windows are closed.
app.on('window-all-closed', () => {
    if (process.platform !== 'darwin') {
        app.quit();
    }
})

Run it with npm run start (you also have an appropriate packages.json), this gets you the previous screenshot.

JS and Electron experts, please criticize and build on it.

Missing parts

We didn’t fully finish the example: we need to automatically bundle the binary into the Electron release.

Then, if you want to communicate from the Lisp app to the Electron window, and the other way around, you’ll have to use the JavaScript layers. Ceramic might help here.

(this section was first released here: https://dev.to/vindarel/common-lisp-gui-with-electron-how-to-28fj)

What about Tauri?

Bundling an app with Tauri will, AFAIK (I just tried quickly), involve the same steps than with Electron. Tauri might still have less tools for it.

WebUI

WebUI is a new kid in town. It is in development, it has bugs. You can view it as a wrapper around a browser window (or webview.h).

However it is ligthweight, it is easy to build and we have Lisp bindings.

A few more words about it:

Use any web browser or WebView as GUI, with your preferred language in the backend and modern web technologies in the frontend, all in a lightweight portable library.

written in pure C
one header file
multi-platform & multi-browser
cross-platform webview
we can call JS from Common Lisp, and call Common Lisp from JS.

Think of WebUI like a WebView controller, but instead of embedding the WebView controller in your program, which makes the final program big in size, and non-portable as it needs the WebView runtimes. Instead, by using WebUI, you use a tiny static/dynamic library to run any installed web browser and use it as GUI, which makes your program small, fast, and portable. All it needs is a web browser.

your program will always run on all machines, as all it needs is an installed web browser.

Sounds compelling right?

The other good news is that Common Lisp was one of the first languages it got bindings for. How it happened: I was chating in Discord, mentioned WebUI and BAM! @garlic0x1 developed bindings:

https://github.com/garlic0x1/cl-webui/

thank you so much! (@garlic0x1 has more cool projects on GitHub you can browse. He’s also a contributor to Lem)

Here’s a simple snippet:

(defpackage :webui/examples/minimal
  (:use :cl :webui)
  (:export :run))
(in-package :webui/examples/minimal)

(defun run ()
  (let ((w (webui-new-window)))
    (webui-show w "<html>Hello, world!</html>")
    (webui-wait)))

I would be the happiest lisper in the world if I didn’t have an annoying issue. See #1. I can run my example just fine, but nothing happens the second time :/ I don’t know if it’s a WebUI thing, the bindings, my system, my build of WebUI… so I’ll give this more time.

Fortunately though, the third solution of this blog post is my favourite! o/

CLOG Frame (webview.h for all)

CLOG Frame is part of the CLOG framework. However, it is not tied to CLOG… nor to Common Lisp!

CLOG Frame is a short C++ program that builds an executable that takes an URL and a PORT as CLI parameters and opens a webview.h window.

It’s easy to build and works just fine.

It’s a great approach. We don’t need to develop CFFI bindings for webview.h. However such bindings would still be nice to have. I did a cursory search and didn’t find a project that seems to work. But please don’t take my word on it. Do you want to try this latest cl-webview, or have a go at the bindings?

Back to our matter.

This is CLOG Frame: 20 lines!

#include <iostream>
#include <sstream>
#include <string>
#include "webview.h"

int main(int argc,char* argv[]) {
  webview::webview w(true, nullptr);
  webview::webview *w2 = &w;
  w.set_title(argv[1]);
  w.set_size(std::stoi(argv[3]), std::stoi(argv[4]), WEBVIEW_HINT_NONE);
  w.bind("clogframe_quit", [w2](std::string s) -> std::string {
    w2->terminate();
    return "";
  });
  std::ostringstream o;
  o << "http://127.0.0.1:" << argv[2];
  w.navigate(o.str());
  w.run();
  return 0;
}

Compile it on GNU/Linux like this and don’t you worry, it takes a fraction of a second:

c++ clogframe.cpp -ldl `pkg-config --cflags --libs gtk+-3.0 webkit2gtk-4.0` -o clogframe

(see its repo for other platforms)

this gives you a clogframe binary. Put it in your $PATH or remember its location. It’s just a short C++ binary, so it weights 197Kb.

Now, back to your web app that you wrote in Common Lisp and that is waiting to be shipped to users.

Start your web app. Say it is started on port 4284.

From the Lisp side, open a CLOG Frame window like this

(uiop:launch-program (list "./clogframe"
                           "Admin"
                           (format nil "~A/admin/" 4284)
                           ;; window dimensions (strings)
                           "1280" "840"))

and voilà.

Now for the cross-platform part, you’ll need to build clogframe and your web app on the target OS (like with any CL app). Webview.h is cross-platform. Leave us a comment when you have a good CI setup for the three main OSes (I am studying 40ants/ci and make-common-lisp-program for now).

CLOG Frame should be a (popular) project on its own IMO! (@dbotton might make it independant eventually)

Please share any experience you might have on the topic 👍

cl-ansi-term: print tables with style, and other script utilities

Fri, 22 Nov 2024 11:56:40 +0100

I am not the original author of cl-ansi-term, but I revived it lately. In particular, I added useful stuff to print data in tables:

print list of lists (where the first one is the list of headers)
print horizontal or vertical tables
- the header keys are either the first row, either the first column
print hash-tables, plists, alists
filter keys to display (include, exclude)
limit the number of columns
they can be styled:
- with or without borders
- choose the columns’ width
- choose the borders’ elements (“-|+”)
- choose the headers’ and the cells’ style (color, bold…).

For example:

(progn
  (defparameter d (serapeum:dict :a 1.1 :b 2.2 :c 3.3))

  (banner "A single hash-table")
  (table d)

  (banner "A single hash-table, in columns")
  (vtable d)

  (banner "A single hash-table, ignoring column :B")
  (table d :exclude :b)

  (banner "A single hash-table, vertically ignoring column :B")
  (vtable d :exclude :b)

  (banner "A list of hash-tables")
  (table (list d d d))

  (banner "A list of hash-tables, ignoring column :B")
  (table (list d d d) :keys '(:a :c))

  (banner "A list of hash-tables, in columns")
  (vtable (list d d d))

  (banner "same, ignoring the column :b")
  (vtable (list d d d) :exclude :b))

prints:

--------------------------------------------------------------------------------
     A single hash-table
--------------------------------------------------------------------------------


+---------+---------+---------+
|A        |B        |C        |
+---------+---------+---------+
|1.1      |2.2      |3.3      |
+---------+---------+---------+

--------------------------------------------------------------------------------
     A single hash-table, in columns
--------------------------------------------------------------------------------


+---------+---------+
|A        |1.1      |
+---------+---------+
|B        |2.2      |
+---------+---------+
|C        |3.3      |
+---------+---------+

--------------------------------------------------------------------------------
     A single hash-table, ignoring column :B
--------------------------------------------------------------------------------


+---------+---------+
|A        |C        |
+---------+---------+
|1.1      |3.3      |
+---------+---------+

--------------------------------------------------------------------------------
     A single hash-table, vertically ignoring column :B
--------------------------------------------------------------------------------


+---------+---------+
|A        |1.1      |
+---------+---------+
|C        |3.3      |
+---------+---------+

--------------------------------------------------------------------------------
     A list of hash-tables
--------------------------------------------------------------------------------


+---------+---------+---------+
|A        |B        |C        |
+---------+---------+---------+
|1.1      |2.2      |3.3      |
+---------+---------+---------+
|1.1      |2.2      |3.3      |
+---------+---------+---------+
|1.1      |2.2      |3.3      |
+---------+---------+---------+

--------------------------------------------------------------------------------
     A list of hash-tables, ignoring column :B
--------------------------------------------------------------------------------


+---------+---------+
|A        |C        |
+---------+---------+
|1.1      |3.3      |
+---------+---------+
|1.1      |3.3      |
+---------+---------+
|1.1      |3.3      |
+---------+---------+

--------------------------------------------------------------------------------
     A list of hash-tables, in columns
--------------------------------------------------------------------------------


+---------+---------+---------+---------+
|A        |1.1      |1.1      |1.1      |
+---------+---------+---------+---------+
|B        |2.2      |2.2      |2.2      |
+---------+---------+---------+---------+
|C        |3.3      |3.3      |3.3      |
+---------+---------+---------+---------+

--------------------------------------------------------------------------------
     same, ignoring the column :b
--------------------------------------------------------------------------------


+---------+---------+---------+---------+
|A        |1.1      |1.1      |1.1      |
+---------+---------+---------+---------+
|C        |3.3      |3.3      |3.3      |
+---------+---------+---------+---------+

or again

TERM> (table (list d d d) :exclude :b  :border-style nil)
A         C
1.1       3.3
1.1       3.3
1.1       3.3

Real example

Remember, the scripts I use in production. I’m usually fine with big data output in the REPL, until:

until I want a cleaner output in the production script, so I can see quicker what’s going on.
when I want to filter and study the data a bit more.

In this case I extract data from my DB and I get a list of plists:

((:|isbn| "3760281971082" :|quantity| -1 :|price| 12.8d0 :|vat| NIL
  :|distributor| NIL :|discount| NIL :|type_name| NIL :|type_vat| NIL
  :|price_bought| NIL :|price_sold| 12.8d0 :|quantity_sold| 1 :|sold_date|
  "2024-04-03 09:27:12")
 (:|isbn| "9791094298169" :|quantity| 4 :|price| 15.0d0 :|vat| NIL
  :|distributor| NIL :|discount| NIL :|type_name| "book" :|type_vat| NIL
  :|price_bought| NIL :|price_sold| 15.0d0 :|quantity_sold| 1 :|sold_date|
  "2024-04-03 10:06:58")
 …)

With the table and vtable functions, I can explore data in a clearer fashion.

(uiop:add-package-local-nickname :sera :serapeum)
(term:table (sera:take 15 *sells*)
                          :keys '(:|isbn| :|quantity| :|price|)
                          :plist t
                          :column-width '(15 10 10))

+--------------+---------+---------+
|isbn          |quantity |price    |
+--------------+---------+---------+
|3760281971082 |-1       |12.8d0   |
+--------------+---------+---------+
|9791094298169 |4        |15.0d0   |
+--------------+---------+---------+
|3700275724249 |-126     |2.8d0    |
+--------------+---------+---------+
|9782372600842 |1        |10.0d0   |
+--------------+---------+---------+
|9782372600736 |0        |10.0d0   |
+--------------+---------+---------+
|9782221256770 |1        |19.0d0   |
+--------------+---------+---------+
|3700275734392 |171      |3.95d0   |
+--------------+---------+---------+
|3662846007789 |2        |16.95d0  |
+--------------+---------+---------+
|9782368292907 |1        |8.95d0   |
+--------------+---------+---------+
|9782095022679 |1        |12.95d0  |
+--------------+---------+---------+
|3662846007871 |5        |5.9d0    |
+--------------+---------+---------+
|9782092588949 |2        |5.95d0   |
+--------------+---------+---------+
|3700275724249 |-126     |2.8d0    |
+--------------+---------+---------+
|3700275734392 |171      |3.95d0   |
+--------------+---------+---------+
|3770017095135 |0        |29.99d0  |
+--------------+---------+---------+

Yes, this calls for more features: align the numbers, automatically adapt the cells’ width (DONE), style cells individually (DONE), etc.

(I’m sure we could have an explorer window, watching for changes, displaying data in a real table with interactive features… I can feel we’re close… CLOG frame and malleable systems someone?)

The use case is cleaner output for scripts.

Other libraries exist with other goals:

https://github.com/AccelerationNet/data-table “A Common Lisp data structure representing tabular data (think CSVs and database results)”
https://github.com/40ants/teddy “A data framework for Common Lisp”
https://github.com/telephil/cl-ascii-table/ “Common Lisp API to present tabular data in ASCII-art tables”

Here are some of other cl-ansi-term’s utilities:

ordered and un-ordered lists:

(term:o-list '((:one one-a (:one-b :one-b-1 :one-b-2)) :two))
1. ONE
   1. ONE-A
   2. ONE-B
      1. ONE-B-1
      2. ONE-B-2
2. TWO

Horizontal lines

(term:hr :filler "=")
================================================================================

printing stuff, align on screen:

(term:cat-print '(:abc :def :ghi) :align :center)
;; =>

                                   ABCDEFGHI

vspace for vertical space (default: 3 newlines)

banner:

(banner "My title" :space 1)

--------------------------------------------------------------------------------
     My title
--------------------------------------------------------------------------------

Stylesheets and colorized text

The library allows to use styles.

Start by defining your stylesheet.

(term:update-style-sheet
 '((:header :cyan   :underline)
   (:mark   :red    :reverse)
   (:term   :yellow :bold)))

:header, :mark and :term are now your own vocabulary. Anytime you use functions that accept a style, reference them.

Example:

(term:table (list '(:name :age) '(:me 7)) :header-style :header)

To see colors in a “dumb” terminal like in Emacs Slime, install the package slime-repl-ansi-color, “require” it and enable it ith M-x slime-repl-ansi-color-mode.

You can also disable styles in non-interactive terminals with term::*enable-effects-on-dumb-terminals*.

Happy lisping.

Running my 4th Common Lisp script in production© - you can do it too

Tue, 22 Oct 2024 19:19:26 +0200

Last week I finished a new service written in Common Lisp. It now runs in production© every mornings, and it expands the set of services I offer to clients.

It’s the 4th service of this kind that I developed: - they are not big - but have to be done nonetheless, and the quicker the better (they each amount to 1k to 2k lines of Lisp code), - they are not part of a super advanced domain that requires Common Lisp superpowers - I am the one who benefits from CL during development, - I could have written them in Python - and conversely nothing prevented me from writing them in Common Lisp.

So here lies the goal of this post: illustrate that you don’t need to need a super difficult problem to use Common Lisp. This has been asked many times, directly to me or on social media :)

At the same time, I want to encourage you to write a little something about how you use Common Lisp in the real world. Sharing creates emulation. Do it! If you don’t have a blog you can simply write in a new GitHub repository or in a Gist and come share on /r/lisp. We don’t care. Thanks <3

We’ll briefly see what my scripts do, what libraries I use, how I deploy them, what I did along the way.

Needless to say that I dogfooded my CIEL (beta) meta-library and scripting tool for all those projects.

Table of Contents

Scripts n°4 and 2 - shaping and sending data - when you can write Lisp on the side

My latest script needs to read data from a DB, format what’s necessary according to specifications, and send the result by SFTP.

In this case I read a DB that I own, created by a software that I develop and host. So I could have developed this script in the software itself, right? I could have, but I would have been tied to the main project’s versioning scheme, quirks, and deployment. I rather had to write this script on the side. And since it can be done on the side, it can be done in Common Lisp.

I have to extract products and their data (price, VAT…), aggregate the numbers for each day, write this to a file, according to a specification.

To read the DB, I used cl-dbi. I didn’t format the SQL with SxQL this time like in my web apps (where I use the Mito light ORM), but I wrote SQL directly. I’m spoiled by the Django ORM (which has its idiosyncrasies and shortcomings), so I double checked the different kinds of JOINs and all went well.

I had to group rows by some properties, so it was a great time to use serapeum:assort. I left you an example here: https://dev.to/vindarel/common-lisps-group-by-is-serapeumassort-32ma

Dates have to be handled in different formats. I used local-time of course, and I still greatly appreciate its lispy formatter syntax:

(defun date-yymmddhhnnss (&optional date stream)
  (local-time:format-timestring stream
                                (or date (local-time:now))
                                :format
                                '((:year 4)
                                  (:month 2)
                                  (:day 2)
                                  (:hour 2)
                                  (:min 2)
                                  (:sec 2)
                                  )))

the 2 in (:month 2) is to ensure the month is written with 2 digits.

Once the file is written, I have to send it to a SFTP server, with the client’s codes.

I wrote a profile class to encapsulate the client’s data as well as some functions to read the credentials from either environment variables, the file system, or a lisp variable. I had a top-level profile object for ease of testing, but I made sure that my functions formatting or sending data required a profile parameter.

(defun send-stock (profile &key date) …)
(defun write-stock (profile filename) …)

Still nothing surprising, but it’s tempting to only use global parameters for a one-off script. Except the program grows and you pay the mess later.

SFTP

To send the result through SFTP, I had to make a choice. The SFTP command line doesn’t make it possible to give a password as argument (or via an environment variable, etc). So I use lftp (in Debian repositories) that allows to do that. In the end, we format a command like this:

lftp sftp://user:****@host  -e "CD I/; put local-file.name; bye"

You can format the command string and run it with uiop:run-program: no problem, but I took the opportunity to release another utility:

https://github.com/vindarel/lftp-wrapper

First, you create a profile object. This one-liner reads the credentials from a lispy file:

(defvar profile (make-profile-from-plist (uiop:read-file-form "CREDS.lisp-expr"))

then you define the commands you’ll want to run:

(defvar command (put :cd "I/" :local-filename "data.csv"))
;; #<PUT cd: "I/", filename: "data.csv" {1007153883}>

and finally you call the run method on a profile and a command. Tada.

Deploying

Build a binary the classic way (it’s all on the Cookbook), send it to your server, run it.

(during a testing phase I have deployed “as a script”, from sources, which is a bit quicker to pull changes and try again on the server)

Set up a CRON job.

No Python virtual env to activate in the CRON environment…

Add command line arguments the easy way or with the library of your choice (I like Clingon).

Script n°2 and simple FTP

My script #2 at the time was similar and simpler. I extract the same products but only take their quantities, and I assemble lines like

EXTRACTION STOCK DU 11/04/2008
....978202019116600010000001387
....978270730656200040000000991

For this service, we have to send the file to a simple FTP server.

We have a pure Lisp library for FTP (and not SFTP) which works very well, cl-ftp.

It’s a typical example of an old library that didn’t receive any update in years and so that looks abandoned, that has seldom documentation but whose usage is easy to infer, and that does its job as requested.

For example we do this to send a file:

(ftp:with-ftp-connection (conn :hostname hostname
                                   :username username
                                   :password password
                                   :passive-ftp-p t)
      (ftp:store-file conn local-filename filename))

I left you notes about cl-ftp and my SFTP wrapper here:

https://dev.to/vindarel/ftp-and-sftp-clients-for-common-lisp-1c3b

Scripts n°3 and n°1 - specialized web apps

A recent web app that I’m testing with a couple clients extends an existing stock management system.

This one also was done in order to avoid a Python monolith. I still needed additions in the Python main software, but this little app can be independent and grow on its own. The app maintains its state and communicates it with a REST API.

It gives a web interface to their clients (so my clients’ clients, but not all of them, only the institutional) so that they can:

search for products
add them in shopping carts
validate the cart, which sends the data to the main software and notifies the owner, who will work on them.

The peculiarities of this app are that:

there is no user login, we use unique URLs with UUIDs in the form: http://command.client.com/admin-E9DFOO82-R2D2-007/list?id=1
I need a bit of file persistence but I didn’t want the rigidity of a database so I am using the clache library. Here also, not a great activity, but it works©. I persist lists and hash-tables. Now that the needs grow and the original scope doesn’t cut it any more, I wonder how long I’ll survive without a DB. Only for its short SQL queries VS lisp code to filter data.

I deploy a self-contained binary: code + html templates in the same binary (+ the implementation, the web server, the debugger…), with Systemd.

I wrote more on how to ship a standalone binary with templates and static assets with Djula templates here:

https://lisp-journey.gitlab.io/blog/lisp-for-the-web-build-standalone-binaries-foreign-libraries-templates-static-assets/

I can connect to the running app with a Swank server to check and set parameters, which is super helpful and harmless.

It is possible to reload the whole app from within itself and I did it with no hiccups for a couple years, but it isn’t necessary the most reliable, easiest to set up and fastest method. You can do it, but nobody forces you to do this because you are running CL in production. You can use the industry’s boring and best practices too. Common Lisp doesn’t inforce a “big ball of mud” approach. Develop locally, use Git, use a CI, deploy a binary…

Every thing that I learned I documented it along the way in the Cookbook ;)

Another app that I’ll mention but about which I also wrote earlier is my first web app. This one is open-source. It still runs :)

In this project I had my friend and colleague contribute five lines of Lisp code to add a theme switcher in the backend that would help him do the frontend. He had never written a line of Lisp before. Of course, he did so by looking at my existing code to learn the existing functions at hand, and he could do it because the project was easy to install and run.

(defun get-template(template &optional (theme *theme*))
  "Loads template from the base templates directory or from the given theme templates directory if it exists."
  (if (and (str:non-blank-string-p theme)
           (probe-file (asdf:system-relative-pathname "abstock" (str:concat "src/templates/themes/" theme "/" template))))
      ;; then
      (str:concat "themes/" theme "/" template)
      ;; else :D
      template))

He had to annotate the if branches :] This passed the code review.

Lasting words

The 5th script/app is already on the way, and the next ones are awaiting that I open their .docx specification files. This one was a bit harder but the Lisp side was done sucessfully with the efficient collaboration of another freelance lisper (Kevin to not name him).

All those tasks (read a DB, transform data…) are very mundane.

They are everywhere. They don’t always need supercharged web framework or integrations.

You have plenty of opportunities to make yourself a favor, and use Common Lisp in the wild. Not counting the super-advanced domains where Lisp excels at ;)

Links

https://lispcookbook.github.io/cl-cookbook/
awesome-cl
companies using Common Lisp in production (at least the ones we know)
Common Lisp course in videos – it helps me, and you ;) I added 9 videos about CLOS last month, and more are coming. It’s 86 minutes of an efficient code-first approach, out of 7+ hours of total content in the course. After this chapter you know enough to read the sources of the Hunchentoot web server or of the Kandria game.

I have done some preliminary Common Lisp exploration prior to this course but had a lot of questions regarding practical use and development workflows. This course was amazing for this! I learned a lot of useful techniques for actually writing the code in Emacs, as well as conversational explanations of concepts that had previously confused me in text-heavy resources. Please keep up the good work and continue with this line of topics, it is well worth the price! [Preston, October of 2024]

Common Lisp: equality functions explained (=, eq, equal, string= et all)

Fri, 23 Aug 2024 12:53:29 +0200

Common Lisp has various equality functions: =, eq, eql, equal, equalp, string-equal, char-equal… but what are the differences?? We tell you everything, with examples.

As usual, this is best read on the Common Lisp Cookbook (a new page added on August, 2024). This is where it will get the updates.

In short:

= is only for numbers and equal is the equality predicate that works on many things.
you can’t overload built-in operators such as = or equal for your own classes, unless you use a library.
when you manipulate strings with functional built-ins (remove-if, find…) and you are surprised to get no results, you probably forgot the :test key argument: (find "foo" '("hello" "foo") :test #'equal).

Table of Contents

`=` is for numbers (beware of `NIL`)

The = function compares the value of two or more numbers:

(= 2 2) ;; => T
(= 2 2.0 2 2) ;; => T
(= 2 4/2) ;; => T

(= 2 42) ;; => NIL

but = is only for numbers. In the below example we get an error with the interactive debugger. We show the error message, the condition type, and the backtrace, from SBCL.

(= 2 NIL)
;; => ERROR:
The value
  NIL
is not of type
  NUMBER
when binding SB-KERNEL::Y

   [Condition of type TYPE-ERROR]

Restarts:
  …

Backtrace:
  0: (SB-KERNEL:TWO-ARG-= 2 NIL) [external]
  1: (SB-VM::GENERIC-=)
  2: (= 2 NIL)

Note how SB-KERNEL::Y refers to an internal variable of the compiler. No, you don’t have a Y in your code.

As a consequence, if your equality check with numbers might contain NILs, you can use equalp, or encapsulate your variables with (or … 0), or do prior checks with (null …).

`eq` is low-level. Think pointers, position in memory.

(eq x y) is true if and only if x and y are the same identical object.

eq works for symbols and keywords.

Those are true:

(eq :a :a)
(eq 'a 'a)

If we compare an object with itself, it is eq:

(let ((x '(a . b)))
  (eq x x))
;; => T

eq does not work to compare numbers, lists, strings and other compound objects. It looks like it can, but it isn’t specified to be true for all implementations.

As such, eq works for numbers on my implementation, but it might not on yours:

(eq 2 2) ;; => T or NIL, this is not specified

An implementation might allocate the exact same position in memory for the same number, but it might not. This isn’t dictated by the standard.

Likewise, these might depend on the implementation:

(eq '(a . b) '(a . b)) ;; might be true or false.
(eq #\a #\a) ;; true or false

Comparing lists or strings are false:

(eq (list 'a) (list 'a)) ;; => NIL
(eq "a" "a") ;; => NIL

those strings (vectors of characters) are not equal by eq because the compiler might have created two different string objects in memory.

`eql` is a better `eq` also for numbers of same types and characters.

The eql predicate is true if its arguments are eq, or if they are numbers of the same type with the same value, or if they are character objects that represent the same character.

In terms of usefulness, we could say that eq < eql.

Now this number comparison is true:

(eql 3 3) ;; => T

but beware, this one isn’t because 3 and 3.0 are not of the same type (integer and single float):

(eql 3 3.0) ;; => NIL

for complex numbers:

(eql #c(3 -4) #c(3 -4)) ;; is true.
(eql #c(3 -4.0) #c(3 -4)) ;; is false (because of -4.0 and -4)

Comparing two characters works:

(eql #\A #\A) ;; => T

And we still can’t compare lists or cons cells:

(eql (cons 'a 'b) (cons 'a 'b)) ;; => NIL

`equal` is also for strings (for objects whose printed representation is similar).

The equal predicate is true if its arguments are structurally similar (isomorphic) objects. A rough rule of thumb is that two objects are equal if and only if their printed representations are the same.

Again, conceptually, we could say that eq < eql < equal.

We can still not compare numbers of different types:

(equal 3 3.0) ;; => NIL

but we can now compare lists and cons cells. Indeed, their printed representation is the same. No matter this time if they are different objects in memory.

(equal (cons 'a 'b) (cons 'a 'b)) ;; => T

(equal (list 'a) (list 'a)) ;; => T

We can compare strings!

(equal "Foo" "Foo") ;; => T

No matter if they are different objects in memory:

(equal "Foo" (copy-seq "Foo")) ;; => T

Case is important. Indeed, “FOO” doesn’t print the same as “foo”:

(equal "FOO" "foo") ;; => NIL

`equalp` is case-insensitive for strings and for numerical value of numbers.

Two objects are equalp if they are equal; if they are characters and satisfy char-equal, which ignores alphabetic case and certain other attributes of characters; if they are numbers and have the same numerical value, even if they are of different types; or if they have components that are all equalp.

Continuing with our ordering, we could say that eq < eql < equal < equalp.

We can compare two numbers, looking at their value, even if they have different types:

(equalp 3 3.0) ;; => T

Now look at our string comparison:

(equalp "FOO" "foo") ;; => T

equalp is case *in*sensitive for strings because a string is a sequence of characters, equalp compares all of its components and it uses char-equal for characters, which ignores the characters’ case.

Other comparison functions

`null`

The function null returns true if its one argument is NIL.

`eql` is used by default by many CL built-ins

This is a common issue for newcomers who manipulate strings. Sometimes, you use a CL built-in function and you are puzzled why you get no result.

Look at this:

(find "foo" (list "test" "foo" "bar"))
;; NIL

we want to know if the string “foo” exists in the given list. We get NIL. What’s happening?

This CL built-in function, as all that work for sequences, use eql for testing each elements. But (eql "foo" "foo") won’t work for strings. We need to use another test function.

All of those functions accept a :test keyword parameter, that allows you to change the test function:

(find "foo" (list "test" "foo" "bar") :test #'equal)
;; => "foo"

We can also use equalp to ignore the string case:

(find "FOO" (list "test" "foo" "bar") :test #'equalp)
;; => "foo"

You will find more examples about those built-in functions in data-structures.

`char-equal`

We have a special operator to compare characters:

char-equal ignores alphabetic case and certain other attributes of characters

strings and `string-equal`

string-equal has a specific function signature to compare strings and substrings (you can specify the start and end boundaries for the comparison), but be aware that it uses char-equal, so the comparison is case-*in*sensitive. And it works with symbols.

(string-equal :foo "foo") ;; => T
(string-equal :foo "FOO") ;; => T

This is its docstring:

STRING-EQUAL

This is a function in package COMMON-LISP

Signature
(string1 string2 &key (start1 0) end1 (start2 0) end2)

Given two strings (string1 and string2), and optional integers start1,
start2, end1 and end2, compares characters in string1 to characters in
string2 (using char-equal).

Compare trees with `tree-equal`

Here you have it:

tree-equal returns T if X and Y are isomorphic trees with identical leaves

Compare function table: to compare against (this), use (that) function

To compare against...      Use...

Objects/Structs            EQ

NIL                        EQ (but the function NULL is more concise and probably cheaper)

T                          EQ (or just the value but then you don't care for the type)

Precise numbers            EQL

Floats                     =

Characters                 EQL or CHAR-EQUAL

Lists, Conses, Sequences   EQ (if you want the exact same object)
                           EQUAL (if you just care about elements)

Strings                    EQUAL (case-sensitive), EQUALP (case-insensitive)
                           STRING-EQUAL (if you throw symbols into the mix)

Trees (lists of lists)     TREE-EQUAL (with appropriate :TEST argument)

How to compare your own objects AKA built-in functions are not object-oriented

Use eq to check that two objects are identical, that they are the same object in memory

If you want to compare your own objects with a logic of your own (for example, two “person” objects will be considered equal if they have the same name and surname), you can’t specialize a built-in function for this. Use your own person= or similar function, or use a library (see our links below).

While this can be seen as a limitation, not using generic functions has the advantage of being (much) faster.

As an example, let’s consider the person class from the CLOS tutorial:

(defclass person ()
  ((name
    :initarg :name
    :accessor name)))

Let’s create two person objects, they have the same name but are two different objects:

(defparameter *p1* (make-instance 'person :name "me"))
(defparameter *p2-same-name* (make-instance 'person :name "me"))

Use eq to compare two objects:

(eq *p1* *p1*) ;; => T
(eq *p1* *p2-same-name*) ;; => NIL

We use our own person= method to compare different objects and decide when they are equal:

(defmethod person= (p1 p2)
  (string= (name p1) (name p2)))

(person= *p1* *p2-same-name*)  ;; => T

If you really want to use = or equal, use a library, see below.

Credits

CLtL2: Equality Predicates
the compare table: Leslie P. Polzer on Stack-Overflow

Oh no, I started a Magit-like plugin for the Lem editor

Thu, 14 Mar 2024 13:11:34 +0100

Lem is an awesome project. It’s an editor buit in Common Lisp, ready to use out of the box for Common Lisp, that supports more languages and modes (Python, Rust, Elixir, Go, JavaScript, TypeScript, Haskell, Java, Nim, Dart, OCaml, Scala, Swift, shell, asm, but also markdown, ascii, JSON, HTML and CSS, SQL…) thanks to, in part, its built-in LSP support.

I took the challenge to add an interactive interface for Git, à la Magit, because you know, despite all its features (good vim mode, project-aware commands, grep, file tree view and directory mode, multiple cursors, tabs…), there’s so much an editor should do to be useful all day long.

Now, for a Git project (and to a lower extent, Fossil and Mercurial ones) we can see its status, stage changes, commit, push & pull, start an interactive rebase…

I like the shape it is taking, and frankly, what I have been able to assemble in a continuously successful hack session is a tribute to what @cxxxr and the early contributors built. Lem’s codebase is easily explorable (more so in Lem itself of course, think Emacs in steroïds with greater Common Lisp power), clear, and fun. Come to the Discord or watch the repository and see how new contributors easily add new features.

I didn’t even have to build an UI interface, fortunately. I started with the built-in interactive grep mode, and built from there.

Enough talk, what can we do with Lem/legit as of today? After that, we’ll discuss some implementation details.

Disclaimer: there’s room for collaboration ;)

Table of Contents

Lem/legit - manual

NOTE: you'd better read the latest manual on Lem's repository: https://github.com/lem-project/lem/blob/main/extensions/legit/README.md

legit’s main focus is to support Git operations, but it also has preliminary support for other VCSs (Fossil, Mercurial).

We can currently open a status window, stage and unstage files or diff hunks, commit our changes or again start an interactive rebase.

Its main source of inspiration is, obviously, Magit.

Status

legit is in development. It isn’t finished nor complete nor at feature parity with Magit nor suitable for mission-critical work. Use at your own risk.

However it should run a few operations smoothly.

Load

legit is built into Lem but it isn’t loaded by default. To load it, open a Lisp REPL (M-x start-lisp-repl) or evaluate Lisp code (M-:) and type:

(ql:quickload "lem/legit")

Now you can start it with C-x g or M-x legit-status.

Help

Press ? or C-x ? to call legit-help.

M-x legit-status

The status windows show us, on the left:

the current branch
the untracked files.
the unstaged changes, staged changes,
the latest commits.

It also warns us if a rebase is in process.

and the window on the right shows us the file diffs or the commits’ content.

Refresh the status content with g.

We can navigate inside legit windows with n, p, M-n and M-p (go to next/previous section).

To change windows, use the usual M-o key from Lem.

Quit with q or C-x 0 (zero).

Stage or unstage files, diff hunks (s, u)

Stage changes with “s”.

When your cursor is on an Unstaged change file, you can see the file changes on the right, and you can stage the whole file with s.

You can also go to the diff window on the right, navigate the diff hunks with n and p and stage a hunk with s.

Unstage a change with u.

Discard changes to a file

Use k. Be careful, you can loose your changes.

Commit

Pressing c opens a new buffer where you can write your commit message.

Validate with C-c C-c and quit with M-q (or C-c C-k).

Branches, push, pull

Checkout a branch with b b (“b” followed by another “b”).

Create a new branch with b c.

You can push to the current remote branch with P p and pull changes (fetch) with F p.

NOTE: after pressing "P" or "F", you will not see an intermediate window giving you choices. Just press "P p" one after the other.

Interactive rebase

You can start a Git interactive rebase. Place the cursor on a commit you want to rebase from, and press r i.

You will be dropped into the classical Git rebase file, that presents you commits and an action to apply on them: pick the commit, drop it, fixup, edit, reword, squash…

For example:

pick 26b3990f the following commit
pick 499ba39d some commit

# Commands:
# p, pick <commit> = use commit
# r, reword <commit> = use commit, but edit the commit message
# e, edit <commit> = use commit, but stop for amending
# s, squash <commit> = use commit, but meld into previous commit
# f, fixup <commit> = like "squash", but discard this commit's log message
# x, exec <command> = run command (the rest of the line) using shell
# b, break = stop here (continue rebase later with 'git rebase --continue')
# d, drop <commit> = remove commit
# l, label <label> = label current HEAD with a name
# t, reset <label> = reset HEAD to a label
# m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]
# .       create a merge commit using the original merge commit's
# .       message (or the oneline, if no original merge commit was
# .       specified). Use -c <commit> to reword the commit message.
#
# These lines can be re-ordered; they are executed from top to bottom.
#
# If you remove a line here THAT COMMIT WILL BE LOST.
#
# However, if you remove everything, the rebase will be aborted.
#
# Note that empty commits are commented out

legit binds keys to the rebase actions:

use p to “pick” the commit (the default)
f to fixup

and so on.

Validate anytime with C-c C-c and abort with C-c C-k.

NOTE: at the time of writing, "reword" and "edit" are not supported.

NOTE: the interactive rebase is currently Unix only. This is due to the short shell script we use to control the Git process. Come join us if you know how to "trap some-fn SIGTERM" for Windows plateforms.

Abort, continue, skip

In any legit window, type r a to abort a rebase process (if it was started by you inside Lem or by another process), r c to call git rebase --continue and r s to call git rebase --skip.

Fossil

We have basic Fossil support: see current branch, add change, commit.

Mercurial

We have basic Mercurial support.

Customization

In the lem/porcelain package:

*git-base-arglist*: the Git program, to be appended command-line options. Defaults to (list "git").

=> you can change the default call to the git binary.

Same with *fossil-base-args* and *hg-base-arglist* (oops, a name mismatch).

*nb-latest-commits*: defaults to 10
*branch-sort-by*: when listing branches, sort by this field name. Prefix with “-” to sort in descending order. Defaults to “-creatordate”, to list the latest used branches first.
*file-diff-args*: defaults to (list "diff" "--no-color"). Arguments to display the file diffs. Will be surrounded by the git binary and the file path. For staged files, –cached is added by the command.

If a project is managed by more than one VCS, legit takes the first VCS defined in *vcs-existence-order*:

(defvar *vcs-existence-order*
  '(
    git-project-p
    fossil-project-p
    hg-project-p
    ))

where these symbols are functions with no arguments that return two values: a truthy value if the current project is considered a Git/Fossil/Mercurial project, and a keyword representing the VCS: :git, :fossil, :hg.

For example:

(defun hg-project-p ()
  "Return t if we find a .hg/ directory in the current directory (which should be the project root. Use `lem/legit::with-current-project`)."
  (values (uiop:directory-exists-p ".hg")
          :hg))

Variables and parameters in the lem/legit package. They might not be exported.

*legit-verbose*: If non nil, print some logs on standard output (terminal) and create the hunk patch file on disk at (lem-home)/lem-hunk-latest.patch.

=> to help debugging

see sources in /extensions/legit/

Implementation details

Calls

Repository data is retrieved with calls to the VCS binary. We have a POC to read some data directly from the Git objects (proactively looking for best efficiency) using cl-git.

Basically, we get Git status data with git status --porcelain=v1. This outputs something like:

 A project/settings.lisp
 M project/api.lisp
?? project/search/datasources

we output this a to a string and we parse it.

Interactive rebase

The interactive rebase currently uses a Unix-only shell script.

When you run git rebase --interactive, the Git program creates a special file in .git/rebase-merge/git-rebase-merge-todo, opens it with your $EDITOR in the terminal, lets you edit it (change a “pick” to “fixup”, “reword” etc), and on exit it interprets the file and runs the required Git operations. What we want is to not use Git’s default program, edit the file with Lem and our special Legit mode that binds keys for quick actions (press “f” for “fixup” etc). So we bind the shell’s $EDITOR to a dummy editor, this shell script:

function ok {
    exit 0
}

trap ok SIGTERM
echo "dumbrebaseeditor_pid:$$"

while :
do
        sleep 0.1
done

This script doesn’t simulate an editor, it waits, so we can edit the rebase file with Lem, but this script catches a SIGTERM signal and exits successfully, so git-rebase is happy and terminates the rebase and all is well.

But that’s Unix only.

On that matter Magit seems to be doing black magic.

Displaying data, actionable links

The basic function to write content to a buffer is

(insert-string point s :read-only t)

And this is how you make actionable links:

(put-text-property start end :visit-file-function function))

where :visit-file-function is any keyword you want, and function is any lambda function you want. So, how to make any link useful? Create a lambda, make it close over any variables you want, “store” it in a link, and later on read the attribute at point with

(text-property-at point :visit-file-function)

where point can be (buffer-point (window-buffer *my-window*)) for instance.

Now create a mode, add keybindings and you’re ready to go.

;; Legit diff mode: view diffs.
;; We use the existing patch-mode and supercharge it with our keys.
(define-major-mode legit-diff-mode lem-patch-mode:patch-mode
    (:name "legit-diff"
     :syntax-table lem-patch-mode::*patch-syntax-table*
     :keymap *legit-diff-mode-keymap*)
  (setf (variable-value 'enable-syntax-highlight) t))

;; git commands.
;; Some are defined on peek-legit too.
(define-key *global-keymap* "C-x g" 'legit-status)

or a minor mode:

(define-minor-mode peek-legit-mode
    (:name "Peek"
     :keymap *peek-legit-keymap*)
  (setf (not-switchable-buffer-p (current-buffer)) t))

;; Git commands
;; Some are defined on legit.lisp for this keymap too.
(define-key *peek-legit-keymap* "s" 'peek-legit-stage-file)
(define-key *peek-legit-keymap* "u" 'peek-legit-unstage-file)
(define-key *peek-legit-keymap* "k" 'peek-legit-discard-file)

TODOs

Much needs to be done, if only to have a better discoverable UX.

First:

interactive rebase: support reword, edit.
show renamed files

and then:

visual submenu to pick subcommands
view log
stage only selected region (more precise than hunks)
unstage/stage/discard multiple files
many, many more commands, settings and switches
mouse context menus

Closing words

You’ll be surprised by all Lem’s features and how easy it is to add features.

I believe it doesn’t make much sense to “port Magit to Lem”. The UIs are different, the text displaying mechanism is different, etc. It’s faster to re-implement the required functionality, without the cruft. And look, I started, it’s possible.

But, sad me, I didn’t plan to be involved in yet another side project, as cool and motivating as it might be :S

Celebrating 1001 learners on my Common Lisp course 🥳

Wed, 14 Feb 2024 17:05:44 +0100

I just got 1001 learners on my Common Lisp course on Udemy. Thanks everybody for your support, here or elsewhere!

Starting with CL was honestly not easy. The first thing I did was writing the “data structures” page on the Cookbook, bewildered that it didn’t exist yet. A few years and a few projects later, this course allows me to share more, learn more, have fun, and have some rewards to keep the motivation up.

I know the course isn’t complete by any means, I want to add many chapters, both advanced topics but easier material for newcomers as well (beware, my course isn’t for total beginners in a Lisp language). The next one, and soon©, will be all about CLOS. In the meantime, I don’t abandon you, I enhance the Cookbook, I publish some videos on Youtube (last one: web development in Common Lisp, part 1 and part 2), I work on starter kits or on newcomer-friendly libraries ;)

Thanks again,

don’t hesitate to share the link with a friend or a colleague ;)

Here’s a link with a coupon until March, 13th 2024. (student? Get in touch for a free link)

more information on Github

Latest string manipulation functions in the STR library

Fri, 22 Dec 2023 16:52:51 +0100

We just released cl-str v0.21. It’s been a while since the last release, and many enhancements make it more useful than ever. Let’s review the changes, the newest first.

But first, I want to you thank everyone who contributed, by sending pull requests or feedback. Special thanks to @kilianmh who suddenly appeared one day, helped with new features as well as grunt work, and who is now a co-maintainer.

`split` by regex

The latest addition sent by ccQpein is that str:split now accepts a :regex key argument to split by regular expressions. The functions rsplit and split-omit-nulls have it too.

(str:split "[,|;]" "foo,bar;baz" :regex t)
;; => ("foo" "bar" "baz")

That’s handy for advent of code ;)

You can also use ppcre:split, this is the function that str:split relies on anyways, except that by default, str:split ensures that the split argument is not a regex. We need this:

(ppcre:split `(:sequence ,(string separator)) s …)

(and eventually, we remove null elements if :omit-nulls t was set)

replace by regex

str:replace-all, str:replace-first and str:replace-using got a :regex keyword too:

(str:replace-all "(?i)fo+" "frob" "FOO bar FOO" :regex t)
;; => "frob bar frob"

The `ensure` functions

These were added in March.

The “ensure-” functions return a string that has the specified prefix or suffix, appended if necessary.

ensure encapsulates the other two.

ensure-prefix, ensure-suffix `(start/end s)`

Ensure that s starts with start/end (or ends with start/end, respectively).

Return a new string with its prefix (or suffix) added, if necessary.

Example:

(str:ensure-prefix "/" "abc/") => "/abc/" (a prefix was added)
;; and
(str:ensure-prefix "/" "/abc/") => "/abc/" (does nothing)

We also have a couple functions to find the prefixes or the suffixes, please see our README.

ensure-wrapped-in `(start/end s)`

Ensure that s both starts and ends with start/end.

Return a new string with the necessary added bits, if required.

It simply calls str:ensure-suffix followed by str:ensure-prefix.

See also str:wrapped-in-p and uiop:string-enclosed-p prefix s suffix.

(str:ensure-wrapped-in "/" "abc") ;; => "/abc/"  (added both a prefix and a suffix)
(str:ensure-wrapped-in "/" "/abc/") ;; => "/abc/" (does nothing)

ensure `(s &key wrapped-in prefix suffix)`

This str:ensure function looks for the following key parameters, in order:

:wrapped-in: if non nil, call str:ensure-wrapped-in. This checks that s both starts and ends with the supplied string or character.
:prefix and :suffix: if both are supplied and non-nil, call str:ensure-suffix followed by str:ensure-prefix.
:prefix: call str:ensure-prefix
:suffix: call str:ensure-suffix.

Example:

(str:ensure "abc" :wrapped-in "/")  ;; => "/abc/"
(str:ensure "/abc" :prefix "/")  ;; => "/abc"  => no change, still one "/"
(str:ensure "/abc" :suffix "/")  ;; => "/abc/" => added a "/" suffix.

These functions accept strings and characters:

(str:ensure "/abc" :prefix #\/)

warn: if both :wrapped-in and :prefix (and/or :suffix) are supplied together, :wrapped-in takes precedence and :prefix (and/or :suffix) is ignored.

`:char-bag` parameter to trim, trim-left, trim-right

This was added in January.

str:trim removes all characters in char-bag (default: whitespaces) at the beginning and end of s.

If supplied, char-bag has to be a sequence (e.g. string or list of characters).

(str:trim "cdoooh" :char-bag (str:concat "c" "d" "h")) => "ooo"

fit a string to some length

This is older, it was added in February of 2022.

Fit this string to the given length:

if it’s too long, shorten it (showing the ellipsis),
if it’s too short, add paddding (to the side pad-side, adding the character pad-char).

As such, it accepts the same key arguments as str:shorten and str:pad: ellipsis, pad-side, pad-char…

CL-USER> (str:fit 10 "hello" :pad-char "+")
"hello+++++"

CL-USER> (str:fit 10 "hello world" :ellipsis "…")
"hello wor…"

If, like me, you want to print a list of data as a table, see:

cl-ansi-term

CL-USER> (ql:quickload "cl-ansi-term")
CL-USER> (term:table '(("name" "age" "email")
              ("me" 7 "some@blah")
              ("me" 7 "some@with-some-longer.email"))
             :column-width '(10 4 20))
+---------+---+-------------------+
|name     |age|email              |
+---------+---+-------------------+
|me       |7  |some@blah          |
+---------+---+-------------------+
|me       |7  |some@with-some-l(…)|
+---------+---+-------------------+

cl-ascii-table

CL-USER> (ql:quickload "cl-ascii-table")
CL-USER> (let ((table (ascii-table:make-table '("Id" "Name" "Amount") :header "Infos")))
  (ascii-table:add-row table '(1 "Bob" 150))
  (ascii-table:add-row table '(2 "Joe" 200))
  (ascii-table:add-separator table)
  (ascii-table:add-row table '("" "Total" 350))
  (ascii-table:display table))

.---------------------.
|        Infos        |
+----+-------+--------+
| Id | Name  | Amount |
+----+-------+--------+
|  1 | Bob   |    150 |
|  2 | Joe   |    200 |
+----+-------+--------+
|    | Total |    350 |
+----+-------+--------+
NIL

fixed `string-case`

The str:string-case macro was missing an implicit progn, so with more than one s-expression in the clauses, it didn’t fail… but it didn’t work as expected either.

Fixed for LispWorks

Characters are named differently, like #\NewLine. We are still awaiting input on one issue.

We reimplemented str:replace-using to fix it on LispWorks.

Misc

We added type declaration e.g. for concat, join.

str:ends-with-p now works with a character.

Small breaking change: fixed str:prefixp when used with a smaller prefix: “f” was not recognized as a prefix of “foobar” and “foobuz”, only “foo” was. Now it is fixed. Same for str:suffixp.

We added str:ascii-p and str:ascii-char-p (in 2021).

More functions now work with characters as well.

We sped up str:join (measured: 4x). We use with-output-to-string and a loop instead of format’s iteration directive.

We use uninterned symbols in defpackage.

We deprecated predicates ending with “?” (but they are still there).

We made casing-functions consistent to inbuilt cl casing functions (we use cl-change-case, but the functions also allow symbols and characters (not only strings) and return NIL when given NIL).

We added :ignore-case to str:count-substring.

We switched the testing framework from prove to fiveam (that was grunt work by the new maintainer yay o/ )

That’s it, thanks again for helping make this lil’ lib useful since day 1.

The “str” library defines many more functions. Look at our table of content on the README: https://github.com/vindarel/cl-str

Install it with

(ql:quickload "str")

Common Lisp on the web: enrich your stacktrace with request and session data

Fri, 13 Oct 2023 16:51:07 +0200

A short post to show the usefulness of Hunchentoot-errors and to thank Mariano again.

This library adds the current request and session data to your stacktrace, either in the REPL (base case) or in the browser.

TLDR;

Use it like this:

;; (ql:quickload "hunchentoot-errors)
;;
;; We also use easy-routes: (ql:quickload "easy-routes")

(defclass acceptor (easy-routes:easy-routes-acceptor hunchentoot-errors:errors-acceptor)
  ()
  (:documentation "Our Hunchentoot acceptor that uses easy-routes and hunchentoot-errors, for easier route definition and enhanced stacktraces with request and session data."))

then (make-instance 'acceptor :port 4242).

Base case

Imagine you have a bug in your route:

(easy-routes:defroute route-card-page ("/card/:slug" :method :GET :decorators ((@check-roles admin-role)))
    (&get debug)
  (error "oh no"))

When you access localhost:4242/card/100-common-lisp-recipes, you will see this in the REPL:

[2023-10-13 16:48:21 [ERROR]] oh no
Backtrace for: #<SB-THREAD:THREAD "hunchentoot-worker-127.0.0.1:53896" RUNNING {10019A21A3}>
0: (TRIVIAL-BACKTRACE:PRINT-BACKTRACE-TO-STREAM #<SB-IMPL::CHARACTER-STRING-OSTREAM {1006E9ED43}>)
1: (HUNCHENTOOT::GET-BACKTRACE)
2: ((FLET "H0" :IN HUNCHENTOOT:HANDLE-REQUEST) #<SIMPLE-ERROR "oh no" {1006E9EBE3}>)
3: (SB-KERNEL::%SIGNAL #<SIMPLE-ERROR "oh no" {1006E9EBE3}>)
4: (ERROR "oh no")
5: (MYWEBAPP/WEB::ROUTE-CARD-PAGE "100-common-lisp-recipes")
6: ((:METHOD HUNCHENTOOT:ACCEPTOR-DISPATCH-REQUEST (EASY-ROUTES:EASY-ROUTES-ACCEPTOR T)) #<MYWEBAPP/WEB::ACCEPTOR (host *, port 4242)> #<HUNCHENTOOT:REQUEST {1006C55F33}>) [fast-method]
7: ((:METHOD HUNCHENTOOT:HANDLE-REQUEST (HUNCHENTOOT:ACCEPTOR HUNCHENTOOT:REQUEST)) #<MYWEBAPP/WEB::ACCEPTOR (host *, port 4242)> #<HUNCHENTOOT:REQUEST {1006C55F33}>) [fast-method]
[…]

And, by default, you see a basic error message in the browser:

Show errors

Set this:

(setf hunchentoot:*show-lisp-errors-p* t)

Now you can see a backtrace in the browser window, which is of course super useful during development:

BTW, if you unset this one:

(setf hunchentoot:*show-lisp-backtraces-p* nil)  ;; t by default

You will see the error message, but not the backtrace:

And I remind you that if you set *catch-errors-p* to nil, you’ll get the debugger inside your IDE (Hunchentoot will not catch the errors, and will pass it to you).

Now with request and session data

Now create your server with our new acceptor, inheriting hunchentoot-errors.

You’ll see the current request and session paramaters both in the REPL:

[…]
19: (SB-THREAD::NEW-LISP-THREAD-TRAMPOLINE #<SB-THREAD:THREAD "hunchentoot-worker-127.0.0.1:48756" RUNNING {100AAAFC43}> NIL #<CLOSURE (LAMBDA NIL :IN BORDEAUX-THREADS::BINDING-DEFAULT-SPECIALS) {100AAAFBEB}> NIL)
20: ("foreign function: call_into_lisp")
21: ("foreign function: new_thread_trampoline")

HTTP REQUEST:
  uri: /card/100-common-lisp-recipes
  method: GET
  headers:
    HOST: localhost:4242
    USER-AGENT: Mozilla/5.0 (X11; Linux x86_64; rv:103.0) Gecko/20100101 Firefox/103.0
    ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
    ACCEPT-LANGUAGE: fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3
    ACCEPT-ENCODING: gzip, deflate, br
    DNT: 1
    CONNECTION: keep-alive
    COOKIE: "..."
    UPGRADE-INSECURE-REQUESTS: 1
    SEC-FETCH-DEST: document
    SEC-FETCH-MODE: navigate
    SEC-FETCH-SITE: none
    SEC-FETCH-USER: ?1

SESSION:
  :USER: #<MYWEBAPP.MODELS:USER {100EA8C753}>

127.0.0.1 - [2023-10-13 17:32:18] "GET /card/100-common-lisp-recipes HTTP/1.1" 500 5203 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:103.0) Gecko/20100101 Firefox/103.0"

and in the browser:

(notice the #<USER {…}> at the bottom? You’ll need a commit from today to see it, instead of # only)

Final words

These Hunchentoot variables were kinda explained on the Cookbook/web.html, I’ll augment that.

Clack users can use the clack-errors midleware.

Who wants to send a PR for colourful stacktraces?

I published 17 videos about Common Lisp macros - learn Lisp with a code-first tutorial 🎥 ⭐

Fri, 15 Sep 2023 17:07:23 +0200

For those who don’t know and who didn’t see the banner :D I am creating a Common Lisp course on the Udemy platform (with complementary videos on Youtube). I wanted to do something different and complementary than writing on the Cookbook.

I worked on new videos this summer and I just finished editing the subtitles. I have added 17 videos (worth 1h30+ of code-driven content) about Common Lisp macros!

We cover a lot of content: quote, backquote and comma, “,@”, comparison with C macros, comparison with functions, GENSYM and variable capture, useful patterns (call-with…), compile-time computing, read-time evaluation… (full summary below)

find the course here: https://www.udemy.com/course/common-lisp-programming/?couponCode=LISPMACROSPOWER (various videos are free to watch, so you can judge, and learn a couple things) (I can send free links to students, plz PM)

I recorded the last one, about the MACROSTEP tool, inside the Lem editor. It’s short, you should have a look at how this new editor looks like. (I’m very excited about it. Did I say I started develop a Magit-like plugin for it?)

Who is this course for?

The whole course is for beginners in Lisp, although not total beginners in programming. This chapter is, logically, a bit more difficult than the others. If you didn’t write small Common Lisp programs yet, be gentle with yourself and stop if you don’t understand. (you can ask questions in the Udemy forum, of course) In your case I would advise to watch the introductory one, the comparison with C macros, the video on QUOTE, the “functions VS macros” one, and then carry on at your rhythm. Be sure to work on the previous chapters before tackling this one.

Content

This is what we see on the topic of macros. For a full overview of the course, what I want to do next (if you subscribe now, you’ll get new content for the same price) and read others’ feedback, see its GitHub project page (there are six more chapters including getting started, functions, iteration, condition handling…).

Table of Contents

7.1 A quick intro (FREE PREVIEW)

Macros do not evaluate their arguments and expand to new code at compile time. What does that mean? A quick intro before diving deeper.

7.2. A comparison with C macros (FREE PREVIEW)

Lisp macros are NOT manipulating text, unlike C. Text leads to many unnecessary problems. We have a fun tour of a trivial need yet complicated issue in C that is easily done in Common Lisp.

7.3 QUOTE (FREE PREVIEW)

QUOTE does not evaluate its argument.

What we see: how to use QUOTE outside macros. Data takes the shape of code. We pair it with eval and we go full circle. We introduce the need to extrapolate values inside a quote.

7.4 Backquote and comma

What we see: how we extrapolate variable values. How they can help create data structures. Real world examples.

7.5 How to spot you are using a macro

Four tips to recognize if you are using a function or a macro, and why it matters.

7.6 Functions vs macros

Macros do NOT replace functions!

What we see: they are not higher-level functions. The subtle but logic need to re-compile functions using macros.

Introducing MACROEXPAND.

Keeping compile-time computing in mind (more on that later). A look at a function’s disassembly. So… you might not need a macro yet ;)

7.7 COMMA SPLICE ,@ the third most important macro mechanism

What we see: when use it, understanding the common error messages, passing body forms to our macro. Our first macro model.

7.8 &body and other macro parameters. Our second macro model.

What we see: how &body differs to &rest. Macro parameters: lots of possibilities, but some conventions carry meaning. Our own DOLIST macro. Our second macro model you can follow.

7.9 Putting this together: with-echo macro. Macroexpand in use.

We build our first macro with backquote and comma-splice, even a quote followed by a comma. We use macroexpand.

7.10 GENSYM -the simple fix to the most dangerous macros gotcha

What we see: what is variable capture and how to avoid it. Writing our own REPEAT macro. A little discussion about Common Lisp VS Scheme macros. GENSYM can be used outside macros too.

At this point you know enough to write all common macros. See the exercises for easy and not-so-easy ones.

7.11 CALL-WITH pattern: simplifying macros

We saw there can be subtle pitfalls when we write a macro. This pattern allows to offload most of the work to a function, which presents many advantages. We demo with our REPEAT macro.

7.12 Compile time computing

When writing macros, we have the full power of Common Lisp at compile time. This gives great tools to the developer: early type errors and warnings, faster runtime.

What we see: a simple example, writing a scientific macro for conversion of unit at compile time, existing libraries for that, introduction to dispatching macro characters and reader macros.

7.13 Lists VS AST

What we see: other languages don’t have macros but can manipulate Abstract Syntax Trees. Code as lists of symbols is not the same, we would need a third-party library to manipulate a Lisp AST proper. This doesn’t prevent us to develop crazy macros though, see this library adding Haskell-like type checking on top of Common Lisp, in pure CL macros.

7.14 Two example macros for compile-time computing

defstar allows to specify a function’s arguments’ types, Serapeum’s ecase-of does exhaustiveness type checking. At compile time, of course.

7.15 SYMBOL-MACRO

A symbol macro is not your everyday Lisp development tool, but it expands your toolbet. Again.

7.16 Read-time evaluation with #.

Macros occur at compile-time. But Common Lisp blurs the lines between read time, compile time and run time. This allows to execute code at READ time.

7.17 EDITOR TOOL: macrostep (FREE PREVIEW, Lem demo)

Macrostep is an editor extension that helps understand our macro expansions. It is only available in Sly and Lem. We demo with the Lem editor.

Thanks

Thanks for your support, it does make a difference (I am self employed, I don’t earn millions and I’d love to spend *even more time* on CL resources and projects). If you want to learn what I do for the Lisp community and why you should buy my course, read more on Github.

My complementary Lisp videos are on Youtube.

Don’t hesitate to share the link with a friend or a colleague :) Thanks, and happy lisping.

A demo about web development has been recorded and is coming.

ps: we just got a Dockerfile for CIEL, which is then easier to test, thanks to a “student” of my course. Thanks, @themarcelor. It will be on Dockerhub in due time.

The Udemy course by @vindarel is the best introductory material for a fast and practical intro to Common Lisp.

(thanks <3)

A wonderful course for someone with cursory knowledge of lisp. I’ve dipped my feet many times now, but always struggled to wrap my head around everything. This course really helped give me greater confidence in how to start a project. I really enjoyed the focus on having an executable early. The Lisp-2 reveal was beautiful and made me finally understand the difference. Thanks a lot!

Simon, August of 2023. (thanks <3 )

Pretty GUIs now: nodgui comes with a pre-installed nice looking theme

Thu, 01 Jun 2023 19:03:35 +0200

Being able to load a custom theme is great, but it would be even better if we didn’t have to manually install one.

Well, recent changes in nodgui from yesterday and today just dramatically improved the GUI situation for Common Lisp[0].

nodgui now ships the yaru theme

@cage commited the Yaru theme from ttkthemes in nodgui’s repository, and we added QoL improvements. To use it, now you can simply do:

(with-nodgui ()
  (use-theme "yaru")
  …)

(with-nodgui (:theme "yaru")
  …)

(setf nodgui:*default-theme* "yaru")
(with-nodgui ()
  …)

Yaru looks like this:

No, it isn’t native, but it doesn’t look like the 50s either.

See my previous post for more themes, screenshots and instructions to load a third-party theme. Forest Light is nice too!

Try the demos

Try the demos with this theme:

(setf nodgui:*default-theme* "yaru")
(nodgui.demo:demo)
;; or
(nodgui.demo:demo :theme "yaru")
;; a precise demo
(nodgui.demo::demo-widget :theme "yaru")

Themes directory

@cage also made it easier to load a theme.

I have added the special variable *themes-directory* (default is the directory themes under the directory where the asdf system is) where the library looks for themes.

Each theme must be placed in their own directory as a subdirectory of the aforementioned variable, the name of the directory must be the name of the theme; moreover the name of the TCL file that specify the file must be named as the same of the theme with the extension “tcl” appended

For example, the theme “foo” has to be: “foo/foo.tcl”

Provided these conditions are met using a new theme should be as simple as type (nodgui:use-theme "foo"), without (nodgui: eval-tcl-file).

Otherwise, just clone a theme repository somewhere, and call

(eval-tcl-file "path/to/the/theme.tcl")
(use-theme "theme")

I can very well imagine using small GUI tools built in Tk and this theme. I’ll have to try nogui’s auto-complete widget too. If you do build a little something, please share, it will help and inspire me and the ones after you.

https://peterlane.netlify.app/ltk-examples/
@cage announces new releases on Mastodon. @cage@stereophonic.space.

[0]: be more grandiose if you can.

Pretty GUI in Common Lisp with nodgui's Tk themes

Sat, 27 May 2023 00:22:04 +0200

Do you think Tcl/Tk GUIs are doomed to look outdated?

Fear not!

A treeview widget:

The official example of Forest Light:

The ttkthemes gallery

Plus, Tk itself has a little choice of built-in themes:

We can use these themes with nodgui, the Ltk fork.

In June of 2020, @cage added a little function to load a .tcl file:

(defun eval-tcl-file (file-path)
  "This function will feed the TCL interpreter with the contents
   of the file `path'.
   Please, as this function will load  and execute a script, ensure to
   load files only from trusted sources otherwise severe security problem
   may arise."
  (assert (stringp file-path))
  (format-wish "source {~a}" file-path))

As a consequence, we can load a .tcl script that defines a theme, and use it. Themes generally consist of a .tcl script and a directory of png or gif images (when images are not defined in-line).

Considering we cloned the ttkthemes repo locally:

  (with-nodgui ()
    (eval-tcl-file "ttkthemes/ttkthemes/png/yaru/yaru.tcl")
    (use-theme "yaru")

and that’s all there is to it.

For now, some themes are not supported. Scalable themes are not supported, the .gif based themes of ttkthemes won’t load (the “scid” and “smog” themes in ttkthemes, the Sun Valley theme didn’t work). This could change when tksvg lands in Debian (or maybe, if you install it yourself? I didn’t try), or with the next release of Tcl/Tk that will include SVG support (read #13).

Frankly, that was a great news of the day. Yes, I think some themes are pleasant to the eyes! This makes me want to use little Tk UIs here and there.

Here’s the code for the little media player of the screenshots. It is based on Peter Lane’s extensive examples.

Kuddos to @cage o/

i18n in my Lisp web app with Djula templates and gettext

Mon, 08 May 2023 14:01:34 +0200

I finally added translations to my Lisp web app \o/

I wanted to do it with gettext and Djula templates. There seemed to be some support for this, but it turned out… not straightforward. After two failed attempts, I decided to offer a little 90 USD bounty for the task (I announced it on the project’s issues and on Discord, watch them out for future bounties ;) ).

@fstamour took the challenge and is the person I’ll be eternally grateful for :D He kindly set up everything, answered my questions and traced down annoying bugs. BTW, I recommend you have a look at his ongoing breeze project (towards refactoring tools for CL) and local-gitlab.

Many thanks go as usual to @mmontone for incorporating changes to Djula after our feedback. Here’s Djula documentation:

https://mmontone.github.io/djula/djula/Internationalization.html#Backends

Djula’s gettext backend is based of the rotatef/gettext library. It worked fine. I left some feedback there anyways.

Why gettext

GNU gettext is the canonical tool to bring translations to software projects. Using it ensures we have access to its range of localization features and it unlocks the possibility to use modern web-based translation tools (like Weblate), according you have the pretention to have external translators for your project.

I looked at other Lisp libraries.

cl-i18n

an i18n library. Load translations from GNU gettext text or binary files or from its native format. Localisation helpers of plural forms.

It may ship improvements uppon gettext, but @fstamour ultimately chose gettext over it:

I ended up with so much less code with gettext than with cl-i18n and I found gettext’s code much easier to read if the documentation was lacking.

(BTW, @cage has been really helpful in answering many questions, hello o/ ) He explained:

Seems that the library you pointed out does not support any files but MO (binary) files. cl-18n can parse a couple more of formats like its own and include an extractor for translatable strings in source files, so can be used without any of the gettext toolchain. But they address the same problem in more or less the same way. :)

translate also is not gettext-compatible, it has a function to find missing translations, it got a Djula backend last April. Look, it is this easy to add a backend:

;; translation-translate.lisp
(in-package :djula)

(defmethod backend-translate ((backend (eql :translate)) string language &rest args)
  (apply #'translate:translate string language args))

cl-locale, a “Simple i18n library for Common Lisp”, works with hand-written dictionaries, it also is not gettext-compatible, it has a Djula backend but it has no tool to collect all the translatable strings.

Djula is a very nice templating library that works with HTML templates, much like Django templates. It has support for 2 translation backends, although I found it hard to start with. It should be better now, but you’re welcome to improve things further.

To translate a string in a template, we enclose it between {_ _} marks like so:

<p> {_ "Please login to continue" _} </p>

We will setup what’s necessary to collect those strings and handle them with gettext.

Extracting strings from .lisp source files

We need to extract strings from .lisp source files and from HTML templates.

xgettext already allows to collect strings for a lot of languages. It understands the Lisp syntax, we only need to tell it what is the marker used to mark strings to translate. We will use the underscore function, as it is the convention for many languages out there:

(_ "welcome")

We have to setup the gettext library:

(setf (gettext:textdomain) "bookshops")
;;                          ^^ a meaningful name for gettext's catalogue.

(gettext:setup-gettext #.*package* "bookshops")

This creates new functions under the hood in the current package:

(defmacro setup-gettext (package default-domain)
  (setf package (find-package package))
  (check-type default-domain string)
  `(progn
     (defun ,(intern "GETTEXT" package) (msgid &optional domain category locale)
       (gettext* msgid (or domain ,default-domain) category locale))
     (defun ,(intern "_" package) (msgid &optional domain category locale)
       (gettext* msgid (or domain ,default-domain) category locale))
     (defun ,(intern "NGETTEXT" package) (msgid1 msgid2 n &optional domain category locale)
       (ngettext* msgid1 msgid2 n (or domain ,default-domain) category locale))
     (defun ,(intern "N_" package) (msgid)
       msgid)
     (defun ,(intern "CATALOG-META" package) (&optional domain category locale)
       (catalog-meta* (or domain ,default-domain) category locale))))

So yes, it creates the _ function. It does this in a macro so that the function will populate our catalogue by default. You can now export it:

(defpackage :bookshops.i18n
  (:use :cl)
  (:import-from :gettext #:*current-locale*)
  (:export
   #:_
   #:n_
   #:*current-locale*
   #:list-loaded-locales
   #:set-locale
   #:with-locale
   #:update-djula.pot)
  (:documentation "Internationalisation utilities"))

We can now call xgettext:

xgettext --language=lisp --from-code=UTF-8 --keyword=_ --output=locale/ie.pot --sort-output

the --keyword argument (-K) tells it we are using the underscore. Hey, we also want to collect the N_ ones (for ngettext, it handles grammatical forms that depend on a number (typically, plurals)):

xgettext -k_ -kN_

OK, now we want to find all our .lisp sources and extract strings from them all. We’ll search them with a call to find . -iname "*.lisp" …. You have an example in Djula’s doc, here’s how we did (ahem, how Francis did) with a Makefile target:

# List .lisp files under our src/ directory, unless they contain a #
SRC := $(shell find src/ -name '*.lisp' -a ! -name '*\#*')
HTML := $(shell find src/ -name '*.html' -a ! -name '*\#*')
DEPS := $(SRC) $(HTML) bookshops.asd # and some more...

# list of supported locales
LOCALES := fr_fr
# Example of how the variable should look after adding a new locale:
# LOCALES := fr_FR en_GB

.PHONY: tr
tr: ${MO_FILES}

PO_TEMPLATE_DIR := locale/templates/LC_MESSAGES
PO_TEMPLATE := ${PO_TEMPLATE_DIR}/bookshops.pot

# Rule to extract translatable strings from SRC
${PO_TEMPLATE_DIR}/lisp.pot: $(SRC)
	mkdir -p $(@D)
	xgettext -k_ -kN_ --language=lisp -o $@ $^

# and then, come the rules to extract strings from HTML templates
# and build everything.

Extracting strings from HTML templates

Now, we need to fire a Lisp and call the Djula function that knows how to collect marked strings.

The Djula doc shows how to do it with djula:xgettext-templates:

sbcl --eval '(ql:quickload :my-project)'
     --eval '(djula::xgettext-templates
               :my-project-package
               (asdf:system-relative-pathname :my-project "i18n/xgettext.lisp"))'
     --quit

This function receives 2 arguments: your project package and the output file, where to store results. It stores them in a .lisp file in a regular gettext syntax, so this .lisp file is then read by a regular xgettext command (looking for _ strings), and this command ultimately creates the .pot file:

find src -iname "*.lisp" | xargs xgettext --from-code=UTF-8 --keyword=_ --output=i18n/my-project.pot --sort-output

We did it a bit differently with two other functions, in order to keep track of the source filename of each string (our source here):

#|
This could technically be just
(mapcan #'djula.locale:file-template-translate-strings
        (djula:list-asdf-system-templates "bookshops" "src/web/templates"))

But I (fstamour) made it just a bit more complex in order to keep track of the source (just the
filename) of each translatable strings. Hence why the hash-table returned is named `locations`.
|#
(defun extract-translate-strings ()
  "Extract all {_ ... _} string from the djula templates."
  (loop
    :with locations = (make-hash-table :test 'equal)
    :for path :in (djula:list-asdf-system-templates "bookshops" "src/web/templates")
    :for strings = (djula.locale:file-template-translate-strings path)
    :do (loop :for string :in strings
              :unless (gethash string locations)
                :do (setf (gethash string locations) path))
    :finally (return locations)))


(defun update-djula.pot ()
  "Update djula.pot from *.html files."
  (with-open-file (s (asdf:system-relative-pathname "bookshops" "locale/templates/LC_MESSAGES/djula.pot")
                     :direction :output
                     :if-exists :supersede
                     :if-does-not-exist :create)
    (let* ((locations (extract-translate-strings))
           (strings (alexandria:hash-table-keys locations)))
      (loop
        :for string :in strings
        :for location = (gethash string locations)
        :do
           (format s "~%#: ~a~%#, lisp-format~%msgid ~s~%msgstr \"\" ~%"
                   (enough-namestring location (asdf:system-relative-pathname "bookshops" ""))
                   string)))))

So this is our Makefile:

# Rule to extract translatable strings from djula templates
${PO_TEMPLATE_DIR}/djula.pot: $(HTML) src/i18n.lisp
	$(LISP) --non-interactive \
		--eval '(ql:quickload "deploy")' \
		--eval '(ql:quickload "cl+ssl")' \
		--eval '(asdf:load-asd (truename "bookshops.asd"))' \
		--eval '(push :djula-binary *features*)' \
		--eval '(ql:quickload :bookshops)' \
		--eval '(bookshops.i18n:update-djula.pot)'

# Rule to combine djula.pot and lisp.pot into bookshops.pot
${PO_TEMPLATE}: ${PO_TEMPLATE_DIR}/djula.pot ${PO_TEMPLATE_DIR}/lisp.pot
	msgcat --use-first $^ > $@

# Rule to generate or update the .po files from the .pot file
locale/%/LC_MESSAGES/bookshops.po: ${PO_TEMPLATE}
	mkdir -p $(@D)
	[ -f $@ ] || msginit --locale=$* \
          -i $< \
          -o $@ \
	&& msgmerge --update $@ $<

# Rule to create the .mo files from the .po files
locale/%/LC_MESSAGES/bookshops.mo: locale/%/LC_MESSAGES/bookshops.po
	mkdir -p $(@D)
	msgfmt -o $@ $<

Ultimately, this is the one make target we, as a developer, have to use:

.PHONY: tr
tr: ${MO_FILES}

Loading the translations

Once gettext is run and we added a couple translations, we have to load them inside our lisp app. We use gettext:preload-catalogs, as in:

;; Only preload the translations into the image if we're not deployed yet.
(unless (deploy:deployed-p)
  (format *debug-io* "~%Reading all *.mo files...")
  (gettext:preload-catalogs
   ;; Tell gettext where to find the .mo files
   #.(asdf:system-relative-pathname :bookshops "locale/")))

This is a top-level instruction, we want it to work on our machine during development or when building the binary (situations where asdf will find the required directories), but not when we run the binary (the location wanted by asdf would not exist on another machine), and we can do this with the help of Deploy (Cookbook recipe).

The gettext hash-table is saved into the binary, we correctly find our translations when we deploy it.

During development

Set the current locale:

(defun set-locale (locale)
  "Setf gettext:*current-locale* and djula:*current-language* if LOCALE seems valid."
  ;; It is valid to set the locale to nil.
  (when (and locale
             (not (member locale (list-loaded-locales)
                          :test 'string=)))
    (error "Locale not valid or not available: ~s" locale))
  (setf *current-locale* locale
        djula:*current-language* locale))


(defmacro with-locale ((locale) &body body)
  "Calls BODY with gettext:*current-locale* and djula:*current-language* set to LOCALE."
  `(let (*current-locale*
         djula:*current-language*)
     (set-locale ,locale)
     ,@body))

BOOKSHOPS> (djula:set-locale "fr_fr")   ;; <-- same as the ones declared in the Makefile

The change takes effect immediately.

However, run this when developping to reload the translations into gettext’s catalogue:

#+ (or)
(progn
  ;; Clear gettext's cache (it's a hash table)
  (clrhash gettext::*catalog-cache*)
  (gettext:preload-catalogs
   ;; Tell gettext where to find the .mo files
   #.(asdf:system-relative-pathname :bookshops "locale/")))

I’ll add a file watcher to automatically reload them later, when I work more with the system.

and:

;; Run this to see the list of loaded message for a specific locale
#+ (or)
(gettext::catalog-messages
 (gethash '("fr_fr" :LC_MESSAGES "bookshops")  ;; yes, a list for the HT key.
	  gettext::*catalog-cache*))

and:

;; Test the translation of a string
#+ (or)
(with-locale ("fr_fr")
  (_ "Please login to continue"))

Usage

From our readme:

make tr takes care of extracting the strings (generating .pot files) and generating or updating (with msgmerge) .po and .mo files for each locale. The .mo files are loaded in the lisp image at compile-time (or run-time, when developing the application).

How to add a new locale?

Add the new locale to the LOCALES variable in the makefile.
Call make tr. This will generate the .po file (and directory) for the new locale.

How to add a translation for an existing string?

Update the .po file for the locale.
1. Find the msgid that corresponds to the string you want to translate.
2. Fill the msgstr.
Call make tr to update the .mo file for the locale.

Another blog post I wish I had read a couple years ago o/

You are welcome to make everything even easier to use.

Happy lisping!

How to parse command line arguments in Common Lisp (bis)

Wed, 19 Apr 2023 11:44:56 +0200

In 2018, I wrote a blog post and the Cookbook page on how to build Common Lisp binaries, and how to parse command-line arguments with the unix-opts library.

But since then, new libraries were created an they are pretty good! They are simpler to use, and have much more features. I had a good experience with Clingon: its usage is clear, its documentation is very good, it is very flexible (it has hooks and generic functions waiting to have an :around method) and @dnaeon is not at his first great CL project.

You might give adopt a look, or maybe defmain though I felt a little something was missing.

So I updated the guide to use Clingon. Let’s go.

=> This article is best read on the Common Lisp Cookbook where it will receive updates.

As a reminder to this often-asked question, my SBCL standalone binaries, with dozens of dependencies (SBCL’s compiler and debugger (very useful to load code during the application’s lifecycle), a web server, static assets and other libraries) weight about 30MB and start in ±0.4s, with SBCL compression. Without compression, it’s more about 130MB and 0.01s.

Parsing command line arguments

SBCL stores the command line arguments into sb-ext:*posix-argv*.

But that variable name differs from implementations, so we want a way to handle the differences for us.

We have (uiop:command-line-arguments), shipped in ASDF and included in nearly all implementations. From anywhere in your code, you can simply check if a given string is present in this list:

(member "-h" (uiop:command-line-arguments) :test #'string-equal)

That’s good, but we also want to parse the arguments, have facilities to check short and long options, build a help message automatically, etc.

We chose the Clingon library, because it may have the richest feature set:

it handles subcommands,
it supports various kinds of options (flags, integers, booleans, counters, enums…),
it generates Bash and Zsh completion files as well as man pages,
it is extensible in many ways,
we can easily try it out on the REPL
etc

Let’s download it:

(ql:quickload "clingon")

As often, work happens in two phases:

we first declare the options that our application accepts, their kind (flag, string, integer…), their long and short names and the required ones.
we ask Clingon to parse the command-line options and run our app.

Declaring options

We want to represent a command-line tool with this possible usage:

$ myscript [-h, --help] [-n, --name NAME]

Ultimately, we need to create a Clingon command (with clingon:make-command) to represent our application. A command is composed of options and of a handler function, to do the logic.

So first, let’s create options. Clingon already handles “–help” for us, but not the short version. Here’s how we use clingon:make-option to create an option:

(clingon:make-option
 :flag                ;; <--- option kind. A "flag" does not expect a parameter on the CLI.
 :description "short help"
 ;; :long-name "help" ;; <--- long name, sans the "--" prefix, but here it's a duplicate.
 :short-name #\h      ;; <--- short name, a character
 ;; :required t       ;; <--- is this option always required? In our case, no.
 :key :help)          ;; <--- the internal reference to use with getopt, see later.

This is a flag: if “-h” is present on the command-line, the option’s value will be truthy, otherwise it will be falsy. A flag does not expect an argument, it’s here for itself.

Similar kind of options would be:

:boolean: that one expects an argument, which can be “true” or 1 to be truthy. Anything else is considered falsy.
:counter: a counter option counts how many times the option is provided on the command line. Typically, use it with -v / --verbose, so the user could use -vvv to have extra verbosity. In that case, the option value would be 3. When this option is not provided on the command line, Clingon sets its value to 0.

We’ll create a second option (“–name” or “-n” with a parameter) and we put everything in a litle function.

;; The naming with a "/" is just our convention.
(defun cli/options ()
  "Returns a list of options for our main command"
  (list
   (clingon:make-option
    :flag
    :description "short help."
    :short-name #\h
    :key :help)
   (clingon:make-option
    :string              ;; <--- string type: expects one parameter on the CLI.
    :description "Name to greet"
    :short-name #\n
    :long-name "name"
    :env-vars '("USER")     ;; <-- takes this default value if the env var exists.
    :initial-value "lisper" ;; <-- default value if nothing else is set.
    :key :name)))

The second option we created is of kind :string. This option expects one argument, which will be parsed as a string. There is also :integer, to parse the argument as an integer.

There are more option kinds of Clingon, which you will find on its good documentation: :choice, :enum, :list, :filepath, :switch and so on.

Top-level command

We have to tell Clingon about our top-level command. clingon:make-command accepts some descriptive fields, and two important ones:

:options is a list of Clingon options, each created with clingon:make-option
:handler is the function that will do the app’s logic.

And finally, we’ll use clingon:run in our main function (the entry point of our binary) to parse the command-line arguments, and apply our command’s logic. During development, we can also manually call clingon:parse-command-line to try things out.

Here’s a minimal command. We’ll define our handler function afterwards:

(defun cli/command ()
  "A command to say hello to someone"
  (clingon:make-command
   :name "hello"
   :description "say hello"
   :version "0.1.0"
   :authors '("John Doe <john.doe@example.org")
   :license "BSD 2-Clause"
   :options (cli/options) ;; <-- our options
   :handler #'null))  ;; <--  to change. See below.

At this point, we can already test things out on the REPL.

Testing options parsing on the REPL

Use clingon:parse-command-line: it wants a top-level command, and a list of command-line arguments (strings):

CL-USER> (clingon:parse-command-line (cli/command) '("-h" "-n" "me"))
#<CLINGON.COMMAND:COMMAND name=hello options=5 sub-commands=0>

It works!

We can even inspect this command object, we would see its properties (name, hooks, description, context…), its list of options, etc.

Let’s try again with an unknown option:

CL-USER> (clingon:parse-command-line (cli/command) '("-x"))
;; => debugger: Unknown option -x of kind SHORT

In that case, we are dropped into the interactive debugger, which says

Unknown option -x of kind SHORT
   [Condition of type CLINGON.CONDITIONS:UNKNOWN-OPTION]

and we are provided a few restarts:

Restarts:
 0: [DISCARD-OPTION] Discard the unknown option
 1: [TREAT-AS-ARGUMENT] Treat the unknown option as a free argument
 2: [SUPPLY-NEW-VALUE] Supply a new value to be parsed
 3: [RETRY] Retry SLIME REPL evaluation request.
 4: [*ABORT] Return to SLIME's top level.

which are very practical. If we needed, we could create an :around method for parse-command-line, handle Clingon’s conditions with handler-bind and use its restarts, to do something different with unknown options. But we don’t need that yet, if ever: we want our command-line parsing engine to warn us on invalid options.

Last but not least, we can see how Clingon prints our CLI tool’s usage information:

CL-USER> (clingon:print-usage (cli/command) t)
NAME:
  hello - say hello

USAGE:
  hello [options] [arguments ...]

OPTIONS:
      --help          display usage information and exit
      --version       display version and exit
  -h                  short help.
  -n, --name <VALUE>  Name to greet [default: lisper] [env: $USER]

AUTHORS:
  John Doe <john.doe@example.org

LICENSE:
  BSD 2-Clause

We can tweak the “USAGE” part with the :usage key parameter of the lop-level command.

Handling options

When the parsing of command-line arguments succeeds, we need to do something with them. We introduce two new Clingon functions:

clingon:getopt is used to get an option’s value by its :key
clingon:command-arguments gets use the free arguments remaining on the command-line.

Here’s how to use them:

CL-USER> (let ((command (clingon:parse-command-line (cli/command) '("-n" "you" "last"))))
           (format t "name is: ~a~&" (clingon:getopt command :name))
           (format t "free args are: ~s~&" (clingon:command-arguments command)))
name is: you
free args are: ("last")
NIL

It is with them that we will write the handler of our top-level command:

(defun cli/handler (cmd)
  "The handler function of our top-level command"
  (let ((free-args (clingon:command-arguments cmd))
        (name (clingon:getopt cmd :name)))  ;; <-- using the option's :key
    (format t "Hello, ~a!~%" name)
    (format t "You have provided ~a more free arguments~%" (length free-args))
    (format t "Bye!~%")))

We must tell our top-level command to use this handler:

;; from above:
(defun cli/command ()
  "A command to say hello to someone"
  (clingon:make-command
   ...
   :handler #'cli/handler))  ;; <-- changed.

We now only have to write the main entry point of our binary and we’re done.

By the way, clingon:getopt returns 3 values:

the option’s value
a boolean, indicating wether this option was provided on the command-line
the command which provided the option for this value.

Main entry point

This can be any function, but to use Clingon, use its run function:

(defun main ()
  "The main entrypoint of our CLI program"
  (clingon:run (cli/command)))

To use this main function as your binary entry point, see above how to build a Common Lisp binary. A reminder: set it in your .asd system declaration:

:entry-point "my-package::main"

And that’s about it. Congratulations, you can now properly parse command-line arguments!

Go check Clingon’s documentation, because there is much more to it: sub-commands, contexts, hooks, handling a C-c (see also the Cookbook for that), developing new options such as an email kind, Bash and Zsh completion…

Thanks for reading and thanks again to @dnaeon.

These Years in Common Lisp: 2022 in review

Mon, 09 Jan 2023 19:54:29 +0100

And 2022 is over. The Common Lisp language and environment are solid and stable, yet evolve. Implementations, go-to libraries, best practices, communities evolve. We don’t need a “State of the Ecosystem” every two weeks but still, what happened and what did you miss in 2022?

This is my pick of the most exciting, fascinating, interesting or just cool projects, tools, libraries and articles that popped-up during that time (with a few exceptions that appeared in late 2021).

This overview is not a “State of the CL ecosystem” (HN comments (133)) that I did in 2020, for which you can find complementary comments on HN.

I think this article (of sorts) is definitely helpful for onlookers to Common Lisp, but doesn’t provide the full “story” or “feel” of Common Lisp, and I want to offer to HN my own perspective.

And, suffice to say, I tried to talk about the most important things, but this article (of sorts) is by no means a compilation of all new CL projects or all the articles published on the internet. Look on Reddit, Quicklisp releases, Github, and my favourite resources:

Awesome-cl - a curated list of libraries (there might be more than you think)
the CL Cookbook

If I had to pick 3 achievements they would be:

SBCL developments: SBCL is now callable as a shared library. See below in “Implementations”.
a new 3D graphics project: Kons-9: “The idea would be to develop a system along the lines of Blender/Maya/Houdini, but oriented towards the strengths of Common Lisp”. And the project progresses at a good pace.
CLOG, the Common Lisp Omnificent GUI. It’s like a GUI framework to create web apps. Based on websockets, it offers a light abstraction to create fully-dynamic web applications, in Common Lisp. It has lots of demos to create websites, web apps, games, and it ships a complete editor. For development, we can connect our Lisp REPL to the browser, and see changes on the fly. The author had a similar commercial product written in Ada, discovered Common Lisp, and is now super active on this project.

Let’s go for more.

Thanks to @k1d77a, @Hexstream, @digikar and @stylewarning for their feedback.

Table of Contents

Documentation

A newcomer to Lisp came, asked a question, and suddenly he created a super useful rendering of the specification. Check it out!

🚀 Common Lisp CommunitySpec (CLCS) - a rendition of the Common Lisp ANSI Specification draft.
- Github: https://github.com/fonol/cl-community-spec
- It is readable, it has syntax highlighting for code snippets, an interactive search bar, it looks modern, it is open.

But that’s not all, he also started work on a new Common Lisp editor, built in Rust and Tauri, see below.

We continue to enrich the Common Lisp Cookbook. You are welcome to join, since documention is best built by newcomers for newcomers.

New CL Cookbook EPUB and PDF release - mainly readability and file format improvements - thanks to the 13 contributors

A resurrected project:

🚀 Learn Lisp the Hard Way is back online

Also:

the revamped Common Lisp Document Repository (CDR) site, “a repository of documents that are of interest to the Common Lisp community. The most important property of a CDR document is that it will never change”
- comments
I am creating a Common Lisp video course on Udemy 🎥
- read more about my motivation and follow the project on Github: https://github.com/vindarel/common-lisp-course-in-videos/
- the course has some free videos. If you are a student, drop me a line for a free link.
- direct link to the course.
- Thanks for your support!

Implementations

We saw achievements in at least 7 8 implementations.

SBCL continues to ship monthly. In 2022:
- 🚀 SBCL is now callable as a shared library. See sbcl-librarian below.
- 🚀 sb-simd - SIMD programming in SBCL.
- core compression uses zstd instead of zip: compression is about 4 times faster, decompression about two times, compression saves ±10% of size.
- trace now supports tracing macro functions, compiler-macro functions, individual methods and local functions (flet and labels) (SBCL 2.2.5)
- the SBCL repository reached 20,000 commits.
- Prebuilt SBCL binary for Android (Termux) (unofficial)
Clasp 2.0.0 released
- This is Common Lisp on LLVM.
- Christian Schafmeister talk - brief update about his “molecular lego” supported by his Lisp compiler
- there’s less funding than in the 80s, but still funding: “CLASP was supported by The Defense Threat Reduction Agency, The National Institutes of Health, The National Science Foundation”.
ECL:
- LQML: a lightweight ECL binding to QML (both Qt5 and Qt6) derived from EQL5
- tested on the following platforms: Linux, macOS, android, iOS.
- ECL targetting WASM via Emscripten - preliminary support
LispWorks Personal Edition updated to version 8.0.1, incl. native Apple Silicon version
- additional important LispWorks 8 patch for macOS 12.6
Major New release of Allegro Common Lisp Express Edition for 10.1
- Browser-based IDE for Linux and macOS
- no syntax highlighting for the editor though :S
- Applications built using Common Graphics can use browsers for delivery. Firefox, Chrome, Safari, Edge and many other browsers are supported.
- New platforms: aarch64, x86_64
- download: https://franz.com/downloads/clp/survey
ABCL 1.9.0 was released in May.
- “ABCL 1.9.0 has been best tested on the openjdk8, openjdk11, and openjdk17 runtimes. It will run other places but those are the best supported.”
GNU Common Lisp 2.6.13 released, after 8 years.

New implementation! It’s 2022 and people start new CL implementations.

NPT - an implementation of ANSI Common Lisp in C.
- comments

See also:

LCL, Lua Common Lisp - The goal of this project is to provide an implementation of Common Lisp that can be used wherever an unmodified Lua VM is running.
- not a complete implementation.

They are doing great work to revive a Lisp machine:

2022 Medley Interlisp Annual Report

Medley Interlisp is a project aiming to restore the Interlisp-D software environment of the Lisp Machines Xerox produced since the early 1980s, and rehost it on modern operating systems and computers. It’s unique in the retrocomputing space in that many of the original designers and implementors of major parts of the system are participating in the effort.

Paolo Amoroso blog post: my encounter with Medley Interlisp.

Jobs

I won’t list expired job announces, but this year Lispers could apply for jobs in: web development(WebCheckout, freelance announces), cloud service providers (Keepit), big-data analysis (Ravenpack, and chances are they are still hiring)), quantum computing (HLR Laboratories), AI (Mind AI, SRI International), real-time data aggregration and alerting engines for energy systems (3E); for a startup building autism tech (and using CLOG already); there have been a job seeking to rewrite a Python backend to Common Lisp (RIFFIT); there have been some bounties; etc.

Prior Lisp experience was not 100% necessary. There were openings for junior and senior levels, remote and not remote (Australia for “a big corp”, U.S., Spain, Ukraine…).

Comes a question:

Any Ideas for strategies to find jobs with lisp?

I remind the reader that most Lisp jobs do not have a public job posting, instead candidates are often found organically on the community channels: IRC, Twitter, Discord, Reddit… or teams simply train their new developer.

In 2022 we added a few companies to the ongoing, non-official list on awesome-lisp-companies. If your company uses Common Lisp, feel free to tell us on an issue or in the comments!

For example, Feetr.io “is entirely Lisp”.

Lisp was a conscious decision because it allows a small team to be incredibly productive, plus the fact that it’s a live image allows you to connect to it over the internet and poke and prod the current state, which has really allowed a much clearer understanding of the data.

They post SLY screenshots on their Twitter^^

Evacsound (HN):

We’re using CL in prod for an embedded system for some years now, fairly smooth sailing. It started out as an MVP/prototype so implementation was of no concern, then gained enough velocity and market interest that a rewrite was infeasible. We re-train talent on the job instead.

Pandorabots, or barefootnetworks, designing the Intel Tofino programmable switches, and more.

Projects

Ultralisp now supports tags. We can browse a list of projects under a tag.
- Ultralisp is a Quicklisp distribution that ships every five minutes.
- see also CLPM for a new package manager.

Language libraries

Typo: A portable type inference library for Common Lisp, by Marco Heisig.
Testing:
- fiveam-matchers - An extensible, composable matchers library for FiveAM.
- testiere - TDD-like dev for Common Lisp. Tests are included at the top of a defun/t form. When you recompile your functions interactively, the tests are run.
- melisgl/try test framework. - “it is what we get if we make tests functions and build a test framework on top of the condition system.”
- journal - A Common Lisp library for logging, tracing, testing and persistence
- 40ants-critic - a wrapper around LISP-CRITIC which provides a better interface to analyze ASDF systems and a command-line interface.
CLEDE - the Common Lisp Emacs Development Environment
- “The idea is to supply features that other language with and static analyzer have, like refactoring and code generation.”
easy-macros - An easy way to write 90% of your macros.
Quicklisp doctor - a program that examines the quicklisp installation.
more-cffi: Additional helper macros for the cffi project

Editors, online editors, REPLs, plugins

🚀 Parrot - A cross-platform Common Lisp editor.
- built in Rust with Tauri, CodeMirror, the Slynk server.
- “A hobby project started in Summer 2022. It aims to be an editor for Common Lisp (SBCL), that mostly works out of the box.”
- in development, untested on Linux and Mac.
Alive LSP for VSCode v0.1.9 · Add initial step debugger support
Common Lisp at Riju, a fast online playground for every programming language.
Codewars (code training platform) now has Common Lisp (SBCL 2.0.9)
Mobile app “cl-repl” (LQML) to replace “CL REPL” (EQL5)
☆ slime-star - SLIME configuration with some extensions pre-installed.
- a Lisp System Browser
- SLIME Doc Contribs
- Quicklisp Systems browsers
- Quicksearch utility
- Slime breakpoints - Inspect objects from their printed representation in output streams.
- custom utilities and menus.
parachute-browser: A lightweight UI for using the Parachute testing framework in LispWorks

New releases:

Lem editor 1.10.0: lsp-mode by default, multiple cursors, sql mode, and more.
- Lem is a general purpose editor written in Common Lisp. It works for many languages thanks to its LSP client.

Concurrency

🚀 New version of the Sento Actor Framework released with a few new goodies in future handling. Nicer syntax and futures can now be mapped.
- in v2.2.0: stashing and replay of messages.
- in v1.12: “Shutdown and stop of actor, actor context and actor system can now wait for a full shutdown/stop of all actors to really have a clean system shutdown.”
Writing distributed apps with cl-etcd

See also lisp-actors, which also does networking. It looks like more of a research project, as it doesn’t have unit-tests nor documentation, but it was used for the (stopped) Emotiq blockchain.

Discussions:

Databases

More choices: awesome-cl#databases.

Delivery tools

There has been outstanding work done there. It is also great to see the different entities working on this. That includes SBCL developers, Doug Katzman of Google, and people at HRL Laboratories (also responsible of Coalton, Haskell-like on top of CL).

Have you ever wanted to call into your Lisp library from C? Have you ever written your nice scientific application in Lisp, only to be requested by people to rewrite it in Python, so they can use its functionality? Or, maybe you’ve written an RPC or pipes library to coordinate different programming languages, running things in different processes and passing messages around to simulate foreign function calls.

[…] If you prefer using SBCL, you can now join in on the cross-language programming frenzy too.

🎉 sbcl-librarian - An opinionated interface for creating C- and Python-compatible shared libraries in Common Lisp with SBCL. Requires SBCL version >2.1.10.
- introductory blogpost: Using Lisp libraries from other programming languages - now with SBCL.
- HN comments (67)
🚀 alien-works-delivery - WIP system for delivering Common Lisp applications as executable bundles. For now it only supports AppImage format for Linux and MSIX for Windows, but .APK for Android and later MacOSX and iOS bundle formats are planned too.
Support for compiling Common Lisp code using bazel.io, by the CL team at Google.

Games

Kandria launches on Steam on the 11th of January, in two days!

🎥 Kandria trailer.

Graphics, GUIs

We saw the release of fresh bindings for Gtk4.

🚀 Writing beautiful GUI with Common Lisp and GTK4

We had bindings for Qt5… but they are still very rough, hard to install so far.

CommonQt 5 - Qt5 bindings.

Also:

I made a Wayland client from scratch for Common Lisp
vk - CFFI bindings to Vulkan

History:

Izware Mirai is available on the Internet Archive (no source code)
- Hacking Lisp on Mirai (screencast)

But an awesome novelty of 2022 is Kons-9.

Kons-9, a new 3D graphics project

🚀 A new 3D graphics project: Kons-9.

reddit announce on August
- HN comments (80)

The idea would be to develop a system along the lines of Blender/Maya/Houdini, but oriented towards the strengths of Common Lisp.

I’m an old-time 3D developer who has worked in CL on and off for many years.

I don’t consider myself an expert […] A little about me: • wrote 3D animation software used in “Jurassic Park” • software R&D lead on “Final Fantasy: The Spirits Within” movie • senior software developer on “The Hobbit” films.

kons-9, a Common Lisp 3D Graphics Project
🎥 trailer
author’s blog posts
and see also his screencasts below.

Interfaces with other languages

py4cl2-cffi: CFFI based alternative to py4cl2.
- it does one big new thing: it supports passing CL arrays by reference. That means we actually have access to numpy, scipy, and friends.
- “If py4cl2-cffi reaches stability, and I find that the performance of (i) cffi-numpy, (ii) magicl, as well as (iii) a few BLAS functions I have handcrafted for numericals turn out to be comparable, I might no longer have to reinvent numpy.” @digikar
Small update to RDNZL (CL .NET bridge by Edi Weitz)
- forked project, added support for Int16, fixed Int64, re-building the supporting DLLs.
- see also: Bike
jclass: Common Lisp library for Java class file manipulation

For more, see awesome-cl.

Numerical and scientific

🚀 new Lisp Stats release
- “emphasis on plotting and polishing of sharp edges. data-frames, array operations, documentation.”
- HN comments (55)
- ” I’ve been using lisp-stat in production as part of an algorithmic trading application I wrote. It’s been very solid, and though the plotting is (perhaps was, in light of this new release) kinda unwieldy, I really enjoyed using it. Excited to check out the newest release.”
- “For example, within Lisp-Stat the statistics routines [1] were written by an econometrician working for the Austrian government (Julia folks might know him - Tamas Papp). It would not be exaggerating to say his job depending on it. These are state of the art, high performance algorithms, equal to anything available in R or Python. So, if you’re doing econometrics, or something related, everything you need is already there in the tin.”
- “For machine learning, there’s CLML, developed by NTT. This is the largest telco in Japan, equivalent to ATT in the USA. As well, there is MGL, used to win the Higgs Boson challenge a few years back. Both actively maintained.”
- “For linear algebra, MagicCL was mention elsewhere in the thread. My favourite is MGL-MAT, also by the author of MGL. This supports both BLAS and CUBLAS (CUDA for GPUs) for solutions.”
- “Finally, there’s the XLISP-STAT archive. Prior to Luke Tierney, the author of XLISP-Stat joining the core R team, XLISP-STAT was the dominate statistical computing platform. There’s heaps of stuff in the archive, most at least as good as what’s in base R, that could be ported to Lisp-Stat.”
- “Common Lisp is a viable platform for statistics and machine learning. It isn’t (yet) quite as well organised as R or Python, but it’s all there.”
numericals - Performance of NumPy with the goodness of Common Lisp
- numericals is “a high performance basic math library with vectorized elementary and transcendental functions. It also provides support for a numpy-like array object through dense-arrays and static-dispatch through the CLTL2 API.”
- the post is about numericals, dense-arrays, magicl, numcl, py4cl/2…
- the author’s comparison and wish-list of features for a Common Lispy approach to a (better) Numpy
MGL-MAT - a library for working with multi-dimensional arrays which supports efficient interfacing to foreign and CUDA code. BLAS and CUBLAS bindings are available.
hbook - Text-based histograms in Common Lisp inspired by the venerable HBOOK histogramming library from CERN.

New releases:

Maxima 5.46 was released.
- “Maxima is a Computer Algebra System comparable to commercial systems like Mathematica and Maple. It emphasizes symbolic mathematical computation: algebra, trigonometry, calculus, and much more.”
- see its frontends, for example WxMaxima.

Call to action:

📢 Uncle Stats Wants You

Web

Screenshotbot (Github) was released. It is “a screenshot testing service to tie with your existing Android, iOS and Web screenshot tests”.

It is straightforward to install with a Docker command. They offer more features and support with their paid service.

LicensePrompt was released. It is “a single place to track all recurring software and IT expenses and send relevant reminders to all interested people”. It’s built in CL, interface with HTMX.

Libraries:

jingle: Common Lisp web framework with bells and whistles (based on ningle)
- jingle demo: OpenAPI 3.x spec, Swagger UI, Docker and command-line interface app with jingle.
ciao: Ciao is an easy-to-use Common Lisp OAuth 2.0 client library. It is a port of the Racket OAuth 2.0 Client to Common Lisp.
stepster: a web scraping library, on top of Plump and Clss (new in QL)
openrpc: Automatic OpenRPC spec generation, automatic JSON-RPC client building
HTTP/2 implementation in Common Lisp

Skeletons:

cl-cookieweb: my project skeleton to start web projects. Demo in video. I am cheating, the bulk of it was done in 2021.
- “Provides a working toy web app with the Hunchentoot web server, easy-routes, Djula templates, styled with Bulma, based on SQLite, with migrations and an example table definition.”
- if you don’t know where to start for web dev in CL, enjoy all the pointers of this starter kit and find your best setup.
- see also this web template by @dnaeon, and check out all his other Lisp libraries.

Bindings:

👍 lisp-pay: Wrappers around various Payment Processors (Paypal, Stripe, Coinpayment)
lunamech-matrix-api: Implementation of the Matrix API, LunaMech a Matrix bot

Apps:

Ackfock - a platform of mini agreements and mini memos of understanding (built with CLOG, closed source).
todolist-cl: a nice looking todolist with a web UI, written in Common Lisp (and by a newcomer to CL, to add credit)

I don’t have lots of open-source apps to show. Mines are running in production and all is going well. I share everything on my blog posts. I also have an open-source one in development, but that’s for the 2023 showcase :D

CLOG

🚀 The awesome novelty of 2022 I spoke of in the introduction is CLOG, the Common Lisp Omnificent GUI:

I know of one open-source consequent CLOG app: mold-desktop, in development.

I’m developing a programmable desktop and a bookmarks manager application with CLOG. I think I know one of the things that make CLOG user interfaces so easy to develop. It is that they are effortlessly composable. That’s it for now :)

@mmontone

New releases

There are lots of awesome projects in music composition, including OpusModus and OpenMusic which saw new releases. I also like to cite ScoreCloud, a mobile app built with LispWorks, where you whistle, sing or play your instrument, and the app writes the music score O_o

🎵 Opusmodus 3.0, Music Composition System, macOS Intel & Apple Silicon native, based on LispWorks, just released
🎵 OpenMusic 7.0, now also native for M1 Macs, visual programming language designed for music composition
Consfigurator, a Common Lisp based declarative configuration management system, reaches v1.0
April 1.0 released - APL in Common Lisp.
cl-cmark approaching stable release
cl-git: a Common Lisp CFFI interface to the libgit2 library
tinmop 0.9.9.1, a terminal client for gopher, gemini, kami and pleroma.
- look here for how to build full-screen terminal applications in Common Lisp.
clingon - new release, new features (command-line options parser)
- a full-featured options parser. Supports sub-commands, bash and zsh completions, many arguments types…

See awesome-cl and Cliki for more.

(re) discoveries

The math pastebin Mathb.in is written in Common Lisp
TIL that PTC’s Creo Elements/Direct 3D CAD modeling software has a free version for Windows. “7+ million lines of Common Lisp code”, used by Eterna for their watches.
- that’s a huge software. 7 million lines. There’s a free version to try out!

Articles

Graphics

Tooling

Debugging Lisp: trace options, break on conditions
- HN comments (28), I learned, again, new tips.
Developing Common Lisp using GNU Screen, Rlwrap, and Vim
- again, learned new tips. The --remember flag of rlwrap allows to TAB-complete whatever was previously typed at the prompt. That’s dumb autocompletion, but autocompletion nonetheless. My summary.
Configuring Slime cross-referencing
SLIME Compilation Tips
How to save lisp and die from Slime or Sly. trivial-dump-core
- that may be super useful, at least if answers a question everybody has one time or another. I should try it more.
How to approach a Lisp sandbox environment
log4cl questions
A StumpWM debugger
Windows environment for SBCL
Securing Quicklisp through mitmproxy
- because Quicklisp doesn’t use HTTPS. Here’s how to add security. The new CLPM uses HTTPS.
Lisp in Vim

Scripting

Day 3: Roswell: Common Lisp scripting, by E. Fukamachi.
Qlot tutorial with Docker
- Qlot v1.0.0 was officially released.
Day 4: Roswell: How to make Roswell scripts faster
Day 5: Roswell: Hidden feature of “-s” option
Writing scripts in lisp?
- a legitimate question. Many tips.
- where I show my very new and un-released CIEL scripting facility. Batteries included for Common Lisp. That’s for 2023 but you can be an early adopter :p
Lisp for Unix-like systems
- in particular, read this answer by /u/lispm to learn about past and present attempts and solutions.
- will the OP manage to make WCL work? “WCL: Delivering efficient Common Lisp applications under Unix”, an implementation “tailored to zillions of small Lisp programs. Make sure that much of Lisp is shared memory. Be able to create shared memory libraries/applications”.
(SBCL) Is there a way to detect if lisp code was run by –script vs interactively?

Around the language

🚀 Using Coalton to Implement a Quantum Compiler
- Coalton brings Haskell-like type checking on top of Common Lisp. Groundbreaking. They use it for their quantum compiler.
- it is still under development and there is their Discord to talk about it.
Eliminating CLOS for a 4.5x speedup
- sometimes it’s easy, use a struct.
The empty list. “Someone has been getting really impressively confused and cross on reddit about empty lists, booleans and so on in Common Lisp, which led us to a discussion about what the differences really are. Here’s a summary which we think is correct.”
Are there any tutorials or examples people would recommend to get started with unit testing common lisp?
Fun with Macros: Do-File
Series tips and tricks
STATIC-LET, Or How I Learned To Stop Worrying And Love LOAD-TIME-VALUE

History:

Old LISP programs still run in Common Lisp

Common Lisp web development and the road to a middleware
🚀 Lisp for the web: building one standalone binary with foreign libraries, templates and static assets
- what I managed to do here, with the help of the community, represents a great step forward for my Lisp web stack.
- I can build a standalone binary for my web app, containing all static assets (a patch was required for the Djula HTML templating engine), so I can just rsync it to my server and it works.
- a standalone binary is easy to integrate into an Electron window. Stay tuned.
Lisp for the web: deploying with Systemd, gotchas and solutions
- all the little gotchas I faced are now google-able.
Woo: a high-performance Common Lisp web server, by E. Fukamachi.
HTTP over unix sockets in Common Lisp. By the same author:
- Using TailwindCSS in Common Lisp web apps without Node.js tooling
- Using SVGs in Common Lisp web apps with Djula
- Create a Common Lisp Web app using ningle
- Using environment variables with cl-dotenv in Common Lisp
- Mito: An ORM for Common Lisp
- Running docker commands from Common Lisp REPLs
- he also put up several little examples of a Common Lisp web app with HTMX, such as cl-beer. His stack: Caveman, Djula templates, HTMX for the interactivity. Thanks again, they are super useful.
A new static site generator
Web Development with CL Backend and ClojureScript Frontend
Update on gRPC Server - gRPC now has the ability to create servers based on protocol buffer service descriptors.
🚀 Almost 600,000 entries per second from Lisp to Apache Accumulo over Apache Thrift
Writing an interactive web app in Common Lisp: Hunchentoot then CLOG
Review of CL Json Libraries, updated 15 Jan 2022

Call for action:

📢 JACL call for collaboration

Screencasts and podcasts

New videos by me:

🚀 Debugging Lisp: fix and resume a program from any point in stack 🎥
How to request the GitHub API: demo of Dexador, Jonathan, Shasht (and Serapeum), livecoding in Emacs and SLIME.
How to create, run, build and load a new Lisp project, with my fully-featured project skeleton.
Interactively Fixing Failing Tests in Common Lisp

by Gavin Freeborn:

KONS-9 series:

CLOG series:

CLOG Extra 3 - The CLOG Project System for CLOG and NON-CLOG projects
CLOG Extra 4 - All about panels
Repl Style. Dev visually with CLOG Builder
- “This is amazing work! Having Pharo/Smalltalk capabilities with Common Lisp is fascinating.”

CL study group:

Roswell (part II)

Others:

and of course, find 3h48+ of condensed Lisp content on my Udemy video course! (I’m still working on new content, as a student you get updates).

Aside screncasts, some podcasts:

Other discussions

Community

💌 Impressions of Common Lisp and the Lisp Community
- “So, thank you. I’m glad I found this group (and some other Lisp groups) on Reddit. You guys are a real blessing.”
CL Community(?) Norms
In what domains is common lisp used in 2022?
What are you working on?
A brief interview with Common Lisp creator Dr. Scott Fahlman

Learning Lisp

Advice on professional development in Common Lisp?
Trying to get into Lisp, Feeling overwhelmed
Looking for good common lisp projects on github to read?
- this GitHub advanced search searches for defclass in 4 users projects: Shinmera, 40ants, Ruricolist, edicl (maintained Edi Weitz libraries):
Teach lisp to high schoolers?
Why buy LispWorks?
- it’s expensive (there’s a free limited edition), but all this feedback is much interesting.
Writing robust software in CL
- talking Erlang, Actors libraries.
- Moira - Monitor and restart background threads.
What should a new programmer writing in Common Lisp know about security?

Common Lisp VS …

what were the lessons you learned from programming in Lisp that carried over to other languages (more on the imperative side)?
For serious/industrial/business use, in what ways do you think Common Lisp beat Clojure, and vice versa?
- and: When should I choose Common Lisp over Clojure (for business), and vice versa?
Common Lisp VS Racket (openly biased over CL, with testimonies of lispers knowing both)
- HN comments (143): “I’m a heavy user of Common Lisp, and I only dabble in Racket from time to time. While Common Lisp is my tool of choice for a lot of reasons stated in the post, since the post largely skews favorably toward Common Lisp, I’ll offer two things that Racket shines at.”
Why not: from Common Lisp to Julia
- HackerNews comments (200) (for this post and the post it responds to)
- The author of the first post said he was “migrating from Common Lisp to Julia as [his] primary programming language”. He was starting “to grow increasingly frustrated with various aspects of the language” and the CL ecosystem. Many of his arguments were harsh towards CL and didn’t mention existing better alternatives.
- Two months later, in another blog post, he admits the previous article “was unfair to both languages”, and while “[he] is still mostly using Julia”, “Common Lisp is still on the table for [him]“.
- On LiberaChat, he says he’s back from Julia to Common Lisp: “As of yesterday, I’m back to making libraries in CL. I cannot consider Julia as a serious language anymore, for it has deceived me after 7 years of research.”
- Welcome back and thanks to him for the past and future CL libraries.
Is Lisp too malleable to use for serious development?
Why Lisp? (version française). “We will show how our own LispE interpreter was implemented.”
- HN comments (200)

Thanks everyone, happy lisping and see you around!

Debugging Lisp: fix and resume a program from any point in stack 🎥

Tue, 06 Dec 2022 12:47:04 +0100

You are doing god’s work on a time-intensive computation, but your final step errors out :S Are you doomed to start everything from zero, and wait again for this long process? No! Find out.

I show this with Emacs and Slime, then with the Lem editor (ready-to-use for CL, works with many more languages thanks to its LSP client).

(This video is so cool :D Sound on)

We use the built-in Common Lisp interactive debugger that lets us restart one precise frame from the call stack. Once you get the debugger:

press “v” on a frame to go to the corresponding source file
fix the buggy function, compile it again (C-c C-c in Slime)
come back to the debugger, place the point on the frame you want to try again, press “r” (sldb-restart-frame, see your editor’s menu)
see everything succeed. You did not restart everything from zero.
if you don’t see some function calls and variables in the debugger, compile your code with max debug settings (C-u C-c C-c in Slime).

For more:

https://lispcookbook.github.io/cl-cookbook/debugging.html - see how to break and step, how to trace function calls, etc.

Debugging Lisp: trace options, break on conditions

Fri, 02 Dec 2022 11:08:27 +0100

Those are useful Common Lisp debugging tricks. Did you know about trace options?

We see how trace accepts options. Especially, we see how we can break and invoke the interactive debugger before or after a function call, how we can break on a condition (“this argument equals 0”) and how we can enrich the trace output. But we only scratch the surface, more options are documented on their upstream documentation:

INFO: You'd better read this on the Common Lisp Cookbook, that's where it will receive updates.

Table of Contents

Trace - basics

But let’s first see a recap’ of the trace macro. Compared to the previous Cookbook content, we just added that (trace) alone returns a list of traced functions.

trace allows us to see when a function was called, what arguments it received, and the value it returned.

(defun factorial (n)
  (if (plusp n)
    (* n (factorial (1- n)))
    1))

To start tracing a function, just call trace with the function name (or several function names):

(trace factorial)

(factorial 2)
  0: (FACTORIAL 3)
    1: (FACTORIAL 2)
      2: (FACTORIAL 1)
        3: (FACTORIAL 0)
        3: FACTORIAL returned 1
      2: FACTORIAL returned 1
    1: FACTORIAL returned 2
  0: FACTORIAL returned 6
6

(untrace factorial)

To untrace all functions, just evaluate (untrace).

To get a list of currently traced functions, evaluate (trace) with no arguments.

In Slime we have the shortcut C-c M-t to trace or untrace a function.

If you don’t see recursive calls, that may be because of the compiler’s optimizations. Try this before defining the function to be traced:

(declaim (optimize (debug 3)))  ;; or C-u C-c C-c to compile with maximal debug settings.

The output is printed to *trace-output* (see the CLHS).

In Slime, we also have an interactive trace dialog with M-x slime-trace-dialog bound to C-c T.

But we can do many more things than calling trace with a simple argument.

Trace options - break and invoke the debugger

trace accepts options. For example, you can use :break t to invoke the debugger at the start of the function, before it is called (more on break below):

(trace factorial :break t)
(factorial 2)

We can define many things in one call to trace. For instance, options that appear before the first function name to trace are global, they affect all traced functions that we add afterwards. Here, :break t is set for every function that follows: factorial, foo and bar:

(trace :break t factorial foo bar)

On the contrary, if an option comes after a function name, it acts as a local option, only for its preceding function. That’s how we first did. Below foo and bar come after, they are not affected by :break:

(trace factorial :break t foo bar)

But do you actually want to break before the function call or just after it? With :break as with many options, you can choose. These are the options for :break:

:break form  ;; before
:break-after form
:break-all form ;; before and after

TIP: form can be any form that evaluates to true. You can add any custom logic here.

Note that we explained the trace function of SBCL. Other implementations may have the same feature with another syntax and other option names. For example, in LispWorks it is “:break-on-exit” instead of “:break-after”, and we write (trace (factorial :break t)).

Below are some other options but first, a trick with :break.

break on a condition

The argument to an option can be any form. Here’s a trick, on SBCL, to get the break window when we are about to call factorial with 0. (sb-debug:arg 0) refers to n, the first argument.

CL-USER> (trace factorial :break (equal 0 (sb-debug:arg 0)))
;; WARNING: FACTORIAL is already TRACE'd, untracing it first.
;; (FACTORIAL)

Running it again:

CL-USER> (factorial 3)
  0: (FACTORIAL 3)
    1: (FACTORIAL 2)
      2: (FACTORIAL 1)
        3: (FACTORIAL 0)

breaking before traced call to FACTORIAL:
   [Condition of type SIMPLE-CONDITION]

Restarts:
 0: [CONTINUE] Return from BREAK.
 1: [RETRY] Retry SLIME REPL evaluation request.
 2: [*ABORT] Return to SLIME's top level.
 3: [ABORT] abort thread (#<THREAD "repl-thread" RUNNING {1003551BC3}>)

Backtrace:
  0: (FACTORIAL 1)
      Locals:
        N = 1   <---------- n is still 1, we break before the call with 0.

Other options

Trace on conditions

:condition enables tracing only if the condition in form evaluates to true.

:condition form
:condition-after form
:condition-all form

If :condition is specified, then trace does nothing unless Form evaluates to true at the time of the call. :condition-after is similar, but suppresses the initial printout, and is tested when the function returns. :condition-all tries both before and after.

Trace if called from another function

:wherein can be super useful:

:wherein Names

If specified, Names is a function name or list of names. trace does nothing unless a call to one of those functions encloses the call to this function (i.e. it would appear in a backtrace.) Anonymous functions have string names like “DEFUN FOO”.

Enrich the trace output

:report Report-Type

If Report-Type is trace (the default) then information is reported by printing immediately. If Report-Type is nil, then the only effect of the trace is to execute other options (e.g. print or break). Otherwise, Report-Type is treated as a function designator and, for each trace event, funcalled with 5 arguments: trace depth (a non-negative integer), a function name or a function object, a keyword (:enter, :exit or :non-local-exit), a stack frame, and a list of values (arguments or return values).

See also :print to enrich the trace output:

In addition to the usual printout, the result of evaluating Form is printed at the start of the function, at the end of the function, or both, according to the respective option. Multiple print options cause multiple values to be printed.

Example:

(defparameter *counter* 0)
(defun factorial (n)
  (incf *counter*)
  (if (plusp n)
    (* n (factorial (1- n)))
    1))

CL-USER> (trace factorial :print *counter*)
CL-USER> (factorial 3)
(FACTORIAL 3)
  0: (FACTORIAL 3)
  0: *COUNTER* = 0
    1: (FACTORIAL 2)
    1: *COUNTER* = 1
      2: (FACTORIAL 1)
      2: *COUNTER* = 2
        3: (FACTORIAL 0)
        3: *COUNTER* = 3
        3: FACTORIAL returned 1
      2: FACTORIAL returned 1
    1: FACTORIAL returned 2
  0: FACTORIAL returned 6
6

Closing remark

As they say:

it is expected that implementations extend TRACE with non-standard options.

and we didn’t list all available options or parameters, so you should check out your implementation’s documentation.

For more debugging tricks see the Cookbook and the links in it, the Malisper series have nice GIFs.

I am also preparing a short screencast to show what we can do inside the debugger, stay tuned!

Lisp for the web: building one standalone binary with foreign libraries, templates and static assets

Mon, 28 Nov 2022 18:43:43 +0100

In our previous entry, we saw how to deploy our web application with Systemd, either from sources or with a binary. Now we’ll speak more about this building process to produce one binary that contains everything for our web app. We’ll tackle 3 issues:

ship foreign libraries alongside your binary, such as libreadline.so or libsqlite3.so,
include your Djula templates into your binary,
serve static files from your binary, without reading the filesystem,
and we’ll see my Gitlab CI recipe.

This allows us to create a binary that is really easy to deploy, to ship to users or to embed in an external process such as an Electron window (more on that later). Coming from Python and JS, what a dream!

Now, I want to thank the people that helped me figure these issues out and who wrote, fixed and extended these libraries: special shout-out to @mmontone for writing a Djula patch so quickly, @shinmera for Deploy and answering my many questions on Discord, @zulu.inoe for finding Hunchentoot answers, and everybody else on Discord for their help (@gavinok, @fstamour et all, sorry if I forgot) and all who dare asking questions to let everybody learn!

Table of Contents - Ship foreign libraries: the need of Deploy - Configuring Deploy: ignore libssl, verbosity - Remember: your program runs on another user’s machine. - Telling ASDF to calm down - Embed HTML Djula templates in your binary - Serve static assets - Gitlab CI - Closing remarks

Ship foreign libraries: the need of Deploy

Deploy is the way to go. If you used asdf:make in your .asd system definition to create a binary already, you just need to change two things:

;; my-project.asd
:defsystem-depends-on (:deploy)  ;; so you need to quickload deploy sometime before.
:build-operation "deploy-op"  ;; instead of program-op for asdf:make

and those two lines stay the same:

:build-pathname "my-application-name"
:entry-point "my-package:my-start-function"

Here’s my Makefile target, where I quickload Deploy before loading my app and calling asdf:make:

LISP ?= sbcl

build:
	$(LISP)	--non-interactive \
		--eval '(ql:quickload "deploy")' \
		--load openbookstore.asd \
		--eval '(ql:quickload :openbookstore)' \
		--eval '(asdf:make :openbookstore)'

This creates a bin/ directory with our binary and the foreign libraries:

  -rwxr-xr-x  1 vindarel vindarel 130545752 Nov 25 18:48 openbookstore
  -rw-rw-r--  1 vindarel vindarel    294632 Aug  3 13:06 libreadline.so.7.0
  -rw-rw-r--  1 vindarel vindarel    319528 Aug 23 18:01 libreadline.so.8.0
  -rw-rw-r--  1 vindarel vindarel   1212216 Aug 24 16:42 libsqlite3.so.0.8.6
  -rw-rw-r--  1 vindarel vindarel    116960 Aug  3 13:06 libz.so.1.2.11

We need to deploy this directory.

When we start the binary, Deploy tells us what it is doing:

$ ./bin/openbookstore --datasource argentina lisp
 ==> Performing warm boot.
   -> Runtime directory is /home/vindarel/projets/openbookstore/openbookstore/bin/
   -> Resource directory is /home/vindarel/projets/openbookstore/openbookstore/bin/
 ==> Running boot hooks.
 ==> Reloading foreign libraries.
   -> Loading foreign library #<LIBRARY READLINE>.
   -> Loading foreign library #<LIBRARY SQLITE3-LIB>.
   -> Loading foreign library #<LIBRARY LIBSSL>.
   -> Loading foreign library #<LIBRARY LIBCRYPTO>.
 ==> Launching application.
OpenBookStore version 0.2-d2ac5f2
[…]
==> Epilogue.
==> Running quit hooks.

We can configure Deploy.

Configuring Deploy: ignore libssl, verbosity

You can silence the Deploy statuses by pushing :deploy-console into the *features* list, before calling asdf:make. Add this to the Makefile:

--eval '(push :deploy-console *features*)'

Now all seems well, you rsync your app to your server, run it and… you get a libssl error:

=> Deploying files to /home/vindarel/projets/myapp/commandes-collectivites/bin/
Unhandled SIMPLE-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING
                                    {10007285B3}>:
  #<LIBRARY LIBCRYPTO> does not have a known shared library file path.

Nicolas (@shinmera) explained that we typically want to import libssl or libcrypto from the target system, that “deploying these libraries without them blowing up on Linux is hard”. To do this, we ask Deploy to not handle them. In the .asd:

#+linux (deploy:define-library cl+ssl::libssl :dont-deploy T)
#+linux (deploy:define-library cl+ssl::libcrypto :dont-deploy T)

As a consequence, you now need to quickload or require :cl+ssl before loading the .asd file, because of the cl+ssl::libssl/libcrypto symbols at the top level.

Nicolas built all this for his needs when working on his Trial game engine and on his Kandria game (soon on Steam!), check them out!

Remember: your program runs on another user’s machine.

By this I mean that if you took the habit to use functions that locate a project’s source directory (asdf:system-source-directory, asdf:system-relative-pathname, for example when asking Hunchentoot to serve static assets, more on that below), then you need to re-write them, because your binary runs on another machine and it doesn’t run from sources, so ASDF, Quicklisp and friends are not installed, and your project(s) don’t have source directories, they are embedded in the binary.

Use a deploy:deployed-p runtime check if needed.

Telling ASDF to calm down

Now, we are very happy and confident, what could possibly go wrong? We run our app once again on our naked VPS:

$ ./bin/myapp
 ==> Performing warm boot.
   -> Runtime directory is /home/debian/websites/app/myapp/bin/
   -> Resource directory is /home/debian/websites/app/myapp/bin/
 ==> Running boot hooks.
 ==> Reloading foreign libraries.
   -> Loading foreign library #<LIBRARY LIBSSL>.
   -> Loading foreign library #<LIBRARY LIBRT>.
   -> Loading foreign library #<LIBRARY LIBOSICAT>.
   -> Loading foreign library #<LIBRARY LIBMAGIC>.
 ==> Launching application.
WARNING:
   You are using ASDF version 3.3.4.15 from
   #P"/home/vindarel/common-lisp/asdf/asdf.asd" and have an older version of ASDF
   (and older than 2.27 at that) registered at
   #P"/home/vindarel/common-lisp/asdf/asdf.asd".

  [ long message ellided ]

;
; compilation unit aborted
;   caught 1 fatal ERROR condition
An error occured:
 Error while trying to load definition for system asdf from pathname
 /home/vindarel/common-lisp/asdf/asdf.asd:
    Couldn't load #P"/home/vindarel/common-lisp/asdf/asdf.asd": file does not
    exist. ==> Epilogue.
 ==> Running quit hooks.

Now ASDF wants to do what, update itself? Whatever it tries to do, it crashes. Yes, this happens on the target host, when we run the binary. Damn!

The solution is easy, but it had to be documentend or google-able… Add this in your .asd to tell ASDF to not try to upgrade itself:

(deploy:define-hook (:deploy asdf) (directory)
  ;; Thanks again to Shinmera.
  (declare (ignorable directory))
  #+asdf (asdf:clear-source-registry)
  #+asdf (defun asdf:upgrade-asdf () nil))

By the way, if you want a one-liner to upgrade ASDF to 3.3.5 so that you can use package-local nicknames, check this lisp tip

Embed HTML Djula templates in your binary

Our binary now runs fine on our server: super great. But our app has another issue.

I like very much Djula templates, maintained by @mmontone. It is a traditional, no-surprises HTML templating system, very similar to Django templates. It is easy to setup, it is very easy to create custom filters, it has good error messages, both in the browser window and on the Lisp REPL. It’s one of the most downloaded Quicklisp libraries. Like Django templates, its philosophy is that it doesn’t allow a lot of computations in the template. It encourages to prepare your data on the back-end, so it is straightforward to process them in the templates. Sometimes it is limiting, so for more flexibility I’d look at Ten. It isn’t as much used and tested though (and I didn’t try it myself). If you want lispy templates, look at Spinneret. You can say goodby to copy-pasting nice-looking HTML examples, though.

However, by using Spinneret you would have not faced the following issue:

Djula reads templates from your file system.

and when your application runs on someone else’s machine, this is undefined behaviour.

Until now, you had to deploy your web app from sources or, at least, you had to send the HTML files to the server. This was the case until I talked about this issue to Mariano. He sent a patch the day after.

Now, we can choose: by default, Djula reads the HTML files from disk: very well. But now, when we build our binary, we can ask Djula to build the templates in memory, so they are saved into the Lisp binary.

Normally, you only need to tell Djula where to find templates (“add a template directory”), then to compile them into a variable:

;; normal, file-system case.
(djula:add-template-directory (asdf:system-relative-pathname "webapp" "templates/"))
(defparameter +base.html+ (djula:compile-template* "base.html"))

;; and then, we render the template with (djula:render-template* nil +base.html+ …)

This uses a filesystem-template-store. In addition, it recompiles templates on change. This can be turned off as we’ll see.

For our binary, we need to set Djula’s *current-store* to a memory-template-store AND we need to turn off the djula:*recompile-templates-on-change* setting. Then, we need to compile all the templates of our application, and save our binary.

I actually do all this at the top-level of my web.lisp file. By default I load the app for development, and if we find a custom “feature”, that is added by the “build” target of the Makefile, we compile templates in memory.

So, in order:

in the “build” target of my Makefile, I push a new setting in the *features* list:
```
--eval '(push :djula-binary *features*)'
```
in my web.lisp, I check for this setting (with #+djula-binary) and I create either a filetemplate store or a memory store. This is written at the top-level so it will be executed when we load the file. We can probably come up with better ergonomics.

This will be executed when I quickload my app in the build target of the Makefile, following the one above.

    --eval '(ql:quickload :openbookstore)'

(setf djula:*current-store*
      (let ((search-path (list (asdf:system-relative-pathname "openbookstore"
                                                              "src/web/templates/"))))
        #-djula-binary
        (progn
          (uiop:format! t "~&Openbookstore: compiling templates in file system store.~&")
          ;; By default, use a file-system store and reload templates during development.
          (setf djula:*recompile-templates-on-change* t)
          (make-instance 'djula:filesystem-template-store
		         :search-path search-path))

        ;; But, if this setting is set to NIL at load time, from the Makefile,
        ;; we are building a standalone binary: use an in-memory template store.
        ;;
        ;; We also need to NOT re-compile templates on change.
        #+djula-binary
        (progn
          (uiop:format! t "~&Openbookstore: compiling templates in MEMORY store.~&")
          (setf djula:*recompile-templates-on-change* nil)
          (make-instance 'djula:memory-template-store :search-path search-path))))

compile all the templates. If you used a web framework (or started to develop yours), you might have used a shortcut: calling a render function which takes the name of a template as a string for argument. I’m thinking about Caveman:

@route GET "/"
(defun index ()
  (render #P"index.tmpl"))

This strings denotes the name of the template. For a standalone binary, we need to compile the template before. That’s why Djula shows how to define and compile our templates:

(defparameter +base.html+ (djula:compile-template* "base.html"))

You need this line for every template of your application:

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; Compile and load templates (as usual).
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defparameter +base.html+ (djula:compile-template* "base.html"))
(defparameter +dashboard.html+ (djula:compile-template* "dashboard.html"))
(defparameter +search.html+ (djula:compile-template* "search.html"))
(defparameter +stock.html+ (djula:compile-template* "stock.html"))
(defparameter +card-page.html+ (djula:compile-template* "card-page.html"))
(defparameter +card-stock.html+ (djula:compile-template* "card-stock.html"))
(defparameter +card-create.html+ (djula:compile-template* "card-create.html"))
(defparameter +card-update.html+ (djula:compile-template* "card-edit.html"))
;; and so on.

let asdf:make (and Deploy) save your binary. To try it, rename your templates/ directory to something else and run your app.

Addendum.

An .asd system definition can reference static files, so they are part of the build process and included into the delivered application. That’s how you can ship a README file:

  :components ((:static-file "README.md")
               ...)

I did the same for my templates. To be honest, I don’t recall if there is a solid reason, since they are compiled and saved into the image anyhow with the compilation step above. I show this anyways, it looks like a good practice to me:

            (:module "src/web/templates"
                        :components
                        ;; Order is important.
                        ((:static-file "login.html")
                         (:static-file "404.html")
                         (:static-file "base.html")
                         (:static-file "dashboard.html")
                         (:static-file "history.html")
                         (:static-file "search.html")
                         (:static-file "sell.html")
                         ...))

now we need to programmatically get the list of files from this src/web/templates module and to compile everything:

(let ((paths (djula:list-asdf-system-templates "bookshops" "src/web/templates")))
  (loop for path in paths
     do (uiop:format! t "~&Compiling template file: ~a…~&" path)
       (djula:compile-template* path))
  (values t :all-done))

This snippet and the general instructions are documented: https://mmontone.github.io/djula/djula/Deployment.html#Deployment

Feel free to show how you do it.

Bonus: here’s the list-asdf-system-templates function. We use asdf functions to get a system name, its components, their names…

(defun list-asdf-system-templates (asdf-system component)
  "List djula templates in ASDF-SYSTEM at COMPONENT.
  A list of template PATHNAMEs is returned."
  (let* ((sys (asdf:find-system asdf-system))
         (children (asdf:component-children sys))
         (module (or (find component children :key #'asdf:component-name :test #'equal)
                     (error "Component ~S not found in system named ~S.~&Available components are: ~S" component asdf-system (mapcar #'asdf:component-name children))))
         (alltemplates (remove-if-not (lambda (x) (typep x 'asdf:static-file))
                                      (asdf:module-components module))))
    (mapcar (lambda (it) (asdf:component-pathname it))
            alltemplates)))

Serve static assets

At that point in time, I figured out static assets would need to be worked on too. Hopefully, the people on Discord helped me and it was quickly solved.

This is how I served static assets with Hunchentoot. We use a “folder dispatcher and handler”:

https://common-lisp-libraries.readthedocs.io/hunchentoot/#create-folder-dispatcher-and-handler

(defun serve-static-assets ()
  "Serve static assets under the /src/static/ directory when called with the /static/ URL root."
  (push (hunchentoot:create-folder-dispatcher-and-handler
         "/static/" (merge-pathnames *default-static-directory*
                                     (asdf:system-source-directory :openbookstore) ;; => NOT src/
                                     ))
        hunchentoot:*dispatch-table*))

But when your app is on another machine… hence the need to ship the static assets into the standalone binary, and to ask Hunchentoot to serve them.

What we do is pretty obvious: save our static files into a data structure, so this one is saved in the image, but we use a couple Lisp tricks so I comment the code below.

You’ll see that this time I hardcoded the file names and I didn’t declare them on the .asd file… clearly there is room for improvement, be my guest.

You can find the file I use for my application here.

;;; pre-web.lisp
;;; Parameters and functions required before loading web.lisp
;;;
;;; We read the content of our static files and put them into variables, so that they can be saved in the Lisp image.
;;; We define %serve-static-file to simply return their content (as string),
;;; and because we use with the #. reader macro, we need to put these functions in another file than web.lisp.

;;; Where my static files are:
(defparameter *default-static-directory* "src/static/"
  "The directory where to serve static assets from (STRING). If it starts with a slash, it is an absolute directory. Otherwise, it will be a subdirectory of where the system :abstock is installed.
  Static assets are reachable under the /static/ prefix.")

;;; We simply use a hash-table that maps a file name to its content, a a string.
;;; I love Serapeum's dict which is a readable hash-table, that's what I use:
(defparameter *static-files-content* (dict)
  "Content of our JS and CSS files.
  Hash-table with file name => content (string).")

;;; I read all my static files and I save them into the hash-table:
(defun %read-static-files-in-memory ()
  "Save the JS and CSS files in a variable in memory, so they can be saved at compile time."
  (loop for file in (list "openbookstore.js"
                          "card-page.js")
     with static-directory = (merge-pathnames *default-static-directory*
                                              (asdf:system-source-directory :bookshops))
     for content = (uiop:read-file-string (merge-pathnames file static-directory))
     do (setf (gethash file *static-files-content*) content)
     finally (return *static-files-content*)))

;; AT COMPILE TIME, read the content of our static files.
(%read-static-files-in-memory)

(defun %serve-static-file (path)
  "Return the content as a string of this static file.
  For standalone binaries delivery."
  ;; "alert('yes, compiled in pre-web.lisp');"  ;; JS snippet to check if this dispatcher works.
  (gethash path *static-files-content*))  ;; this would not work without the #. reader macro.

It is inside “web.lisp” that I set other rules for Hunchentoot. If it recognizes my static files, we simply return their content, as a string.

I don’t know if this works well with very big or with numerous files. But I-want-a-standalone-binary! For serious needs, I’d serve the static files with a proper server… I guess.

We use the #. reader macro to get our files’ content at compile time, this is why we needed to define our helper functions in another file, that is loaded before this one.

;;; web.lisp
(defun serve-static-assets-for-release ()
  "In a binary release, Hunchentoot can not serve files under the file system: we are on another machine and the files are not there.
  Hence we need to get the content of our static files into memory and give them to Hunchentoot."
  (push
   (hunchentoot:create-regex-dispatcher "/static/openbookstore\.js"
                                        (lambda ()
                                          ;; Returning the result of the function calls silently fails. We need to return a string.
                                          ;; Here's the string, read at compile time.
                                          #.(%serve-static-file "openbookstore.js")))
   hunchentoot:*dispatch-table*)

  (push
   (hunchentoot:create-regex-dispatcher "/static/card-page\.js"
                                        (lambda ()
                                          #.(%serve-static-file "card-page.js")))
   hunchentoot:*dispatch-table*))

Finally, it is inside my start-app function that I decide how to serve my static assets:

  (hunchentoot:start *server*)
  (if (deploy:deployed-p)
      ;; Binary release: don't serve files by reading them from disk.
      (serve-static-assets-for-release)
      ;; Normal setup, running from sources: serve static files as usual.
      (serve-static-assets))
  (uiop:format! t "~&Application started on port ~a.~&" port)

Find my web.lisp file here.

Gitlab CI

I build my binary on Gitlab.

image: clfoundation/sbcl

# uncomment to run the jobs in parallel. They are now run one after the other.
# stages:
  # - test
  # - build

# We need to install some system dependencies,
# to clone libraries not in Quicklisp,
# and to update ASDF to >= 3.3.5 in order to use local-package-nicknames.
before_script:
  - apt-get update -qy
  - apt-get install -y git-core sqlite3 tar
  # The image doesn't have Quicklisp installed by default.
  - QUICKLISP_ADD_TO_INIT_FILE=true /usr/local/bin/install-quicklisp
  # clone libraries not in Quicklisp or if we need the latest version.
  - make install
  # Upgrade ASDF (UIOP) to 3.3.5 because we want package-local-nicknames.
  - mkdir -p ~/common-lisp/asdf/
  - ( cd ~/common-lisp/ && wget https://asdf.common-lisp.dev/archives/asdf-3.3.5.tar.gz  && tar -xvf asdf-3.3.5.tar.gz && mv asdf-3.3.5 asdf )
  - echo "Content of ~/common-lisp/asdf/:" && ls ~/common-lisp/asdf/

qa:
  allow_failure: true
  # stage: test
  script:
    # QA tools:
    # install Comby:
    - apt-get install -y sudo
    # - bash <(curl -sL get.comby.dev)
    - bash <(curl -sL https://raw.githubusercontent.com/vindarel/comby/set-release-1.0.0/scripts/install.sh)
    # install Colisper for simple lisp checks:
    - git clone https://github.com/vindarel/colisper ~/colisper
    - chmod +x ~/colisper/colisper.sh
    - cd src/ && ~/colisper/colisper.sh

build:
  # stage: build
  script:
    - make build
  artifacts:
    name: "openbookstore"
    # Publish the bin/ directory (todo: rename, include version…)
    paths:
      - bin/

Closing remarks

I am so excited by the possibilities this brings.

I knew it was possible to do this in CL but I admit I thought it would be simpler… it turned out it is not a very crowded path. Now the steps are documented and google-able, here and everywhere else I could leave a comment, but it will be nice to come up with shorter and friendlier ready-to-use utilities. In a new web framework? And again, please share how you do all of this in the comments.

Having this standalone binary dramatically simplifies my deployment process. With a small web app, running from sources was easy (once you set up Quicklisp, and ASDF, and…). But with a growing application, that uses my local forks or code not yet pushed to GitHub, deployment was becoming tedious, and it is now greatly simplified. rsync, systemctl restart and done.

Its only limitation is that you need the same libc version on the target OS as on your local machine. So, back in august I could build on my machine and send the result to my VPS, but I upgraded my Debian-ish system, and left the server with its (very) old Ubuntu version, so I can’t run the binary from my machine there any more… I must resort to a CI pipeline that uses a matrix of Ubuntu versions, or build with Docker or a virtual machine. Or run from sources… Maybe soon I will build (truly) static executables: they are coming to SBCL.

I repeat again the good side: my friends (on Debian so far) can download the app, run bin/openbookstore and it works \o/

One last thing I’d like to do is to be able to double-click an executable to start the app, and to have one single file (and not an archive that extracts as a directory, although it is not too bad!). This looks possible with Makeself.

If you want to try the standalone binary on your GNU/Linux system (does it actually work on other distros?), download the artifacts of the latest passing build on the pipelines page, or grab it with this direct link. Un-zip, run bin/bookshops and go to localhost address shown in the output (and also create an admin user as shown in the readme). You can leave me a comment here, on Gitter, on Discord or with a good ol’ email.

Stay tuned, OpenBookStore is still a work in progress but it will be a Common Lisp application flagship ;)

Lisp for the web: deploying with Systemd, gotchas and solutions

Tue, 30 Aug 2022 16:57:37 +0200

How do you run your Common Lisp (web) application on your server? Nowadays most GNU/Linux distros have Systemd. I recently used it more, with a mix of applications running from source, from a binary, running locally or on my VPS. I had to bypass a few gotchas, so let’s recap’ what you need to know.

Also stay tuned: next, we’ll see how to build a standalone binary for your Common Lisp application with Deploy (so that we handle foreign libraries like libssl), how to include your Djula HTML templates as well as your static assets. This, in turns, makes it straightforward to ship your web app into an Electron desktop window.

INFO: dear readers, here's the best price I can set for my Udemy course "Learn Lisp effectively" , it will be available for 5 days! Thanks all for your support (NB: I'm working on the chapter on condition handling and new chapters are available for existing students).

Before Systemd

Let’s say we can run our app like this: we load the system definition, its dependencies, and we call the entry point.

sbcl --load my-app.asd \
     --eval '(ql:quickload :my-app)' \
     --eval '(my-app:start-app)'

Now we need to put the app on the background, we must ensure that if it fails, it is restarted, if the server is restarted, our app too, we must ensure we get to see the logs, etc.

Maybe you run your app inside of Emacs on your VPS… maybe you run your app inside tmux. This works and it is convenient to get back to the Lisp REPL, but this doesn’t ensure a restart on failure.

Systemd: daemonizing, restarting in case of crashes, handling logs

Systemd (or the service system of your distro) can help with all that.

Write a service file like this:

$ emacs -nw /etc/systemd/system/my-app.service
[Unit]
Description=Lisp app example

[Service]
WorkingDirectory=/path/to/your/app
# Next, your command, with the full path to SBCL.
# This works, locally. Be sure to see the last section.
ExecStart=/usr/bin/sbcl --load run.lisp
# or: use a path to your binary.
Type=simple
Restart=always
RestartSec=10
# Use environment variables:
Environment="SECRET=pGNqduRFkB4K9C2vijOmUDa2kPtUhArN"

# Start or restart at boot:
[Install]
WantedBy=basic.target

When you run your app on the terminal, ensure that its webserver (Hunchentoot here) stays listening correctly on the foreground (otherwise see below):

$ sbcl --load run.lisp
Hunchentoot server is started.
Listening on localhost:9003.

Now run this command to start the service:

sudo systemctl start my-app.service

to check its status use this:

systemctl status my-app.service

Systemd handles logs for you. We make our app write to stdout or stderr, Systemd writes logs:

journalctl -u my-app.service

use -f -n 30 to see live updates of logs, where -n is the number of lines you want as context.

The following tells Systemd to handle crashes and to restart the app:

Restart=always

and this makes it start the app after a reboot:

[Install]
WantedBy=basic.target

to enable it:

sudo systemctl enable my-app.service

Now keep in mind a couple things.

Make it stay on the foreground

The first gotcha is that your app must stay on the foreground.

If you run your app from source, you might have nothing to do, you’ll get a Lisp REPL, from which you can interact with your running application. Awesome.

But, if you build a binary, you might see this error when you run it with Systemd:

* ;
; compilation unit aborted
; caught 1 fatal error condition" error.

This puzzled me: I thought I had a Lisp prompt (the * ;) and that my program crashed, but no. I knew it, that’s simply Lisp quitting too early. Don’t rush and double check that your binary runs correctly.

What you must do can be found elsewhere (the Cookbook!): in your main function where you start your app, in this example with Hunchentoot, put its thread in the foreground:

;; with bordeaux-threads. Also sb-ext: join-thread, thread-name, list-all-threads.
(bt:join-thread (find-if (lambda (th)
                            (search "hunchentoot" (bt:thread-name th)))
                          (bt:all-threads)))

Let it crash: `--disable-debugger`

We want our app to crash so that it can be re-started automatically: you’ll want the --disable-debugger flag with SBCL, when you run your app from sources.

Relying on Quicklisp

When you run your apps locally, you most probably rely on Quicklisp being installed and being started in your init file (~/.sbclrc):

;;; The following lines added by ql:add-to-init-file:
#-quicklisp
(let ((quicklisp-init (merge-pathnames "quicklisp/setup.lisp"
                                       (user-homedir-pathname))))
  (when (probe-file quicklisp-init)
    (load quicklisp-init)))

There are 2 gotchas.

Systemd will, by default, run your app as root, so:

if it happened you did install Quicklisp on your production machine, you probably didn’t install it as root, so Systemd won’t find the init file that initializes Quicklisp (and so your startup scripts will fail).
- you can use SBCL’s --userinit flag to tell the username where to find the init file.
- you can set the Systemd user with User=xyz in the [service] section (disclaimer: untested).
the Quicklisp snippet will fail at (user-homedir-pathname), for a clash on usernames too, so Quicklisp won’t find its setup.lisp file. I replaced this function call with a hard path (/home/vindarel/), until I used a standalone binary.

That’s it. Now you can deploy in peace. I hope I saved you some hours. Now these issues are better google-able \o/

See you around and stay tuned.

State of Common Lisp Web Development - an overview

Fri, 24 Jun 2022 07:51:49 +0100

Caution [from 2017: what a road since then]: this is a draft. I take notes and write more in other resources (the Cookbook, my blog).

update, Jan 2025: I published a new resources specialized in web development in Common Lisp:

https://web-apps-in-lisp.github.io/.

update, January 2024: see my new video on Youtube, a demo on how to assemble Hunchentoot, easy-routes, Djula templates (with ready-to-use Bulma themes), with error handling and common traps: https://www.youtube.com/watch?v=h_noB1sI_e8

update, June 2022: see my web project skeleton, it illustrates and fixes common issues: https://github.com/vindarel/cl-cookieweb

update july, 5th 2019: I put this content into the Cookbook: https://lispcookbook.github.io/cl-cookbook/web.html, fixing a long-standing request.

new post: why and how to live-reload one’s running web application: https://lisp-journey.gitlab.io/blog/i-realized-that-to-live-reload-my-web-app-is-easy-and-convenient/

new project skeleton: lisp-web-template-productlist: Hunchentoot + easy-routes + Djula templates + Bulma CSS + a Makefile to build the project

Web application environments

Clack, Lack

Clack is to Lisp what WSGI is to Python. However it is mostly undocumented and not as battle-proofed as Hunchentoot.

Web frameworks

Hunchentoot

The de-facto web server, with the best documentation (cough looking old cough), the most websites on production. Lower level than a web framework (defining routes seems weird at first). I think worth knowing.

Its terminology is different from what we are used to (“routes” are not called routes but we create handlers), part I don’t know why and part because the Lisp image-based development allows for more, and thus needs more terminology. For example, we can run two applications on different URLs on the same image.

https://edicl.github.io/hunchentoot/

edit: here’s a modern looking page: https://digikar99.github.io/common-lisp.readthedocs/hunchentoot/

Caveman

A popular web framework, or so it seems by the github stars, written by a super-productive lisper, with nice documentation for basic stuff but lacking for the rest, based on Clack (webserver interface, think Python’s WSGI), uses Hunchentoot by default.

I feel like basic functions are too cumbersome (accessing url parameters).

https://github.com/fukamachi/caveman

Snooze

By the maintainer of Sly, Emacs’ Yasnippet,…

Defining routes is like defining functions. Built-in features that are available as extensions in Clack-based frameworks (setting to get a stacktrace on the browser, to fire up the debugger or to return a 404,…). Definitely worth exploring.

https://github.com/joaotavora/snooze

Radiance

Radiance, with extensive tutorial and existing apps.

It doesn’t look like a web framework to me. It has ready-to-use components:

admin page (but what does it do?)
auth system
user: provide user accounts and permissions
image hosting
there is an email marketing system in development…

cl-rest-server

a library for writing REST Web APIs in Common Lisp.

Features: validation via schemas, Swagger support, authentication, logging, caching, permission checking…

It seems complete, it is maintained, the author seems to be doing web development in CL for a living. Note to self: I want to interview him.

Wookie

https://github.com/orthecreedence/wookie

An asynchronous web server, by an impressive lisper, who built many async libraries. Used for the Turtl api backend. Dealing with async brings its own set of problems (how will be debugging ?).

Nice api to build routes, good documentation: http://wookie.lyonbros.com/

Weblocks (solving the Javascript problem)

Weblocks allows to create dynamic pages without a line of JavaScript, all in Lisp. It was started years ago and it saw a large update and refactoring lately.

It isn’t the easy path to web development in CL but there’s great potential IMO.

It doesn’t do double data binding as in modern JS frameworks. But new projects are being developed…

See our presentation below.

http://40ants.com/weblocks/quickstart.html

Tasks

Accessing url parameters

It is easy and well explained with Hunchentoot or easy-routes in the Cookbook.

Lucerne has a nice with-params macro that makes accessing post or url query parameters a breeze:

@route app (:post "/tweet")
(defview tweet ()
  (if (lucerne-auth:logged-in-p)
      (let ((user (current-user)))
        (with-params (tweet)
          (utweet.models:tweet user tweet))
        (redirect "/"))
      (render-template (+index+)
                       :error "You are not logged in.")))

Snooze’s way is simple and lispy: we define routes like methods and parameters as keys:

(defroute lispdoc (:get :text/* name &key (package :cl) (doctype 'function))
   ...

matches /lispdoc, /lispdoc/foo and /lispdoc/foo?package=arg.

On the contrary, I find Caveman’s and Ningle’s ways cumbersome.

Ningle:

(setf (ningle:route *app* "/hello/:name")
      #'(lambda (params)
          (format nil "Hello, ~A" (cdr (assoc "name" params :test #'string=)))))

The above controller will be invoked when you access to “/hello/Eitaro” or “/hello/Tomohiro”, and then (cdr (assoc “name” params :test #‘string=)) will be “Eitaro” and “Tomohiro”.

and it doesn’t say about query parameters. I had to ask:

(assoc "the-query-param" (clack.request:query-parameter lucerne:*request*) :test 'string=)

Caveman:

Parameter keys contain square brackets (”[” & “]”) will be parsed as structured parameters. You can access the parsed parameters as _parsed in routers.

(defroute "/edit" (&key _parsed)
  (format nil "~S" (cdr (assoc "person" _parsed :test #'string=))))
;=> "((\"name\" . \"Eitaro\") (\"email\" . \"e.arrows@gmail.com\") (\"birth\" . ((\"year\" . 2000) (\"month\" . 1) (\"day\" . 1))))"

Session an cookies

Data storage

SQL

Mito works for MySQL, Postgres and SQLite3 on SBCL and CCL.

https://lispcookbook.github.io/cl-cookbook/databases.html

We can define models with a regular class which has a mito:dao-table-class :metaclass:

(defclass user ()
  ((name :col-type (:varchar 64)
         :initarg :name
         :accessor user-name)
   (email :col-type (:varchar 128)
          :initarg :email
          :accessor user-email))
  (:metaclass mito:dao-table-class)
  (:unique-keys email))

We create the table with ensure-table-exists:

(ensure-table-exists 'user)
;-> ;; CREATE TABLE IF NOT EXISTS "user" (
;       "id" BIGSERIAL NOT NULL PRIMARY KEY,
;       "name" VARCHAR(64) NOT NULL,
;       "email" VARCHAR(128),
;       "created_at" TIMESTAMP,
;       "updated_at" TIMESTAMP
;   ) () [0 rows] | MITO.DAO:ENSURE-TABLE-EXISTS

Persistent datastores

Migrations

Mito has migrations support and DB schema versioning for MySQL, Postgres and SQLite3, on SBCL and CCL. Once we have changed our model definition, we have commands to see the generated SQL and to apply the migration.

We inspect the SQL: (suppose we just added the email field into the user class above)

(mito:migration-expressions 'user)
;=> (#<SXQL-STATEMENT: ALTER TABLE user ALTER COLUMN email TYPE character varying(128), ALTER COLUMN email SET NOT NULL>
;    #<SXQL-STATEMENT: CREATE UNIQUE INDEX unique_user_email ON user (email)>)

and we can apply the migration:

(mito:migrate-table 'user)
;-> ;; ALTER TABLE "user" ALTER COLUMN "email" TYPE character varying(128), ALTER COLUMN "email" SET NOT NULL () [0 rows] | MITO.MIGRATION.TABLE:MIGRATE-TABLE
;   ;; CREATE UNIQUE INDEX "unique_user_email" ON "user" ("email") () [0 rows] | MITO.MIGRATION.TABLE:MIGRATE-TABLE
;-> (#<SXQL-STATEMENT: ALTER TABLE user ALTER COLUMN email TYPE character varying(128), ALTER COLUMN email SET NOT NULL>
;    #<SXQL-STATEMENT: CREATE UNIQUE INDEX unique_user_email ON user (email)>)

Crane advertises automatic migrations, i.e. it would run them after a C-c C-c. Unfortunately Crane has some issues, it doesn’t work with sqlite yet and the author is busy elsewhere. It didn’t work for me at first try.

Let’s hope the author comes back to work on this in a near future.

Forms

There are a few libraries, see the awesome-cl list. At least one is well active.

Debugging

On an error we are usually dropped into the interactive debugger by default.

Snooze gives options:

use the debugger,
print the stacktrace in the browser (like clack-errors below, but built-in),
display a custom 404.

clack-errors. Like a Flask or Django stacktrace in the browser. For Caveman, Ningle and family.

By default, when Clack throws an exception when rendering a page, the server waits for the response until it times out while the exception waits in the REPL. This isn’t very useful. So now there’s this.

It prints the stacktrace along with some request details on the browser. Can return a custom error page in production.

clack-pretend

Are you tired of jumping to your web browser every time you need to test your work in Clack? Clack-pretend will capture and replay calls to your clack middleware stack. When developing a web application with clack, you will often find it inconvenient to run your code from the lisp REPL because it expects a clack environment, including perhaps, cookies or a logged-in user. With clack-pretend, you can run prior web requests from your REPL, moving development back where it belongs.

Testing

Testing with a local DB: example of a testing macro.

We would use envy to switch configurations.

Misc

Oauth, Job queues, etc

Templating engines

HTML-based

Djula: as Django templates. Good documentation. Comes by default in Lucerne and Caveman.

We also use a dot to access attributes of dict-like variables (plists, alists, hash-tables, arrays and CLOS objects), such a feature being backed by the access library.

We wanted once to use structs and didn’t find how to it directly in Djula, so we resorted in a quick helper function to transform the struct in an alist.

Defining custom template filters is straightforward in Djula, really a breeze compared to Django.

Eco - a mix of html with lisp expressions.

Truncated example:

<body>
      <% if posts %>
        <h1>Recent Posts</h1>
        <ul id="post-list">
          <% loop for (title . snippet) in posts %>
            <li><%= title %> - <%= snippet %></li>
          <% end %>
        </ul>
        ...

Lisp-based

I prefer the semantics of Spinneret over cl-who. It also has more features (like embeddable markdown, warns on malformed html, and more).

Javascript

Parenscript

Parenscript is a translator from an extended subset of Common Lisp to JavaScript. Parenscript code can run almost identically on both the browser (as JavaScript) and server (as Common Lisp). Parenscript code is treated the same way as Common Lisp code, making the full power of Lisp macros available for JavaScript. This provides a web development environment that is unmatched in its ability to reduce code duplication and provide advanced meta-programming facilities to web developers.

https://common-lisp.net/project/parenscript/

JSCL

A Lisp-to-Javascript compiler bootstrapped from Common Lisp and executed from the browser.

https://github.com/jscl-project/jscl

https://t-cool.github.io/jscl-playground/

Ajax

Is it possible to write Ajax-based pages only in CL?

The case Webblocks - Reblocks, 2017

Weblocks is an “isomorphic” web frameworks that solves the “Javascript problem”. It allows to write the backend and an interactive client interface in Lisp, without a line of Javascript, in our usual Lisp development environment.

The framework evolves around widgets, that are updated server-side and are automatically redisplayed with transparent ajax calls on the client.

It is being massively refactored, simplified, rewritten and documented since 2017. See the new quickstart:

http://40ants.com/weblocks/quickstart.html

Writing a dynamic todo-app resolves in:

defining a widget class for a task:

(defwidget task ()
        ((title
          :initarg :title
          :accessor title)
         (done
          :initarg :done
          :initform nil
          :accessor done)))

doing the same for a list of tasks:

(defwidget task-list ()
        ((tasks
          :initarg :tasks
          :accessor tasks)))

saying how to render these widgets in html by extending the render method:

(defmethod render ((task task))
        "Render a task."
        (with-html
              (:span (if (done task)
                         (with-html
                               (:s (title task)))
                       (title task)))))

(defmethod render ((widget task-list))
        "Render a list of tasks."
        (with-html
              (:h1 "Tasks")
              (:ul
                (loop for task in (tasks widget) do
                      (:li (render task))))))

telling how to initialize the Weblocks app:

(defmethod weblocks/session:init ((app tasks))
         (declare (ignorable app))
         (let ((tasks (make-task-list "Make my first Weblocks app"
                                      "Deploy it somewhere"
                                      "Have a profit")))
           (make-instance 'task-list :tasks tasks)))

and then writing functions to interact with the widgets, for example adding a task:

(defmethod add-task ((task-list task-list) title)
        (push (make-task title)
              (tasks task-list))
        (update task-list))

Adding an html form and calling the new add-task function:

(defmethod render ((task-list task-list))
        (with-html
          (:h1 "Tasks")
          (loop for task in (tasks task-list) do
            (render task))
          (with-html-form (:POST (lambda (&key title &allow-other-keys)
                                         (add-task task-list title)))
            (:input :type "text"
                    :name "title"
                    :placeholder "Task's title")
            (:input :type "submit"
                    :value "Add"))))

Shipping

We can build our web app from sources, no worries, that works.

Building

We can build an executable also for web apps. That simplifies a deployment process drastically.

We can even get a Lisp REPL and interact with the running web app, in which we can even install new Quicklisp dependencies. That’s quite incredible, and it’s very useful, if not to hot-reload a web app (which I do anyways but that might be risky), at least to reload a user’s configuration file.

This is the general way:

(sb-ext:save-lisp-and-die #p"name-of-executable" :toplevel #'main :executable t)

But this is an SBCL-specific command, so we can be generic and use asdf:make, with a couple lines inside our system .asd declaration. See the Cookbook: https://lispcookbook.github.io/cl-cookbook/scripting.html#with-asdf

Now if you run your binary, your app will start up all fine, but it will quit instantly. We need to put the web server thread on the foreground:

(defun main ()
    ;; with bordeaux-threads. Also sb-ext: join-thread, thread-name, list-all-threads.
    (bt:join-thread (find-if (lambda (th)
                                (search "hunchentoot" (bt:thread-name th)))
                              (bt:all-threads))))

When I run it, Hunchentoot stays listening at the foreground:

$ ./my-webapp
Hunchentoot server is started.
Listening on localhost:9003.

I can use a tmux session (tmux, then C-b d to detach it) or better yet, start the app with Systemd, see below.

But we need something more for real apps, we need to ship foreign libraries. Deploy to the rescue. Edit your .asd file slightly to have:

:defsystem-depends-on (:deploy)  ;; (ql:quickload "deploy") before
:build-operation "deploy-op"     ;; instead of "program-op" as above
:build-pathname "my-webapp"  ;; doesn't change
:entry-point "my-webapp:main"  ;; doesn't change

Build the app again with asdf:make, and see how Deploy discovers and ships in the required foreign libraries: libssl.so, libosicat.so, libmagic.so…

It puts the final binary and the .so libraries in a bin/ directory. This is what you’ll have to ship.

I can now build my web app, send it to my VPS and see it live \o/

One more thing. We don’t want to ship libssl and libcrypto, so we ask Deploy to not ship them. We prefer to rely on the target OS.

;; .asd
;; Deploy may not find libcrypto on your system.
;; But anyways, we won't ship it to rely instead
;; on its presence on the target OS.
(require :cl+ssl)  ; sometimes necessary.
#+linux (deploy:define-library cl+ssl::libssl :dont-deploy T)
#+linux (deploy:define-library cl+ssl::libcrypto :dont-deploy T)

There’s also a hack if you have an issue with ASDF trying to update itself… see the skeleton: https://github.com/vindarel/cl-cookieweb

To be exhaustive, here’s how to catch a user’s C-c and stop our app correctly. By default, you would get a full backtrace. Yuk.

(defun main ()
  (start-app :port 9003)
  ;; with bordeaux-threads
  (handler-case (bt:join-thread (find-if (lambda (th)
                                             (search "hunchentoot" (bt:thread-name th)))
                                         (bt:all-threads)))
    (#+sbcl sb-sys:interactive-interrupt
      #+ccl  ccl:interrupt-signal-condition
      #+clisp system::simple-interrupt-condition
      #+ecl ext:interactive-interrupt
      #+allegro excl:interrupt-signal
      () (progn
           (format *error-output* "Aborting.~&")
           (clack:stop *server*)
           (uiop:quit 1)) ;; portable exit, included in ASDF, already loaded.
    ;; for others, unhandled errors (we might want to do the same).
    (error (c) (format t "Woops, an unknown error occured:~&~a~&" c)))))

To see:

a Debian package for every Quicklisp system: http://margaine.com/2015/12/22/quicklisp-packagecloud-debian-packages.html.

Multiplatform delivery with Electron (Ceramic)

Ceramic makes all the work for us.

It is as simple as this:

;; Load Ceramic and our app
(ql:quickload '(:ceramic :our-app))

;; Ensure Ceramic is set up
(ceramic:setup)
(ceramic:interactive)

;; Start our app (here based on the Lucerne framework)
(lucerne:start our-app.views:app :port 8000)

;; Open a browser window to it
(defvar window (ceramic:make-window :url "http://localhost:8000/"))

;; start Ceramic
(ceramic:show-window window)

and we can ship this on Linux, Mac and Windows.

Ceramic applications are compiled down to native code, ensuring both performance and enabling you to deliver closed-source, commercial applications.

(so no need to minify our JS)

with one more line:

(ceramic.bundler:bundle :ceramic-hello-world
                                 :bundle-pathname #p"/home/me/app.tar")
Copying resources...
Compiling app...
Compressing...
Done!
#P"/home/me/app.tar"

This last line was buggy for us.

Deployment

When you build a self-contained binary (see above, “Shipping”), deployment gets easy.

Manually

sbcl --load <my-app> --eval '(start-my-app)'

For example, a run Makefile target:

run:
	sbcl --load my-app.asd \
	     --eval '(ql:quickload :my-app)' \
	     --eval '(my-app:start-app)'  ;; given this function starts clack or hunchentoot.

this keeps sbcl in the foreground. We can use tmux to put it in background, or use Systemd.

Then, we need of a task supervisor, that will restart our app on failures, start it after a reboot, handle logging. See the section below and example projects.

with Clack

$ clackup app.lisp
Hunchentoot server is started.
Listening on localhost:5000.

with Docker

So we have various implementations ready to use: sbcl, ecl, ccl… with Quicklisp well configured.

https://lispcookbook.github.io/cl-cookbook/testing.html#gitlab-ci

On Heroku

See heroku-buildpack-common-lisp and the Awesome CL#deploy section.

Systemd: daemonizing, restarting in case of crashes, handling logs

Generally, this depends on your system. But most GNU/Linux distros now come with Systemd. Write a service file like this:

$ /etc/systemd/system/my-app.service
[Unit]
Description=stupid simple example

[Service]
WorkingDirectory=/path/to/your/app
ExecStart=/usr/bin/sbcl --load run.lisp  # your command, full path
Type=simple
Restart=always
RestartSec=10

One gotcha is that your app must be run on the foreground. See the threads snippet above in Shipping, and add it when running the app from sources too. Otherwise, you’ll see a “compilation unit aborted, caught 1 fatal error condition” error. That’s simply your Lisp quitting too early.

Run this command to start the service:

sudo systemctl start my-app.service

to check its status:

systemctl status my-app.service

Systemd handles logging for you. We write to stdout or stderr, it writes logs:

journalctl -u my-app.service

use -f -n 30 to see live updates of logs.

This tells Systemd to handle crashes and to restart the app:

Restart=always

and it can start the app after a reboot:

[Install]
WantedBy=basic.target

to enable it:

sudo systemctl enable my-app.service

Debugging SBCL error: ensure_space: failed to allocate n bytes

If you get this error with SBCL on your server:

mmap: wanted 1040384 bytes at 0x20000000, actually mapped at 0x715fa2145000
ensure_space: failed to allocate 1040384 bytes at 0x20000000
(hint: Try "ulimit -a"; maybe you should increase memory limits.)

then disable ASLR:

sudo bash -c "echo 0 > /proc/sys/kernel/randomize_va_space"

Connecting to a remote Swank server

Little example here: http://cvberry.com/tech_writings/howtos/remotely_modifying_a_running_program_using_swank.html.

It defines a simple function that prints forever:

;; a little common lisp swank demo
;; while this program is running, you can connect to it from another terminal or machine
;; and change the definition of doprint to print something else out!
;; (ql:quickload '(:swank :bordeaux-threads))

(require :swank)
(require :bordeaux-threads)

(defparameter *counter* 0)

(defun dostuff ()
  (format t "hello world ~a!~%" *counter*))

(defun runner ()
  (bt:make-thread (lambda ()
                    (swank:create-server :port 4006)))
  (format t "we are past go!~%")
  (loop while t do
       (sleep 5)
       (dostuff)
       (incf *counter*)))

(runner)

On our server, we run it with

sbcl --load demo.lisp

we do port forwarding on our development machine:

ssh -L4006:127.0.0.1:4006 username@example.com

this will securely forward port 4006 on the server at example.com to our local computer’s port 4006 (swanks accepts connections from localhost).

We connect to the running swank with M-x slime-connect, typing in port 4006.

We can write new code:

(defun dostuff ()
  (format t "goodbye world ~a!~%" *counter*))
(setf *counter* 0)

and eval it as usual with M-x slime-eval-region for instance. The output should change.

There are more pointers on CV Berry’s page.

Hot reload

When we run the app as a script we get a Lisp REPL, so we can hot-reload the running web app. Here we demonstrate a recipe to update it remotely.

Example taken from Quickutil.

It has a Makefile target:

hot_deploy:
	$(call $(LISP), \
		(ql:quickload :quickutil-server) (ql:quickload :swank-client), \
		(swank-client:with-slime-connection (conn "localhost" $(SWANK_PORT)) \
			(swank-client:slime-eval (quote (handler-bind ((error (function continue))) \
				(ql:quickload :quickutil-utilities) (ql:quickload :quickutil-server) \
				(funcall (symbol-function (intern "STOP" :quickutil-server))) \
				(funcall (symbol-function (intern "START" :quickutil-server)) $(start_args)))) conn)) \
		$($(LISP)-quit))

It has to be run on the server (a simple fabfile command can call this through ssh). Beforehand, a fab update has run git pull on the server, so new code is present but not running. It connects to the local swank server, loads the new code, stops and starts the app in a row.

Demo: fetching the GitHub REST API with Common Lisp

In this video I show you how to create a new project and we explore the Lisp ecosystem to make API calls, notably how to fire HTTP requests and parse JSON. We develop in Emacs and SLIME and in the end we get a Lisp library AND an application for the command line. Hope you enjoy the ride.

New video: how to request a REST API in Common Lisp: fetching the GitHub API 🎥

Sat, 11 Jun 2022 11:43:31 +0200

A few weeks ago, I put together a new Lisp video. It’s cool, sound on, ‘til the end ;)

I want to show how to (quickly) do practical, real-world stuff in Lisp. Here, how to request a web API. We create a new full-featured project with my project skeleton, we study the GitHub API, and we go ahead.

I develop in Emacs with Slime, but in the end we also build a binary, so we have a little application that works on the command line (note that I didn’t use SBCL’s core compression, we could have a lighter binary of around 30MB).

I use the libraries: Dexador for HTTP requests, the Jonathan and then the Shasht JSON libraries, as well as Access and Serapeum to help with accessing, creating and viewing hash-tables.

Thanks for the comments already :)

tks for that… it’s amazing.

excellent

Thank you, that was a great video! I plan to get the udemy course now based on this.

Here are my previous ones:

I want to do more but damn, it takes time. You can push me for more :)

Last notes:

I had fun with the soundtrack :)

I edit the video with the latest Kdenlive with the snap package and it’s great for me.

Video: Create a Common Lisp project from scratch with our project generator 🎥

Wed, 11 May 2022 11:43:31 +0200

In this video I want to demo real-world Lisp stuff I had trouble finding tutorials for:

how to create a CL project:
- what’s in the .asd file?
- what’s a simple package definition?
- how do we load everything in our editor (Emacs and SLIME here)?
how to set up tests?
- and how to run them from the terminal?
- and (WTF) how to get the correct exit code????
how to build a binary in order to run our app from the terminal?
- and so, how to get command-line arguments?
- but also, can I run my app from sources, without building a binary?
what’s Roswell and how do I share my app with it?

Let’s find out (it’s 5 minutes long):

I use my project generator: cl-cookieproject. See setup, alternatives, limitations and TODOs there. You will also find a similar generator for web projects.

If you find it useful, share the video!

Writing an interactive web app in Common Lisp: Hunchentoot then CLOG

Thu, 28 Apr 2022 15:23:14 +0200

We want a web app to display a list of data and have an input field to interactively filter it.

We’ll start with a simple, regular web app built with Hunchentoot. We’ll have a search input to filter our data, and we’ll see that to be more interactive, typically to filter out the results as the user types, we’ll need more than basic HTTP requests. We’ll need some JavaScript. But we’ll reach this level of interactivity with CLOG (and no JavaScript).

DISCLAIMER: this post is my entry for the CLOG contest!

Let’s install our first libraries: Hunchentoot for the web server, Djula for the HTML templates, str for a string utility.

#+(or)
(ql:quickload '("hunchentoot" "djula" "str"))

We create a package for our experiments, and we “enter” it. I use UIOP’s define-package because it throws less warnings than defpackage when we add and remove symbols. It also has more features (:reexport) that I don’t use here.

(uiop:define-package :clog-contest
    (:use :cl))

(in-package :clog-contest)

OK. We define a route. It takes one GET parameter, for demo purposes.

(hunchentoot:define-easy-handler (root-route :uri "/") (name)
  (format nil "Hey~@[ ~A~]!" name))

Start the server:

(defvar *server* (make-instance 'hunchentoot:easy-acceptor :port 6789))
(hunchentoot:start *server*)

and access http://localhost:6789/ Now let’s create our products. We quickly define a class containing an ID, a title and a price.

(defclass product ()
  ((id :initarg :id :accessor product-id :type integer
       :documentation "Unique ID")
   (title :initarg :title :accessor product-title :type string)
   (price :initarg :price :accessor product-price :type integer)))

(defvar *product-id* 1
  "Stupid counter to increment our unique product ID.
  Normally this is given by a DB.")

(defparameter *products* '() "A list of products.")

We are going to create random testing products, so let’s have a couple helpers to create random titles and prices.

(defun random-price ()
  "Return an integer between 1 and 10.000 (price is expressed in cents)."
  (1+ (random 9999)))

(defparameter *title-part-1* (list "pretty" "little" "awesome" "white" "blue"))
(defparameter *title-part-2* (list "book" "car" "laptop" "travel" "screwdiver"))
(defun random-title ()
  (let ((index (random (length *title-part-1*)))
        (index-2 (random (length *title-part-2*))))
    (format nil "~a ~a" (elt *title-part-1* index) (elt *title-part-2* index-2))))

try it out:

#+(or)
(random-title)

We get titles like “white book”, “little car”, etc. Now, for testing purposes, we create a 100 dummy product instances:

(defun gen-test-products (&optional (nb 100))
  (dotimes (i nb)
    (push (make-instance 'product
                         :id (incf *product-id*)
                         :title (random-title)
                         :price (random-price))
          *products*)))

(defun reset-test-products ()
  (setf *products* nil))

Try it and we get:


*products*
(#<PRODUCT {1005B29363}> #<PRODUCT {1005B29113}> #<PRODUCT {1005B28EC3}>
 #<PRODUCT {1005B28C73}> #<PRODUCT {1005B28A23}> #<PRODUCT {1005B287D3}>
 …)

Implement the print-object method if you want nice-looking product literals. See the Cookbook.
Now let’s display the products in the browser. We will redefine our route. We make sure to extract the view logic in functions.


(defun print-product (it &optional (stream nil))
  "Print a product title and price on STREAM (return a new string by default)."
  (format stream "~a - ~f~&"
          (str:fit 20 (product-title it))  ;; the fit function was merged recently.
          (/ (product-price it) 100)))

(defun print-products (products)
  "Return a list of products as a string (dummy, for tests purposes)."
  (with-output-to-string (s)
    (format s "Products:~&")
    (dolist (it products)
      (print-product it s))))


CL-USER> (print-products (subseq *products* 0 10))

"Products:
pretty car           -   22.26
awesome travel       -   13.87
little screwdiver    -   35.6
white laptop         -   6.08
little book          -   27.57
white laptop         -   42.63
blue travel          -   93.8
blue car             -   29.99
pretty car           -   38.95
little screwdiver    -   46.99

We tell our route to display the list of products like this:


(hunchentoot:define-easy-handler (root-route :uri "/") ()
  (print-products *products*))

We see something, but it’s stupid to return text to the browser. We need templates. We’ll use Djula templates. And we’ll steal some ready-to-use pretty HTML :)

I’ll use Bulma CSS because it’s simple, modern (flexbox) and just because. I don’t know all CSS frameworks out there. I’ll do my shopping in this showcase of Bulma templates: https://bulmatemplates.github.io/bulma-templates/ IIRC I took the “Modal Cards” one and simplified it a bit. Our final result is: Let’s create a templates/ directory and create:

base.html
products.html, that inherits the base.
- inside this template, we “include” the search form that we wrote in another template: search-form.html. It’s a way to factorize HTML code that we can re-use here and there.

The base template loads Bulma from a CDN, creates a navbar, defines a “content” block that our other templates will override, and a footer.

Our products template “extends” base.html and creates the “content” block. There we loop over a list of products given by our Hunchentoot root and display them. But before that happens, we need to install and configure Djula to find and compile our templates. We tell Djula to look for templates in the templates/ directory.

(djula:add-template-directory "templates/")

Note that normally, I do that relatively to an .asd file, that we didn’t create yet:


(djula:add-template-directory
 (asdf:system-relative-pathname "myproject" "src/templates/"))

If you have an issue with the path on the Lisp REPL, on SLIME you can do ,cd (a comma command) to change the current working directory. Now we define our templates:

(defparameter +base.html+ (djula:compile-template* "base.html"))
(defparameter +products.html+ (djula:compile-template* "products.html"))

As a result, you can see they are compiled templates:


CLOG-CONTEST> +products.html+
#<DJULA::COMPILED-TEMPLATE /home/vince/bacasable/bacalisp/lisp-tutorial-clog-contest/templates/products.html {2073D30B}>

OK, our route needs to return a template and give data to it. Our route returns djula:render-template*.

(hunchentoot:define-easy-handler (root-route :uri "/") ()
  (djula:render-template* +products.html+ nil
                          :products *products*))

Nice! We display all products. We will accept a search query to filter them. Pagination is for later.

We have an input field that defines an HTML form, that calls the search endpoint. We need:

to define the /search route
to write a dummy function to search in our products list.


(defun search-products (query &optional (products *products*))
  "Search for QUERY in the products' title.
  This would be a DB call."
  (loop for product in products
     when (str:containsp (str:downcase query) (str:downcase (product-title product)))
     collect product))

Try it:

#+(or)
(search-products "awesome")

Now the search route. It accepts one GET parameter, q for “query”.


(hunchentoot:define-easy-handler (search-route :uri "/search") (q)
  (let* ((products (search-products *products* q)))
    (djula:render-template* +products.html+ nil
                            :title (format nil "My products - ~a" q)
                            :query q
                            :products products
                            :no-results-p (zerop (length products)))))

Try it, it works :) I agree, the search algorithm is simplistic. What about multiple words, accents, typos, non-exact searches (stemming)…?

Your search query is seen in the URL parameters: http://localhost:6789/search?q=travel That is usually a good thing. In modern single-page applications, you loose this, or you have to handle the URL construction yourself.

The search required a page reload. If your app is fast, it might not be an issue. However, if we wanted the search to be more interactive, for example showing results as we type, we would need to use JavaScript. Enters CLOG.

CLOG

Can we make our app interactive with CLOG?

Well, we can, and what’s even cooler is that the development process is itself very interactive. CLOG sends changes to the page through websockets as you add or edit functionalities. As such we can see changes in real time. For example, change a colour:


(setf (background-color *body*) :red)

and BAM, it’s red.

Let’s create another package for this new app. I’ll “use” functions and macros provided by the :clog package, as well as our previously defined :clog-contest ones (duh… we didn’t :export any yet).


(uiop:define-package :clog-contest-with-clog
    (:use :cl :clog
          :clog-contest))

(in-package :clog-contest-with-clog)

The very first steps you can do to grasp CLOG’s interactive fun is to make changes to a browser window while on the CLOG REPL.


(ql:quickload "clog")
CL-USER> (in-package clog-user)
CLOG-USER> (clog-repl)
CLOG-USER> (setf (background-color *body*) "red")
CLOG-USER> (create-div *body* :content "Hello World!")

And voilà. A browser window was opened for you.

You will also find many demos here: https://github.com/rabbibotton/clog/tree/main/tutorial You can run them with (clog:run-tutorial 1) (by their number id).

For the following, I invite you to have a look at CLOG’s common elements: https://rabbibotton.github.io/clog/clog-manual.html#toc-8-common-clog-elements

Typically, to create a div on a DOM element, we use create-div.

The first thing we want to start our CLOG app is the initialize function. Its signature:


initialize (on-new-window-handler &key (host 0.0.0.0) (port 8080) (server hunchentoot)
 (extended-routing nil) (long-poll-first nil) (boot-file /boot.html)
 (boot-function nil) (static-boot-html nil) (static-boot-js nil)
 (static-root (merge-pathnames ./static-files/ (system-source-directory clog))))

Inititalize CLOG on a socket using HOST and PORT to serve BOOT-FILE
as the default route to establish web-socket connections and static
files located at STATIC-ROOT. […]

The following calls our add-products function with a body (CLOG object) as argument.

(defun start-tutorial ()
  "Start tutorial."
  (initialize 'add-products)
  (open-browser))

OK so what do we want to do? We want to create a search input field, and to display our products below. When the user types something, we want to immediately filter the products, and re-display them.

A first version where we only display products would be this:


(defun add-products (body)
  (let* ((result-div (create-div body :content "")))
    (display-products result-div (subseq clog-contest::*products* 0 10))))

And the display-products function is below:


(defun display-products (body products)
  "Display these products in the page.
  Create a div per product, with a string to present the product.
  We don't create nice-looking Bulma product cards here."
  (dolist (it products)
      (create-div body :content
                  (format nil "~a - ~a"
                          (clog-contest::product-id it)
                          (clog-contest::print-product it)))))

Now we want to handle the interactivity. The event to watch is the key up event. In CLOG, we have the set-on-key-up method. It takes: a CLOG object (the DOM object it watches for events) and a handler function. This function takes two arguments: the CLOG object and the event.

In our add-products function below, we create the search input and we listen the key-up event:


(defun add-products (body)
  "Create the search input and a div to contain the products.
  Bind the key-up event of the input field to our filter function."
  (let* ((form (create-form body))
         (input (create-form-element form :input :name "query"
                                     :label
				     (create-label form :content "Filter product: ")))
         (result-div (create-div body :content "" )))

    (set-on-key-up input
                   (lambda (obj event)
                     (format t ":key-up, value: ~a~&" (value obj)) ; logging
                     (setf (text result-div) "") ; this is how we erase the current content.
                     (handle-filter-product result-div obj event)))

    (display-products result-div clog-contest::*products*)))

Below, to find out what is typed in the search input, we use (value obj).


(defun handle-filter-product (div obj event)
  "Search and redisplay products."
  ;TODO: wait a little latency
  (declare (ignorable event))
  (let ((query (value obj)))
    (if (> (length query) 2)
        (display-products div (clog-contest::search-products query))
        (print "waiting for more input"))))

It works \o/

There are some caveats that need to be worked on:

if you type a search query of 4 letters quickly, our handler waits for an input of at least 2 characters, but it will be fired 2 other times. That will probably fix the blickering.

And, as you noticed:

we didn’t copy-paste a nice looking HTML template, so we have a bit of work with that :/

CLOG is not at all limited to websites like this. You can create games (there is a Snake demo), multiplayer applications (there is a chat demo)… all this by doing everything in the backend, in Common Lisp, with a lot of interactivity under the fingertips. Try it out!

By the way, this post was written in a literate style with Erudite. Everything is written in a .lisp file, and exported to markdown. Read about it here and see its source on GitHub. You can wget this source, open it in your editor and compile the snippets along the way.

For more web stuff, see:

https://github.com/CodyReichert/awesome-cl#web-frameworks traditional web frameworks as well as Weblocks (Reblocks), CLOG, ISSR…
https://lispcookbook.github.io/cl-cookbook/web.html
a demo: https://github.com/vindarel/demo-web-live-reload
a demo, very similar to our product list built with Djula and Bulma CSS, a bit more complete: https://github.com/vindarel/lisp-web-template-productlist
ISSR todo-list demo: https://github.com/vindarel/demo-ISSR-djula
my web project skeleton: https://github.com/vindarel/cl-cookieweb
a demo of Caveman and htmx, a neat JS library that brings easy interactivity for many use cases: cl-beers. “htmx gives you access to AJAX, CSS Transitions, WebSockets and Server Sent Events directly in HTML, using attributes, so you can build modern user interfaces with the simplicity and power of hypertext”. It doesn’t always replace a JS framework, but is worth considering.

Resources

Thu, 21 Apr 2022 07:51:49 +0100

lisp-lang.org
Common Lisp Cookbook
awesome-cl
- learning and tutorials
  - Lisp project of the day
  - Google’s Lisp koans
  - Lisp tips a GitHub repository where anybody can post tips an tricks.

search libraries on

http://quickdocs.org/

Individual sites:

sjl’s road to Lisp
Martin Cracauer’s Gentle Introduction to compile-time computing - excellent article series

Screencasts:

Of course, see my Udemy Lisp course!

I also post videos on Youtube, check this out:

How to quickly create a Common Lisp project with my project generator: see an ASDF system definition, how to build a binary, run the project from sources, load everything in Emacs, run the test suite…
How to interactively fix failing tests - very short video to showcase the interactive debugger and that we can re-compile a failing function and resume the execution from where it failed to see it finally pass.

Those ones ar great too:

Little Bits of Lisp - short videos on various topics: inspecting a condition, basics of lisp’s evaluation model,…
Common Lisp Tutorials, of which Emacs and Slime - useful keyboard shortcuts
Programming a message bus in Common Lisp - shows the interactive nature of lisp, good use of the debugger, test-driven development to shape the api, bordeaux-threads.
Marco Baringer’s SLIME Tutorial Video - a video showing Marco develop a package and explaining Slime and Lisp features, with many little important explanations (1 hour). It only has some rotten bits, for example it installs packages with asdf-install and not Quicklisp.
Shinmera playlists: Treehouse (game dev), Demos (of small programs).
Pushing pixels with Lisp, and by the same author:
- CEPL demo - working with OpenGL
- Baggers’ channels.
Cando: computational chemistry with Common Lisp on LLVM with Jupyter notebooks
McClim interactive GUI demos. Code examples. Presentation of Clim listener, Clim debugger, drawing objects into the GUI repl.
videos of the European Lisp Symposium

and more on Cliki.

Some games:

and more games (Lisp Game Jam 2023 submissions).

Kandria - a nice platform game launched on Steam. Check it out!
nano-towers - a simple tower defense game written in Common Lisp with the EON framework based on Raylib, submitted for the Spring Lisp Game Jam 2024.

dmomd - A rogue-like RPG with turn based movement and combat.

The Maze and Lost Cat (in-browser)

Thoughtbound

Jettatura - a challenging first-person dungeon-crawler (DRPG) with round-based combat. On Steam and planned for 2022, Q2.
Orb

Notalone

Moppu

The price of a cup of coffee

Fruktorama, Tetris-like with fruits.
Spycursion - “a sandbox “edutainment” MMO centered around hacking and espionage which takes place in a near-future world”. [stalled]

I Am Creating a Common Lisp Video Course on Udemy (free video previews) 🎥

Fri, 15 Apr 2022 15:23:14 +0200

Everyone, let me celebrate a little bit: I am creating a Common Lisp video course on the Udemy platform. I’m several dozen hours in already and it’s taking a good shape! It is so much more time consuming to create videos than to write a tutorial O_o But I like what’s in there already, although there isn’t everything I want to teach, of course. I’m working on more content. Everything will come in time, and meanwhile you can buy the course: you’ll get future content for “free” ;) Yes the course is to sell, hopefully it will help me concentrate more on my CL activities (BTW, dear reader, here’s a 50% off coupon for April, 2022, and if you are a student drop me a line). Currently 4 videos are freely viewable, and I’m also posting new videos on Youtube (that one is on how to create a new Common Lisp project with my project generator) but more on that later. You currently have more than 3 hours of learning material.

You probably know me from my blog and my Lisp activities. I am vindarel, I contribute to community-based learning resources such as the Cookbook and I use CL in production©. Honestly, I missed a Cookbook-like resource as a beginner. I dumped there a lot of content and must-know tricks that I either learned the hard way, either learned by chance. But I still want Common Lisp to be easier to learn and, for that, there is the video media. Consequently, I truely think this course is today’s most efficient way to learn Common Lisp.

So, what can you learn already in my course?

I want it to be practical: you will learn Lisp the language in order to build real-world stuff.

Chapter 1 is how to get started

1.1. We start by installing SBCL on our machine (showed for Unix, links for Windows). This one is 15m long and is 🆓 free to watch. We see how to start our Lisp, how to write “hello world”, we understand the output, we add readline support to the SBCL default REPL in the terminal, we disable the interactive debugger, and we have a few words on Lisp implementations and GNU CLISP in particular.

1.2. We see how to run Lisp code, the simplest way. We write a code snippet with a simple text editor and we run it with sbcl’s --script and --load flags. We use the LOAD function.

1.3. We see how to use Portacle (the ready-to-use, multiplatform image shipping Emacs, SBCL, Quicklisp, Git and a couple handy Emacs packages).

Chapter 2 is about Lisp basics

2.1. I heard Lisp beginners who needed a recap on the Lisp syntax and the evualation model. This one is also 🆓 available to everyone. We see: the prefix notation, that everything is an expression, the evaluation model (and the exception of macros). Code is data is code… right?

2.2. How to define variables, at the toplevel or locally. How to lexically re-bind dynamic variables, the gotcha, the alternative.

2.3. and conditionals (if, when, #+or…)

That’s it for now for this chapter (yes, we’ll see data structures, but for now I refer you to the Cookbook, on the page that I also authored. You are armed to read it.).

Chapter 3 is about iteration. It is made of shorter videos that typically sum up in 5 minutes

…what took me a long time to learn or discover, a way longer time to admit.

3.1 Iterating over lists and vectors (with 🆓 free preview). loop, dolist and other libraries of the ecosystem.

3.2 then: Iterating over a hash-table keys and values. We see 5 different ways in 5 minutes.

3.3 Iterating a fixed or infinite number of times, and we take the opportunity to build our first read-eval-print-loop.

3.4. Here, we take a high level overview of loop and we study some gotchas. We see a practical example from an answer to last year’s Advent Of Code.

Chapter 4 teaches everything you need to know about functions

(with a sneak peak into CLOS):

4.1. How to create named functions, how to handle all types of arguments (🆓 free preview). We see defun, returned values, required arguments, optional arguments, key arguments, how to set a default value, how to know if an argument was supplied, &rest, example of apply, feature flags…

4.2. Referencing functions, redefining functions locally, accessing documentation

4.3. Multiple Return Values (they are NOT like returning a list or a tuple!!!)

4.4. Higher Order Functions: how to give functions as arguments, member, the :test keyword, map and mapcar, lambda, how to generate functions, what are symbols, setf symbol-function. A word on currying and being a Lisp-2.

4.5. Closures

4.6. setf functions

4.7. Generic Functions (quick intro to CLOS): they allow to write functions that dynamically dispatch on the type of their arguments. What we see (quickly): defmethod, defgeneric.

and, lastly (for now):

Chapter 5 shows how to work with projects

5.1 How to work with an existing project. ASDF, Quicklisp, SLIME shortcuts, some useful asdf functions…

5.2. How to create a new project. As a bonus, see again my new video on Youtube: Common Lisp: how to create a new project (demo of my project generator).

5.3. What are systems and packages anyways? Demo. Gotcha when creating a package. Finding all external symbols of a package.

Soooo

Who is this course for by the way?

I must warn that this course is not for total newcomers in programming. You should know what variables and functions are. You should knew that Common Lisp is a language of the Lisp family! (I tell more in the presentation video, but still).

Lisp newbies are welcome because I introduce Lisp basics (syntax, evaluation model) and I show how to install everything. It would help if you know what is a language of the Lisp family.

This course is mainly targetting young(ish) profesional developers like me, who feel they deserve a more fun, comfy, compiled and fast programming language.

It is for Python or JavaScript programmers frustrated by the unstability of their ecosystem,

for students of computer science who want to discover why Lisp still has un-matched alien technology inside (and maybe for *your* students?),

for Clojurists who want to transition quickly to a bare-metal Lisp,

or simply for your friend or colleague to whom you are trying to sell the power of Lisp ;)

Closing thoughts

If you are already a programmer: you can watch the videos at speed 1.25 or higher, but try to not skip content. You can start by the chapter of your choice. Use the captions, they are manually edited.

Thanks kindly for your support in that new journey of mine! Hope you’ll enjoy the content (I know you’ll learn a few tricks).

Learn Common Lisp now! It's a tool for a lifetime.

And have fun!

🎥 Common Lisp: from novice to effective developer 🎥

project’s GitHub
- request: debugging chapter
a known issue: I started with a meh microphone. I bought a new one, sound is now good, but older videos were not re-captured yet.

ADDENDUM: the Lisp philosophy revealed

Udemy auto-generates captions for your videos. You can manually edit them… but sometimes I was tempted not to, as they can reveal some hidden truth^^

I can see my new prince here

reveals my respect for “print”.

Survivor needs votes.

I never thought about that O_o (that’s for “the variable name is unbound”)

CL might be more active than you think:

Google and China’s also actively contributes, as we said.

for “Google engineers also actively contribute to the SBCL implementation”. My cute accent is well interpreted beyond hopes (and it is not wrong, hello fellow chinese lispers o/ ).

and yes, I use communism prediction.

“I use CL in production”…

Lisp is used in more places than you think:

I want to give you a few tips for when you are dealing with cruise people

aka “for when you are dealing with macros”. And yes:

my crew is special

indeed :p “the loop macro is special”.

We knew a so called Lisp curse… there’s a conspiracy too!

the conspiracy control key

is “C-c C-k”.

and we can create with controversy

instead of “and we can quit with C-c C-c”.

you have to respect the water

you have to respect the order.

discipline is nested in your code.

for “deeply nested in your code”

I’ll show you from Oslo

= “I’ll show you from Emacs and Slime”, that’s quite an interpretation but Emacs is a great place to live in too.

And I also like that one:

I’m actually a terrific

that is the translation of the name “Eitaro Fukamachi”. Add “coder” and that matches :)

Who's using Common Lisp ?

Tue, 05 Apr 2022 07:51:49 +0100

Everyone says “Nobody uses Lisp” and Lispers say “Yes they do, there’s ITA, and, um, Autocad, and, uh, oh yeah, Paul Graham wrote Viaweb in Lisp!” Not very helpful for either side.

We now have a list: awesome-lisp-companies. It isn’t official nor exhaustive, but it’s way better than the past situaton.

Of course, see also:

lisp-lang.org’s success stories for a showcase of projects and companies in aerospace, AI & Machine Learning, Science, Graphics etc.
Franz.com success stories
LispWorks’ success stories

In a nutshell, what’s Common Lisp good for? Let’s quote Kent Pitman’s famous answer:

But please don’t assume this is an exhaustive list, and please don’t assume Lisp is only useful for Animation and Graphics, AI, Bioinformatics, B2B and Ecommerce, Data Mining, EDA/Semiconductor applications, Expert Systems, Finance, Intelligent Agents, Knowledge Management, Mechanical CAD, Modeling and Simulation, Natural Language, Optimization, Research, Risk Analysis, Scheduling, Telecom, and Web Authoring just because these are the only things they happened to list. Common Lisp really is a general language capable of a lot more than these few incidental application areas, even if this web page doesn’t totally bring that out.

(and this list doesn’t mention that it was used for auto-piloting the Deep Space 1 spaceship by the NASA for several days).

Here’s a shorter list of companies using CL in production©.

Quantum computing companies use CL, especially SBCL:

Rigetti

They already sponsored a Quicklisp development. They chose Common Lisp (SBCL). Video. Their Lisp even runs 40% faster than their C code.

D-wave systems, “quantum processor development”. “The software is implemented in Common Lisp (SBCL) and is an integral part of the quantum computing system.” lispjobs announce.

And there’s at least also HRL Laboratories in the Quantum space (I am not counting defunkt Quantum companies). They are “one of the world’s premier physical science and engineering research laboratories. [They] engage in the development of a full quantum device and computation stack, including an optimizing compiler for a quantum programming language”. Uses SBCL.

Grammarly is a famous English language writing-enhancement platform.

Ravenpack is “the leading big data analytics provider for financial services”. reddit announce.

ITA Software is still Google’s leading airfaire search and scheduling platform. They use SBCL, and contribute to its development.

Kina knowledge is a small company that “automates the processing of documents”: “We use Common Lisp extensively in our document processing software core for classification, extraction and other aspects of our service delivery and technology stack”. See their Lisp Interview: questions to Alex Nygren

Doremir Music Research AB develops ScoreCloud, a music notation software (a LispWorks product). It lets you play an instrument, sing or whistle into the app, and it writes the music score. Futuristic indeed.

I’ll stop here and I’ll let you discover the awesome-lisp-companies list!

Lisp Interview: Arnold Noronha of Screenshotbot: from Facebook and Java to Common Lisp.

Mon, 06 Dec 2021 19:08:30 +0100

I have come to like asking questions to people running companies that use CL, and I have Arnold in my radar for quite some time.

He contributed a while back to my cl-str library, and at that time, I don’t recall how many Lisp projects he had in his Github, but not as much as today. Since then, he created ScreenShotBot (an open-source screenshot testing service) and he released a few very useful Lisp (and Elisp) libraries. Wow! I’ll try to investigate what happened :)

First, ScreenShotBot. It’s an open-source project and a company. Its website shows a team of four. Can you tell us what’s the state of the project, of the company, what are your goals with it? Who are your clients?

Sure. So the team of four isn’t fully active at the moment. We’re all still friends, but Screenshotbot didn’t take off the way we wanted it to, so currently it’s just me and Reuben. I’m the primary developer, Reuben ¹ is actually my brother, he manages marketing. But he’s tech savvy. I’ll talk about how he contributes in code without knowing any Lisp in a second.

Lili ² and Francesco ³ are awesome designers we worked with in building our marketing pages.

Reuben and I are also prototyping some other ideas at the moment.

By the way, I think cl-str was probably one of the first CL libraries I contributed to. It’s also easily my most used library :)

You pinned the project facebook/screenshot-tests-for-android on your GH profile. Maybe you wrote about it already. What’s your relation with this project? (we read you “built the infrastructure for running Screenshot Tests at Facebook, which still runs tests from iOS, Android and React at Scale.”)

I was the original author of that project when I worked at Facebook. It’s the de-facto screenshot testing library for Android. (The other one, called Shot, uses this library under the hood.) It’s maintained by other engineers at Facebook at the moment.

At Facebook, I also ended up building the infrastructure to run these screenshot tests, and that infrastructure ended up being used for iOS and React and ElectronJS etc. The infrastructure stored the screenshots, bisected, sent tasks, notified on diff reviews etc.

With Screenshotbot I wanted to build a similar infrastructure that can be used outside of Facebook.

So… do you come from Facebook and from Java ? (ah, you worked at Google too!)

Yes Java! Also did C++ and lots of Python at Facebook.

What made you start a new project/company, and…

Speaking of Java.. Java is ridiculously slow to work with. Especially on Android, every single change required you to re-build the entire app. My initial goal when I started my own company was to solve this problem. I wanted to build CL style interactive development for Java, but without forcing developers to switch from Java. (Take a look at an early demo here: https://www.jipr.io/demo).

I haven’t completely abandoned this project, I might get back to this soon. But this is a good segue to your next question.

(the famous question) how did you end up with Common Lisp?

So I’ve been toying with Lisp for a while. I started using StumpWM around 2010, and have been using it ever since (I think I was looking for Emacs level customizability for my WM which CL and StumpWM gave me). My own personal website runs on CL, probably since about 2015. Using CL encouraged me to build small tools for personal use without much friction. (For instance, I have my own weight tracker tool, tracking my car mileage, tracking my investment portfolio etc.) I just had my server running on my desktop, and Emacs was perpetually connected to the Lisp server, so making changes would take less than a minute.

Even before I started using CL professionally, I attempted to build a shitty Lisp interpreter way back in 2012 ⁴. The goal here was mostly to learn about compilers.

Now, when I started working on Jipr, at some point I realized I needed to define some kind of “bytecode”. But then I thought, why not just use Lisp as the “bytecode”, i.e. the Java gets compiled to Lisp? I needed a Lisp-like language that works with Android and Java to build the actual Java interpreter. At that time, I could not find any CL that provided me the ability, so I boldly took the Lisp that I built in 2012, and actually built a shit tonne of real code on top of it. (The demo I linked to earlier used my homegrown Lisp.)

At some point, that stopped scaling. Every small change required lots of debugging because of the lack of SLIME, and other debugging tools. I gave up for a while, until one day I realized I could actually pay LispWorks, and they have already built Android support for me.

Paying for Lispworks was the best thing I’ve done in my startup journey. Initially, I just wanted the Android support for building Jipr, and I was able to remove all references of my homegrown Lisp with a month of effort. Their support is spectacular. And having access to Java libraries meant I could move a lot faster on my future ideas, including Screenshotbot.

Are you the only (Lisp) developer on Screenshotbot, did you have to make a colleague work on the Lisp codebase and how did that go?

I’m the solo Lisp developer. However: I’ve worked on multiple websites collaborating with my wife (who has some experience with Python, but not technical), and my brother (who is tech savvy, but not a programmer). Both helped me build websites in a Lisp codebase because of Markup! I helped them set up Atom + SLIMA (with SBCL, not Lispworks since I couldn’t afford another license), and then since it looked like HTML from that point, they would mostly work with a theme that we would purchase to build complex UIs.

When they took over the UI, that left me more time to work on building the core components and interactions for the apps.

We can find you on reddit. Besides it, is there another Lisp circle where you are active? (a pro mailing list?) Do you have interaction with the “pro” side of the CL community?

Hmm, I’m definitely active on Reddit. I follow the LispWorks HUG, but I’m not a super active participant. Do you have suggestions of where else I should be active on?

If I recall correctly, you use LispWorks, and you appreciate its Java FFI. Can you give details on your stack? When do you turn to the Java side?

I briefly described this above. For Jipr, which is literally a Java interpreter, Java interop was a necessity (the interpreter which runs in Lisp, keeps calling back and forth into Java). And it helps that Lispworks also has Android support (which CCL and ABCL don’t have, the other two implementations that support Java).

For Screenshotbot, the app has multiple integrations with externals tools such as Slack, Asana, etc. Sometimes, interacting with the services’ APIs aren’t that hard, and can be written in plain old Lisp. But often, I need to prototype a solution quickly, and having access to existing well-tested Java libraries are a necessity. (For instance, I had a few sales calls where the client was using a specific external tool for which I didn’t have the integration, and I would have to work overnight to build the integration).

Over-reliance on Java can eventually cause pain and suffering (I have to restart the Lisp process if I make a Java change), so I avoid writing any Java code, and instead talk to the Java libraries directly, even if it’s slower. Usually these Java libraries are not in the hot-path anyway, so performance isn’t critical. Occasionally I rewrite code that relied on Java to just use plain CL.

Anymore feedback on LispWorks? It seems you also extensively use Emacs and Slime.

Yeah, I definitely don’t use the LispWorks IDE. :) I tried it for a while, but it wasn’t for me. Sly (I prefer Sly over Slime these days), is definitely a more “complete” experience for me.

I’m the kind of hacker that is always optimizing their workflow. Emacs just makes it way more easier to do that. Slite is obviously one of my bigger examples, but there are always smaller things. (My most recent one was automatically guessing which package to import a symbol from, so I can press C-c i on a symbol, and it’ll give me a list of package candidates using ido-completing-read.)

Perhaps there are ways to do this with the LispWorks IDE, but I’m more experienced with Emacs Lisp, and there’s a wealth of documentation and blogs about how to do different things in Emacs.

How’s your CI and deployment story going, is everything fine, is there anything particular to CL?

I have a mono-repo. Almost all my Lisp code is in one single repo. Some of my newer open source libraries get copied over from the mono-repo to GitHub using Copybara.

I run my own Jenkins server, but plan to switch to Buildbot. Jenkins is annoying beyond a certain point. But it’s a great starting point for anybody who doesn’t have CI.

I have two desktops, and use my older desktop exclusively as a Jenkins worker. But Lispworks adds a complication because of licensing issues, so my primary desktop runs all the Lispworks jobs.

Deployment is interesting. I heavily use bknr.datastore ⁵, which while awesome, adds a little pain point to deployment. Restarting a service can take a 10-20s because we have to load all the objects to memory. So I avoid restarting the service.

Lispworks also adds a quirk. A deployed Lispworks image doesn’t have a compiler, so I can’t just git pull and (asdf:load-system ...). So instead, from my desktop I build a bundled fasl file, and have scripts to upload it to my server and have the server load the fasl file.

Finally, bknr.datastore has some issues which reloading code, which causes existing indices to become empty. I haven’t debugged why this happens, but I’m close. I have workaround scripts for these that can correct a bad index, but because of the potential of bringing my datastore into a bad state, deployment is pretty manual at the moment.

Some CL libraries you particularly like? Some you wish existed?

Apart from the usual suspects (cl-str, alexandria, cl-json, hunchentoot etc):

bknr.datastore: This is a game changer. If you’re the type of person that likes prototyping things quickly, this is for you. If you need to scale things, you can always do it when you have to. (In my experience, not all ideas reach the stage where you need to scale over a 100 servers.) It does take a bit of getting used to (particularly dealing with how to recover old snapshots, or replaying bad transactions). I’m using my own patches for Lispworks that aren’t merged upstream yet (I have a PR for it). I think the index failing issue might be in my own patches, but I don’t know for sure yet.
dexador (as opposed to Drakma): For the longest time I avoided Dexador because I thought Drakma is the more well established library. But Drakma for all its history, still doesn’t handle SSL correctly at least on LispWorks. And dexador does.
Qlot also sounds awesome, and I want to start using it.

Some I wish existed:

A carefully designed algorithms library, with all the common algorithms, and consistent APIs.
…in particular graph algorithms. There are libraries out there, and I have used them, but they are clunky and not very well documented.
Better image processing libraries. Opticl is fine, but confusing to work with. Maybe it’s just me.
A modern and complete Selenium library. I use the Java library with the Java FFI.
An extensible test matchers library, similar to Java’s Hamcrest. There’s a cl-hamcrest, but it’s not very extensible, and you can’t configure the test result output IIRC. I attempted a solution here (copied over as part of my mono-repo): https://github.com/screenshotbot/screenshotbot-oss/tree/main/src/fiveam-matchers, but it’s not ready for publishing yet.

I also think there’s a verbosity problem to be solved with classes and methods, that still needs to be solved in the Lispy way. For instance, in Java or python, the method names don’t have to be explicitly imported, but in CL we have to import each method that we need to use, which makes it hard to define what the object’s “interface” is. I am not proposing a solution here, I’m just identifying this as a problem that slows me down.

Is that a baby alligator you caught yourself on your GH profile picture?

It’s one of those pictures they take of you at a tourist trap alligator tour. :)

The alligator’s jaw is taped shut.

Anything more to add?

Nothing I can think of :)

Thanks and the best for your projects!

notes:

Arnold’s GitHub: https://github.com/tdrhq/
ModernInterpreters: https://github.com/moderninterpreters

Lisp for the web: pagination and cleaning up HTML with LQuery

Thu, 25 Nov 2021 11:39:18 +0100

I maintain a web application written in Common Lisp, used by real world© clients© (incredible I know), and I finally got to finish two little additions:

add pagination to the list of products
cleanup the HTML I get from webscraping (so we finally fetch a book summary, how cool) (for those who pay for it, we can also use a third-party book database).

The HTML cleanup part is about how to use LQuery for the task. Its doc shows the remove function from the beginning, but I have had difficulty to find how to use it. Here’s how. (see issue #11)

Cleanup HTML with lquery

https://shinmera.github.io/lquery/

LQuery has remove, remove-attr, remove-class, remove-data. It seems pretty capable.

Let’s say I got some HTML and I parsed it with LQuery. There are two buttons I would like to remove (you know, the “read more” and “close” buttons that are inside the book summary):

(lquery:$ *node* ".description" (serialize))
   ;; HTML content…
        <button type=\"button\" class=\"description-btn js-descriptionOpen\"><span class=\"mr-005\">Lire la suite</span><i class=\"far fa-chevron-down\" aria-hidden=\"true\"></i></button>
        <button type=\"button\" class=\"description-btn js-descriptionClose\"><span class=\"mr-005\">Fermer</span><i class=\"far fa-chevron-up\" aria-hidden=\"true\"></i></button></p>")

On GitHub, @shinmera tells us we can simply do:

($ *node* ".description" (remove "button") (serialize))

Unfortunately, I try and I still see the two buttons in the node or in the output. What worked for me is the following:

first I check that I can access these HTML nodes with a CSS selector:

(lquery:$ *NODE* ".description button" (serialize))
;; => output

now I use remove. This returns the removed elements on the REPL, but they are corrcetly removed from the node (a global var passed as parameter):

(lquery:$ *NODE* ".description button" (remove) (serialize))
;; #("<button type=\"button\" class=\"description-btn js-descriptionOpen\"><span class=\"mr-005\">Lire la suite</span><i class=\"far fa-chevron-down\" aria-hidden=\"true\"></i></button>"

Now if I check the description field:

(lquery:$ *NODE* ".description" (serialize))
;; …
;; </p>")

I have no more buttons \o/

Now to pagination.

Pagination

This is my 2c, hopefully this will help someone do the same thing quicker, and hopefully we’ll abstract this in a library…

On my web app I display a list of products (books). We have a search box with a select input in order to filter by shelf (category). If no shelf was chosen, we displayed only the last 200 most recent books. No need of pagination, yet… There were only a few thousand books in total, so we could show a shelf entirely, it was a few hundred books by shelf maximum. But the bookshops grow and my app crashed once (thanks, Sentry and cl-sentry). Here’s how I added pagination. You can find the code here and the Djula template there.

The goal is to get this and if possible, in a re-usable way:

I simply create a dict object with required data:

the current page number
the page size
the total number of elements
the max number of buttons we want to display
etc

(defun make-pagination (&key (page 1) (nb-elements 0) (page-size 200)
                         (max-nb-buttons 5))
  "From a current page number, a total number of elements, a page size,
  return a dict with all of that, and the total number of pages.

  Example:

(get-pagination :nb-elements 1001)
;; =>
 (dict
  :PAGE 1
  :NB-ELEMENTS 1001
  :PAGE-SIZE 200
  :NB-PAGES 6
  :TEXT-LABEL \"Page 1 / 6\"
 )
"
  (let* ((nb-pages (get-nb-pages nb-elements page-size))
         (max-nb-buttons (min nb-pages max-nb-buttons)))
    (serapeum:dict :page page
                   :nb-elements nb-elements
                   :page-size page-size
                   :nb-pages nb-pages
                   :max-nb-buttons max-nb-buttons
                   :text-label (format nil "Page ~a / ~a" page nb-pages))))

(defun get-nb-pages (length page-size)
  "Given a total number of elements and a page size, compute how many pages fit in there.
  (if there's a remainder, add 1 page)"
  (multiple-value-bind (nb-pages remainder)
      (floor length page-size)
    (if (plusp remainder)
        (1+ nb-pages)
        nb-pages)))
#+(or)
(assert (and (= 30 (get-nb-pages 6000 200))
             (= 31 (get-nb-pages 6003 200))
             (= 1 (get-nb-pages 1 200))))

You call it:

(make-pagination :page page
    :page-size *page-length*
    :nb-elements (length results))

then pass it to your template, which can {% include %} the template given above, which will create the buttons (we use Bulma CSS there).

When you click a button, the new page number is given as a GET parameter. You must catch it in your route definition, for example:

(easy-routes:defroute search-route ("/search" :method :get) (q shelf page)
   …)

Finally, I updated my web app (while it runs, it’s more fun and why shut it down? It’s been 2 years I do this and so far all goes well (I try to not upgrade the Quicklisp dist though, it went badly once, because of external, system-wide dependencies)) (see this demo-web-live-reload).

That’s exactly the sort of things that should be extracted in a library, so we can focus on our application, not on trivial things. I started that work, but I’ll spend more time next time I need it… call it “needs driven development”.

Happy lisping.

Lisp Interview: questions to Alex Nygren of Kina Knowledge, using Common Lisp extensively in their document processing stack

Fri, 22 Oct 2021 12:49:17 +0200

Recently, the awesome-lisp-companies list was posted on HN, more people got to know it (look, this list is fan-cooked and we add companies when we learn about one, often by chance, don’t assume it’s anything “official” or exhaustive), and Alex Nygren informed us that his company Kina Knowledge uses Common Lisp in production:

We use Common Lisp extensively in our document processing software core for classification, extraction and other aspects of our service delivery and technology stack.

He very kindly answered more questions.

Thanks for letting us know about Kina Knowledge. A few more words if you have time? What implementation(s) are you using?

We use SBCL for all our Common Lisp processes. It’s easier with the standardization on a single engine, but we also have gotten tied to it in some of our code base due to using the built in SBCL specific extensions. I would like, but have no bandwidth, to evaluate CCL as well, especially on the Windows platform, where SBCL is weakest. Since our clients use Windows systems attached to scanners, we need to be able to support it with a client runtime.

Development is on MacOS with Emacs or Ubuntu with Emacs for CL, and then JetBrains IDEs for Ruby and JS and Visual Studio for some interface code to SAP and such. We develop the Kina UI in Kina itself using our internal Lisp, which provides a similar experience to Emacs/SLY.

What is not Lisp in your stack? For example, in “Kina extracts information from PDF, TIFFs, Excel, Word and more” as we read on your website.

Presently we use a Rails/Ruby environment for driving our JSON based API, and some legacy web functions. However, increasingly, once the user is logged in, they are interacting with a Common Lisp back end via a web socket (Hunchentoot and Hunchensocket) interacting with a Lisp based front end. Depending on the type of information extraction, the system uses Javascript, Ruby and Common Lisp. Ideally, I’d like to get all the code refactored into a prefix notation, targeting Common Lisp or DLisp (what we call our internal Lisp that compiles into Javascript).

What’s your position on open-source: do you use open-source Lisp libraries, do you (plan to) open-source some?

Yes. We recently put our JSON-LIB (https://github.com/KinaKnowledge/json-lib) out on Github, which is our internal JSON parser and encoder and we want to open source DLisp after some clean-up work. Architecturally, DLisp can run in the browser, or in sandboxed Deno containers on the server side, so we can reuse libraries easily. It’s not dependent on a server-side component though to run.

Library wise, we strictly try and limit how many third party (especially from the NPM ecosystem) libraries we are dependent on, especially in the Javascript world. In CL, we use the standard stuff like Alexandria, Hunchentoot, Bordeaux Threads, and things like zip.

How did hiring and forming lisp or non-lisp developers go? Did you look for experienced lispers or did you seek experienced engineers, even with little to no prior Lisp background?

Because we operate a lot in Latin America, I trained non-lisper engineers who speak Spanish on how to program Lisp, specifically our DLisp, since most customizations occur specifically for user interface and workflows around document centric processes, such as presenting linked documents and their data in specific ways. How the lisp way of thinking really depended on their aptitude with programming, and their English capabilities to understand me and the system. The user system is multilingual, but the development documentation is all in English. But it was really amazing when I saw folks who are experienced with Javascript and .Net get the ideas of Lisp and how compositional it can be as you build up towards a goal.

Besides, with DLisp, you can on the fly construct a totally new UI interaction - live - in minutes and see changes in the running app without the dreadful recompile-and-reload everything cycle that is typical. Instead, just recompile the function (analogous to C-c, C-c in Emacs), in the browser, and see the change. Then these guys would go out and interact with clients and build stuff. I knew once I saw Spanish functions and little DSLs showing up in organizational instances that they were able to make progress. I think it is a good way to introduce people to Lisp concepts without having to deal with the overhead of learning Emacs at the same time. I pushed myself through that experience when I first was learning CL, and now use Emacs every day for a TON of work tasks, but at the beginning it was tough, and I had to intentionally practice getting to the muscle memory that is required to be truly productive in a tool.

How many lispers are working together, how big a codebase do you manage?

Right now, in our core company we have three people, two here in Virginia and one in Mexico City. We use partners that provide services such as scanning and client integration work. We are self-funded and have grown organically, which is freeing because we are not beholden to investor needs. We maintain maximum flexibility, at the expense of capital. Which is OK for us right now. Lisp allows us to scale dramatically and manage a large code base. I haven’t line counted recently, but it exceeds 100K lines across server and client, with > 50% in Lisp.

Do you sometimes wish the CL (pro) world was more structured? (we have a CL Foundation but not so much active).

I really like the Common Lisp world. I would like it to be more popular, but at the same time, it is a differentiator for us. It is fast - our spatial classifier takes only milliseconds to come to a conclusion about a page (there is additional time prior to this step due to the OpenCV processing - but not too much) and identify it and doesn’t require expensive hardware. Most of our instances run on ARM-64, which at least at AWS, is 30% or so cheaper than x86-64. The s-expression structures align to document structures nicely and allow a nice representation that doesn’t lose fidelity to the original layouts and hierarchies. I am not as active as I would like to be in the Common Lisp community, mainly due to time and other commitments. I don’t know much about the CL foundation.

And so, how did you end up with CL?

Our UI was first with the DLisp concepts. I was intrigued by Clojure for the server portion, but I couldn’t come to terms with the JVM and the heavyweight of it. The server-side application was outgrowing the Rails architecture in terms of what we wanted to do with it, and, at the time, 4 years ago, Ruby was slower. In fact, Ruby had become a processing bottleneck for us (though I am certain the code could have been improved too). I liked the idea of distributing binary applications as well, which we needed to do in some instances, and building a binary runtime of the software was a great draw, too.

I also liked how well CL is thought out, from a spec standpoint. It is stable both in terms of performance and change. I had been building components with TensorFlow and Python 3, but for what I wanted to do, I couldn’t see how I could get there with back propagation and the traditional “lets calculate the entire network state”. If you don’t have access to high end graphic cards, it’s just too slow and too heavy. I was able to get what we needed to do in CL after several iterations and dramatically improve speed and resource utilization. I am very happy with that outcome. We are in what I consider to be a hard problem space: we take analog representations of information, a lot of it being poor quality and convert it to clean, structured digital information. CL is the core of that for us.

Here is an example of our UI, where extractions and classification can be managed. This is described in DLisp which interacts with a Common Lisp back end via a web socket.

Here is the function for the above view being edited in Kina itself. We do not obfuscate our client code, and all code that runs on our clients’ computers is fully available to view and, with the right privileges, to modify and customize. You can see the Extract Instruction Language in the center pane, which takes ideas from the Logo language in terms of a cursor (aka the turtle) that can be moved around relative to the document. We build this software to be used by operations teams and having a description language that is understandable by non-programmers such as auditors and operations personnel, is very useful. You can redefine aspects of the view or running environment and the change can take effect on the fly. Beyond the Javascript boot scaffolding to get the system started up in the browser, everything is DLisp communicating with Common Lisp and, depending on the operation, Rails.

I hope this information is helpful!

It is, thanks again!

Discovering the Lispworks IDE

Mon, 24 May 2021 13:57:41 +0200

LispWorks is a Common Lisp implementation that comes with its own Integrated Development Environment (IDE) and its share of unique features, such as the CAPI GUI toolkit. It is proprietary and provides a free limited version.

Here, we will mainly explore its IDE, asking ourselves what it can offer to a seasoned lisper used to Emacs and Slime. The short answer is: more graphical tools, such as an easy to use graphical stepper, a tracer, a code coverage browser or again a class browser. Setting and using breakpoints was easier than on Slime.

LispWorks also provides more integrated tools (the Process browser lists all processes running in the Lisp image and we can stop, break or debug them) and presents many information in the form of graphs (for example, a graph of function calls or a graph of all the created windows).

Note: you will find updated versions of this review on the Cookbook.

LispWorks features

We can see a matrix of LispWorks features by edition and platform here: http://www.lispworks.com/products/features.html.

We highlight:

32-bit, 64-bit and ARM support on Windows, MacOS, Linux, Solaris, FreeBSD,
CAPI portable GUI toolkit: provides native look-and-feel on Windows, Cocoa, GTK+ and Motif.
- comes with a graphical “Interface Builder” (think QtCreator) (though not available on MacOS (nor on mobile))
LispWorks for mobile runtime, for Android and iOS,
optimized application delivery: LispWorks can use a tree shaker to remove unused lisp code from the delivered applicatiion, thus shipping lighter binaries than existing open-source implementations.
ability to deliver a dynamic library,
a Java interface, allowing to call from Lisp to Java or the other way around,
an Objective-C and Cocoa interface, with drag and drop and multi-touch support,
a Foreign Language Interface,
TCP/UDP sockets with SSL & IPv6 support,
natived threads and symmetric multiprocessing, unicode support, and all other Common Lisp features, and all other LispWorks Enterprise features.

And, of course, a built-in IDE.

LispWorks is used in diverse areas of the industry. They maintain a list of success stories. As for software that we can use ourselves, we find ScoreCloud amazing (a music notation software: you play an instrument, sing or whistle and it writes the music) or OpenMusic (opensource composition environment).

Free edition limitations

The download instructions and the limitations are given on the download page.

The limitations are the following:

There is a heap size limit which, if exceeded, causes the image to exit. A warning is provided when the limit is approached.

What does it prevent us to do? As an illustration, we can not load this set of libraries together in the same image:

(ql:quickload '("alexandria" "serapeum" "bordeaux-threads" "lparallel" "dexador" "hunchentoot" "quri" "ltk" "cl-ppcre" "mito"))

There is a time limit of 5 hours for each session, after which LispWorks Personal exits, possibly without saving your work or performing cleanups such as removing temporary files. You are warned after 4 hours of use.
It is impossible to build a binary. Indeed, the functions save-image, deliver (the function to create a stand-alone executable), and load-all-patches are not available.
Initialization files are not loaded. If you are used to initializing Quicklisp from your ~/.sbclrc on Emacs, you’ll have to load an init file manually every time you start LispWorks ((load #p"~/.your-init-file)).

For the record, the snippet provided by Quicklisp to put in one’s startup file is the following:

;; provided you installed quicklisp in ~/quicklisp/
(let ((quicklisp-init (merge-pathnames "quicklisp/setup.lisp" (user-homedir-pathname))))
  (when (probe-file quicklisp-init)
    (load quicklisp-init)))

You’ll have to paste it to the listener window (with the C-y key, y as “yank”).

Layered products that are part of LispWorks Professional and Enterprise Editions (CLIM, KnowledgeWorks, Common SQL and LispWorks ORB) are not included. But we can try the CAPI toolkit.

The installation process requires you to fill an HTML form to receive a download link, then to run a first script that makes you accept the terms and the licence, then to run a second script that installs the software.

Licencing model

LispWorks actually comes in four paid editions. It’s all explained by themselves here: http://www.lispworks.com/products/lispworks.html. In short, there is:

a Hobbyist edition with save-image and load-all-patches, to apply updates of minor versions, without the obvious limitations, for non-commercial and non-academic use,
a HobbyistDV edition with the deliver function to create executables (still for non-commercial and non-academic uses),
a Professional edition, with the deliver function, for commercial and academic uses,
an Enterprise one, with their enterprise modules: the Common SQL interface, LispWorks ORB, KnowledgeWorks.

At the time of writing, the licence of the hobbyist edition costs 750 USD, the pro version the double. They are bought for a LW version, per platform. They have no limit of time.

NB: Please double check their upstream resources and don't hesitate to contact them.

LispWorks IDE

The LispWorks IDE is self-contained, but it is also possible to use LispWorks-the-implementation from Emacs and Slime (see below).

The editor

The editor offers what’s expected: a TAB-completion pop-up, syntax highlighting, Emacs-like keybindings (including the M-x extended command). The menus help the discovery.

We personally found the editing experience a bit “raw”. For example: - indention after a new line is not automatic, one has to press TAB again. - the auto-completion is not fuzzy. - there are no plugins similar to Paredit or Lispy, nor a Vim layer.

We also had an issue, in that the go-to-source function bound to M-. did not work out for built-in Lisp symbols. This is probably a free edition limitation too.

The editor provides an interesting tab: Changed Definitions. It lists the functions and methods that were redefined since, at our choosing: the first edit of the session, the last save, the last compile.

See also:

the Editor User Guide.

Keybindings

Most of the keybindings are similar to Emacs, but not all. Here are some differences:

to compile a function, use C-S-c (control, shift and c), instead of C-c C-c.
to compile the current buffer, use C-S-b (instead of C-c C-k).

Similar ones include:

C-g to cancel what you’re doing,
C-x C-s to save the current buffer,
M-w and C-y to copy and paste,
M-b, M-f, C-a, C-e… to move around words, to go to the beginning or the end of the line,
C-k to kill until the end of the line, C-w to kill a selected region,
M-. to find the source of a symbol,
C-x C-e to evaluate the current defun,
…

Some useful functions don’t have a keybinding by default, for example:

clear the REPL with M-x Clear Listener
Backward Kill Line

It is possible to use classical keybindings, à la KDE/Gnome. Go to the Preferences menu, Environment and in the Emulation tab.

There is no Vim layer.

Searching keybindings by name

It is possible to search for a keybinding associated to a function, or a function name from its keybinding, with the menu (Help -> Editing -> Key to Command / Command to Key) or with C-h followed by a key, as in Emacs. For example type C-h k then enter a keybinding to get the command name. See more with C-h ?.

Tweaking the IDE

It is possible to change keybindings. The editor’s state is accessible from the editor package, and the editor is built with the CAPI framework, so we can use the capi interface too. Useful functions include:

`
editor:bind-key
editor:defcommand
editor:current-point
editor:with-point  ;; save point location
editor:move-point
editor:*buffer-list*
editor:*in-listener* ;; returns T when we are in the REPL
…

Here’s how you can bind keys:

;; Indent new lines.
;; By default, the point is not indented after a Return.
(editor:bind-key "Indent New Line" #\Return :mode "Lisp")

;; Insert pairs.
(editor:bind-key "Insert Parentheses For Selection" #\( :mode "Lisp") ;;
(editor:bind-key "Insert Double Quotes For Selection" #\" :mode "Lisp")

Here’s how to define a new command. We make the ) key to go past the next closing parenthesis.

(editor:defcommand "Move Over ()" (p)
  "Move past the next close parenthesis.
Any indentation preceeding the parenthesis is deleted."
  "Move past the next close parenthesis."
  ;; thanks to Thomas Hermann
  ;; https://github.com/ThomasHermann/LispWorks/blob/master/editor.lisp
  (declare (ignore p))
  (let ((point (editor:current-point)))
    (editor:with-point ((m point))
      (cond ((editor::forward-up-list m)
	     (editor:move-point point m)
             (editor::point-before point)
             (loop (editor:with-point ((back point))
                     (editor::back-to-indentation back)
                     (unless (editor:point= back point)
                       (return)))
                   (editor::delete-indentation point))
	     (editor::point-after point))
	    (t (editor:editor-error))))))

(editor:bind-key "Move Over ()" #\) :mode "Lisp")

And here’s how you can change indentation for special forms:

(editor:setup-indent "if" 1 4 1)

See also:

a list of LispWork keybindings: https://www.nicklevine.org/declarative/lectures/additional/key-binds.html

The listener

The listener is the REPL.

Its interactive debugger is primarily textual but you can also interact with it with graphical elements. For example, you can use the Abort button of the menu bar, which brings you back to the top level. You can invoke the graphical debugger to see the stacktraces and interact with them. See the Debugger button at the very end of the toolbar.

If you see the name of your function in the stacktraces (you will if you wrote and compiled your code in a file, and not directly wrote it in the REPL), you can double-click on its name to go back to the editor and have it highlight the part of your code that triggered the error.

NB: this is equivalent of pressing M-v in Slime.

The stepper. Breakpoints.

The stepper is one of the areas where LispWorks shines.

When your are writing code in the editor window, you can set breakpoints with the big red “Breakpoint” button (or by calling M-x Stepper Breakpoint). This puts a red mark in your code.

The next time your code is executed, you’ll get a comprehensive Stepper pop-up window showing:

your source code, with an indicator showing what expression is being evaluated
a lower pane with two tabs:
- the backtrace, showing the intermediate variables, thus showing their evolution during the execution of the program
- the listener, in the context of this function call, where you can evaluate expressions
and the menu bar with the stepper controls: you can step into the next expression, step on the next function call, continue execution until the position of the cursor, continue the execution until the next breakpoint, etc.

That’s not all. The non-visual, REPL-oriented stepper is also nice. It shows the forms that are being evaluated and their results.

In this example, we use :s to “step” though the current form and its subforms. We are using the usual listener, we can write any Lisp code after the prompt (the little -> here), and we have access to the local variables (X).

CL-USER 4 > (defun my-abs (x) (cond ((> x 0) x) ((< x 0) (- x)) (t 0)))
CL-USER 5 > (step (my-abs -5))
(MY-ABS -5) -> :s
   -5 -> :s
   -5
   (COND ((> X 0) X) ((< X 0) (- X)) (T 0)) <=> (IF (> X 0) (PROGN X) (IF (< X 0) (- X) (PROGN 0)))
   ;; Access to the local variables:
   (IF (> X 0) (PROGN X) (IF (< X 0) (- X) (PROGN 0))) -> (format t "Is X equal to -5? ~a~&" (if (equal x -5) "yes" "no"))
Is X equal to -5? yes
   (IF (> X 0) (PROGN X) (IF (< X 0) (- X) (PROGN 0))) -> :s
      (> X 0) -> :s
         X -> :s
         -5
         0 -> :s
         0
      NIL
      (IF (< X 0) (- X) (PROGN 0)) -> :s
         (< X 0) -> :s
            X -> :s
            -5
            0 -> :s
            0
         T
         (- X) -> :s
            X -> :s
            -5
         5
      5
   5
5

Here are the available stepper commands (see :?):

:s       Step this form and all of its subforms (optional +ve integer arg)
:st      Step this form without stepping its subforms
:si      Step this form without stepping its arguments if it is a function call
:su      Step up out of this form without stepping its subforms
:sr      Return a value to use for this form
:sq      Quit from the current stepper level
:bug-form <subject> &key <filename>
         Print out a bug report form, optionally to a file.
:get <variable> <command identifier>
         Get a previous command (found by its number or a symbol/subform within it) and put it in a variable.
:help    Produce this list.
:his &optional <n1> <n2>
         List the command history, optionally the last n1 or range n1 to n2.
:redo &optional <command identifier>
         Redo a previous command, found by its number or a symbol/subform within it.
:use <new> <old> &optional <command identifier>
         Do variant of a previous command, replacing old symbol/subform with new symbol/subform.

The class browser

The class browser allows us to examine a class’s slots, parent classes, available methods, and some more.

Let’s create a simple class:

(defclass person ()
  ((name :accessor name
         :initarg :name
         :initform "")
   (lisper :accessor lisperp
           :initform t)))

Now call the class browser:

use the “Class” button from the listener,
or use the menu Expression -> Class,
or put the cursor on the class and call M-x Describe class.

It is composed of several panes:

the class hierarchy, showing the superclasses on the left and the subclasses on the right, with their description to the bottom,
the superclasses viewer, in the form of a simple schema, and the same for subclasses,
the slots pane (the default),
the available initargs,
the existing generic functions for that class
and the class precedence list.

The Functions pane lists all methods applicable to that class, so we can discover public methods provided by the CLOS object system: initialize-instance, print-object, shared-initialize, etc. We can double-click on them to go to their source. We can choose not to include the inherited methods too (see the “include inherited” checkbox).

You’ll find buttons on the toolbar (for example, Inspect a generic function) and more actions on the Methods menu, such as a way to see the functions calls, a menu to undefine or trace a function.

See more:

Chapter 8 of the documentation: the Class Browser

The function call browser

The function call browser allows us to see a graph of the callers and the callees of a function. It provides several ways to filter the displayed information and to further inspect the call stack.

NB: The Slime functions to find such cross-references are slime-who-[calls, references, binds, sets, depends-on, specializes, macroexpands].

After loading a couple packages, here’s a simple example showing who calls the string-trim function.

It shows functions from all packages, but there is a select box to restrict it further, for example to the “current and used” or only to the current packages.

Double click on a function shown in the graph to go to its source. Again, as in many LispWorks views, the Function menu allows to further manipulate selected functions: trace, undefine, listen (paste the object to the Listener)…

The Text tab shows the same information, but textually, the callers and callees side by side.

We can see cross references for compiled code, and we must ensure the feature is on. When we compile code, LispWorks shows a compilation output likes this:

;;; Safety = 3, Speed = 1, Space = 1, Float = 1, Interruptible = 1
;;; Compilation speed = 1, Debug = 2, Fixnum safety = 3
;;; Source level debugging is on
;;; Source file recording is  on
;;; Cross referencing is on

We see that cross referencing is on. Otherwise, activate it with (toggle-source-debugging t).

See more:

Chapter 15: the function call browser

The Process Browser

The Process Browser shows us a list of all threads running. The input area allows to filter by name. It accepts regular expressions. Then we can stop, inspect, listen, break into these processes.

See more:

Chapter 28: the Process Browser

Using LispWorks from Emacs and Slime

To do that, start LispWorks normally, start a Swank server and connect to it from Emacs (Swank is the backend part of Slime).

First, let’s load the dependencies:

(ql:quickload "swank")
;; or
(load "~/.emacs.d/elpa/slime-20xx/swank-loader.lisp")

Start a server:

(swank:create-server :port 9876)
;; Swank started at port: 9876.
9876

From Emacs, run M-x slime-connect, choose localhost and 9876 for the port.

You should be connected. Check with: (lisp-implementation-type). You are now able to use LispWorks’ features:

(setq button
      (make-instance 'capi:push-button
                     :data "Button"))
(capi:contain button)

Lem can now be started as a full featured Lisp REPL

Wed, 24 Mar 2021 12:43:13 +0100

The Lem editor, which supports Common Lisp as well as other languages, works by default in the terminal with a ncurses frontend (it also has an experimental Electron frontend). It ships a nice Lisp REPL: it has good fuzzy completion, enough keyboard shortcuts, an interactive debugger, a completion menu, etc.

It is now possible to run Lem straight in its Lisp REPL. Run it with:

lem --eval "(lem-lisp-mode:start-lisp-repl t)"

The optional argument (t) was added recently (thanks, @cxxxr) and allows to start the REPL in fullscreen.

Here is it in action:

The other terminal-based REPL alternatives are:

cl-repl (wants to be full-featured, à la ipython)
sbcli (handy, but minimalistic)

but IMO, they now fall short compared to Lem’s features.

Installation

To install it, see its wiki. In short, do

 ros install cxxxr/lem

or use lem-docker.

To install Roswell, do one of

brew install roswell  # macos or linuxbrew
yay -S roswell
scoop install roswell  # windows

# Debian package:
curl -sOL `curl -s https://api.github.com/repos/roswell/roswell/releases/latest | jq -r '.assets | .[] | select(.name|test("deb$")) | .browser_download_url'`
sudo dpkg -i roswell.deb

See its releases.

Usage (quickref)

Lem has Emacs-like keybindings, as well as a vi emulation (M-x vi-mode). Unfortunately, that is not documented much.

So, to open a file, press C-x C-f (you get the file selection dialog shown above). To save it, it’s C-x C-s.

To switch to the REPL: C-c C-z.

To compile and load a buffer: C-c C-k. To compile a function: C-c C-c.

To switch windows: C-x o. To make a window fullscreen: C-x 1. To split it vertically: C-x 3 and horizontally: C-x 2.

To switch buffers: C-x b.

To run an interactive command: M-x (alt-x).

See this Emacs & Slime cheatsheet to find more: https://lispcookbook.github.io/cl-cookbook/emacs-ide.html#appendix

Lem works out of the box for several programming languages (Python, Rust, Scala…). It also has an HTML mode, a directory mode… Lem needs to be discovered!

chat: https://gitter.im/lem-developers/community

State of the Common Lisp ecosystem, 2020 🎉

Mon, 08 Feb 2021 11:29:11 +0200

This is a description of the Common Lisp ecosystem, as of January, 2021, from the perspective of a user and contributor.

The purpose of this article is both to give an overview of the ecosystem, and to help drive consolidation in each domain.

Each application domain has recommendations for consolidating that part of the ecosystem, and pointers for interesting future work.

This article is derived from Fernando Borretti’s State of the Common Lisp ecosystem from 2015, hence the introduction that sounded familiar. This new one will be an opportunity to look at what was achieved, or what is still lacking.

Disclaimer: This article is not a list of every project or article of interest that came out in the last years. I wrote an overview of 2018 closer to that goal here. More libraries can be discovered on the Awesome-cl list, on GitHub and on Cliki.

Acknowledgements I would like to thank @borodust, @ambrevar and @digikar for their kind feedback.

Table of Contents

Application domains

Command line

There used to be several options to ease building and distribution of command line programs, but now Roswell has gained most momentum, and that’s a good thing. Roswell is an implementation manager, installer and a script runner, and one of its neat features is support for very easily compiling tiny scripts into executables.

Now, GNU Guix has gained many CL libraries, and becomes a contender to Roswell. Guix can be used as a package manager on top of your Unix distribution. It brings reproducible builds, rollbacks, the ability to install exact versions of any library (including system dependencies), contained environments and user profiles. It makes it easy too to install the latest version of a CL implementation and libraries and, to a certain extent, to share scripts. See the article A Lisp REPL as my main shell for insights.

To parse command line arguments, unix-opts shows decent activity. As a reminder, the CLI arguments are stored portably in uiop:command-line-arguments.

Adams is a new UNIX system administration tool, not unlike Chef or Ansible.

Consolidation

More features to the sripting libraries.

Future work

The Lem editor has built a great user interface and REPL on top of ncurses, with the cl-charms library. It would be great to re-use its components, so that Lispers could easily build similar rich terminal-based interfaces.

Databases

Mito is an ORM for Common Lisp with migrations, relationships and PostgreSQL support. It is based on cl-dbi (a uniform interface to the various database server-specific libraries such as cl-postgres and cl-mysql) and SxQL (a DSL for building safe, automatically parameterized SQL queries).

It also has a tutorial in the Cookbook: Cookbook/databases.

There are of course more libraries in that field. Some new ones since 2015 are:

cl-yesql (by the author of Serapeum, Spinneret and other great libraries) is based on Clojure’s Yesql.

vivace-graph is a graph database and Prolog implementation, taking design and inspiration from CouchDB, neo4j and AllegroGraph.

Vsevolod Dyomkin, the author of Rutils, the Programming Algorithms book and other libraries, is writing cl-agraph, a minimal client to Franz Inc’s AllegroGraph. AllegroGraph is a “horizontally distributed, multi-model (document and graph), entity-event knowledge graph technology”. It is proprietary and has a free version with a limit of 5 million triples. Surely one of those Lisp hidden gems we should know more about.

A general migration tool was lacking. We now have cl-migratum, a “system which provides facilities for performing database schema migrations, designed to work with various databases”.

And of course, pgloader is still a Common Lisp success story.

Achievement

Among the emerging ORMs, Mito is the one actively maintained that Lispers seem to have chosen. Good. CLSQL certainly still works, but we don’t hear about it and it looks outdated. So, Mito it is.

Consolidation

Mito has 11 contributors and is actively watched, but it probably should have another(s) core maintainers.

Future work

Bindings for the new databases coming out.

Concurrency

In the last year, Manfred Bergmann developed cl-gserver. It is a “message passing” library/framework with actors similar to Erlang or Akka. It is an important achievement.

Its v1 features:

actors can use a shared pool of message dispatchers which effectively allows to create millions of actors.
the possibility to create actor hierarchies. An actor can have child actors. An actor now can also “watch” another actor to get notified about its termination.

Many other libraries exist in this area:

BordeauxThreads - Portable, shared-state concurrency
- the “de-facto” concurrency library.
lparallel - A library for parallel programming.
- also solid, battle-tested and popular, aka de-facto.
calispel - CSP-like channels for common lisp. With blocking, optionally buffered channels and a “CSP select” statement. ISC-style.
- “It is complete, flexible and easy to use. I would recommend Calispel over Lparallel and ChanL.” @Ambrevar. discussion
ChanL - Portable, channel-based concurrency.
cl-async - A library for general-purpose, non-blocking programming.
- works atop libuv
Moira - Monitor and restart background threads. In-lisp process supervisor.
trivial-monitored-thread - a Common Lisp library offering a way of spawning threads and being informed when one any of them crash and die.
lfarm - distributing work across machines (on top of lparallel and usocket).
cl-gearman - a library for the Gearman distributed job system.
- Alexander Artemenko used it instead of lfarm for Ultralisp: https://40ants.com/lisp-project-of-the-day/2020/06/0095-cl-gearman.html, because “lfarm is not well suited to environments where worker hosts can go down and return back later”.
swank-crew - distributed computation framework implemented using Swank Client.
cl-coroutine - a coroutine library. It uses the CL-CONT continuations library in its implementation.
CMTX: high performance transactional memory for Common Lisp.
- In our opinion, a library not well known and under-appreciated.

(see awesome-cl#parallelism-and-concurrency)

Consolidation

Bordeaux-Threads is the “de-facto” library, but there is some choice paralysis between Lparallel, Calispel, Bordeaux-Threads and SBCL’s contribs. Use the libraries in the wild and write about them.

File formats

There exist Common Lisp libraries for all the major file formats:

XML: Plump (and Lquery), as well as CXML, which can parse large files incrementally.
JSON: Jonathan, cl-json or more. With utilities:
- json-pointer - A JSON Pointer implementation.
- json-mop - A metaclass for bridging CLOS and JSON objects (remind that JSON libraries can already serialize your own objects).
- json-schema
YAML: cl-yaml
CSV: cl-csv

Achievement

New in 2015, Jonathan is now a good first choice for an easy to use and fast JSON encoder and decoder.

Consolidation

There is not a predominant JSON library. This leads to choice paralysis.

They all represent null values differently. We need a library that “does the right thing”. See maybe the massive web-toolkit for its JSON handling ?

It distinguishes null, false and [] from Lisp’s NIL thus supports identical transformation between JSON values. It provides object constructor and accessor to build and access nesting JSON objects.

Give the XPath library some love and documentation.

Future Work

Still valid from 2015:

A YAML parser so that cl-yaml doesn’t depend on the libyaml library would make distribution far simpler.

GUI

A usual complain in Common Lisp land is the lack of a complete, cross-platform GUI solution. Ltk is a very good library, but Tk is limited. Qtools is great, but is only for Qt4.

A lot has happened, and is still happening (if you watch the right repositories, you know that a Qt5 wrapper is in the works (ECL already has Qt5 bindings: EQL5, with an Android port)).

edit: see also EQL5-sailfish for Sailfish OS. Here are two example apps.

Matthew Kennedy wrote excellent FFI bindings to the IUP Portable User Interface library: IUP. IUP is cross-platform (Windows, macOS, GNU/Linux, with new Android, iOS, Cocoa and Web Assembly drivers), has many widgets (but less than Qt), has a small API and is actively developed. IUP was created at the PUC university of Rio de Janeiro.

Nicolas Hafner started Alloy, a new user interface protocol and toolkit implementation, which he uses in his Kandria game.

Very recently, David Botton released CLOG, “the Common Lisp Omnificent GUI”:

CLOG uses web technology to produce graphical user interfaces for applications locally or remotely. CLOG can take the place, or work alongside, most cross-platform GUI frameworks and website frameworks. The CLOG package starts up the connectivity to the browser or other websocket client (often a browser embedded in a native template application.)

It is complete enough for most uses.

There are more GUI libraries and frameworks: https://github.com/CodyReichert/awesome-cl#Gui (and more under the works). In particular, LispWorks’ CAPI is still presented as the best in town by the ones who tried it.

Consolidation

Since roughly October, 2020, Nicolas Hafner works full time on Kandria. Supporting his work, through GitHub sponsors or ko-fi would be 1) a great sign of recognition and 2) useful for the ecosystem, especially for Alloy.

I wrote an introduction to these frameworks in the Cookbook: Cookbook/gui. More examples or demo projects would be welcome.

There are two actively maintained diverged forks of the GTK bindings. A reunification effort is required.

Future work

Write a desktop application with IUP/your toolkit of choice for everyday use and make it a Common Lisp flagship.

Study other approaches to GUI bindings. What about gtk-server? GObject introspection? An effort started for Qt: giqt (in which we recognize @ambrevar from the Nyxt browser).

Machine Learning

It seems that not much changed since 2015, but libraries are still being developed:

CLML, developed at Mathematical Systems Inc., a Japanese company.
MGL

used by its author to win the Higgs Boson Machine Learning Challenge

mgl-mat - a library for working with multi-dimensional arrays which supports efficient interfacing to foreign and CUDA code. BLAS and CUBLAS bindings are available.

Others are less active:

Antik - a foundation for scientific and engineering computation in Common Lisp. It is designed not only to facilitate numerical computations, but to permit the use of numerical computation libraries and the interchange of data and procedures, whether foreign (non-Lisp) or Lisp libraries.
- more than 2000 commits, last update 2 years ago.

System

To quote Fernando:

UIOP, ASDF’s portable compatibility layer, contains a large set of tools for portably doing everything from querying the hostname to running external programs to manipulating environment variables.

We should not require cl-fad anymore (but we need Osicat, which unlike UIOP is POSIX friendly).

Built on top of UIOP, Paul M. Rodriguez’s cmd brings in short and handy helpers to run and pipe programs.

Web Development

Backend

Common Lisp’s main web servers are Hunchentoot and Clack. Since 2015, Clack’s documentation state barely improved and is still lacking.

Clack is the equivalent of WSGI/Rack. It has existed since 2009. It is an HTTP server abstraction, that allows the user to write web applications (or, more reasonably, web application frameworks) without depending on a particular server. Some web frameworks are built on top of it, for example Caveman2.

The importance of using Clack cannot be understated: If you build an application directly on, say, Hunchentoot, you’re tied to Hunchentoot, and if a new, faster server – like Woo – comes out, you have to rewrite the entire application to use it. If you write a plugin for Clack – like clack-errors – it is automatically usable by all applications, regardless of framework, that are built on Clack, reducing useless duplication of code.

With Clack, switching from Hunchentoot to Woo, and enjoying the incredible speedup, is a simple matter of installing libev and changing a keyword argument.

This still holds true, but the situation didn’t improve much. In comparison, Hunchentoot is very well documented (and you can read its documentation on a better looking readthedocs here), and it is “fast enough”.

About Hunchentoot: Mariano Montone wrote easy-routes, a little but handy route handling facility on top of Hunchentoot. It brings:

dispatch by HTTP method,
arguments extraction from the URL path,
“decorators” to, for example, quickly add authorization checks,
integration with the Djula framework to generate URLs from route names.

Achievement

Several Clack plugins were written, such as a single-sign on middleware.

Consolidation

Write more documentation for Clack. While Lispers know about it, they don’t necessarily adopt it because of the lack of documentation. We can expand this getting started guide.

Future work

Build a batteries-included framework.

Frontend

Many HTML generators and template libraries exist (see the list below). However, some new and good ones appeared lately:

TEN, by Djula’s maintainer, brings the completness of Djula with the usability of Eco (by Fernando Borretti), aka: you write Django-like HTML templates but you can interleave any Lisp code.
markup - a JSX-like templating engine, where HTML tags are Common Lisp code. Comes with an Emacs package.

Other HTML generators and templating engines include:

spinneret - Common Lisp HTML5 generator.
cl-who - The venerable HTML generator.
Djula - A port of Django’s template engine to Common Lisp.
cl-closure-template - Implementation of Google’s Closure templates. [LLGPL][8].
clip - An HTML template processor where the templates are written in HTML.

We have nice other building blocks, such as a nice form handling library (cl-forms) and libraries to create Open-API interfaces. An integrated, opinionated all-in-one solution could be a productivity boom.

Consolidation

Djula is easy to work with. It could do with more built-in filters.

As in 2015:

The foundation is finished, now it’s time to write higher-level layers. An extensible administration framework for Clack applications, like Django’s Admin, would be a good example.

JavaScript

The two “historical” Common Lisp to JavaScript compilers are:

Parenscript, a DSL that compiles a subset of Common Lisp to idiomatic JavaScript, and
JSCL, a CL-to-JS compiler designed to be self-hosting from day one. JSCL is not complete (yet), it lacks CLOS, format and loop.

Two new are in development:

Valtan, a CL to JS compiler.
JACL, JavaScript Assisted Common LispIt has a recording from ELS 2020.

Consolidation

Help develop one of the existing CL-to-JS implementations. Why not have a look at JSCL’s issues?

Bring some new macros to ParenScript for new JavaScript idioms, as Paren6. For example, allow to write async and await.

Isomorphic web frameworks

Weblocks is an already old framework that allows to write dynamic web applications without writing JavaScript (it isn’t as dynamic as modern JS frameworks, there is no “double data binding”). Its server-based components use Ajax if available or fallback to plain HTTP and update the DOM. It is a framework in the vein of Smalltalk’s Seaside.

Weblocks was getting old and unmaintained but Alexander Artemenko greatly updated and refactored it in his Reblocks branch. He uses it for the Ultralisp website, and more apps. You can reach users and developers on Gitter.

Recently, a very new web framework appeared: ISSR, for Interactive Server-Side rendering. It links a client to the server with a websocket connection and updates the DOM selectively. It is thus not unlike Phoenix’s LiveView or Hotwire.

See this todo-app tutorial.

Achievement

Reviving Weblocks and releasing CLOG and ISSR are great achievements. However, work is only started to create a community of users around them.

Languages interop

New solutions arose to interoperate with other runtimes.

APL

April brings the APL programming language (a subset thereof) to Common Lisp. Replace hundreds of lines of number-crunching code with a single line of APL.

C, C++, Objective C

We had CFFI (a portable foreign function interface for CL), C2FFI (Clang-based FFI wrapper generator), then cl-autowrap, a c2ffi-based wrapper generator that makes creating C bindings real quick.

Pavel Korolev is developing CLAW, started as a fork of cl-autowrap, which brings C++ support. For practice he generated bindings to GLM or to the Filament rendering engine.

Achievement

It will be a great achievement when CLAW is officially ready to use. This is not yet the case (though the GLM bindings basically do their hello world on Android, which is an achievement per se).

Clojure

ABCLJ provides a “dead easy Clojure to Common lisp interop”:

instead of rewriting the whole Clojure langugage on CL I’m embedding ABCL in Clojure. Since both are implemented in Java and Clojure has an awesome java interop is easy to have full access on the ABCL Common Lisp environment. This way we have complete support for both Clojure and Common Lisp.

But why?

The reason I wanted to see Clojure and Common Lisp working with each other was to use CL programs/libraries on Clojure, especially Maxima and ACL2. Since ABCL already compiles and runs Maxima it should be possible but we are very far from it 🤷.

There are others of attempts to shorten the gap between clojure and common lisp like Cloture and clclojure. Once they are complete Clojure will benefit from native binaries and excelent compilers like SBCL, however they are far from complete.

On the topic, see this talk by Alan Dipert: “Common Lisp for the curious Clojurian”.

Abstract:

“If I had to be stranded with something other than Clojure, I’d be happiest with a good Common Lisp and its source code.” - Rich Hickey, 2011

Common Lisp (CL) and Clojure are both dialects of Lisp. Rich Hickey, the creator of Clojure, learned CL and used it professionally before creating Clojure.

What can Clojure do that CL can’t, and vice versa? Why would anyone use CL today, when both Clojure and ClojureScript exist?

In this talk, I will try to answer these questions and more, from the perspective of a long-time Clojurian with a growing passion for CL.

Python

py4cl is the new lib in town. It allows Common Lisp code to access Python libraries. It is basically the inverse of cl4py.

See also async-process and, while we’re at it, my comparison of Python VS Common Lisp, where we look at the differences of workflows and ecosystems.

Achievement

Calling to Python and hooking into its ecosystem is easier than ever.

Future work

Improving CL libraries such as Numcl (a Numpy clone) is what’s required to drive Common Lisp forward.

.Net Core

Bike is a cross-platform .Net Core interface.

Development

Implementations

All implementations saw new releases, except CLisp, whose development however continues.

Active implementations include: ABCL, CCL, CLASP, ECL, LispWorks, AllegroCL, SBCL. And to a certain extent, GNU CLisp, SICL (which is the newest one) and Corman Lisp (a CL development environment for Windows) (regenerated here).

ABCL jumped to v1.8.0 to support openjdk15.

SBCL still ships monthly releases. It turned 20 and keeps improving (RISC-V port, M1 port, block compilation, more compile-time type checking…). We can read a blog on the party held in Vienna here. Did you know that Doug Katzman of Google fame contributes to SBCL?

(edit:)

Doug Katzman talked about his work at Google getting SBCL to work with Unix better. For those of you who don’t know, he’s done a lot of work on SBCL over the past couple of years, not only adding a lot of new features to the GC and making it play better with applications which have alien parts to them, but also has done a tremendous amount of cleanup on the internals and has helped SBCL become even more Sanely Bootstrappable. That’s a topic for another time, and I hope Doug or Christophe will have the time to write up about the recent improvements to the process, since it really is quite interesting.

Anyway, what Doug talked about was his work on making SBCL more amenable to external debugging tools, such as gdb and external profilers. It seems like they interface with aliens a lot from Lisp at Google, so it’s nice to have backtraces from alien tools understand Lisp. It turns out a lot of prerequisite work was needed to make SBCL play nice like this, including implementing a non-moving GC runtime, so that Lisp objects and especially Lisp code (which are normally dynamic space objects and move around just like everything else) can’t evade the aliens and will always have known locations.

Editors

Here too, great progress has been made. While a usual complain of non-Lispers was the lack of editor support besides Emacs (and Vim), we now nearly reach choice paralysis:

Portacle is the easiest way to get started with Emacs. It is portable and multi-platform, ready-to-use in three clicks. It ships Emacs, SBCL, Slime, Quicklisp and git.
SLIMA is the Atom extension. It is nearly as good as Slime for Emacs.
VSCode has two extensions: commonlisp-vscode, using the Language Server Protocol, and Alive, more recent, using a Lisp backend (Swank) as traditional extensions.
Sublime Text got a good extension: Slyblime is an implementation of SLY and it uses the same backend (SLYNK). It ships advanced features including a debugger with stack frame inspection.
Lem is an editor written in Common Lisp. It allows to start developing in CL at once, and it supports other languages.
we have a Jupyter kernel for CL.
the Dandelion Eclipse plugin was re-discovered. While it isn’t as feature-rich as others (no interactive debugger for example), it has its users. It specifically targets beginners.

Last but not least, if you want to play in your iPhone or iPad, the CodePlayground app got Lisp support via CCL.

Consolidation

SLY might need more praise. It has sound features such as SLY stickers and the new SLY stepper.

Developer utilities

Life continues to improve for the developper. We will cite some new tools:

cl-flamegraph is a wrapper around SBCL’s statistical profiler to generate FlameGraph charts from Common Lisp programs.
tracer is a tracing profiler for Common Lisp, with output suitable for display in Chrome’s/Chromium’s Tracing Viewer.
GTFL is a graphical terminal for Lisp, meant for Lisp programmers who want to debug or visualize their own algorithms. It is a graphical trace in the browser.
Lisp REPL core dumper is a portable wrapper to generate Lisp cores on demand to start a REPL blazingly fast. It can preload provided systems to help build a collection of specialized Lisp cores.
- if you are used to working in different environments that require their own set of libraries, this core dumper (optionally along with SLY’s mrepl) can make switching from one another easier and faster.

Package Management

Quicklisp is the de-facto package manager. However, we now have:

Ultralisp, a Quicklisp distribution that builds every 5 minutes. We can add our project in two clicks.
CLPM, a new package manager that is compatible with Quicklisp, that allows to pin exact versions of dependencies, that is usable from the command line and that supports HTTPS.

Not forgetting Qlot, to install Quicklisp libraries relative to a directory.

Last but not least, as said earlier, many CL libraries were packaged for Guix (most notably by Pierre Neidhart of Nyxt).

Achievement

Ultralisp solves the 1-month release schedule of Quicklisp (which is a feature, but not to everyone’s taste) and makes it straightforward and quick to publish a library. CLPM by tackling a different approach solves other Quicklisp limitations. Both are great achievements.

Ultralisp also has a search box that searches a symbol on all its registered libraries. Very useful.

Future work

Alexander is working on allowing every Ultralisp user to create his own Quicklisp dist in a few clicks.

Build System

Same as 2015, ASDF is the de-facto build system.

Every project has an .asd file, called a system definition file, which defines project metadata (author, maintainer, homepage, etc.) and the components.

This, to me, is one of the major selling points of Common Lisp. With languages like Python, every file imports whatever it needs, and your project becomes a massive graph of interdependent files. In ASDF, you basically list the files in your project in the order in which they are defined. Or, you can specify the dependencies between the files, and let ASDF figure out a linear ordering. The point is that dependencies are explicit, and clearly spelled out.

Type system

Quoting Fernando:

There’s not much to say here, except that Common Lisp has a pretty great type system that is not exploited nearly enough.

And to our greatest pleasure, SBCL’s type system continues to improve. For example, SBCL 1.5.9 now gives type warnings when a slot declared type doesn’t match its initform. It continued to improve on SBCL 2.0 and onwards.

Moreover, the Coalton library is bringing a dialect of ML on top of CL, in order to write statically typed programs similar in spirit to Standard ML, OCaml, and Haskell.

Consolidation

Help develop Coalton.

Testing, CI

Fernando cited FiveAM and recommended it along with the much newer Prove. Prove has a couple issues and is now deprecated by its author, and its younger brother Rove is not in par yet.

So, use FiveAM.

Moreover, Common Lisp has good support for the CI/CD services out there.

CL Foundation’s Docker images have integrated best practices over the years and are recommended: https://common-lisp.net/project/cl-docker-images/

CI-Utils regroups utilities for various platforms (Travis, Circle, Gitlab, Github, Appveyor, Bitbucket, Azure) and test frameworks.

We got a comprehensive blog post for GitHub actions: part1 and part2.

For Travis CI, you can also see cl-travis (for ABCL, Allegro CL, SBCL, CMUCL, CCL and ECL).

You will find an example for Gitlab CI on the Cookbook.

Consolidation

Rove or Parachute would be great alternatives if developed a bit further.

Further work

Integration with the CI services’ advanced features such as Gitlab’s auto DevOps.

Community

Online presence

Common Lisp is very well documented through its standard, the Common Lisp Hyper Spec and many books. However, we felt it was lacking good on-line material. Good news is, the situation improved tremendously in the last three or four years.

New common-lisp.net website

https://common-lisp.net was written anew. It looked dated. This is now fixed. Well done!

Cookbook

The Common Lisp Cookbook on GitHub got revived by many new contributors, included myself. It got many new content and a new UI. It is also now available in ePub and PDF, for free or as a “pay what you want” option.

Consolidation

Write content on the Cookbook. Don’t write tutorials on your blog. Everyone can help, even new Lispers (and in fact: mostly new Lispers can write content best suited to the Cookbook’s target audience).

Future work

Make it look world-class with a real and modern theme.

Help revive the minispec ?

awesome-cl

The awesome-cl list saw continuous updates and is now a great solution to have an overview of the ecosystem and choose a library.

One of its goals is to break choice paralysis by recommending libraries, with its “+1” marks.

Consolidation

Help furnish and curate it.

A first Common Lisp User survey was run, we can consult its results here on Google docs and read comments on reddit as well as feedback on the questions here.

I agree with /u/defunkydrummer here:

Note that many, many things that people wish to see, are already available, so perhaps we, as a community, are not fully communicating the state of our ecosystem even to our insiders (!)

Several popular libraries have been ported to readthedocs, so the reading experience is more pleasant: https://common-lisp-libraries.readthedocs.io/.

Michal “phoe” Herda organized many online Lisp meetings, and we can find the videos on Youtube: https://www.youtube.com/c/OnlineLispMeetings/videos

Alexander Artemenko started lisp project of the day, a blog to review a library every day for a month, and he is now at post #219. Lately he reviewed many documentation builders for CL.

On a sadder note, Quickdocs closed :(

New books

We got 3 new books on Common Lisp in 2020:

Programming Algorithms, originally published by Vsevolod Dyomkin on his website, then self-published in paperback and then published by Apress.
the Common Lisp Condition System, by Michal “phoe” Herda, was also published by himself and then by Apress.
The Cookbook that was made available in ePub and PDF :)

And also:

the book Calendrical calculations, 4th edition, by Edward M. Reingold, Nachum Dershowitz, Cambridge Press. It provides Lisp sources.
Building Problem Solvers, by Kenneth Forbus and Johan de Kleer, MIT Press, was made available.

Companies

We now have a curated list of companies using CL: awesome-cl-companies. Before that list, the situation was embarassing:

Everyone says “Nobody uses Lisp” and Lispers say “Yes they do, there’s ITA, and, um, Autocad, and, uh, oh yeah, Paul Graham wrote Viaweb in Lisp!” Not very helpful for either side. It’s about time there was a better resource.

Peter Christensen in his first list

Some additions of this year include GraphMetrix (automation of document extraction and publishing for construction, property and logistics), Doremir Music Research AB (developing ScoreCloud, a music notation software: you sing, it writes the score), Keepit (a cloud-to-cloud backup service provider), Mind AI (an artificial intelligence engine and ecosystem), Virtual Insurance Products Ltd (insurance MGA with a bespoke business to business web platform) or again the Mimix Company (creators of MSL and Nebula, new tools for working with facts and documents).

Growth

We are able to compare the number of downloads of the 100 most popular Quicklisp libraries between 2015 and 2020:

We can observe a 3x growth in five years. Of course, these figures need to be taken with a grain of salt, what they really represent is subject to interpretation. What is the role of Continuous Integration here?

Check it yourself: snippet, JSFiddle.

Last words

Many things are happening in the CL universe. Stay tuned!

The article source.

The Common Lisp Cookbook Is Now Available in EPUB and PDF

Tue, 19 Jan 2021 11:32:25 +0100

I am glad to announce that the Common Lisp Cookbook is now available in ePub and PDF.

It is available for free, and you can pay what you want[1] to say a loud “thank you” and to further support its development. Thanks!

This EPUB represents the work on the span of three years where I have been constantly reading, experimenting, asking, discovering tips, tools, libraries and best-practices, built-in or not, all of which should have been easily accessible but were not. Now they are. Reviving the Cookbook project resonated in the community, as other lispers sent great contributions.

Donate and download the EPUB version

=> https://lispcookbook.github.io/cl-cookbook/ <=

[1]: above 6 USD actually.

Another Common Lisp app in production

Mon, 14 Dec 2020 22:54:28 +0100

A quick post to celebrate the birth of another Common Lisp application running in production©. This time, it is not open source, but I can describe it.

It is used by bookshops in France and Belgium to upload their catalogue to online platforms. And no, they don’t know, and don’t need to know, the language it is implemented in!

It is a simple application that reads data from an existing DB, builds a text file with special rules, sends the file to an FTP server, and does it every day. I used cl-dbi with raw SQL queries, cl-ftp (does its job perfectly), and a CRON job. I built a binary that I sent to my server. It is a stand-alone application that reads a DB that is created by a bigger Python/Django web app (that I also develop). I didn’t want to make this one more bloated, so given the goals are complementary but orthogonal, I went with a stand-alone tool.

That’s it. One more!

Another tool I am running connects to a SOAP service, shows data on a website (with Sentry configured in production), sells products with Stripe (edit: it’s postponed :/ ) and sends emails with Sendgrid. And I (generally) update it while it runs by connecting to the Lisp REPL. Just throwing out buzzwords to you.

While I’m at it, let me stress one point, to answer in advance a kind of feedback I already had: no, the resulting application doesn’t use any Lisp superpower and yes, I could have written it in Python. It turns out Lisp is as suited as Python for this task (or then it is more suited, since it is faster), the point is I benefited from Lisp’s superpowers during development (by using the superior REPL, being able to build a binary and all that). In conclusion: there are tons of places where Lisp can be used for professional needs out there.

Oh. In doing it, I built those ~~two~~ three utilities:

cl-sendgrid, a trivial interface to the Sendgrid API (to send emails).
progressons, a progress bar that holds on one line and works on the terminal as well as on Slime. It works for me©. My next goal is to make it output a prettier bar with unicode bars.
termp, a trivial utility that checks if we are on a real or on a dumb terminal (by checking the TERM environment variable). So you can quit or error out.

More Lisp repositories on Github !

Pro mailing list: on Common Lisp and parallel GC

Mon, 14 Dec 2020 22:41:32 +0100

I recently enjoyed this discussion on the pro mailing list. It started with a call of recommendations on music software, and the discussion evolved in discussing parallel garbage collection. By the way, can you site an implementation that has parallel GC?

Pascal Costanza:

«When moving our elPrep software away from Common Lisp, we evaluated C++, Go and Java as potential candidates, and Go turned out to provide the best balance between performance and memory use. We are still using Common Lisp for prototyping, and then translate to Go. These two languages are actually much more similar than it appears at first. […]»

«This was primarily for the lack of good parallel, concurrent garbage collectors in Common Lisp implementations. The CL version of elPrep was actually still a tad faster than any of the C++, Go, or Java versions, but we had to work hard to avoid long GC pauses. elPrep allocates a lot of memory, and the pause time hurts a lot. We solved this by, basically, disabling the garbage collector, and reusing memory manually as much as possible, which turned the program into almost a manually memory-managed affair.»

«Manual memory management became a huge burden because we wanted to add more and more components to the software, and then it becomes almost impossible to predict object lifetimes.»

«We evaluated Go and Java for their concurrent, parallel GCs, and C++ for its reference counting. Interestingly, reference counting is often described as more efficient than GC, but in our case that’s not true: Because there is a huge object graph at some stage that needs to be deallocated, reference counting incurs more or less the same pause that a non-concurrent GC does. That’s why we don’t expect Rust to fare better here either.»

«Again, we’re still prototyping in Common Lisp, which is a huge win, because this makes us much more productive.»

«In my opinion, prototyping in Common Lisp, and then translating to a different programming language for creating the final product, is a perfectly valid professional use of Common Lisp. It’s useful to know which programming languages may be good targets for such an approach. This is, of course, not ideal, because this can easily be misunderstood as a statement that Common Lisp is not fit for purpose. However, I don’t see it that way, and you cannot control people’s perceptions. In our particular case, our manager is on board with this approach, and this allows us to pay for regular licenses for LispWorks. The approach works really well for us.»

Didier Verna:

«I’d be curious to know if there are particularities in CL itself that make this difficult, or if it’s simply because there’s no manpower to improve the GCs we have currently.»

Stelian Ionescu:

«It’s strictly a lack of manpower. Most CL implementations have GCs that were state-of-the-art 25 years ago: they’re either mark-and-sweep or copying & generational, and have to perform all collection while application threads are paused (i.e. stop-the-world), hence the collection pauses that are proportional to the heap size.»

«The newer GCs of Go and the JVM (ZGC and Shenandoah) are not generational and employ techniques such as pointer coloring and store/load barriers by instrumenting all object read/write operations instead of using virtual memory protection (which tends to have a non-indifferent performance penalty), and because they rely heavily on atomic operations to maintain heap consistency the stop-the-world phase is much shorter and only required to update the internal GC metadata. The result is that instead of 99th percentile pauses of 10+ seconds that we see with QPX or other allocation-heavy applications, these newer GCs show 99th percentile pauses of < 10ms, and perhaps medians going from ~500ms to 2ms (YMMV).»

«Here’s a pretty good description of the difference between the two new JVM collectors and how they compare to the older ones: https://www.youtube.com/watch?v=WU_mqNBEacw.»

Martin Cracauer:

«No, it’s as possible as in other languages. Some people don’t want to pay the overall performance penalty for concurrent GC (as in total CPU time/energy spent for any given piece of work).»

«This particularly applies to applications that are query-based, and hence want to be as fast as possible in the non-GC part, and can GC between queries. ITA’s QPX is an example (although they do desire concurrent GC for better monitoring in the production environment).»

«Parallel GC is no problem and implemented.»

Pascal Costanza:

«Which CL implementations have a parallel GC?»

Jeff Caldwell:

«From Franz’s doc on Allegro CL: https://franz.com/support/documentation/10.0/doc/gc.htm#multi-threading-2»

Martin Cracauer:

«Clasp (via Boehm GC and MPS).»

«I thought SBCL was there, but I just checked, not yet. I think Google is pushing for a parallel GC instead, because of response times to their production monitoring.»

«Another untapped source of performance is userfaultfd(2) in the Linux kernel. It allows those GCs that implement a write barrier using page protections SIGSEGV to use the faster userfaultfd interface instead (as opposed to those using a bitmap). This won’t help concurrent GC, but parallel GC would benefit even more than single-thread GC because it uses faster system calls. Proof of concept is here: https://www.cons.org/cracauer/cracauer-userfaultfd.html»

and:

Don’t the latest incarnations of ECL use the Bohem GC?

Daniel Kochmański:

«They do, we plan to resurrect the homegrown gc as an alternative though.»

Lisp Interview: more questions to CLPM author. Common Lisp at university for temporal reasoning and risk-bounded planning

Sun, 15 Nov 2020 08:22:59 +0100

Some days ago on reddit/r/lisp, we got to (re)discover CLPM, the Common Lisp Package Manager.

Its author, Eric Timmons aka daewok, was kind enough to give more context, and to answer some more questions of mine, about his use of Common Lisp in his university group.

Below I’ll give an overview of CLPM, stress on how it differs from Quicklisp, and then paste the interview.

/Note/: it’s the same content as on reddit, but saved from oblivion!

Table of Contents

CLPM

CLPM

Context

CLPM author here. I’m so happy (and shocked!) that people have found CLPM, especially given how little advertising I’ve done for it! I’m a PhD student in a group that does a large amount of our coding in Common Lisp. A big part of why I wrote CLPM, for better or worse, is that my group has not done a great job at versioning or maintaining backward compatibility of our various libraries throughout the years. I’m a very applications focused person and it was incredibly frustrating when I needed to deploy some of our code that had worked in the past but found that it bit rotted. And then when I would eventually get everything rolled back to the correct point in time I had no way to release a fix that was no longer applicable on the master branch (normally because the relevant API no longer existed or had been modified in a non-backward compatible way). So I hoped that encouraging/requiring actual version numbers would help us better communicate and reason about our changes over time (and be able to release minor changes to older versions of the code) and the locking capabilities would help save us in situations like giving a demo on short notice.

I use CLPM as my daily driver as do a couple of the other Lisp-heavy students. It’s been going well so far, but I was planning to convert and get feedback from a few more before attempting to spread CLPM beyond my group. That’s unfortunately taken a lot longer than I wanted due to COVID, my personal life taking more time than normal (no worries, it’s for good reasons!), and just general research related tasks continuing to pop up.

Now that the cat’s out of the bag though, I’d be happy to hear any feedback on it! I’m especially interested in the perspectives of people outside my group since I’ve been holding their hand in getting it set up and explaining my reasoning/goals to them almost every step of the way.

Feature set and comparison to Quicklisp

CLPM’s project page (here’s a non-official GitHub mirror, if that helps you browse the repository) lists the project goals. Here’s my comment and comparison to Quicklisp.

Support and encourage explicitly versioned systems

When a package upgrade introduces regressions, we should be able to use an older version.

But, that is currently not possible with the Quicklisp client, we must refer to other tools (Qlot) or manual workarounds.

CLPM allows to use monthly-based releases, just as Quicklisp and from Quicklisp, but it also started a new source registry for Common Lisp libraries, which would:

allow to precisely pin dependencies. It is possible to do so in ASDF, but this propriety is not used in Quicklisp (or barely, or not by most of the library authors, because Quicklisp comes as monthly distributions anyways).
allow to get the library’s home URL, which surprisingly isn’t in Quicklisp’s metadata (last time I checked, I might be wrong). We have to look at the quicklisp-projects repository.
it would enforce the libraries to be on version control. Currently Quicklisp also accepts source files (as archives).

Support installing multiple package versions, globally and locally

CLPM allows to manage dependencies per project (per directory), and globally. With Quicklisp, it’s only globally. Otherwise we must rely to Qlot, or load projects more manually.

While I personally find the Quicklisp approach great, simple to use, sufficient in most cases and a better default than always pinning dependencies manually, comes a point in a software life when we need project-local dependencies.

Support CI/CD workflows - ship pre-built binaries

CLPM is distributed in both binary and source form. Source for hackers or people who want to use a different feature set and binary for quick and easy installation in other cases.

That’s simpler to install, to use on CI systems, or to make you software’s users install the dependencies.

Currently we can use Roswell to install Quicklisp libraries (and software) from the command line, but its installation isn’t crystal straightforward or super fast either.

Minimize footprint in development images and deployments

When you use CLPM and you build a binary of your program, the binary won’t contain CLPM (or only if you choose to). When we use Quicklisp, the built image contains Quicklisp (which can be very useful, I use it to live-reload running web apps).

Support HTTPS

Quicklisp currently doesn’t download packages through HTTPS.

Cryptographic signature verification is coming.

You can buy a preview of the Common Lisp Cookbook in ePub

Fri, 13 Nov 2020 22:48:15 +0100

See here: https://ko-fi.com/s/01fee22a32

Let me try something: I propose you here to buy the ePub, even though it is meant to be available for free.

I contributed quite a lot to the Cookbook and I found since the beginning that having an EPUB and/or a PDF version would be very useful. Some years later, nobody did it, and I finally wrote a script to bundle all the pages together and generate an ePub, and then a PDF. It isn’t finished though, it needs more editing, and it would also be great to write a proper LaTeX style for the PDF. As I have one version on disk, I thought on giving potential supporters the opportunity to read it and fund the remaining work, or simply give a sign of encouragement. It’s also the opportunity for me to try this new Ko-fi feature, and to practice speaking about financial support…

Let’s turn it another way: if you support me, here’s a reward for you, an exclusivity. And in any case, hold on, this ePub will eventually be available for everybody.

Thanks!

ps: About money: yes, it does a difference right now, because I don’t have a fixed, nor big, income. I am trying to live on my free software activities. I have a few clients, but not enough, so I’m trying to diversify and get some earnings from my Lisp work. I even pay lisp contributors to write free software (we are currently adding Stripe payments to a web application. When it’s done, you’ll know everything about it). So, the money has an impact.

Composing queries with Mito, or doing without Django lazy querysets and Q objects

Thu, 24 Sep 2020 18:15:44 +0200

When I didn’t know Lisp at all, I skimmed at CLSQL’s and Mito’s documentation and I didn’t find a mention of “lazy”, “querysets” (a Django term!) nor a mention of any means to compose queries. I had no idea how I would replace querysets, F and Q objects and the many functions for DB queries that were being added into newer Django versions. I concluded that the Lisp ecosystem was lagging behind.

Table of Contents

Then I began to understand. And today I got the chance to rewrite a Django query involving querysets and Q objects, using regular Lisp. All you have to know is back-quote and comma.

We implement a simple search into a DB. The user enters one or more words and we search against the title and the authors fields. We want to match all words, but each can be either in the title, either in the authors field.

Considering we have two products:

1 - The Lisp Condition System - phoe
2 - Implementing a blockchain in Lisp - F. Drummer

then searching for “lisp cond” must return one result. DWIM.

In Python, we must use Q objects to “OR” the terms with | (you can’t use | without Q):

products = firstquery()
for word in words:
    products = products.objects.filter(Q(title__icontains=word) |
                                       Q(authors__name__icontains=word))\
                               .distinct()

The promise of filter is to be lazy, so when we chain them the ORM constructs one single SQL query.

So what does this query yield as SQL? Funnily, I didn’t find a built-in way to get the generated SQL and I had to use a third-party library. Mmh, I could use special logging. The fact is, we are far from SQL here (and, with the experience, it is NOT a good thing).

It looks like this (searching “hommes femmes” in our test DB):

       SELECT DISTINCT ... FROM "product" LEFT OUTER JOIN ...
       WHERE
       (("product"."title" LIKE %hommes% ESCAPE '\'
         OR "author"."name" LIKE %hommes% ESCAPE '\')
        AND
         ("product"."title" LIKE %femmes% ESCAPE '\'
         OR T5."name" LIKE %femmes% ESCAPE '\'))
       ORDER BY "product"."created" DESC
       LIMIT 3

Does that look complicated? Does that need alien “Q objects”?! It’s just a AND around two OR:

   title like keyword 1 OR author like keyword 1
AND
   title like keyword 2 OR author like keyword 2

Mito is the high-level library, and we compose queries with SXQL. I already had a little query that worked with one keyword:

(defun find-product (&key query (order :asc))
  (mito:select-dao 'product
    (when query
      (sxql:where (:or (:like :title (str:concat "%" query "%"))
                       (:like :authors (str:concat "%" query "%")))))
    (sxql:order-by `(,order :created-at))))

If :query is given, we filter the search. If not, the when is not executed and we return all products.

So what we need to do is iterate over the keywords, produce as many OR and wrap them with a AND. We want something like that (we can try in the REPL):

(:AND
 (:OR (:LIKE :TITLE "%word1%")
      (:LIKE :AUTHORS "%word1%"))
 (:OR (:LIKE :TITLE "%word2%")
      (:LIKE :AUTHORS "%word2%")))

So:

The solution

(sxql:where
 `(:and   ;; <-- backquote
   ,@(loop for word in (str:words query)  ;; <-- comma-splice
        :collect `(:or (:like :title ,(str:concat "%" word "%"))  ;; <-- backquote, comma
                       (:like :authors ,(str:concat "%" word "%"))))))

(using (ql:quickload "str"))

Pay attention to ,@ (comma-splice). Without it, we get a bad level of nesting and two parenthesis before the :OR. We would get a list of clauses, instead of each clause individually. You can try in the REPL.

Note: if you are uneasy with back-quote and comma, see: https://lispcookbook.github.io/cl-cookbook/macros.html

Last words

Django’s filter is similar to using a When, which we were already using on the Lisp side without knowing it was anything special. “Q objects” are easy to replace. So, Python and Django might be easy to getting started with (or it is your feeling, because you must learn the special syntax and its limitations, I bet you had some “WTF?!” moments), but comes a time when your application grows that you pay the price of being far from SQL (not counting the maintenance cost).

With Mito and SXQL, it’s all regular Lisp, we are closer to the metal, the only limitation being to know the language, and a bit of SQL.

So we have a great example of why some Common Lisp libraries have a surprisingly low number of commits. You know, that little voice in your head that wonders if a library is finished or active. The author might not need to develop feature X, thanks to Lisp’s expressiveness. Likewise, many questions don’t need to be asked or upvoted on Stack-Overflow. Though I should have asked years ago.

getting started with a DB: https://lispcookbook.github.io/cl-cookbook/databases.html
more DB choices: https://github.com/CodyReichert/awesome-cl#database

You like what I'm doing?

How are Lisp REPLs different from Python or Ruby REPLs ? (Hackernews, 2020)

Tue, 14 Jul 2020 16:11:43 +0200

Mikelevins, https://news.ycombinator.com/item?id=23811382, July 2020

(some more comments on https://www.reddit.com/r/lisp/comments/hqesvp/explaining_the_advantages_of_the_repl/)

(on terminology: we should maybe call Python’s “REPL” a shell, and put emphasis on image-based development, instead of only saying REPL, for Lisp)

I’ve answered similar questions several times over the past few years, but I don’t mind repeating myself. It offers me a glimmer of hope that my preferred way of working may not fade away, after all.

Consider the standard Common Lisp generic function UPDATE-INSTANCE-FOR-REDEFINED-CLASS (http://clhs.lisp.se/Body/f_upda_1.htm). It reinitializes an object when Lisp detects that the object’s class definition has changed.

Ask yourself this: who would call such a function? Why would anyone ever invent it? Not only did someone invent it, a committee of some of the world’s smartest and most experienced Lisp programmers wrote it into the ANSI standard for the language. What were they up to?

UPDATE-INSTANCE-FOR-REDEFINED-CLASS is not a weird anomaly; it’s part of a carefully-considered set of features and protocols designed to support a specific style of programming. The Lisp runtime calls it for you automatically when it touches an object whose class definition has changed.

If you’ve defined a method specialized for it, then Lisp executes that method to rebuild the touched instance as if it had originally been instantiated from the class’s new definition, and then your program goes its merry way. If you didn’t specialize UPDATE-INSTANCE-FOR-REDEFINED-CLASS for this case, then the Lisp drops you into a breakloop.

A breakloop is an interactive repl with full access to all of the runtime’s memory and all of the language’s features, including visibility into the whole call stack that landed you in the breakloop. You can wander up and down the call stack, inspect anything in the runtime, edit bindings, redefine types and functions, and resume execution either at the point of control where the breakloop started, or at any other point for which the breakloop exposes a restart.

UPDATE-INSTANCE-FOR-REDEFINED-CLASS is not the weird fever dream of a confused eccentric. It’s part of a purposeful system design intended to support a style of programming in which you build a program by interacting with a live runtime and teach it, interaction-by-interaction, how to be the program you want, while it runs.

It’s a particular example of a general approach to programming best exemplified by these old systems. That general approach is the answer to your question: “Can someone knowledgeable explain how are lisp REPLs different from Python / Ruby REPLs? What is the differentiating point of REPL driven development?”

The differentiating point is that the entire language and system is thoughtfully designed from the ground up with the assumption that you’re going to be changing your work in progress while it runs, and that you should be able to change absolutely anything about it as it runs and have a reasonable expectation that it will continue to work while you do it.

I like to call this style of programming “programming as teaching”, and distinguish it from the much more widespread “programming as carpentry”, in which the programmer is, metaphorically speaking, at a workbench banging together artifacts and assembling them to see how they turn out.

To be clear, I do not claim that the teaching approach is objectively better than the carpentry approach. I claim only that I, personally, am happier and measurably more productive using the teaching approach. I know that some other programmers report the same thing, and I suspect that if the teaching style of programming were more widely known, then there would be more programmers who prefer it.

There are several sibling comments that assert that any language can be made to support repl-driven programming, or that offer various languages and systems as examples of repl-driven programming. I’m sure that’s all true, for some relatively restricted version of repl-driven programming, but the gold standard in repl-driven programming is programming as teaching in the style of old-fashioned Lisp and Smalltalk systems. These old systems offer amenities that the younger alternatives touted here do not match. I want more people to be aware of what they’re missing.

Starting in the 1980s, I grew accustomed to systems that could start from cold in about a second, presenting to me a complete interactive development environment with all tools preloaded and ready to work, with the whole dynamic environment of my work in progress in the same state it was in the last time I was working with it. Moreover, I was accustomed to being able to take a single file from one machine to another to reproduce that same whole working environment equally quickly and easily on the new machine.

I could save the entire dynamic state of the running system to an image file, a serialized version of the running system’s memory. I could later start up the system with that image file and be exactly where I was when I saved the image, right down to the positions and contents of all the open windows. I could save an image showing some bug or some strange behavior and give it to a colleague so that they could see it, too, and interact with the restored dynamic state to debug it.

I enjoyed comprehensive whole-system reflection that enabled me to view and edit absolutely everything in the running system while it ran. I could inspect absolutely everything, including the development environment and all its tools, interactively change any variable or field value, redefine any type or function, and continue to work with the changed system without stopping and restarting. (Obviously, if I made a bad change I might break the system, but remember, I could kill it and get back to where I started in a second or so).

I could start some process running–perhaps a 3D animation in a game, or a discrete-event simulation, or whatever–and change any values or definitions I liked to see what changed in the running process, without stopping the process to rebuild. For example, I could tell a rotating copper cube to become a glass icosahedron and reasonably expect to see my changes immediately reflected in the running program. This property is invaluable not only in games, simulations, and any kind of work with a visual-design component, but also in any kind of exploratory programming, where you’re constructing data structures and evaluating expressions interactively to test your ideas.

Similarly, I could build some speculative data structure to explore an idea, and define some functions to operate on it. I could evaluate those expressions to see their results or to change the example structure. I could inspect the structure interactively and edit it in place if I think something different would work better. If I think a problem is caused by some structure or value in it, I could use the inspector to change it and see. If I thought one my my functions was doing something I didn’t expect, I could insert a call to break, to activate a repl from inside the function call that would enable me to inspect and edit the data structure, redefine the function, and continue from there.

Anything the development system could do, I could do by typing an expression into the repl. As an example, nowadays you can still rebuild the whole Clozure Common Lisp environment from the ground up by typing (rebuild-ccl :full t).

The point is not that I would want to rebuld my Lisp from the repl all the time. The point is that the repl doesn’t impose any aribtrary boundaries on what I can do. If the language and development environment can do it, I can do it from the repl. This is one of the properties that distinguishes the whole-system interactive design of these old tools from the more limited repls offered by newer ones. In pretty much every repl I’ve used other than old-style Lisps and Smalltalks I’m all the time stumbling over things you can’t do from the repl.

I mentioned breakloops above. Their absence in younger languages and tools seem to me like some sort of sin, like we’re tragically abandoning some of the best lessons of the past. Few newer development systems have them, but they’re incredibly useful–at least if the language runtime is designed to properly support interactive programming.

A breakloop is a repl with all of the same affordances of the normal repl, but extended with all of the dynamic state of the control path that invoked the breakloop. If an error or an intentional call to break triggers a breakloop somewhere deep in a stack of recursive function calls, you get a repl that can see every frame of that stack, and every variable and value lexically accessible from it. You can browse all of that whole, change values, and redefine functions and types. You can resume execution at your leisure, and any changes you made in the breakloop will be visible in the resumed computation just as if that’s how things were originally.

Proper breakloops don’t just improve error messages; they replace them wholesale with an entire species of programming that lays the whole dynamic state of the system out on the table for you to examine and modify while the program continues to run.

Moreover, everything I just described about breakloops can also be automated. These old systems provide not only interactive tools for rummaging through the dynamic state of a suspended computation, but also APIs for handling them under program control. For example, you can wrap an arbitrary function call in condition handlers that will either drop you into a breakloop and enable you to vivisect the program state, or consult the dynamic state and compute which of several restarts to activate in order to transfer control to a path of your choosing.

I’m banging up against HN’s length limit, but the above, I hope, goes some way toward answering to your question.

Looking for a Lisp Web Developer (not a real job, but nearly)

Fri, 05 Jun 2020 21:38:01 +0200

Dear lispers,

I decided that I can not develop three projects in parallel fast enough, so I’m seeking for a fellow programmer to join the effort.

Disclaimer: this is not a real position, but there is a little budget

I recently presented my online catalogue for bookshops. You will work on something very similar, but bigger. I need help to re-write the existing free software for bookshops in Common Lisp. The existing one is in Python. I have a prototype of the new CL one.

The software specifications are here and are good. We wrote them years ago by consulting people selling books, and we now grow on our experience acquired developing the first Python app as well as on the feedback gathered from clients. The challenge is to build a maintainable, fast, bug-free, easily-deployable application with an user interface that answers the clients’ needs.

I ask you to have some experience in:

Common Lisp
the web: HTML, CSS, web browser API
SQL

You should also have a sufficiently good english to speak with me (a non-native speaker).

I would have tasks for you in June and July, nothing in August, and hopefully more in September and onwards.

Bonus points include, in no particular order:

acquaintance with JavaScript, with a JS framework (preferably Vuejs)
good HTML&CSS design skills
you speak french
you have a good english, or good communication skills in your mother tongue
good backend and “devops” experience
Python experience to install and study an existing project

If this appeals to you, please email me at (reverse "gro.zliam@leradniv") and indicate roughly how you stand in these points, your availability in June, during the upcoming two weeks for a meeting, and we’ll speak further.

Thanks!

A free software for bookshops to show their catalogue online

Sat, 30 May 2020 13:15:47 +0200

I wrote a free software for bookshops to publish their catalogue online. Clients can now browse the available books and order them. It is enough generic so we can show other products too.

https://abstock.gitlab.io/#/en/
sources and bug tracker: https://gitlab.com/vindarel/abstock
Github mirror: https://github.com/vindarel/ABStock
the demo

Here’s how a search result looks like:

Features

The website is made generic enough for different clients, and is made totally hackable with pre- and post- configuration files that load your Lisp logic.

By default we get the following pages:

the welcome screen, with:
- the bookshop’s information,
- the search form. We can search by title, authors, publisher, shelf and ISBN(s).
- a random pre-selection of the books to showcase, if enabled.
an optional special page to showcase a selection of books or other products.
in the search results page, visitors can add a book to their shopping basket.
and in the basket page, they find a confirmation form, which sends the command by email to the shop owner.

There are obvious TODOs, that could be shocking by their absence, but that I actually don’t need yet, so they’ll come right in time :)

online payment
admin page
- simple stats (they are brought in with the email provider, and with Matomo statistics)
i18n, remove a still few hardcoded words

Data

ABStock connects by default to the Abelujo database. Abelujo is a free software for bookshops that I also develop. Booksellers use it for their daily work, registering and selling books.

But we can define our own products. The current possibility is to use a cards.txt file. Each block expects a title, and that’s the only mandatory field. Other recognized fields are:

((:|id| integer)
 :|title|
 :|cover|
 :|isbn|
 :|price|
 :|author|
 :|publisher|
 :|date_publication|
 :|date-publication|
 :|shelf|
 :|shelf_id|
 :|details-url|
 :|summary|
 :|quantity|
 :|repr|
 :|repr2|)

We can define other fields, like likes below, which is actually unused in the application.

[…]

title: Programming Algorithms
author: Vsevolod Domkin
cover: https://d2sofvawe08yqg.cloudfront.net/progalgs/hero?1586867024
details-url: https://leanpub.com/progalgs
shelf: programming
shelf_id: 99
publisher: Leanpub
price: 15
likes: 5

title: Cats
author: mother cat
cover: https://gitlab.com/abstock/abstock.gitlab.io/-/raw/master/logo.png
details-url: https://gitlab.com/abstock/abstock.gitlab.io/-/blob/master/logo.png
shelf: nature
shelf_id: 98
publisher: nature
price: 5
likes: 100

title: Screwdriver
author:
cover: https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2Ftse1.mm.bing.net%2Fth%3Fid%3DOIP.jGOb7dVL1oA9VDzNVPDPpwAAAA%26pid%3DApi&f=1
details-url:
shelf: craftmanship
shelf_id: 97
publisher:
price: 10
likes: 1

(yes there’s a little redundancy with shelf and shelf_id to fix)

That gives:

We could very well load a JSON, a CSV or another database when the need arises.

For now, we think the text loader is enough for you to define your products and try the application.

It is also very easy to host, and in doing so one can realize that to live-reload his Lisp web app is straighforward and very convenient.

Context

I shipped the app the second month of our lockdown period for the client I was working for at that moment, and no need to say it turned 100% helpful. Amazon had hell of an activity, and french alternatives for booksellers (such as lalibrairie.com or placedeslibraires.fr) either had stopped, either couldn’t accept new registrations. So we were left alone, and we did that. His clients were happy, they started passing orders, he organized collection schedules. This happened in a small rural village, where the inhabitants are happy to have, at least, a (nice) bookshop in their village.

Paying one’s rent with Lisp

So yes, I paid my rent with Common Lisp again \o/ And you see, the software is a classical web app. I could have made it with Python or another language that has a web server and a templating library. As I defended before, the app doesn’t exist thanks to CL’s super powers. CL had no particular advantages for this kind of web app, but no disavantages either: it has a good web framework, a good templating library that I liked a lot (Djula: defining custom filters was a breeze), a good SQL wrapper, and that’s all I asked. I use CL’s super powers during development and deployment. Clients wouldn’t see the difference… or would they?

Actually, CL has advantages overall: development speed, ease of deployment, ease of hot-reload, ease to use the language features to bypass a library’s limitation (easy-routes had no built-in to translate a route URL, but it turned possible with a reader macro)… not mentioning ease of maintenance in time, speed, etc.

One production bug I had was due to (me apart for not testing enough) (= 3 nil) throwing an error, so you must had prior checks (or, I just realize, use equalp?). My Sentry dashboard is empty anyways.

Final words (with bonus)

The Big Plan is to Rewrite It (the other software) In Lisp, and the project just moved from R&D to POW… stay tuned, particularly if you don’t know what to do in june and july, I might have a small budget for a helping hand.

Become a Patron!

Today I Realized that to live reload my Lisp web app is straightforward and so convenient

Tue, 12 May 2020 16:02:05 +0200

We all know that we can start a web server in the REPL and develop a web app as interactively as any other app, we know how to connect to a remote Lisp image by starting a Swank server and how to interact with it from our favorite editor on our machine, we know we can build a self-contained binary of the web app and simply run it, but one thing I had not realized, despite being the basics, is that by starting the web app with sbcl --load app.lisp, we are dropped into the regular Lisp REPL, with the web server running in its own thread (as in development mode, but unlike with the binary), and that we can consequently interact with the running app.

As a demonstration, you can clone this repository and run the example like this:

* rlwrap sbcl --load run.lisp

This is SBCL 1.4.5.debian, an implementation of ANSI Common Lisp.
re information about SBCL is available at <http://www.sbcl.org/>.
[…]
; Loading "web-live-reload"
..................................................
Starting the web server on port 7890
Ready. You can access the application!
*

it will load project.asd, install 3 Quicklisp dependencies (you must have Quicklisp installed), start Hunchentoot on port 7890, and drop us into a REPL.

You’ll get this:

The template prints the *config* variable, which you can change in the REPL:

* (in-package :web-live-reload)
* (setf *config*
    '((:key "Name"
       :val "James")
      (:key "phone"
       :val "0098 007")
      (:key "secret language?"
       :val "Lisp")))

refresh, and voilà, your new config is live.

For functions, it is just the same (redefine the function fn that returns a string, if you want to try).

If a file changes (for example after a git pull), compile it with a usual load: (load "src/web.lisp").

You can also reload all the app with (ql:quickload :myproject), which will install the dependencies, without needing to restart the running image.

I was looking for a way to reload a user’s config and personal data from a running website, and this has proved very practical. I have no downtime, it is pure Lisp, it is the workflow I am used to. I am more cautious on using this to recompile the whole app, even though I did it without glitches so far. The thing to not do is to change the global state manually, aka to develop in production!

That’s all, but that made my day.

Bonus points:

after a git pull, the (Djula) templates are automatically updated. No operation is needed to see them live. (you can disable this by pushing :djula-prod into the features set)
you’ll understand and appreciate the difference between defparameter and defvar. Imagine you declare a variable with (defparameter *data* nil) and you populate it with some heavy computation at the application startup. Now if you load the file this declaration is in, you’ll set the data back to nil. If you declare it with defvar, you can live-reload your app and the data doesn’t go away. You can try both cases with the *config* variable.
the app started a Swank server on port 4006, if you want to try on your VPS.

become a Patron!

I Worked Remotely in Common Lisp. Here's My Incredible Story.

Tue, 12 May 2020 12:52:57 +0200

Nearly one year ago, I received an email that asked me if I was available to do remote Lisp work. It was the day before the end of a contract and I had to tell my team if I wanted to continue or not. I made a virtual offering to the Lisp god and I started the Lisp job.

Disclaimer: this post was written on Lisp Advocates’ reddit. Lisp Advocates is a meme, but it’s sort of serious too.

At this time I had been in Lisp for around two years, contributing a couple simple libraries, writing a lot of documentation, blogging, furnishing the reddits, and being enthusiastic and polite. This is what actually gave me the job. I had tried to contribute to a busy CL repository, but the PR was not good enough and that irritated the maintainer, who answered abruptly. Nothing’s more outrageous than receiving contributions right? But I answered with calm and professionalism, and that got noticed by a repository watcher, who decided he could work with me.

That guy already had contacts and a client, and he formed a team around him. Our work was to build a website that would receive many visitors, that would have a client registration form and would have a rather simple admin dashboard, for a team of half a dozen people. The business already existed in the form of a buggy and slow Wordpress site, so the expectations were clear. We were three, we worked together on the same code (with one guy more on the design). I worked on it in a two-months period, but not full time. I’ve had a decent income paid straight and so I paid my rents for a few months thanks to that experience.

What Lisp was good for

The application had no inherent difficulties. It had forms and an admin backend. It was a website for a team of commercial people, as it exists hundreds of thousands. And yeah, Common Lisp was suited for that task. So we see there’s a good margin of progression, business and remote work wise: those thousands of websites for commercial people can very well be done in CL.

Libraries, deployment and Lisp curse

We picked the Caveman framework, the Mito ORM and the cl-markup templating library, with some tests in FiveAM. There was a little bit of JavaScript, less than a thousand lines. I find Caveman a bit convoluted but it was clear and easy. I like Mito very much and wrote material for it. I liked to play with the web server debugging options: usually I received the stacktraces in the debugger in my editor, but I could choose to display them on the browser (as I’m used with Django or Flask). It is this time that I enjoyed so much being able to change the faulty function, recompile it, choose the “try again” restart and see the operation succeed. Now when I’m back on Python I feel the Lisp curse. I’ll never be the same. I’ll never enjoy Python as much as before. Sigh. Anyways, we deployed the app on DigitalOcean with Fast CGI, as documented on Caveman’s README.

The bug

Our most difficult bug that made us loose millions was due to (string-downcase nil) to return “NIL”, the string, instead of nil. Now I use my str library for string manipulation purposes.

All in all, being able to live-debug the software from the earth proved invaluable.

I got also hit by a config of mine that impacted Mito’s results. I had set *print-case* to :downcase in my .sbclrc. I was asking Lisp to DON’T SHOUT AT ME ALL DAY LONG, ‘cause I try to listen to music at the same time. I fixed the Mito bug, but I don’t use this setting anymore.

Voilà. This is my response to LispAdvocates’ call: https://www.reddit.com/r/lispadvocates/comments/ficdvx/tell_us_you_remote_success_story/.

There are of course lots of situations were CL is ready now to get the (remote) job done. There are people who do web dev for years in CL, but we don’t know their story.

Share yours!

ps: stay tuned, ‘cause I deployed another website in production.

Some comments and answers:

Apart from the programmer experience, were there any inherent advantages to using Common Lisp? (Speed I guess?)

CL had no particular advantages, but no disadvantages either (and it is my point!). As I said, it was a site with basic/easy/HTML&JS requirements, so I believe no language would’ve had any particular advantage. Speed was important, it was one of the main requirements. The website felt responsive, the client was very happy about it. For us, it was also easy and fast to deploy, which turned important and impressed the client.

Do you thinK a more standardized framework (from other languages) could have saved you?

No, not with our requirements. Another framework&language would have make us loose millions at the very beginning (still figuratively)

Custom Djula filters

Wed, 08 Apr 2020 11:38:17 +0200

Djula is a Common Lisp port of the Django templating language. It’s good, it’s proven (it’s one of the most downloaded Quicklisp packages), it is easy to use and it has good documentation.

It basically looks like this:

    {% extends "base.html" %}
    {% block title %}Memberlist{% endblock %}
    {% block content %}
      <ul>
      {% for user in users %}
        <li><a href="{{ user.url }}">{{ user.username }}</a></li>
      {% endfor %}
      </ul>
    {% endblock %}

What was missing in the documentation was how to create custom filters. Here’s how, and it’s very simple.

def-filter

Use the def-filter macro. Its general form is:

(def-filter :myfilter-name (value arg)
  (body))

It always takes the variable’s value as argument, and it can have one required or optional argument. For example, this is how those built-in filters are defined:

(def-filter :capfirst (val)
  (string-capitalize (princ-to-string val)))

This is all there is to it. Once written, you can use it in your templates. You can define a filter wherever you want and there is no need to register it or to import it in your templates.

Here’s a filter with a required argument:

(def-filter :add (it n)
  (+ it (parse-integer n)))

and with an optional one:

(def-filter :datetime (it &optional format)
  (let ((timestamp …))))

When you need to pass a second argument, make your filter return a lambda function and chain it with the with filter:

    (def-filter :replace (it regex)
       (lambda (replace)
         (ppcre:regex-replace-all regex it replace)))

    (def-filter :with (it replace)
       (funcall it replace))

Now we can write::

{{ value | replace:foo | with:bar }}

Note: we should most probably be able to define filters with two arguments. There’s an open issue about that.

Error handling

Errors are handled by the macro, but you can handle them and return a template-error condition:

(def-filter :handle-error-filter (it)
   (handler-case
         (do-something)
     (condition (e)
       (template-error "There was an error executing this filter: ~A" e))))

It will be rendered on the browser with a nice stacktrace.

Final words

If you don’t know what template engine to use for your web project, start with it. My only criticism is that accessing variables is not totally flexible. The {{ obj.val }} syntax already works to access objects’ slots, alists, plists, hash-tables and whatnot (it uses the excellent Access library), but it won’t work for some data (like structures), forcing you to a bit of pre-processing before rendering the template. And you can’t use much logic with template tags. However, this is by design. Djula is a port of the Django templating engine after all.

For more flexible templates and still write html (because, you know, we can copy-paste examples easily!), see Eco. See more templates engines in the Awesome-cl list.

Last-minute addition: while I was writing this, Djula’s author released TEN, another templating engine, combining the best of Djula and Eco.

https://mmontone.github.io/djula/doc/build/html/filters.html#custom-filters

GUI Programming in Common Lisp, part 5/5: Nuklear

Fri, 27 Mar 2020 13:19:45 +0100

Nuklear is a small immediate-mode GUI toolkit:

Nuklear is a minimal-state, immediate-mode graphical user interface toolkit written in ANSI C and licensed under public domain. It was designed as a simple embeddable user interface for application and does not have any dependencies, a default render backend or OS window/input handling but instead provides a highly modular, library-based approach, with simple input state for input and draw commands describing primitive shapes as output. So instead of providing a layered library that tries to abstract over a number of platform and render backends, it focuses only on the actual UI.

its Lisp binding is Bodge-Nuklear, and its higher level companions bodge-ui and bodge-ui-window.

Unlike traditional UI frameworks, Nuklear allows the developer to take over the rendering loop or the input management. This might require more setup, but it makes Nuklear particularly well suited for games, or for applications where you want to create new controls.

Previous posts of the serie:

This blog post series was initially written for the Common Lisp Cookbook, you can (and should) read it there:

https://lispcookbook.github.io/cl-cookbook/gui.html

Framework written in: ANSI C, single-header library.
Portability: where C runs. Nuklear doesn’t contain platform-specific code. No direct OS or window handling is done in Nuklear. Instead all input state has to be provided by platform specific code.
Widgets choice: small.
Graphical builder: no.
Other features: fully skinnable and customisable.
Bindings stability: stable
Bindings activity: active
Licence: MIT or Public Domain (unlicence).
Example applications:
- Trivial-gamekit
- Obvius - a resurrected image processing library.
- Notalone - an autumn 2017 Lisp Game Jam entry.

List of widgets

Non-exhaustive list:

buttons, progressbar, image selector, (collapsable) tree, list, grid, range, slider, color picker,
date-picker

Getting started

Disclaimer: as per the author’s words at the time of writing, bodge-ui is in early stages of development and not ready for general use yet. There are some quirks that need to be fixed, which might require some changes in the API.

bodge-ui is not in Quicklisp but in its own Quicklisp distribution. Let’s install it:

(ql-dist:install-dist "http://bodge.borodust.org/dist/org.borodust.bodge.txt" :replace t :prompt nil)

Uncomment and evaluate this line only if you want to enable the OpenGL 2 renderer:

;; (cl:pushnew :bodge-gl2 cl:*features*)

Quickload bodge-ui-window:

(ql:quickload :bodge-ui-window)

We can run the built-in example:

(ql:quickload :bodge-ui-window/examples)
(bodge-ui-window.example.basic:run)

Now let’s define a package to write a simple application.

(cl:defpackage :bodge-ui-window-test
  (:use :cl :bodge-ui :bodge-host))
(in-package :bodge-ui-window-test)

(defpanel (main-panel
            (:title "Hello Bodge UI")
            (:origin 200 50)
            (:width 400) (:height 400)
            (:options :movable :resizable
                      :minimizable :scrollable
                      :closable))
  (label :text "Nested widgets:")
  (horizontal-layout
   (radio-group
    (radio :label "Option 1")
    (radio :label "Option 2" :activated t))
   (vertical-layout
    (check-box :label "Check 1" :width 100)
    (check-box :label "Check 2"))
   (vertical-layout
    (label :text "Awesomely" :align :left)
    (label :text "Stacked" :align :centered)
    (label :text "Labels" :align :right)))
  (label :text "Expand by width:")
  (horizontal-layout
   (button :label "Dynamic")
   (button :label "Min-Width" :width 80)
   (button :label "Fixed-Width" :expandable nil :width 100))
  )

(defun run ()
  (bodge-host:open-window (make-instance 'main-window)))

and run it:

(run)

To react to events, use the following signals:

:on-click
:on-hover
:on-leave
:on-change
:on-mouse-press
:on-mouse-release

They take as argument a function with one argument, the panel. But beware: they will be called on each rendering cycle when the widget is on the given state, so potentially a lot of times.

Interactive development

If you ran the example in the REPL, you couldn’t see what’s cool. Put the code in a lisp file and run it, so than you get the window. Now you can change the panel widgets and the layout, and your changes will be immediately applied while the application is running!

Conclusion

Have fun, don’t hesitate to share your experience and your apps… and contribute to the Cookbook!

GUI Programming in Common Lisp, part 4/5: IUP

Fri, 27 Mar 2020 13:11:36 +0100

IUP is a cross-platform GUI toolkit actively developed at the PUC university of Rio de Janeiro, Brazil. It uses native controls: the Windows API for Windows, Gtk3 for GNU/Linux. At the time of writing, it has a Cocoa port in the works (as well as iOS, Android and WASM ones). A particularity of IUP is its small API.

The Lisp bindings are lispnik/iup. They are nicely done in that they are automatically generated from the C sources. They can follow new IUP versions with a minimal work and the required steps are documented. All this gives us good guarantee over the bus factor.

IUP stands as a great solution in between Tk and Gtk or Qt.

Previous posts of the serie:

This blog post series was initially written for the Common Lisp Cookbook, you can (and should) read it there:

https://lispcookbook.github.io/cl-cookbook/gui.html

Framework written in: C (official API also in Lua and LED)
Portability: Windows and Linux, work started for Cocoa, iOS, Android, WASM.
Widgets choice: medium.
Graphical builder: yes: IupVisualLED
Other features: OpenGL, Web browser (WebKitGTK on GNU/Linux), plotting, Scintilla text editor
Bindings documentation: good examples and good readme, otherwise low.
Bindings stability: alpha (but fully generated and working nicely)
Bindings activity: low
Licence: IUP and the bindings are MIT licenced.

List of widgets

Radio, Tabs, FlatTabs, ScrollBox, DetachBox,
Button, FlatButton, DropButton, Calendar, Canvas, Colorbar, ColorBrowser, DatePick, Dial, Gauge, Label, FlatLabel,
FlatSeparator, Link, List, FlatList, ProgressBar, Spin, Text, Toggle, Tree, Val,
listDialog, Alarm, Color, Message, Font, Scintilla, file-dialog…
Cells, Matrix, MatrixEx, MatrixList,
GLCanvas, Plot, MglPlot, OleControl, WebBrowser (WebKit/Gtk+)…
drag-and-drop

Getting started

Please check the installation instructions upstream. You may need one system dependency on GNU/Linux, and to modify an environment variable on Windows.

Finally, do:

(ql:quickload :iup)

We are not going to :use IUP (it is a bad practice generally after all).

(defpackage :test-iup
  (:use :cl))
(in-package :test-iup)

The following snippet creates a dialog frame to display a text label.

(defun hello ()
  (iup:with-iup ()
    (let* ((label (iup:label :title (format nil "Hello, World!~%IUP ~A~%~A ~A"
                                            (iup:version)
                                            (lisp-implementation-type)
                                            (lisp-implementation-version))))
           (dialog (iup:dialog label :title "Hello, World!")))
      (iup:show dialog)
      (iup:main-loop))))
(hello)

Important note for SBCL: we currently must trap division-by-zero errors (see advancement on this issue). So, run snippets like so:

(defun run-gui-function ()
  #-sbcl (gui-function)
  #+sbcl
  (sb-int:with-float-traps-masked
      (:divide-by-zero :invalid)
    (gui-function)))

How to run the main loop

As with all the bindings seen so far, widgets are shown inside a with-iup macro, and with a call to iup:main-loop.

How to create widgets

The constructor function is the name of the widget: iup:label, iup:dialog.

How to display a widget

Be sure to “show” it: (iup:show dialog).

You can group widgets on frames, and stack them vertically or horizontally (with vbox or hbox, see the example below).

To allow a widget to be expanded on window resize, use :expand :yes (or :horizontal and :vertical).

Use also the :alignement properties.

How to get and set a widget’s attributes

Use (iup:attribute widget attribute) to get the attribute’s value, and use setf on it to set it.

Reacting to events

Most widgets take an :action parameter that takes a lambda function with one parameter (the handle).

(iup:button :title "Test &1"
            :expand :yes
            :tip "Callback inline at control creation"
            :action (lambda (handle)
                      (iup:message "title" "button1's action callback")
                      iup:+default+))

Below we create a label and put a button below it. We display a message dialog when we click on the button.

(defun click-button ()
  (iup:with-iup ()
    (let* ((label (iup:label :title (format nil "Hello, World!~%IUP ~A~%~A ~A"
                                            (iup:version)
                                            (lisp-implementation-type)
                                            (lisp-implementation-version))))
           (button (iup:button :title "Click me"
                               :expand :yes
                               :tip "yes, click me"
                               :action (lambda (handle)
                                         (declare (ignorable handle))
                                         (iup:message "title" "button clicked")
                                         iup:+default+)))
           (vbox
            (iup:vbox (list label button)
                      :gap "10"
                      :margin "10x10"
                      :alignment :acenter))
           (dialog (iup:dialog vbox :title "Hello, World!")))
      (iup:show dialog)
      (iup:main-loop))))

#+sbcl
(sb-int:with-float-traps-masked
      (:divide-by-zero :invalid)
    (click-button))

Here’s a similar example to make a counter of clicks. We use a label and its title to hold the count. The title is an integer.

(defun counter ()
  (iup:with-iup ()
    (let* ((counter (iup:label :title 0))
           (label (iup:label :title (format nil "The button was clicked ~a time(s)."
                                            (iup:attribute counter :title))))
           (button (iup:button :title "Click me"
                               :expand :yes
                               :tip "yes, click me"
                               :action (lambda (handle)
                                         (declare (ignorable handle))
                                         (setf (iup:attribute counter :title)
                                               (1+ (iup:attribute counter :title 'number)))
                                         (setf (iup:attribute label :title)
                                               (format nil "The button was clicked ~a times."
                                                       (iup:attribute counter :title)))
                                         iup:+default+)))
           (vbox
            (iup:vbox (list label button)
                      :gap "10"
                      :margin "10x10"
                      :alignment :acenter))
           (dialog (iup:dialog vbox :title "Counter")))
      (iup:show dialog)
      (iup:main-loop))))

(defun run-counter ()
  #-sbcl
  (counter)
  #+sbcl
  (sb-int:with-float-traps-masked
      (:divide-by-zero :invalid)
    (counter)))

Below we create three list widgets with simple and multiple selection, we set their default value (the pre-selected row) and we place them horizontally side by side.

(defun list-test ()
  (iup:with-iup ()
    (let*  ((list-1 (iup:list :tip "List 1"  ;; tooltip
                              ;; multiple selection
                              :multiple :yes
                              :expand :yes))
            (list-2 (iup:list :value 2   ;; default index of the selected row
                              :tip "List 2" :expand :yes))
            (list-3 (iup:list :value 9 :tip "List 3" :expand :yes))
            (frame (iup:frame
                    (iup:hbox
                     (progn
                       ;; populate the lists: display integers.
                       (loop for i from 1 upto 10
                          do (setf (iup:attribute list-1 i)
                                   (format nil "~A" i))
                          do (setf (iup:attribute list-2 i)
                                   (format nil "~A" (+ i 10)))
                          do (setf (iup:attribute list-3 i)
                                   (format nil "~A" (+ i 50))))
                       ;; hbox wants a list of widgets.
                       (list list-1 list-2 list-3)))
                    :title "IUP List"))
            (dialog (iup:dialog frame :menu "menu" :title "List example")))

      (iup:map dialog)
      (iup:show dialog)
      (iup:main-loop))))

(defun run-list-test ()
  #-sbcl (hello)
  #+sbcl
  (sb-int:with-float-traps-masked
      (:divide-by-zero :invalid)
    (list-test)))

Next is a different toolkit very well suited for games and that enables fully interactive development: Nuklear.

GUI Programming in Common Lisp, part 3/5: Gtk3

Fri, 27 Mar 2020 13:05:22 +0100

We continue our tour of GUI toolkits for CL with Gtk+3 and cl-cffi-gtk.

The previosu posts are:

This blog post series was initially written for the Common Lisp Cookbook, you can (and should) read it there:

https://lispcookbook.github.io/cl-cookbook/gui.html

Gtk+3 (cl-cffi-gtk)

Gtk+3 is the primary library used to build GNOME applications. Its (currently most advanced) lisp bindings is cl-cffi-gtk. While primarily created for GNU/Linux, Gtk works fine under macOS and can now also be used on Windows.

Framework written in: C
Portability: GNU/Linux and macOS, also Windows.
Widgets choice: large.
Graphical builder: yes: Glade.
Other features: web browser (WebKitGTK)
Bindings documentation: very good: http://www.crategus.com/books/cl-gtk/gtk-tutorial.html
Bindings stability: stable
Bindings activity: low activity, active development.
Licence: LGPL
Example applications:
- an Atmosphere Calculator, built with Glade.

Getting started

The documentation is exceptionally good, including for beginners.

The library to quickload is cl-cffi-gtk. It is made of numerous ones, that we have to :use for our package.

(ql:quickload :cl-cffi-gtk)

(defpackage :gtk-tutorial
  (:use :gtk :gdk :gdk-pixbuf :gobject
   :glib :gio :pango :cairo :common-lisp))

(in-package :gtk-tutorial)

How to run the main loop

As with the other libraries, everything happens inside the main loop wrapper, here with-main-loop.

How to create a window

(make-instance 'gtk-window :type :toplevel :title "hello" ...).

How to create a widget

All widgets have a corresponding class. We can create them with make-instance 'widget-class, but we preferably use the constructors.

The constructors end with (or contain) “new”:

(gtk-label-new)
(gtk-button-new-with-label "Label")

How to create a layout

(let ((box (make-instance 'gtk-box :orientation :horizontal :spacing 6))) ...)

then pack a widget onto the box:

(gtk-box-pack-start box mybutton-1)

and add the box to the window:

(gtk-container-add window box)

and display them all:

(gtk-widget-show-all window)

Reacting to events

Use g-signal-connect + the concerned widget + the event name (as a string) + a lambda, that takes the widget as argument:

(g-signal-connect window "destroy"
  (lambda (widget)
    (declare (ignore widget))
    (leave-gtk-main)))

Or again:

(g-signal-connect button "clicked"
  (lambda (widget)
    (declare (ignore widget))
    (format t "Button was pressed.~%")))

Full example

(defun hello-world ()
  ;; in the docs, this is example-upgraded-hello-world-2.
  (within-main-loop
    (let ((window (make-instance 'gtk-window
                                 :type :toplevel
                                 :title "Hello Buttons"
                                 :default-width 250
                                 :default-height 75
                                 :border-width 12))
          (box (make-instance 'gtk-box
                              :orientation :horizontal
                              :spacing 6)))
      (g-signal-connect window "destroy"
                        (lambda (widget)
                          (declare (ignore widget))
                          (leave-gtk-main)))
      (let ((button (gtk-button-new-with-label "Button 1")))
        (g-signal-connect button "clicked"
                          (lambda (widget)
                            (declare (ignore widget))
                            (format t "Button 1 was pressed.~%")))
        (gtk-box-pack-start box button))
      (let ((button (gtk-button-new-with-label "Button 2")))
        (g-signal-connect button "clicked"
                          (lambda (widget)
                            (declare (ignore widget))
                            (format t "Button 2 was pressed.~%")))
        (gtk-box-pack-start box button))
      (gtk-container-add window box)
      (gtk-widget-show-all window))))

Next is IUP, a not very famous but really great toolkit!

GUI Programming in Common Lisp, part 2/5: Qt4 with Qtools

Fri, 27 Mar 2020 12:42:59 +0100

Here’s the second part of our exploration of GUI toolkits for Common Lisp.

The first part and introduction is accessible here:

part 1: Ltk

This blog post series was initially written for the Common Lisp Cookbook, you can (and should) read it there:

https://lispcookbook.github.io/cl-cookbook/gui.html

Qt4 (Qtools)

Do we need to present Qt and Qt4? Qt is huge and contains everything and the kitchen sink. Qt not only provides UI widgets, but numerous other layers (networking, D-BUS…).

Qt is free for open-source software, however you’ll want to check the conditions to ship proprietary ones.

The Qtools bindings target Qt4. The Qt5 Lisp bindings are yet to be created.

Framework written in: C++
Framework Portability: multi-platform, Android, embedded systems, WASM.
Bindings Portability: Qtools runs on x86 desktop platforms on Windows, macOS and GNU/Linux.
Widgets choice: large.
Graphical builder: yes.
Other features: Web browser, a lot more.
Bindings documentation: lengthy explanations, a few examples. Prior Qt knowledge is required.
Bindings stability: stable
Bindings activity: active
Qt Licence: both commercial and open source licences.
Example applications:
- https://github.com/Shinmera/qtools/tree/master/examples
- https://github.com/Shirakumo/lionchat
- https://github.com/shinmera/halftone - a simple image viewer

Getting started

(ql:quickload '(:qtools :qtcore :qtgui))

(defpackage #:qtools-test
  (:use #:cl+qt)
  (:export #:main))
(in-package :qtools-test)
(in-readtable :qtools)

We create our main widget that will contain the rest:

(define-widget main-window (QWidget)
  ())

We create an input field and a button inside this main widget:

(define-subwidget (main-window name) (q+:make-qlineedit main-window)
  (setf (q+:placeholder-text name) "Your name please."))

(define-subwidget (main-window go-button) (q+:make-qpushbutton "Go!" main-window))

We stack them horizontally:

(define-subwidget (main-window layout) (q+:make-qhboxlayout main-window)
  (q+:add-widget layout name)
  (q+:add-widget layout go-button))

and we show them:

(with-main-window
  (window 'main-window))

That’s cool, but we don’t react to the click event yet.

Reacting to events

Reacting to events in Qt happens through signals and slots. Slots are functions that receive or “connect to” signals, and signals are event carriers.

Widgets already send their own signals: for example, a button sends a “pressed” event. So, most of the time, we only need to connect to them.

However, had we extra needs, we can create our own set of signals.

Built-in events

We want to connect our go-button to the pressed and return-pressed events and display a message box.

we need to do this inside a define-slot function,
where we establish the connection to those events,
and where we create the message box. We grab the text of the name input field with (q+:text name).

(define-slot (main-window go-button) ()
  (declare (connected go-button (pressed)))
  (declare (connected name (return-pressed)))
  (q+:qmessagebox-information main-window
                              "Greetings"  ;; title
                              (format NIL "Good day to you, ~a!" (q+:text name))))

And voilà. Run it with

(with-main-window (window 'main-window))

Custom events

We’ll implement the same functionality as above, but for demonstration purposes we’ll create our own signal named name-set to throw when the button is clicked.

We start by defining the signal, which happens inside the main-window, and which is of type string:

(define-signal (main-window name-set) (string))

We create a first slot to make our button react to the pressed and return-pressed events. But instead of creating the message box here, as above, we send the name-set signal, with the value of our input field..

(define-slot (main-window go-button) ()
  (declare (connected go-button (pressed)))
  (declare (connected name (return-pressed)))
  (signal! main-window (name-set string) (q+:text name)))

So far, nobody reacts to name-set. We create a second slot that connects to it, and displays our message. Here again, we precise the parameter type.

(define-slot (main-window name-set) ((new-name string))
  (declare (connected main-window (name-set string)))
  (q+:qmessagebox-information main-window "Greetings" (format NIL "Good day to you, ~a!" new-name)))

and run it:

(with-main-window (window 'main-window))

Building and deployment

It is possible to build a binary and bundle it together with all the necessary shared libraries.

Please read https://github.com/Shinmera/qtools#deployment.

You might also like this Travis CI script to build a self-contained binary for the three OSes.

Next, we’ll have a look at Gtk+3.

GUI Programming in Common Lisp, part 1/5: Tk

Fri, 27 Mar 2020 12:20:36 +0100

Lisp has a long and rich history and so does the development of Graphical User Interfaces in Lisp. In fact, the first GUI builder was written in Lisp (and sold to Apple. It is now Interface Builder).

Lisp is also famous and unrivaled for its interactive development capabilities, a feature even more worth having to develop GUI applications. Can you imagine compiling one function and seeing your GUI update instantly? We can do this with many GUI frameworks today, even though the details differ from one to another.

Finally, a key part in building software is how to build it and ship it to users. Here also, we can build self-contained binaries, for the three main operating systems, that users can run with a double click.

We aim here to give you the relevant information to help you choose the right GUI framework and to put you on tracks. Don’t hesitate to contribute, to send more examples and to furnish the upstream documentations.

This blog post series was initially written for the Common Lisp Cookbook, you can (and should) read it there:

https://lispcookbook.github.io/cl-cookbook/gui.html

Introduction

In this article series, we’ll present the following GUI toolkits:

In addition, you might want to have a look to:

the CAPI toolkit (Common Application Programming Interface), which is proprietary and made by LispWorks. It is a complete and cross-platform toolkit (Windows, Gtk+, Cocoa), very praised by its users. LispWorks also has iOS and Android runtimes. Example software built with CAPI include Opusmodus or again ScoreCloud. It is possible to try it with the LispWorks free demo.
CocoaInterface, a Cocoa interface for Clozure Common Lisp. Build Cocoa user interface windows dynamically using Lisp code and bypass the typical Xcode processes.
McCLIM, a toolkit in 100% Common Lisp.
Alloy, another very new toolkit in 100% Common Lisp, used for example in the Kandria game.
nodgui, a fork of Ltk, with syntax sugar and additional widgets.
eql, eql5, eql5-android, embedded Qt4 and Qt5 Lisp, embedded in ECL, embeddable in Qt. Port of EQL5 to the Android platform.
this demo using Java Swing from ABCL
and, last but not least, Ceramic, to ship a cross-platform web app with Electron.

as well as the other ones listed on awesome-cl#gui and Cliki.

Tk (Ltk)

Tk (or Tcl/Tk, where Tcl is the programming language) has the infamous reputation of having an outdated look. This is not (so) true anymore since its version 8 of 1997 (!). It is probably better than you think:

Tk doesn’t have a great choice of widgets, but it has a useful canvas, and it has a couple of unique features: we can develop a graphical interface fully interactively and we can run the GUI remotely from the core app.

So, Tk isn’t fancy, but it is an used and proven GUI toolkit (and programming language) still used in the industry. It can be a great choice to quickly create simple GUIs, to leverage its ease of deployment, or when stability is required.

The Lisp binding is Ltk.

Written in: Tcl
Portability: cross-platform (Windows, macOS, Linux).
Widgets: this is not the fort of Tk. It has a small set of default widgets, and misses important ones, for example a calendar. We can find some in extensions (such as in Nodgui), but they don’t feel native, at all.
Interactive development: very much.
Graphical builder: no
Other features:
- remote execution: the connection between Lisp and Tcl/Tk is done via a stream. It is thus possible to run the Lisp program on one computer, and to display the GUI on another one. The only thing required on the client computer is tcl/tk installed and the remote.tcl script. See Ltk-remote.
Bindings documentation: short but complete. Nodgui too.
Bindings stability: very stable
Bindings activity: low to non-existent.
Licence: Tcl/Tk is BSD-style, Ltk is LGPL.
Example applications:
- Fulci - a program to organize your movie collections.
- Ltk small games - snake and tic-tac-toe.
- cl-torrents - searching torrents on popular trackers. CLI, readline and a simple Tk GUI.

List of widgets

(please don’t suppose the list exhaustive)

Button Canvas Check-button Entry Frame Label Labelframe Listbox
Menu Menubutton Message
Paned-window
Radio-button Scale
Scrollbar Spinbox Text
Toplevel Widget Canvas

Ltk-megawidgets:
    progress
    history-entry
    menu-entry

Nodgui adds:

treelist tooltip searchable-listbox date-picker calendar autocomplete-listbox
password-entry progress-bar-star notify-window
dot-plot bar-chart equalizer-bar
swap-list

Getting started

Ltk is quick and easy to grasp.

(ql:quickload :ltk)
(in-package :ltk-user)

How to create widgets

All widgets are created with a regular make-instance and the widget name:

(make-instance 'button)
(make-instance 'treeview)

This makes Ltk explorable with the default symbol completion.

How to start the main loop

As with most bindings, the GUI-related code must be started inside a macro that handles the main loop, here with-ltk:

(with-ltk ()
  (let ((frame (make-instance 'frame)))
    …))

How to display widgets

After we created some widgets, we must place them on the layout. There are a few Tk systems for that, but the most recent one and the one we should start with is the grid. grid is a function that takes as arguments the widget, its column, its row, and a few optional parameters.

As with any Lisp code in a regular environment, the functions’ signatures are indicated by the editor. It makes Ltk explorable.

Here’s how to display a button:

(with-ltk ()
  (let ((button (make-instance 'button :text "hello")))
    (grid button 0 0)))

That’s all there is to it.

Reacting to events

Many widgets have a :command argument that accept a lambda which is executed when the widget’s event is started. In the case of a button, that will be on a click:

(make-instance 'button
  :text "Hello"
  :command (lambda ()
             (format t "clicked")))

Interactive development

When we start the Tk process in the background with (start-wish), we can create widgets and place them on the grid interactively.

See the documentation.

Once we’re done, we can (exit-wish).

Nodgui

To try the Nodgui demo, do:

(ql:quickload :nodgui)
(nodgui.demo:demo)

Next, we’ll have a look at a very different beast, Qt4, with Qtools.

Literate Programming in Lisp With Erudite

Sat, 21 Mar 2020 11:01:57 +0100

Erudite is a Common Lisp library to write literate programs. The latest release (march, 2020) brings cool new features, amongst which the ability to capture and print code output.

This page was created with Erudite. You can follow along with its source here. Blogging about a programming language in the language itself is pretty awesome and convenient (no more copy-pasting of code snippets and manual adjustements of the indentation, yay!). It brings us closer to an interactive notebook, even if it isn’t the first goal.

Basic usage

You write a lisp program, as usual. There is no extra step to produce the program sources, since we are inside the sources. This is different than the Org-mode approach for example.

The comments will be the documentation. Comments inside a function are also extracted and cut the function in two.

Erudite can export to Latex, RestructuredText, Markdown, HTML… and actually to and from any format by using a “pass-through” directive.

Top level comments are shown like this. Here’s code:

(defun fibonacci (n &aux (f0 0) (f1 1))
  "docstring"
  (case n
    (0 f0)
    (1 f1)

this is an inline comment (there might be settings to control how it is rendered)

    (t (loop for n from 2 to n
          for a = f0 then b and b = f1 then result
          for result = (+ a b)
          finally (return result)))))

Erudite defines directives to play with the output, such as ignore and eval. Note that directives start with a @ sign, which I cannot use here.

With ignore, we can write lisp code but hide it from the output. And with eval, Erudite connects to a Swank server, captures and prints the output.

There’s also the handy code, to write a snippet inside comments (so it is not part of the Lisp source) and make it appear in the generated document.

For the text markup, we can use Erudite’s syntax (“link”, “section”, “subsection”, “emph”…), or the markup of the output file format.

Evaluating code

With the latest Erudite, we can evaluate code. Note that it’s a work in progress.

The code snippet must be inside the comments too.

Here I call Fibonacci defined above: Fibonacci of 10 is…
55

You might need to create a Swank server first with


(swank:create-server :dont-close t)

and tell Erudite its port if it isn’t 4005.


(setf erudite::*swank-port* 4005)

Rendering the document

Call Erudite like so:


(erudite:erudite #p"literal.md" "literal-erudite.lisp" :output-type :markdown)

We can also use a binary from the shell.

Live rendering

I don’t want to re-run this command everytime I want to see the generated document. I use this snippet to automatically export my document when I save the Lisp source:


(ql:quickload :cl-inotify)

(bt:make-thread
 (lambda ()
   (cl-inotify:with-inotify (inotify t ("literal-erudite.lisp" :close-write))
     (cl-inotify:do-events (event inotify :blocking-p t)
       (format t "~a~&" event)
       (erudite:erudite #p"literal.md" "literal-erudite.lisp" :output-type :markdown))))
 :name "inotify-erudite")

And then I make the markdown file to be live-rendered in the browser. I used impatient-mode for Emacs (see Wikemacs) with the help of M-x auto-revert-mode.

Kuddos to Mariano Montone!

Common Lisp Macros By Example Tutorial

Mon, 02 Mar 2020 21:12:53 +0100

I have recently edited and somewhat expanded the macros page on the Common Lisp Cookbook. I find it may more legible and reader friendly, so I reproduce it below (however, I cut two parts so than you get the essential).

You’d better read it on the Cookbook: https://lispcookbook.github.io/cl-cookbook/macros.html

The word macro is used generally in computer science to mean a syntactic extension to a programming language. (Note: The name comes from the word “macro-instruction,” which was a useful feature of many second-generation assembly languages. A macro-instruction looked like a single instruction, but expanded into a sequence of actual instructions. The basic idea has since been used many times, notably in the C preprocessor. The name “macro” is perhaps not ideal, since it connotes nothing relevant to what it names, but we’re stuck with it.) Although many languages have a macro facility, none of them are as powerful as Lisp’s. The basic mechanism of Lisp macros is simple, but has subtle complexities, so learning your way around it takes a bit of practice.

How Macros Work

A macro is an ordinary piece of Lisp code that operates on another piece of putative Lisp code, translating it into (a version closer to) executable Lisp. That may sound a bit complicated, so let’s give a simple example. Suppose you want a version of setq that sets two variables to the same value. So if you write

(setq2 x y (+ z 3))

when z=8 then both x and y are set to 11. (I can’t think of any use for this, but it’s just an example.)

It should be obvious that we can’t define setq2 as a function. If x=50 and y=-5, this function would receive the values 50, -5, and 11; it would have no knowledge of what variables were supposed to be set. What we really want to say is, When you (the Lisp system) see:

(setq2 v1 v2 e)

then treat it as equivalent to:

(progn
  (setq v1 e)
  (setq v2 e))

Actually, this isn’t quite right, but it will do for now. A macro allows us to do precisely this, by specifying a program for transforming the input pattern (setq2 v₁ v₂ e) into the output pattern (progn ...).

Quote

Here’s how we could define the setq2 macro:

(defmacro setq2 (v1 v2 e)
  (list 'progn (list 'setq v1 e) (list 'setq v2 e)))

It takes as parameters two variables and one expression.

Then it returns a piece of code. In Lisp, because code is represented as lists, we can simply return a list that represents code.

We also use the quote: each quoted symbol evaluates to itself, aka it is returned as is:

(quote foo bar baz) returns (foo bar baz)
the quote character, ', is a shortcut for quote, a special operator (not a function nor a macro, but one of a few special operators forming the core of Lisp).
so, 'foo evaluates to foo.

So, our macro retourns the following bits:

the symbol progn,
a second list, that contains
- the symbol setq
- the variable v1: note that the variable is not evaluated inside the macro!
- the expression e: it is not evaluated either!
a second list, with v2.

We can use it like this:

(defparameter v1 1)
(defparameter v2 2)
(setq2 v1 v2 3)
;; 3

We can check, v1 and v2 were set to 3.

Macroexpand

We must start writing a macro when we know what code we want to generate. Once we’ve begun writing one, it becomes very useful to check effectively what code does the macro generate. The function for that is macroexpand. It is a function, and we give it some code, as a list (so, we quote the code snippet we give it):

(macroexpand '(setq2 v1 v2 3))
;; (PROGN (SETQ V1 3) (SETQ V2 3))
;; T

Yay, our macro expands to the code we wanted!

More interestingly:

(macroexpand '(setq2 v1 v2 (+ z 3)))
;; (PROGN (SETQ V1 (+ z 3)) (SETQ V2 (+ z 3)))
;; T

We can confirm that our expression e, here (+ z 3), was not evaluated. We will see how to control the evaluation of arguments with the comma: ,.

Note: with Slime, you can call macroexpand by putting the cursor at the left of the parenthesis of the s-expr to expand and call the functionM-x slime-macroexpand-[1,all], or C-c M-m:

[|](setq2 v1 v2 3)
;^ cursor
; C-c M-m
; =>
; (PROGN (SETQ V1 3) (SETQ V2 3))

Macros VS functions

Our macro is very close to the following function definition:

(defun setq2-function (v1 v2 e)
  (list 'progn (list 'setq v1 e) (list 'setq v2 e)))

If we evaluated (setq2-function 'x 'y '(+ z 3)) (note that each argument is quoted, so it isn’t evaluated when we call the function), we would get

(progn (setq x (+ z 3)) (setq y (+ z 3)))

This is a perfectly ordinary Lisp computation, whose sole point of interest is that its output is a piece of executable Lisp code. What defmacro does is create this function implicitly and make sure that whenever an expression of the form (setq2 x y (+ z 3)) is seen, setq2-function is called with the pieces of the form as arguments, namely x, y, and (+ z 3). The resulting piece of code then replaces the call to setq2, and execution resumes as if the new piece of code had occurred in the first place. The macro form is said to expand into the new piece of code.

Evaluation context

This is all there is to it, except, of course, for the myriad subtle consequences. The main consequence is that run time for the setq2 macro is compile time for its context. That is, suppose the Lisp system is compiling a function, and midway through it finds the expression (setq2 x y (+ z 3)). The job of the compiler is, of course, to translate source code into something executable, such as machine language or perhaps byte code. Hence it doesn’t execute the source code, but operates on it in various mysterious ways. However, once the compiler sees the setq2 expression, it must suddenly switch to executing the body of the setq2 macro. As I said, this is an ordinary piece of Lisp code, which can in principle do anything any other piece of Lisp code can do. That means that when the compiler is running, the entire Lisp (run-time) system must be present.

We’ll stress this once more: at compile-time, you have the full language at your disposal.

Novices often make the following sort of mistake. Suppose that the setq2 macro needs to do some complex transformation on its e argument before plugging it into the result. Suppose this transformation can be written as a Lisp procedure some-computation. The novice will often write:

(defmacro setq2 (v1 v2 e)
  (let ((e1 (some-computation e)))
    (list 'progn (list 'setq v1 e1) (list 'setq v2 e1))))

(defmacro some-computation (exp) ...) ;; _Wrong!_

The mistake is to suppose that once a macro is called, the Lisp system enters a “macro world,” so naturally everything in that world must be defined using defmacro. This is the wrong picture. The right picture is that defmacro enables a step into the ordinary Lisp world, but in which the principal object of manipulation is Lisp code. Once that step is taken, one uses ordinary Lisp function definitions:

(defmacro setq2 (v1 v2 e)
  (let ((e1 (some-computation e)))
    (list 'progn (list 'setq v1 e1) (list 'setq v2 e1))))

(defun some-computation (exp) ...) ;; _Right!_

One possible explanation for this mistake may be that in other languages, such as C, invoking a preprocessor macro does get you into a different world; you can’t run an arbitrary C program. It might be worth pausing to think about what it might mean to be able to.

Another subtle consequence is that we must spell out how the arguments to the macro get distributed to the hypothetical behind-the-scenes function (called setq2-function in my example). In most cases, it is easy to do so: In defining a macro, we use all the usual lambda-list syntax, such as &optional, &rest, &key, but what gets bound to the formal parameters are pieces of the macro form, not their values (which are mostly unknown, this being compile time for the macro form). So if we defined a macro thus:

(defmacro foo (x &optional y &key (cxt 'null)) ...)

then

If we call it thus …	The parameters’ values are …
`(foo a)`	`x=a`, `y=nil`, `cxt=null`
`(foo (+ a 1) (- y 1))`	`x=(+ a 1)`, `y=(- y 1)`, `cxt=null`
`(foo a b :cxt (zap zip))`	`x=a`, `y=b`, `cxt=(zap zip)`

Note that the values of the variables are the actual expressions (+ a 1) and (zap zip). There is no requirement that these expressions’ values be known, or even that they have values. The macro can do anything it likes with them. For instance, here’s an even more useless variant of setq: (setq-reversible e₁ e₂ d) behaves like (setq e₁ e₂) if d=:normal, and behaves like (setq e₂ e₁) if d=:backward. It could be defined thus:

(defmacro setq-reversible (e1 e2 direction)
  (case direction
    (:normal (list 'setq e1 e2))
    (:backward (list 'setq e2 e1))
    (t (error "Unknown direction: ~a" direction))))

Here’s how it expands:

(macroexpand '(setq-reversible x y :normal))
(SETQ X Y)
T
(macroexpand '(setq-reversible x y :backward))
(SETQ Y X)
T

And with a wrong direction:

(macroexpand '(setq-reversible x y :other-way-around))

We get an error and are prompted into the debugger!

We’ll see the backquote and comma mechanism in the next section, but here’s a fix:

(defmacro setq-reversible (v1 v2 direction)
  (case direction
    (:normal (list 'setq v1 v2))
    (:backward (list 'setq v2 v1))
    (t `(error "Unknown direction: ~a" ,direction))))
    ;; ^^ backquote                    ^^ comma: get the value inside the backquote.

(macroexpand '(SETQ-REVERSIBLE v1 v2 :other-way-around))
;; (ERROR "Unknown direction: ~a" :OTHER-WAY-AROUND)
;; T

Now when we call (setq-reversible v1 v2 :other-way-around) we still get the error and the debugger, but at least not when using macroexpand.

Backquote and comma

Before taking another step, we need to introduce a piece of Lisp notation that is indispensable to defining macros, even though technically it is quite independent of macros. This is the backquote facility. As we saw above, the main job of a macro, when all is said and done, is to define a piece of Lisp code, and that means evaluating expressions such as (list 'prog (list 'setq ...) ...). As these expressions grow in complexity, it becomes hard to read them and write them. What we find ourselves wanting is a notation that provides the skeleton of an expression, with some of the pieces filled in with new expressions. That’s what backquote provides. Instead of the the list expression given above, one writes

  `(progn (setq ,v1 ,e) (setq ,v2 ,e))
;;^ backquote   ^   ^         ^   ^ commas

The backquote (`) character signals that in the expression that follows, every subexpression not preceded by a comma is to be quoted, and every subexpression preceded by a comma is to be evaluated.

You can think of it, and use it, as data interpolation:

`(v1 = ,v1) ;; => (V1 = 3)

That’s mostly all there is to backquote. There are just two extra items to point out.

Comma-splice ,@

First, if you write “,@e” instead of “,e” then the value of e is spliced (or “joined”, “combined”, “interleaved”) into the result. So if v equals (oh boy), then

(zap ,@v ,v)

evaluates to

(zap oh boy (oh boy))
;;   ^^^^^ elements of v (two elements), spliced.
;;          ^^ v itself (a list)

The second occurrence of v is replaced by its value. The first is replaced by the elements of its value. If v had had value (), it would have disappeared entirely: the value of (zap ,@v ,v) would have been (zap ()), which is the same as (zap nil).

Quote-comma ‘,

When we are inside a backquote context and we want to print an expression literally, we have no choice but to use the combination of quote and comma:

(defmacro explain-exp (exp)
  `(format t "~S = ~S" ',exp ,exp))
  ;;                   ^^

(explain-exp (+ 2 3))
;; (+ 2 3) = 5

See by yourself:

;; Defmacro with no quote at all:
(defmacro explain-exp (exp)
  (format t "~a = ~a" exp exp))
(explain-exp v1)
;; V1 = V1

;; OK, with a backquote and a comma to get the value of exp:
(defmacro explain-exp (exp)
  ;; WRONG example
  `(format t "~a = ~a" exp ,exp))
(explain-exp v1)
;; => error: The variable EXP is unbound.

;; We then must use quote-comma:
(defmacro explain-exp (exp)
  `(format t "~a = ~a" ',exp ,exp))
(explain-exp (+ 1 2))
;; (+ 1 2) = 3

Nested backquotes

Second, one might wonder what happens if a backquote expression occurs inside another backquote. The answer is that the backquote becomes essentially unreadable and unwriteable; using nested backquote is usually a tedious debugging exercise. The reason, in my not-so-humble opinion, is that backquote is defined wrong. A comma pairs up with the innermost backquote when the default should be that it pairs up with the outermost. But this is not the place for a rant; consult your favorite Lisp reference for the exact behavior of nested backquote plus some examples.

Building lists with backquote

[…]

Getting Macros Right

I said in the first section that my definition of setq2 wasn’t quite right, and now it’s time to fix it.

Suppose we write (setq2 x y (+ x 2)), when x=8. Then according to the definition given above, this form will expand into

(progn
  (setq x (+ x 2))
  (setq y (+ x 2)))

so that x will have value 10 and y will have value 12. Indeed, here’s its macroexpansion:

(macroexpand '(setq2 x y (+ x 2)))
;;(PROGN (SETQ X (+ X 2)) (SETQ Y (+ X 2)))

Chances are that isn’t what the macro is expected to do (although you never know). Another problematic case is (setq2 x y (pop l)), which causes l to be popped twice; again, probably not right.

The solution is to evaluate the expression e just once, save it in a temporary variable, and then set v1 and v2 to it.

Gensym

To make temporary variables, we use the gensym function, which returns a fresh variable guaranteed to appear nowhere else. Here is what the macro should look like:

(defmacro setq2 (v1 v2 e)
  (let ((tempvar (gensym)))
    `(let ((,tempvar ,e))
       (progn (setq ,v1 ,tempvar)
              (setq ,v2 ,tempvar)))))

Now (setq2 x y (+ x 2)) expands to

(let ((#:g2003 (+ x 2)))
  (progn (setq x #:g2003) (setq y #:g2003)))

Here gensym has returned the symbol #:g2003, which prints in this funny way because it won’t be recognized by the reader. (Nor is there any need for the reader to recognize it, since it exists only long enough for the code that contains it to be compiled.)

Exercise: Verify that this new version works correctly for the case (setq2 x y (pop l1)).

Exercise: Try writing the new version of the macro without using backquote. If you can’t do it, you have done the exercise correctly, and learned what backquote is for!

The moral of this section is to think carefully about which expressions in a macro get evaluated and when. Be on the lookout for situations where the same expression gets plugged into the output twice (as e was in my original macro design). For complex macros, watch out for cases where the order that expressions are evaluated differs from the order in which they are written. This is sure to trip up some user of the macro - even if you are the only user.

What Macros are For

[…]

Yes Google Uses and Hacks on Common Lisp

Mon, 02 Mar 2020 20:47:32 +0100

ITA Software, owned by Google, the airfare search and pricing system that is still used by companies such as Kayak.com or Orbitz, is a well-known example of a successful industrial and large Common Lisp software.

We’re legitimate to wonder if they still run it (they do), if Google develops more CL software (I don’t know), or if they put resources to improve a CL implementation: they do.

According to https://mstmetent.blogspot.com/2020/01/sbcl20-in-vienna-last-month-i-attended.html:

Doug Katzman talked about his work at Google getting SBCL to work with Unix better. For those of you who don’t know, he’s done a lot of work on SBCL over the past couple of years, not only adding a lot of new features to the GC and making it play better with applications which have alien parts to them, but also has done a tremendous amount of cleanup on the internals and has helped SBCL become even more Sanely Bootstrappable. That’s a topic for another time, and I hope Doug or Christophe will have the time to write up about the recent improvements to the process, since it really is quite interesting.

Anyway, what Doug talked about was his work on making SBCL more amenable to external debugging tools, such as gdb and external profilers. It seems like they interface with aliens a lot from Lisp at Google, so it’s nice to have backtraces from alien tools understand Lisp. It turns out a lot of prerequisite work was needed to make SBCL play nice like this, including implementing a non-moving GC runtime, so that Lisp objects and especially Lisp code (which are normally dynamic space objects and move around just like everything else) can’t evade the aliens and will always have known locations.

So now that’s in the wild, Common Lisp can go trendy again: Google uses and dedicates resources for Common Lisp!

Python VS Common Lisp Workflow and Ecosystem

Wed, 06 Nov 2019 15:21:53 +0100

please see Python VS Common Lisp (it’s a static page you can find in the menu).

Snippet: Manipulating Rows and Columns With ClAWK

Mon, 04 Nov 2019 18:40:52 +0100

I just discovered clawk, that seems to originate from lispbuilder-clawk. Its last commit dates from 2011, typical from Lisp and that’s OK, libraries have the right to be done, it has no useful README nor documentation, but we can see its use in the tests, and the library is easily discoverable.

This library seems perfect to manipulate data in rows and columns.

Let’s have a quick look with this dummy txt file:

1 Alice 40 40
2 Bob 39 50

I had a conflict when use-ing clawk, which I resolved by not accepting the change in the debugger.

We parse all lines, give a name to the space-delimited fields, and print them back:

(for-file-lines ("test.txt")
    (with-fields ((a b c d))
        ($print a b c d)))

1 Alice 40 40
2 Bob 39 50
NIL

Let’s multiply the two last fields. If we use the regular * operator, we get a type error because fields are extracted as strings by default. We then use $*:

(for-file-lines ("test.txt")
    (with-fields ((id name payrate hrsworked))
        (declare (ignore id))
        ($print name ($* payrate hrsworked))))
Alice 1600
Bob 1950
NIL

We can change the field separator with a string or a regexp, with the no surprising clawk:*fs* variable (FS in awk):

(for-file-lines ("test.txt")
    (let ((clawk:*fs* "-"))
        (with-fields ((a b c))
        ($print a))))

And… that’s all folks. Another tool to keep in our toolbelt.

more about awk: https://www.gnu.org/software/gawk/manual/html_node/

Python VS Common Lisp, workflow and ecosystem

Wed, 30 Oct 2019 07:51:49 +0100

I learned Java and C at school, I learned Python by myself and it was a relief. After 8 years working and doing side projects in Python and JavaScript (mostly web dev, Django/Flask/AngularJS/Vuejs), I am not satisfied anymore by the overall experience so I’m making Common Lisp my language of choice.

I am not here to compare languages themselves, but their inherent workflow and their ecosystem. This is the article I wish I had read earlier, when I was interested in Lisp but was a bit puzzled, because the Lisp way always seemed different, and I couldn’t find many voices to explain it. The Python way may not be the most practical or effective, Common Lisp might not be a dead language. I find many “workflow fixes”, overall improvements and hackerish possibilities on the CL side, even if sometimes the Python tooling is superior.

Let’s dive in.

and thanks to the proofreaders.

Table of Contents

Development process

Interactivity

In Python, we typically restart everything at each code change, we use breakpoints: this takes some time, I find it too repetitive and boring, it requires to re-manipulate data to re-reach the state we were at to analyze and debug our program. We might figure out a non-standard, more interactive way, but still: a web server needs to restart, object instances don’t get updated after a class definition. We can get a prompt on an error (-m pdb), some tools include it (Werkzeug): a sign that it is a good thing to have. Unfortunately, it is not built-in, as in CL.

In Common Lisp, everything is much more interactive in the REPL. Even developing web apps. On an error, we get an interactive debugger with the stacktrace in our editor, we press v and voilà, we are at the problematic line. We can of course catch errors to avoid the debugger, or disable it with global settings. We can resume the program execution from any stackframe. No process needs to restart. The variables that we define on the REPL stay here. If we change a class definition (say, we remove a field), existing instances get (lazily) updated.

The Lisp REPL is part of the development process, it is not only used for exploration and debugging. It’s fun, it’s a productive boost, and it allows to catch errors earlier, both because we try functions earlier, and because we get type warnings when we compile the file or the current function (yes, we can compile a single function).

Now, the cost is that one must learn to play with this live data. We might come to a state that doesn’t reflect the code anymore, so we’ll write our own “reset” functions or just restart the lisp image.

Here’s a video where the developer defines a dummy interface, makes it fail, develops it, and tests it, all quickly by interacting with the REPL.

In this video I show how we can interactively fix a function and resume the program from this precise point in the call stack, without re-starting the whole, time-consuming operation.

Editing code

Python: we edit code line by line, paragraph by paragraph. We can try out half-backed editor plugins to edit code by semantic units. Sometimes we must even pay attention to add a couple whitespace there, remove one there. We are far from the immediate interactive feedback of the hacker’s vision “Inventing on Principles”.

Common Lisp: we edit code by semantic units. I love emacs’ lispy mode, which is weird at first of course, but so convenient. We can navigate to expressions back and forth, we can delete a whole “if” expression with a keypress, indentation is automatic, etc. There are other emacs plugins. Parinfer is appreciated in other editors too.

Actually, we edit code by parenthesis units, which doesn’t carry as much meaning as an Abstract Syntax Tree. For a real AST, we’d need a code walker (like Concrete-Syntax-Tree). But since Lisp’s syntax is based on parenthesis, in practice the experience is similar.

I had a try on writing a little plugin to help editing Python code by manipulating the AST (red4e). We first need an AST parser. There was a couple for Python 2, another one for Python 3 without type annotations, eventually one emerged a couple years later: these are signs of an unstable language and ecosystem, and it is more work required by the developer. I went the simple way by calling each function into a new Python process, which is of course too slow. traad is a better project, it can do much more but still, it’s difficult to answer questions like cross-referencing: “who calls this function” or “who does this function call”, which are built-in in SLIME. SLIME is like the Language Server Protocol for Common Lisp in Emacs, its backend Swank being editor-agnostic.

Maybe other editors and proprietary ones come with a better experience, at the cost of freedom, money, configuration time and memory and CPU resources. If I have the choice, I prefer to not go this route, and choose a better platform from the start.

Traad is built around a client-http server approach, this is the idea behind LSP… this reminds me of the architecture of SLIME! It has a backend, Swank, and a client (SLIME for Emacs, SLIMA for Atom,…). It thus has a modern architecture since its inception :) It is moreover based on a stable language whose syntax can not rot and has decades of development behind it, so we can be confident about the tool. Saying this because it’s hard to grasp what SLIME is at the beginning.

SLIME itself is tied to Emacs, and thus a newcomer can find the UI unpractical. Swank though can be used outside of Emacs, and it is for example for Atom’s SLIMA, which now has all the most important SLIME features: REPL, integrated debugger, jump to definition, autocompletion, interactive object inspection, and more.

more: https://lispcookbook.github.io/cl-cookbook/editor-support.html (Eclipse, Lem, Jupyter Notebook,…)

Running, testing programs

Python: the default workflow is to run commands in the terminal. Scroll, read the output, copy-paste manually (or use the non-UX-optimal termux or a terminal inside emacs), go back to your editor. Type commands without completion, type the whole path to a single unit test (pytest path/to/test.py::foo), or configure your editor and find a good plugin that is compatible with your test runner (I can’t use the excellent nose-mode :( ).

Common Lisp: the default workflow is to do everything interactively into the REPL, but some people still use a write-compile-run approach. Consequently there is built-in completion for everything. We don’t have to use the shell (except from once in a while to run global tests or build the system) and that’s a good thing. There is an interactive debugger. We can interactively fix and re-run code and tests.

Here’s a quick demo on how to interactively fix failing tests:

Running and debugging on a remote server: in Python, we usually rsync sources and run tests manually, or start vim/emacs under tmux on the server. We have to kill the app to reload it. In Common Lisp, we can connect to the running, remote instance, write changes from the comfort of our editor locally, hit C-c C-c on a function to compile it and see changes on the remote image. CL has more hackerish capacities here, no doubt, and I find it attractive :)

more information on (remote) debugging: https://lispcookbook.github.io/cl-cookbook/debugging.html
a demo with a web app: https://lisp-journey.gitlab.io/blog/i-realized-that-to-live-reload-my-web-app-is-easy-and-convenient/
watch Baggers working with OpenGL: https://www.youtube.com/watch?v=a2tTpjGOhjw&index=20&list=RDxzTH_ZqaFKI
a Minecraft game engine that you can change while playing: https://github.com/gmasching/sucle

Typing

Python: we catch a lot of type errors in production, and/or we have to write a lot more unit tests. Hope we agree on this.

Now we can improve the situation somehow with type annotations, however it has the cons of being an after-thought: it is not stable (differences between Python versions), not well integrated (we have to run another command, choose between mypy, the new typing module, pyre), it is not interactive, we need to configure our IDE, it adds a start-up penalty (which might or might not be important).

In Common Lisp, particularly with SBCL, we get a lot of type errors or warnings at compile time. We can compile a single function, and thus have an immediate feedback. We’re closer (not there, just closer) to the “if it compiles, it works” situation (we know it runs, since we constantly compile and try the functions). We can also create our compound types and add type declarations to variables and functions. It’s great, though it doesn’t do as much static checks as a real typed language.

Adding type declarations in well chosen places such as inner loops also allows to gradually speed up the program where needed.

https://lispcookbook.github.io/cl-cookbook/type.html
Compile-time type checking in the programmable programming language Lisp
will ML embedded into CL help even more ?

Refactoring

Python: we can’t refactor code as we want. Decorators, context managers: they have an interface and they are limited to what they offer. You can’t do things a bit differently, you must comply to the interface. That might be a feature, but I prefer not being restricted. In my experience, this leads to code repetition whereas in CL, we can refactor how we want, and we get a cleaner code.

Common Lisp: there are similar patterns than in Python, but we can escape them. We can use macros, be concise and do what we want. We can have the decorator syntax with the cl-annot library, and any other by writing our reader macros (they can bring triply-quoted docstrings, string interpolation, infix notation, C syntax…). It’s not only macros though. The polymorphism of the object system (or generic dispatch) helps, and Lisp’s “moldability” in a whole allows us to refactor code exactly how we want, to build a “Domain Specific Language” to express what we want. Other language features than macros help here, like closures or multiple values (which are different, and safer for refactoring, than returning a tuple).

Now, speaking about refactoring tools, they are better Python side. I don’t know of a Lisp tool that allows to change all the code-base according to the AST, maybe in a proprietary editor. There are utilities to make local transformations, like “extract this expression into a let variable at the top of the function”, “transform a function to a lambda equivalent” or the contrary, etc.

(edit January, 2020: the language-agnostic tool Comby will be useful here. I used it for syntactic manipulation, for example to replace a (if (…) (progn …)) by a (when (…) …)). See Colisper (a POC).

Libraries

Library management

pip: use virtual environments (virtualenv, virtualenvwrapper, tox, anaconda,… or install per-user), pin dependencies (pip-tools, pipenv, poetry, pyupdate,…). Debug problems due to a third party library that didn’t pin its dependencies strictly enough (happens at the wrong moment).

quicklisp: think of it like Debian’s apt, shipping releases that work together (that load together), and that we upgrade together, when we want. If needed, we can still clone projects into ~/quicklisp/local-projects/ for a system-wide installation, or have project-local dependencies with Qlot. Quicklisp is very slick. Libraries are installed at runtime, during our REPL session. We don’t have to reload the Lisp process.

We are not even limited to Quicklisp any more (it can be limiting because of its one month release cycle). The Ultralisp distribution builds every 5 minutes. clpm is a package manager with a traditional approach. One can publish his own Quicklisp distribution, to provide a set of packages that are known to work together.

State of the libraries

CL might have more libraries than you think, see the Awesome CL list, http://quickdocs.org/ or do a quick search on the net. I know I am constantly surprised.

But sure, the Python ecosystem is huge. A few remarks on the differences:

Quicklisp has around 1500 packages, PyPI over than 170 000. It’s hard to imagine that there are a hundred times more useful libraries :D Even in CL we have duplication of libraries with a dozen of test frameworks.
Quicklisp is a curated distribution, PyPI is not. That means that libraries that don’t compile anymore are rejected (after a notice to the maintainers), and that orphan projects’ URL can be updated to point to a community maintained one.
Anybody can easily publish a library to PyPI on its own. Less so with Quicklisp, one must open an issue (Ultralisp doesn’t have this limitation).
numcl is a Numpy clone.
if needed, you can use py4cl and more to interface with Python.

An important remark, is that Common Lisp is a stable language, and that the libraries play this game (I saw a deprecation feature staying for 12 years). We can still run code that was written in the early 90’s.

Lisp’s simpler, non-rotting syntax plays a good role on stability. Caution: that doesn’t mean the implementations don’t evolve, quite the contrary.

In his appreciated article A Road to Common Lisp, the author writes:

as you learn Common Lisp and look for libraries, try to suppress the voice in the back of your head that says “This project was last updated six years ago? That’s probably abandoned and broken.” The stability of Common Lisp means that sometimes libraries can just be done, not abandoned, so don’t dismiss them out of hand.

hacker news comments

Templates

Ever had headaches with Jinja ? Ever fought against Django templates ? Ever abandoned to clean up a big mess of html and templating code ? Used Jinja macros to factorize code ? Maybe you turned to the good looking Jade (Pug). So you read another documentation, you install tools to integrate it into your project. And now damn, no more cross-files macros. You edit blocks of code whitespace by whitespace. And in the end, your html may still not be valid…

You might use Mako templates, but there’s something you can’t do.

In CL, we can also use a Django-like templating engine, Djula templates (despite its modest number of stars, it is one of the most downloaded projects on Quicklisp). The Mako equivalent would be Ten. However, we can alternatively just use plain old Lisp, for example with Spinneret. As a consequence, we can factorize code as we always do (with spinneret functions or lisp macros). We manipulate code as we always do. It even warns on malformed html and has some neat features (it is clever about headers levels, it can embed markdown, etc).

Stuff like this is less possible with Python, because the language is less flexible. The components libraries I have seen use strings inside Python code.

Deployment, Shipping

Shipping an app, even more a web app, in Python (and JS) is tedious. There are no default way to ship a self-contained executable. Current projects aiming at fixing that can work… and may not.

So the current solution is to turn to containers. They’re the Big Thing, but we still need to spend hours on reading resources, building the Docker file, the deployment pipeline, fixing bugs, updating the stack, accepting security holes, etc. Hours we could put on our app. With Docker though, users still can’t download a binary.

In Common Lisp: we (re)discover the joy of a compiled language. We compile our program to machine code, the binary embeds the run-time, the debugger, the web server, the static assets, and we ship it. We run it on the server and we can access it from the outside straight away.

An SBCL image of a non-trivial web project will weight ± 20 to 30MB (with core compression). For a lighter binary (not that I care personally), we could try ECL (that compiles to C), or use tree-shakers of proprietary implementations (LispWorks, Allegro).

We can still benefit from Docker if needed, of course.

Deployment process in Python: install Python and pip, install pip dependencies and their system requirements and be prepared for errors (or try non-standard tools, like Platter), configure a server for static files (nginx, whitenoise), run a WSGI web server,…

Deployment in CL: build your binary, send it to the server, run it. Configure nginx eventually. We can compile and include assets into the image (see Rock).

Performance

Python is notoriously slow, and passed the hobby project you quickly realize that.

Python has a Global Interpreter Lock.

SBCL compiles to machine code and is garbage collected [1].

We can fine-tune the types in our Lisp programs for the compiler to make the consequent optimizations. We can run in “debugging first” or in “speed first” modes. We can inline code to gain in the cost of function calls.

As a consequence, you may not need memcached in your Lisp project yet.

https://lispcookbook.github.io/cl-cookbook/performance.html
CL can be tuned to be faster than C
interesting stuff: Petalisp - an attempt to generate high performance code for parallel computers by JIT-compiling array definitions. It works on a more fundamental level than NumPy, by providing even more powerful N-dimensional arrays, but just a few building blocks for working on them.
pgloader was re-written from Python to Common Lisp for a 30x speed gain.

[1]: and rest assured, Google improves the GC

Conclusion

I hope I killed some FUD and showed you new ways to make stuff. May that inspire you!

Appendix: FAQ

Some info every Python programmer will come across eventually. Saves you some googling.

Are there no iterators ?

In practice, we mostly rely on closures, but there are libraries to create iterators.

See https://stackoverflow.com/questions/32956033/is-there-a-straightforward-lisp-equivalent-of-pythons-generators

Can I define my own `+` operator like in an OO language ?

By default, no, because the Common Lisp Object System (CLOS) came after the language specification and thus everything isn’t object-based. However there are libraries like generic-cl and, in practice, we quickly forget about this. Different operators is also a means for performance, good type inference and error messages.

To which extent can Lisp be compiled, with all its dynamic nature, garbage collection, macros and what else ?

Many Lisp compilers compile to machine code (SBCL, CCL, CMUCL,…).

Full answer: https://stackoverflow.com/questions/913671/are-there-lisp-native-code-compilers/914383#914383

But what is Common Lisp good for, really ?

We have a ready-to-use citation :)

Please don’t assume Lisp is only useful for Animation and Graphics, AI, Bio-informatics, B2B and Ecommerce, Data Mining, EDA/Semiconductor applications, Expert Systems, Finance, Intelligent Agents, Knowledge Management, Mechanical CAD, Modeling and Simulation, Natural Language, Optimization, Research, Risk Analysis, Scheduling, Telecom, and Web Authoring just because these are the only things they happened to list. – Kent Pitman

Kent Pitman

http://www.nhplace.com/kent/quoted.html

So why is CL not more popular ?

First, some reminders:

popularity doesn’t equal quality, and popularity is hard to measure.
some success stories: http://lisp-lang.org/success/ Aircraft analysis suits, Missile defense, ICAD, music composition, algebra systems, bulk importer for PostgreSQL, grammar checking, 3D editor, knowledge graphs,…
did you know that pgloader was re-written from Python to Common Lisp? (for a x30 speed gain, among other benefits)
CL was used in a spacecraft (and the REPL was used to debug the system live from the earth)
some companies still use and pick CL: https://github.com/azzamsa/awesome-lisp-companies, companies provide professional support (Platform.sh).
Google’s ITA Software still powers airfare search on Orbitz or Kaya.com,
reddit v1 was written in CL! JavaScript was ~~written~~ sketched in CL! (see here, with lisp code still visible in the repository: here)
CL was number 2 on the Tiobe index for years in the 80s!

That being said, my 2 cents since you ask:

I think the CL world missed the web bandwagon for some time (common-lisp.net was horrible for some years), but that’s been fixed.
an enormous code-base existed before GitHub.
we missed visually nice, practical content on the web, even though there are many books. It’s a bit better now.
CL missed a package manager for some time behind other languages, that’s now fixed.
I reckon CL is still quite hard for the web, it doesn’t have a killer web framework (though maybe Weblocks, CLOG or ISSR soon©, all isomorphic web frameworks), hence no hype.
CL seems to be used for big, non-trivial projects, hence it gets no easy hype.
CL has no entity doing marketing today. We saw the Common Lisp foundation pairing with sponsors recently. It did receive a lot of financial and institutional support from the MIT, the NASA, Xerox, Carnegie Mellon University (CMUCL), Lisp vendors (Symbolics, Lucid, Franz…),…
CL worked well with Emacs, Vim, CCL’s built-in editor on macOs, LispWorks’ editor (which has a free version), but this doesn’t satisfy the masses. We now have more options, including Atom (very good support), VSCode (okay support) and Eclipse (basic support).
other reasons: it may be hard (or harder than the concurrence) to grasp and getting started with, Lisp isn’t for everyone, it gets a lot of FUD, and has a so-called Lisp curse!

but that’s all debatable, I wouldn’t focus much on this. Times are good for implementations, that’s what counts.

How do I get started?

You need to install sbcl and run it in your terminal. From there you can write code in the basic REPL (use rlwrap sbcl for a readline wrapper). You can write code to a file and use (load "myfile.lisp") to load and execute it. Then you’ll want to install Quicklisp and to set up a true editor. Here’s a video to get you started in no speed:

However, make you a favor and learn some Lisp for good! I am creating a video course on the Udemy platform, where I teach the language for practical purposes, and where I make you avoid all the gotchas that I faced during my journey.

It’s here: Common Lisp Programming, from novice to effective programmer.

Replic v0.12

Tue, 29 Oct 2019 18:40:07 +0100

We recently pushed our replic library version 0.12, adding a couple of expected features, thanks to the input of our ~~users~~ user:

we can TAB-complete sentences (strings inside quotes)
we can define a different completion method for each arguments of a command.
we added a declarative way to automatically print a function’s result. The default function can be overriden by users (in order too, for example, color output).

So we can do something like this: we create a function (that will become a command on the readline command line):

(defun say (verb name)
  (format t "~a, ~a !~&" verb name))

We define how to TAB-complete its arguments:

(replic.completion:add-completion "say"
                                  (list "greetings" "\"nice to see you\"")
                                  (lambda () *names*))

Now if you type say TAB you get the two greeting choices. After you pick one and press TAB again, you get the names that were given to hello.

What’s beginning to be wanted now is fuzzy completion.

Hope you enjoy.

What is replic’s goal ?

Building a readline application is cool, but readline gives you the basics and you must still build a REPL around it: loop and read commands, catch a =C-c=, a =C-d=, ask confirmation to quit, print the general help, the help of a command, setup the completion of commands, the completion of their arguments, load an init file, colorize output,… =replic= does this for you.

Replic’s goal is that when you have a lisp library, with lisp functions, it should be straightforward to create a terminal application out of it.

Here’s an example in the wild. The lyrics library is cool. It is a lisp library, it must be used on the Lisp REPL. This is the amount of code that was needed to create a terminal application out of it: https://github.com/mihaiolteanu/lyrics/pull/1/files

Compile Time Type Checking in Common Lisp

Tue, 29 Oct 2019 17:12:33 +0100

We often hear that Common Lisp is dynamically typed, which is not wrong, but that leads to the belief that Lisp is as bad as Python concerning types, which is plainly wrong. We don’t hear enough that CL is a compiled language, that we can add type annotations, and that SBCL does thorough type checking. Hence, what we have at hand is awesome: we can compile a whole program or compile a single function and get type warnings. Once again, the feedback is immediate. We can define our own types and get compile-time type warnings.

You use a paramater that must be a list of list of strings of length 3 ? Ok, define the type:

(defun list-of-3tuples-strings-p (list)
  "Return t if LIST is a list composed of 3-tuples, made only of strings."
  (and (consp list)
       (every (lambda (it)
                (and
                 (= 3 (length it))
                 (every #'stringp it)))
              list)))

(deftype alist-of-3tuples-strings ()
  `(satisfies list-of-3tuples-strings-p))

and type the variable as explained below.

It’s useful for development, it’s also great to catch errors in a user’s configuration file. Checks are done when we load a file, and error messages are explicit. We use this now in the Next browser.

We don’t hear a lot about all that, maybe because the information was hard to find, maybe because SBCL was not there at the time Lisp books were written. The following was published to the Common Lisp Cookbook /type.html, so hopefully the issue is solved!

On the topic, don’t miss these:

the article Static type checking in SBCL, by Martin Cracauer
the article Typed List, a Primer - let’s explore Lisp’s fine-grained type hierarchy! with a shallow comparison to Haskell.
the Coalton library (pre-alpha): adding Hindley-Milner type checking to Common Lisp which allows for gradual adoption, in the same way Typed Racket or Hack allows for. It is as an embedded DSL in Lisp that resembles Standard ML or OCaml, but lets you seamlessly interoperate with non-statically-typed Lisp code (and vice versa).

Compile-time type checking

You may provide type information for variables, function arguments etc via the macros declare and declaim. However, similar to the :type slot introduced in CLOS section, the effects of type declarations are undefined in Lisp standard and are implementation specific. So there is no guarantee that the Lisp compiler will perform compile-time type checking.

However, it is possible, and SBCL is an implementation that does thorough type checking.

Let’s recall first that Lisp already warns about simple type warnings. The following function wrongly wants to concatenate a string and a number. When we compile it, we get a type warning.

(defconstant +foo+ 3)
(defun bar ()
  (concatenate 'string "+" +foo+))
; caught WARNING:
;   Constant 3 conflicts with its asserted type SEQUENCE.
;   See also:
;     The SBCL Manual, Node "Handling of Types"

The example is simple, but it already shows a capacity some other languages don’t have, and it is actually useful during development ;) Now, we’ll do better.

Declaring the type of variables

Use the macro declaim.

Let’s declare that our global variable *name* is a string (you can type the following in any order in the REPL):

(declaim (type (string) *name*))
(defparameter *name* "book")

Now if we try to set it with a bad type, we get a simple-type-error:

(setf *name* :me)
Value of :ME in (THE STRING :ME) is :ME, not a STRING.
   [Condition of type SIMPLE-TYPE-ERROR]

We can do the same with our custom types. Let’s quickly declare the type list-of-strings:

(defun list-of-strings-p (list)
  "Return t if LIST is non nil and contains only strings."
  (and (consp list)
       (every #'stringp list)))

(deftype list-of-strings ()
  `(satisfies list-of-strings-p))

Now let’s declare that our *all-names* variables is a list of strings:

(declaim (type (list-of-strings) *all-names*))
(defparameter *all-names* "")

We can compose types:

(declaim (type (or null list-of-strings) *all-names*))

Declaring the input and output types of functions

We use again the declaim macro, with ftype (function …) instead of just type:

(declaim (ftype (function (fixnum) fixnum) add))
;;                         ^^input ^^output [optional]
(defun add (n)
	(+ n  1))

With this we get nice type warnings at compile time.

If we change the function to erroneously return a string instead of a fixnum, we get a warning:

(defun add (n)
	(format nil "~a" (+ n  1)))
; caught WARNING:
;   Derived type of ((GET-OUTPUT-STREAM-STRING STREAM)) is
;     (VALUES SIMPLE-STRING &OPTIONAL),
;   conflicting with the declared function return type
;     (VALUES FIXNUM &REST T).

If we use add inside another function, to a place that expects a string, we get a warning:

(defun bad-concat (n)
    (concatenate 'string (add n)))
; caught WARNING:
;   Derived type of (ADD N) is
;     (VALUES FIXNUM &REST T),
;   conflicting with its asserted type
;     SEQUENCE.

If we use add inside another function, and that function declares its argument types which appear to be incompatible with those of add, we get a warning:

(declaim (ftype (function (string)) bad-arg))
(defun bad-arg (n)
    (add n))
; caught WARNING:
;   Derived type of N is
;     (VALUES STRING &OPTIONAL),
;   conflicting with its asserted type
;     FIXNUM.

This all happens indeed at compile time, either in the REPL, either with a simple C-c C-c in Slime, or when we load a file.

Reddit: ABCL Common Lisp vs Clojure

Mon, 23 Sep 2019 01:47:10 +0200

Not that I’m interested in using the Java platform :D but relevant comparisons between ABCL (Common Lisp on Java) and Clojure are rare. We just had a nice feedback on reddit, so here it is. The question was:

After looking at the quite old benchmarks, ABCL seems to perform alright. Can anyone share their experience with ABCL in terms of performance, stability and memory usage?

I wish I could give you more concrete numbers with an application you could test and see for yourself. Since I can’t do that, I will tell you about my recent work with ABCL. I ported a small Clojure server-side utility to ABCL and can qualitatively tell you that the performance was close to Clojure. After profiling the ABCL version, I believe I can attribute the differences to ABCL’s use of Java reflection for its Java FFI.

I’ve already been successfully deploying Clojure-based applications professionally, and as I’ve gotten more into Common Lisp, I’d like to start deploying Common Lisp based applications as well. I recently posted a patch to the ABCL mailing list and got a very quick response from the maintainer. I really like the quality of the ABCL code base. The compiler itself was very approachable and easy to understand.

I think ABCL really is a worthwhile target in the Common Lisp world because:

Painless Java FFI. You avoid all the instability and signaling issues that crop up when using the JVM combined with SBCL or CCL. If you make a lot of calls, native Java is always going to be faster anyhow than calls over JNI (which is more comparable to reflection).
Use large heaps without worry. Part of the benefit of the JVM is its proven ability to have huge heaps (I’ve been part of projects that had 64GB+ heaps (though honestly I’d rather stay small)).
JVM platform is well supported and tested on a number of OS and hardware platforms

SBCL uses conservative garbage collection and I’m curious how well it would handle really large heaps. CCL uses precise garbage collection but again, I’d like to know how it handles really large heaps. In general, I want all my applications to run with heaps that are naturally in CCL’s or SBCL’s sweet spot, but I’d love to know I could use ABCL if I really ever needed huge heaps. I’m really getting into Common Lisp because I really like the implementation choices. Having a solid Java FFI unfortunately is usually a requirement in my workplace.

To me, ABCL will be /better/ than using Clojure if ABCL’s Java FFI moves away from reflection (when possible). This will close any performance gap with Clojure for most applications. I think this can be done relatively easily in the current ABCL implementation, and I have an idea of how to do it but unfortunately have had no time lately to devote to it. The reason I say “better than Clojure” is that I can write applications that target both ABCL and SBCL/CCL – I can abstract away my Java APIs if I really have to have them (or use LispWorks with a solid Java FFI if I don’t need a ton of Java interoperability). Then when I need fast startup time or low memory footprint, I can use these other CL implementations which are much better suited to it.

The main benefit where I still see Clojure having an edge is if you need a heavy JS-based web interface. I’m not a JS developer, but I was able to successfully use Clojurescript and make a nice looking web application that had pretty seamless interoperability with my Clojure-based server.

Anyhow, I hope this helps you. ABCL is great, I have been very impressed with it and I encourage you to try it out.

by the user somewhat-functional on reddit, september 2019

Slime Tips

Mon, 26 Aug 2019 10:40:46 +0200

Recently on reddit there was a reminder about lisptips.com and slime-tips. I already knew the two, but this time I fully enjoyed the Slime tips. I copy my favourites.

As usual, I enhanced the Cookbook/emacs-ide.html at the same time.

The Slime documentation is here: https://common-lisp.net/project/slime/doc/html/

Documentation lookup

C-c C-d h looks up documentation in CLHS. But it works only on symbols, so there are two more bindings:
C-c C-d # for reader macros
C-c C-d ~ for format directives

Other bindings which may be useful:

C-c C-d d describes a symbol using describe
C-c C-d f describes a function using describe

Synchronizing packages

C-c ~ (slime-sync-package-and-default-directory): When run in a buffer with a lisp file it will change the current package of the REPL to the package of that file and also set the current directory of the REPL to the parent directory of the file.

Calling code

C-c C-y (slime-call-defun): When the point is inside a defun and C-c C-y is pressed,

(I’ll use [] as an indication where the cursor is)

(defun foo ()
 nil[])

then (foo []) will be inserted into the REPL, so that you can write additional arguments and run it.

If foo was in a different package than the package of the REPL, (package:foo ) or (package::foo ) will be inserted.

This feature is very useful for testing a function you just wrote.

That works not only for defun, but also for defgeneric, defmethod, defmacro, and define-compiler-macro in the same fashion as for defun.

For defvar, defparameter, defconstant: [] *foo* will be inserted (the cursor is positioned before the symbol so that you can easily wrap it into a function call).

For defclass: (make-instance ‘class-name ).

Inserting calls to frames in the debugger

C-y in SLDB on a frame will insert a call to that frame into the REPL, e.g.,

(/ 0) =>
…
1: (CCL::INTEGER-/-INTEGER 1 0)
…

C-y will insert (CCL::INTEGER-/-INTEGER 1 0).

(thanks to Slime tips)

Exporting symbols

C-c x (slime-export-symbol-at-point) from the slime-package-fu contrib: takes the symbol at point and modifies the :export clause of the corresponding defpackage form. It also exports the symbol. When called with a negative argument (C-u C-c x) it will remove the symbol from :export and unexport it.

M-x slime-export-class does the same but with symbols defined by a structure or a class, like accesors, constructors, and so on. It works on structures only on SBCL and Clozure CL so far. Classes should work everywhere with MOP.

Customization

There are different styles of how symbols are presented in defpackage, the default is to use uninterned symbols (#:foo). This can be changed:

to use keywords:

(setq slime-export-symbol-representation-function
      (lambda (n) (format ":%s" n)))

or strings:

(setq slime-export-symbol-representation-function
 (lambda (n) (format "\"%s\"" (upcase n))))

Crossreferencing: find who’s calling, referencing, setting a symbol

Slime has a nice cross referencing facility, for example, you can see what calls a particular function or expands a macro. It presents a list of places which reference a particular entity, from there you can recompile the thing which references by pressing C-c C-c on that line. C-c C-k will recompile all the references. This is useful when modifying macros, inline functions, or constants.

The following bindings are also shown in Slime’s menu:

C-c C-w c slime-who-calls callers of a function
C-c C-w m slime-who-macroexpands places where a macro is expanded
C-c C-w r slime-who-references global variable references
C-c C-w b slime-who-bind global variable bindings
C-c C-w s slime-who-sets global variable setters
C-c C-w a slime-who-specializes methods specialized on a symbol

And when the slime-asdf contrib is enabled, C-c C-w d slime-who-depends-on lists dependent ASDF systems

And a general binding: M-? or M-_ *slime-edit-uses** combines all of the above, it lists every kind of references.

Monitoring and controlling threads with Slime

M-x slime-list-threads (you can also access it through the slime-selector, shortcut t) will list running threads by their names, and their statuses.

The thread on the current line can be killed with k, or if there’s a lot of threads to kill, several lines can be selected and k will kill all the threads in the selected region.

g will update the thread list, but when you have a lot of threads starting and stopping it may be too cumbersome to always press g, so there’s a variable slime-threads-update-interval, when set to a number X the thread list will be automatically updated each X seconds, a reasonable value would be 0.5.

How to Check Slots Types at make-instance

Wed, 10 Jul 2019 15:28:08 +0200

In CLOS, a slot can have a :type option, but it doesn’t inforce type checking. It is good practice to use it, for documentation and for compiler optimizations and warnings sometimes (with CCL and SBCL when safety is high), but one shouldn’t rely on it. To comply this need, we can simply create our own constructor functions.

However, the sanity-clause library can do it since a couple of days. The validation error messages are pretty good. Demonstration.

Sanity clause is a data validation/contract library. You might use it for configuration data, validating an api response, or documents from a datastore. In a dynamically typed langauge, it helps you define clearly defined areas of doubt and uncertainty. We should love our users, but we should never blindly trust their inputs.

To make use of it, you define schemas, which can be property lists with symbols for keys and instances of :class:sanity-clause.field:field

We define a class person with slot options from sanity-clause: :field-type, type :members, :required:

(defclass person ()
  ((favorite-dog :type symbol
                 :field-type :member
                 :members (:wedge :walter)
                 :initarg :favorite-dog
                 :required t)
   (age :type (integer 0)
        :initarg :age
        :required t)
   (potato :type string
           :initarg :potato
           :required t))
  (:metaclass sanity-clause.metaclass:validated-metaclass))

Now we try to create a person with make-instance, but we give a bad dog name:

(make-instance 'person :favorite-dog :nope)
; Evaluation aborted on Error converting value for field #<MEMBER-FIELD {1004BFA973}>:
Value "NOPE" couldn't be found in set (WEDGE WALTER)

Now with a bad age:

(make-instance 'person :age -1 :favorite-dog :walter)
; Evaluation aborted on Error validating value -1 in field #<INTEGER-FIELD {1004BFF103}>:
* Value -1 didn't satisfy condition "must be larger than 0"

When a required field is missing:

(make-instance 'person :age 7 :favorite-dog :walter)
; Evaluation aborted on A value for field POTATO is required but none was provided..

And well, it works when all is OK :]

(make-instance 'person :age 1 :favorite-dog :walter :potato "patate")
#<PERSON {10060371E3}>

The usual warnings apply: it’s a new library, we must try it and use it with caution. It however opens up more possibilities. It would be awesome to couple it with an ORM like Mito. This is an open issue.

Resources:

Web Development in Common Lisp: frameworks overview, templating, deployment

Mon, 08 Jul 2019 01:21:47 +0200

We just published a long overdue page on the Cookbook: web development in Common Lisp. We have an ambivalent feeling about it since it isn’t really a recipe as in the other pages. Yet it is valuable content that required a certain amount of digging and tryouts. Indeed, it took us about two years to discover and advertise many projects, to learn, try and put a tutorial together. We also wrote a commercial application. During that time, we were taking notes on our web-dev/ page.

We present Hunchentoot, Clack (briefly), we have an overview of other web frameworks, of templating libraries, we introduce Weblocks, we give recipes for common tasks (such as checking if a user is logged in, encrypting passwords), and we speak about deployment.

Some topics still need to be adressed, so check for updates on the Cookbook !

Prior notice:

Some people sell ten pages long ebooks or publish their tutorial on Gitbook to have a purchase option. I prefer to enhance the collaborative Cookbook (I am by far the main contributor). You can tip me on liberapay if you like: https://liberapay.com/vindarel/. Thanks !

For web development as for any other task, one can leverage Common Lisp’s advantages: the unmatched REPL and exception handling system, performance, the ability to build a self-contained executable, stability, good threads story, strong typing, etc. We can, say, define a new route and try it right away, there is no need to restart any running server. We can change and compile one function at a time (the usual C-c C-c in Slime) and try it. The feedback is immediate. We can choose the degree of interactivity: the web server can catch exceptions and fire the interactive debugger, or print lisp backtraces on the browser, or display a 404 error page and print logs on standard output. The ability to build self-contained executables eases deployment tremendously (compared to, for example, npm-based apps), in that we just copy the executable to a server and run it.

We’ll present here some established web frameworks and other common libraries to help you getting started in developing a web application. We do not aim to be exhaustive nor to replace the upstream documentation. Your feedback and contributions are appreciated.

Table of Contents

Overview

Hunchentoot and Clack are two projects that you’ll often hear about.

Hunchentoot is

a web server and at the same time a toolkit for building dynamic websites. As a stand-alone web server, Hunchentoot is capable of HTTP/1.1 chunking (both directions), persistent connections (keep-alive), and SSL. It provides facilities like automatic session handling (with and without cookies), logging, customizable error handling, and easy access to GET and POST parameters sent by the client.

It is a software written by Edi Weitz (“Common Lisp Recipes”, cl-ppcre and much more), it’s used and proven solid. One can achieve a lot with it, but sometimes with more friction than with a traditional web framework. For example, dispatching a route by the HTTP method is a bit convoluted, one must write a function for the :uri parameter that does the check, when it is a built-in keyword in other frameworks like Caveman.

Clack is

a web application environment for Common Lisp inspired by Python’s WSGI and Ruby’s Rack.

Also written by a prolific lisper (E. Fukamachi), it actually uses Hunchentoot by default as the server, but thanks to its pluggable architecture one can use another web server, like the asynchronous Woo, built on the libev event loop, maybe “the fastest web server written in any programming language”.

We’ll cite also Wookie, an asynchronous HTTP server, and its companion library cl-async, for general purpose, non-blocking programming in Common Lisp, built on libuv, the backend library in Node.js.

Clack being more recent and less documented, and Hunchentoot a de-facto standard, we’ll concentrate on the latter for this recipe. Your contributions are of course welcome.

Web frameworks build upon web servers and can provide facilities for common activities in web development, like a templating system, access to a database, session management, or facilities to build a REST api.

Some web frameworks include:

Caveman, by E. Fukamachi. It provides, out of the box, database management, a templating engine (Djula), a project skeleton generator, a routing system à la Flask or Sinatra, deployment options (mod_lisp or FastCGI), support for Roswell on the command line, etc.
Radiance, by Shinmera (Qtools, Portacle, lquery, …), is a web application environment, more general than usual web frameworks. It lets us write and tie websites and applications together, easing their deployment as a whole. It has thorough documentation, a tutorial, modules, pre-written applications such as an image board or a blogging platform, and more. For example websites, see https://shinmera.com/, reader.tymoon.eu and events.tymoon.eu.
Snooze, by João Távora (Sly, Emacs’ Yasnippet, Eglot, …), is “an URL router designed around REST web services”. It is different because in Snooze, routes are just functions and HTTP conditions are just Lisp conditions.
cl-rest-server is a library for writing REST web APIs. It features validation with schemas, annotations for logging, caching, permissions or authentication, documentation via OpenAPI (Swagger), etc.
last but not least, Weblocks is a venerable Common Lisp web framework that permits to write ajax-based dynamic web applications without writing any JavaScript, nor writing some lisp that would transpile to JavaScript. It is seeing an extensive rewrite and update since 2017. We present it in more details below.

For a full list of libraries for the web, please see the awesome-cl list #network-and-internet and Cliki. If you are looking for a featureful static site generator, see Coleslaw.

Installation

Let’s install the libraries we’ll use:

(ql:quickload '("hunchentoot" "caveman" "spinneret" "djula"))

To try Weblocks, please see its documentation. The Weblocks in Quicklisp is not yet, as of writing, the one we are interested in.

We’ll start by serving local files and we’ll run more than one local server in the running image.

Simple webserver

Serve local files

Hunchentoot

Create and start a webserver like this:

(defvar *acceptor* (make-instance 'hunchentoot:easy-acceptor :port 4242))
(hunchentoot:start *acceptor*)

We create an instance of easy-acceptor on port 4242 and we start it. We can now access http://127.0.0.1:4242/. You should get a welcome screen with a link to the documentation and logs to the console.

By default, Hunchentoot serves the files from the www/ directory in its source tree. Thus, if you go to the source of easy-acceptor (M-. in Slime), which is probably ~/quicklisp/dists/quicklisp/software/hunchentoot-v1.2.38/, you’ll find the root/ directory. It contains:

an errors/ directory, with the error templates 404.html and 500.html,
an img/ directory,
an index.html file.

To serve another directory, we give the option document-root to easy-acceptor. We can also set the slot with its accessor:

(setf (hunchentoot:acceptor-document-root *acceptor*) #p"path/to/www")

Let’s create our index.html first. Put this in a new www/index.html at the current directory (of the lisp repl):

<html>
  <head>
    <title>Hello!</title>
  </head>
  <body>
    <h1>Hello local server!</h1>
    <p>
    We just served our own files.
    </p>
  </body>
</html>

Let’s start a new acceptor on a new port:

(defvar *my-acceptor* (make-instance 'hunchentoot:easy-acceptor :port 4444
                                   :document-root #p"www/"))
(hunchentoot:start *my-acceptor*)

go to p://127.0.0.1:4444/ and see the difference.

Note that we just created another acceptor on a different port on the same lisp image. This is already pretty cool.

Access your server from the internet

Hunchentoot

With Hunchentoot we have nothing to do, we can see the server from the internet right away.

If you evaluate this on your VPS:

(hunchentoot:start (make-instance 'hunchentoot:easy-acceptor :port 4242))

You can see it right away on your server’s IP.

Stop it with (hunchentoot:stop *).

Routing

Simple routes

Hunchentoot

To bind an existing function to a route, we create a “prefix dispatch” that we push onto the *dispatch-table* list:

(defun hello ()
   (format nil "Hello, it works!"))

(push
  (hunchentoot:create-prefix-dispatcher "/hello.html" #'hello)
  hunchentoot:*dispatch-table*)

To create a route with a regexp, we use create-regex-dispatcher, where the url-as-regexp can be a string, an s-expression or a cl-ppcre scanner.

If you didn’t yet, create an acceptor and start the server:

(defvar *server* (make-instance 'hunchentoot:easy-acceptor :port 4242))
(hunchentoot:start *server*)

and access it on [http://localhost:4242/hello.html]http://localhost:4242/hello.html).

We can see logs on the REPL:

127.0.0.1 - [2018-10-27 23:50:09] "get / http/1.1" 200 393 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0"
127.0.0.1 - [2018-10-27 23:50:10] "get /img/made-with-lisp-logo.jpg http/1.1" 200 12583 "http://localhost:4242/" "Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0"
127.0.0.1 - [2018-10-27 23:50:10] "get /favicon.ico http/1.1" 200 1406 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0"
127.0.0.1 - [2018-10-27 23:50:19] "get /hello.html http/1.1" 200 20 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0"

define-easy-handler allows to create a function and to bind it to an uri at once.

Its form follows

define-easy-handler (function-name :uri <uri> …) (lambda list parameters)

where <uri> can be a string or a function.

Example:

(hunchentoot:define-easy-handler (say-yo :uri "/yo") (name)
  (setf (hunchentoot:content-type*) "text/plain")
  (format nil "Hey~@[ ~A~]!" name))

Visit it at p://localhost:4242/yo and add parameters on the url: http://localhost:4242/yo?name=Alice.

Just a thought… we didn’t explicitely ask Hunchentoot to add this route to our first acceptor of the port 4242. Let’s try another acceptor (see previous section), on port 4444: http://localhost:4444/yo?name=Bob It works too ! In fact, define-easy-handler accepts an acceptor-names parameter:

acceptor-names (which is evaluated) can be a list of symbols which means that the handler will only be returned by DISPATCH-EASY-HANDLERS in acceptors which have one of these names (see ACCEPTOR-NAME). acceptor-names can also be the symbol T which means that the handler will be returned by DISPATCH-EASY-HANDLERS in every acceptor.

So, define-easy-handler has the following signature:

define-easy-handler (function-name &key uri acceptor-names default-request-type) (lambda list parameters)

It also has a default-parameter-type which we’ll use in a minute to get url parameters.

There are also keys to know for the lambda list. Please see the documentation.

Caveman

Caveman provides two ways to define a route: the defroute macro and the @route pythonic annotation:

(defroute "/welcome" (&key (|name| "Guest"))
  (format nil "Welcome, ~A" |name|))

@route GET "/welcome"
(lambda (&key (|name| "Guest"))
  (format nil "Welcome, ~A" |name|))

A route with an url parameter (note :name in the url):

(defroute "/hello/:name" (&key name)
  (format nil "Hello, ~A" name))

It is also possible to define “wildcards” parameters. It works with the splat key:

(defroute "/say/*/to/*" (&key splat)
  ; matches /say/hello/to/world
  (format nil "~A" splat))
;=> (hello world)

We must enable regexps with :regexp t:

(defroute ("/hello/([\\w]+)" :regexp t) (&key captures)
  (format nil "Hello, ~A!" (first captures)))

Accessing GET and POST parameters

Hunchentoot

First of all, note that we can access query parameters anytime with

(hunchentoot:parameter "my-param")

It acts on the default *request* object which is passed to all handlers.

There is also get-paramater and post-parameter.

Earlier we saw some key parameters to define-easy-handler. We now introduce default-parameter-type.

We defined the following handler:

(hunchentoot:define-easy-handler (say-yo :uri "/yo") (name)
  (setf (hunchentoot:content-type*) "text/plain")
  (format nil "Hey~@[ ~A~]!" name))

The variable name is a string by default. Let’s check it out:

(hunchentoot:define-easy-handler (say-yo :uri "/yo") (name)
  (setf (hunchentoot:content-type*) "text/plain")
  (format nil "Hey~@[ ~A~] you are of type ~a" name (type-of name)))

Going to http://localhost:4242/yo?name=Alice returns

Hey Alice you are of type (SIMPLE-ARRAY CHARACTER (5))

To automatically bind it to another type, we use default-parameter-type. It can be one of those simple types:

'string (default),
'integer,
'character (accepting strings of length 1 only, otherwise it is nil)
or 'boolean

or a compound list:

'(:list <type>)
'(:array <type>)
'(:hash-table <type>)

where <type> is a simple type.

Error handling

In all frameworks, we can choose the level of interactivity. The web framework can return a 404 page and print output on the repl, it can catch errors and invoke the interactive lisp debugger, or it can show the lisp backtrace on the html page.

Hunchentoot

The global variables to set are *catch-errors-p*, *show-lisp-errors-p* and *show-lisp-backtraces-p*.

Hunchentoot also defines condition classes.

See the documentation: https://edicl.github.io/hunchentoot/#conditions.

Clack

Clack users might make a good use of plugins, like the clack-errors middleware: https://github.com/CodyReichert/awesome-cl#clack-plugins.

Weblocks - solving the “JavaScript problem”©

Weblocks is a widgets-based and server-based framework with a built-in ajax update mechanism. It allows to write dynamic web applications without the need to write JavaScript or to write lisp code that would transpile to JavaScript.

Weblocks is an old framework developed by Slava Akhmechet, Stephen Compall and Leslie Polzer. After nine calm years, it is seeing a very active update, refactoring and rewrite effort by Alexander Artemenko.

It was initially based on continuations (they were removed to date) and thus a lispy cousin of Smalltalk’s Seaside. We can also relate it to Haskell’s Haste, OCaml’s Eliom, Elixir’s Phoenix LiveView and others.

The Ultralisp website is an example Weblocks website in production known in the CL community.

Weblock’s unit of work is the widget. They look like a class definition:

(defwidget task ()
        ((title
          :initarg :title
          :accessor title)
         (done
          :initarg :done
          :initform nil
          :accessor done)))

Then all we have to do is to define the render method for this widget:

(defmethod render ((task task))
        "Render a task."
        (with-html
              (:span (if (done task)
                         (with-html
                               (:s (title task)))
                       (title task)))))

It uses the Spinneret template engine by default, but we can bind any other one of our choice.

To trigger an ajax event, we write lambdas in full Common Lisp:

...
(with-html
          (:p (:input :type "checkbox"
            :checked (done task)
            :onclick (make-js-action
                      (lambda (&key &allow-other-keys)
                        (toggle task))))
...

The function make-js-action creates a simple javascript function that calls the lisp one on the server, and automatically refreshes the HTML of the widgets that need it. In our example, it re-renders one task only.

Is it appealing ? Carry on this quickstart guide here: http://40ants.com/weblocks/quickstart.html.

Templates

Djula - HTML markup

Djula is a port of Python’s Django template engine to Common Lisp. It has excellent documentation.

Caveman uses it by default, but otherwise it is not difficult to setup. We must declare where our templates are with something like

(djula:add-template-directory (asdf:system-relative-pathname "webapp" "templates/"))

A Djula template looks like this, no surprises (forgive the antislash in \%, this is a Jekyll limitation):

{\% extends "base.html" \%}
{\% block title %}Memberlist{\% endblock \%}
{\% block content \%}
  <ul>
  {\% for user in users \%}
    <li><a href="{{ user.url }}">{{ user.username }}</a></li>
  {\% endfor \%}
  </ul>
{\% endblock \%}

Djula compiles the templates before rendering them.

It is, along with its companion access library, one of the most downloaded libraries of Quicklisp.

Spinneret - lispy templates

Spinneret is a “lispy” HTML5 generator. It looks like this:

(with-page (:title "Home page")
     (:header
      (:h1 "Home page"))
     (:section
      ("~A, here is *your* shopping list: " *user-name*)
      (:ol (dolist (item *shopping-list*)
             (:li (1+ (random 10)) item))))
     (:footer ("Last login: ~A" *last-login*)))

The author finds it is easier to compose the HTML in separate functions and macros than with the more famous cl-who. But it has more features under it sleeves:

it warns on invalid tags and attributes
it can automatically number headers, given their depth
it pretty prints html per default, with control over line breaks
it understands embedded markdown
it can tell where in the document a generator function is (see get-html-tag)

Connecting to a database

Please see the databases section. The Mito ORM supports SQLite3, PostgreSQL, MySQL, it has migrations and db schema versioning, etc.

In Caveman, a database connection is alive during the Lisp session and is reused in each HTTP requests.

Checking a user is logged-in

A framework will provide a way to work with sessions. We’ll create a little macro to wrap our routes to check if the user is logged in.

In Caveman, *session* is a hash table that represents the session’s data. Here are our login and logout functions:

(defun login (user)
  "Log the user into the session"
  (setf (gethash :user *session*) user))

(defun logout ()
  "Log the user out of the session."
  (setf (gethash :user *session*) nil))

We define a simple predicate:

(defun logged-in-p ()
  (gethash :user cm:*session*))

and we define our with-logged-in macro:

(defmacro with-logged-in (&body body)
  `(if (logged-in-p)
       (progn ,@body)
       (render #p"login.html"
               '(:message "Please log-in to access this page."))))

If the user isn’t logged in, there will nothing in the session store, and we render the login page. When all is well, we execute the macro’s body. We use it like this:

(defroute "/account/logout" ()
  "Show the log-out page, only if the user is logged in."
  (with-logged-in
    (logout)
    (render #p"logout.html")))

(defroute ("/account/review" :method :get) ()
  (with-logged-in
    (render #p"review.html"
            (list :review (get-review (gethash :user *session*))))))

and so on.

Encrypting passwords

In this recipe we use the de-facto standard Ironclad cryptographic toolkit and the Babel charset encoding/decoding library.

This snippet creates the password hash that should be stored in your database. Note that Ironclad expects a byte-vector, not a string.

(defun password-hash (password)
  (ironclad:pbkdf2-hash-password-to-combined-string
   (babel:string-to-octets password)))

pbkdf2 is defined in RFC2898. It uses a pseudorandom function to derive a secure encryption key based on the password.

The following function checks if a user is active and verifies the entered password. It returns the user-id if active and verified and nil in all other cases even if an error occurs. Adapt it to your application.

(defun check-user-password (user password)
  (handler-case
      (let* ((data (my-get-user-data user))
             (hash (my-get-user-hash data))
             (active (my-get-user-active data)))
        (when (and active (ironclad:pbkdf2-check-password (babel:string-to-octets password)
                                                          hash))
          (my-get-user-id data)))
    (condition () nil)))

And the following is an example on how to set the password on the database. Note that we use (password-hash password) to save the password. The rest is specific to the web framework and to the DB library.

(defun set-password (user password)
  (with-connection (db)
    (execute
     (make-statement :update :web_user
                     (set= :hash (password-hash password))
                     (make-clause :where
                                  (make-op := (if (integerp user)
                                                  :id_user
                                                  :email)
                                           user))))))

Credit: /u/arvid on /r/learnlisp.

Building

Building a self-contained executable

As for all Common Lisp applications, we can bundle our web app in one single executable, including the assets. It makes deployment very easy: copy it to your server and run it.

$ ./my-web-app
Hunchentoot server is started.
Listening on localhost:9003.

See this recipe on scripting#for-web-apps.

Continuous delivery with Travis CI or Gitlab CI

Please see the section on testing#continuous-integration.

Multiplatform delivery with Electron

Ceramic makes all the work for us.

It is as simple as this:

;; Load Ceramic and our app
(ql:quickload '(:ceramic :our-app))

;; Ensure Ceramic is set up
(ceramic:setup)
(ceramic:interactive)

;; Start our app (here based on the Lucerne framework)
(lucerne:start our-app.views:app :port 8000)

;; Open a browser window to it
(defvar window (ceramic:make-window :url "http://localhost:8000/"))

;; start Ceramic
(ceramic:show-window window)

and we can ship this on Linux, Mac and Windows.

There is more:

Ceramic applications are compiled down to native code, ensuring both performance and enabling you to deliver closed-source, commercial applications.

Thus, no need to minify our JS.

Deployment

Deploying manually

We can start our executable in a shell and send it to the bakcground (C-z bg), or run it inside a tmux session. These are not the best but hey, it works©.

Daemonizing, restarting in case of crashes, handling logs with Systemd

This is actually a system-specific task. See how to do that on your system.

Most GNU/Linux distros now come with Systemd, so here’s a little example.

Deploying an app with Systemd is as simple as writing a configuration file:

$ emacs -nw /etc/systemd/system/my-app.service
[Unit]
Description=stupid simple example

[Service]
WorkingDirectory=/path/to/your/app
ExecStart=/usr/local/bin/sthg sthg
Type=simple
Restart=always
RestartSec=10

Then we have a command to start it:

sudo systemctl start my-app.service

a command to check its status:

systemctl status my-app.service

and Systemd can handle logging (we write to stdout or stderr, it writes logs):

journalctl -f -u my-app.service

and it handles crashes and restarts the app:

Restart=always

and it can start the app after a reboot:

[Install]
WantedBy=basic.target

to enable it:

sudo systemctl enable my-app.service

With Docker

There are several Docker images for Common Lisp. For example:

40ants/base-lisp-image is based on Ubuntu LTS and includes SBCL, CCL, Quicklisp, Qlot and Roswell.
container-lisp/s2i-lisp is CentOs based and contains the source for building a Quicklisp based Common Lisp application as a reproducible docker image using OpenShift’s source-to-image.

With Guix

GNU Guix is a transactional package manager, that can be installed on top of an existing OS, and a whole distro that supports declarative system configuration. It allows to ship self-contained tarballs, which also contain system dependencies. For an example, see the Next browser.

Deploying on Heroku and other services

See heroku-buildpack-common-lisp and the Awesome CL#deploy section for interface libraries for Kubernetes, OpenShift, AWS, etc.

Monitoring

See Prometheus.cl for a Grafana dashboard for SBCL and Hunchentoot metrics (memory, threads, requests per second,…).

Connecting to a remote Lisp image

This this section: debugging#remote-debugging.

Hot reload

This is an example from Quickutil. It is actually an automated version of the precedent section.

It has a Makefile target:

hot_deploy:
	$(call $(LISP), \
		(ql:quickload :quickutil-server) (ql:quickload :swank-client), \
		(swank-client:with-slime-connection (conn "localhost" $(SWANK_PORT)) \
			(swank-client:slime-eval (quote (handler-bind ((error (function continue))) \
				(ql:quickload :quickutil-utilities) (ql:quickload :quickutil-server) \
				(funcall (symbol-function (intern "STOP" :quickutil-server))) \
				(funcall (symbol-function (intern "START" :quickutil-server)) $(start_args)))) conn)) \
		$($(LISP)-quit))

Credits

https://lisp-journey.gitlab.io/web-dev/

Pattern Matching in Common Lisp

Sun, 26 May 2019 16:17:08 +0200

New page on the Cookbook: https://lispcookbook.github.io/cl-cookbook/pattern_matching.html All examples come from Trivia’s wiki.

The ANSI Common Lisp standard does not include facilities for pattern matching, but libraries existed for this task and Trivia became a community standard.

Table of Contents

Common destructuring patterns

For an introduction to the concepts of pattern matching, see Trivia’s wiki.

Trivia matches against a lot of lisp objects and is extensible.

The library is in Quicklisp:

(ql:quickload "trivia")

For the following examples, let’s use the library:

(use-package :trivia)

Common destructuring patterns

`cons`

(match '(1 2 3)
  ((cons x y)
  ; ^^ pattern
   (print x)
   (print y)))
;; |-> 1
;; |-> (2 3)

`list`, `list*`

(match '(something #(0 1 2))
  ((list a (vector 0 _ b))
   (values a b)))
SOMETHING
2

list* pattern:

(match '(1 2 . 3)
  ((list* _ _ x)
   x))
3

Note that using list would match nothing.

`vector`, `vector*`

vector checks if the object is a vector, if the lengths are the same, and if the contents matches against each subpatterns.

vector* is similar, but called a soft-match variant that allows if the length is larger-than-equal to the length of subpatterns.

(match #(1 2 3)
  ((vector _ x _)
   x))
;; -> 2

(match #(1 2 3 4)
  ((vector _ x _)
   x))
;; -> NIL : does not match

(match #(1 2 3 4)
  ((vector* _ x _)
   x))
;; -> 2 : soft match.

<vector-pattern> : vector      | simple-vector
                   bit-vector  | simple-bit-vector
                   string      | simple-string
                   base-string | simple-base-string | sequence
(<vector-pattern> &rest subpatterns)

Class and structure pattern

There are three styles that are equivalent:

(defstruct foo bar baz)
(defvar *x* (make-foo :bar 0 :baz 1)

(match *x*
  ;; make-instance style
  ((foo :bar a :baz b)
   (values a b))
  ;; with-slots style
  ((foo (bar a) (baz b))
   (values a b))
  ;; slot name style
  ((foo bar baz)
   (values bar baz)))

`type`, `satisfies`

The type pattern matches if the object is of type. satisfies matches if the predicate returns true for the object. A lambda form is acceptable.

`assoc`, `property`, `alist`, `plist`

All these patterns first check if the pattern is a list. If that is satisfied, then they obtain the contents, and the value is matched against the subpattern.

Array, simple-array, row-major-array patterns

See https://github.com/guicho271828/trivia/wiki/Type-Based-Destructuring-Patterns#array-simple-array-row-major-array-pattern !

Logic based patterns

`and`, `or`

(match x
  ((or (list 1 a)
       (cons a 3))
   a))

matches against both (1 2) and (4 . 3) and returns 2 and 4, respectively.

`not`

It does not match when subpattern matches. The variables used in the subpattern are not visible in the body.

`guards`

The syntax is guard + subpattern + a test form, and the body.

(match (list 2 5)
  ((guard (list x y)     ; subpattern
          (= 10 (* x y)) ; test-form
          (- x y) (satisfies evenp)) ; generator1, subpattern1
   t))

If the subpattern is true, the test form is evaluated, and if it is true it is matched against subpattern1.

The above returns nil, since (- x y) == 3 does not satisfies evenp.

Nesting patterns

Patterns can be nested:

(match '(:a (3 4) 5)
  ((list :a (list _ c) _)
   c))

returns 4.

See more

See special patterns: place, bind and access.

Functions Tutorial: Arguments, Multiple Values, funcall and apply, closures and more

Sun, 26 May 2019 16:08:10 +0200

Here’s an enhanced page for the Cookbook: https://lispcookbook.github.io/cl-cookbook/functions.html

Only the Currying part was untouched (we enhanced it already), the higher-order functions part existed and was rewritten. The rest is new, and it should help you start writing Common Lisp quicker than ever.

Happy lisping !

Table of Contents

Named functions: `defun`

Creating named functions is done with the defun keyword. It follows this model:

(defun <name> (list of arguments)
  "docstring"
  (function body))

The return value is the value returned by the last expression of the body (see below for more). There is no “return xx” statement.

So, for example:

(defun hello-world ()
  ;;               ^^ no arguments
  (print "hello world!"))

Call it:

(hello-world)
;; "hello world!"  <-- output
;; "hello world!"  <-- a string is returned.

Arguments

Base case: required arguments

Add in arguments like this:

(defun hello (name)
  "Say hello to `name'."
  (format t "hello ~a !~&" name))
;; HELLO

(where ~a is the most used format directive to print a variable aesthetically and ~& prints a newline)

Call the function:

(hello "me")
;; hello me !  <-- this is printed by `format`
;; NIL         <-- return value: `format t` prints a string to standard output and returns nil.

If you don’t specify the right amount of arguments, you’ll be trapped into the interactive debugger with an explicit error message:

(hello)

invalid number of arguments: 0

Optional arguments: `&optional`

Optional arguments are declared after the &optional keyword in the lambda list. They are ordered, they must appear one after another.

This function:

(defun hello (name &optional age gender) …)

must be called like this:

(hello "me") ;; a value for the required argument, zero optional arguments
(hello "me" "7")  ;; a value for age
(hello "me" 7 :h) ;; a value for age and gender

Named parameters: `&key`

It is not always convenient to remember the order of the arguments. It is thus possible to supply arguments by name: we declare them using &key <name>, we set them with :name <value> in the function call, and we use name as a regular variable in the function body. They are nil by default.

(defun hello (name &key happy)
  "If `happy' is `t', print a smiley"
  (format t "hello ~a " name)
  (when happy
    (format t ":)~&"))

The following calls are possible:

(hello "me")
(hello "me" :happy t)
(hello "me" :happy nil) ;; useless, equivalent to (hello "me")

and this is not valid: (hello "me" :happy):

odd number of &KEY arguments

A similar example of a function declaration, with several key parameters:

(defun hello (name &key happy lisper cookbook-contributor-p) …)

it can be called with zero or more key parameters, in any order:

(hello "me" :lisper t)
(hello "me" :lisper t :happy t)
(hello "me" :cookbook-contributor-p t :happy t)

Mixing optional and key parameters

It is generally a style warning, but it is possible.

(defun hello (&optional name &key happy)
  (format t "hello ~a " name)
  (when happy
    (format t ":)~&")))

In SBCL, this yields:

; in: DEFUN HELLO
;     (SB-INT:NAMED-LAMBDA HELLO
;         (&OPTIONAL NAME &KEY HAPPY)
;       (BLOCK HELLO (FORMAT T "hello ~a " NAME) (WHEN HAPPY (FORMAT T ":)~&"))))
;
; caught STYLE-WARNING:
;   &OPTIONAL and &KEY found in the same lambda list: (&OPTIONAL (NAME "John") &KEY
;                                                      HAPPY)
;
; compilation unit finished
;   caught 1 STYLE-WARNING condition

We can call it:

(hello "me" :happy t)
;; hello me :)
;; NIL

Default values

In the lambda list, use pairs to give a default value to an optional or a key argument, like (happy t) below:

(defun hello (name &key (happy t))

Now happy is true by default.

Variable number of arguments: `&rest`

Sometimes you want a function to accept a variable number of arguments. Use &rest <variable>, where <variable> will be a list.

(defun mean (x &rest numbers)
    (/ (apply #'+ x numbers)
       (1+ (length numbers))))

(mean 1)
(mean 1 2)
(mean 1 2 3 4 5)

`&allow-other-keys`

Observe:

(defun hello (name &key happy)
  (format t "hello ~a~&" name))

(hello "me" :lisper t)
;; => Error: unknown keyword argument

whereas

(defun hello (name &key happy &allow-other-keys)
  (format t "hello ~a~&" name))

(hello "me" :lisper t)
;; hello me

We might need &allow-other-keys when passing around arguments or with higher level manipulation of functions.

Return values

The return value of the function is the value returned by the last executed form of the body.

There are ways for non-local exits (return-from <function name> <value>), but they are usually not needed.

Common Lisp has also the concept of multiple return values.

Multiple return values: `values` and `multiple-value-bind`

Returning multiple values is not like returning a tuple or a list of results ;) This is a common misconception.

Multiple values are specially useful and powerful because a change in them needs little to no refactoring.

(defun foo (a b c)
  a)

This function returns a.

(defvar *res* (foo :a :b :c))
;; :A

We use values to return multiple values:

(defun foo (a b c)
  (values a b c))

(setf *res* (foo :a :b :c))
;; :A

Observe here that *res* is still :A.

All functions that use the return value of foo need no change, they still work. If we had returned a list or an array, this would be different.

We destructure multiple values with multiple-value-bind (or mvb+TAB in Slime for short):

(multiple-value-bind (res1 res2 res3)
    (foo :a :b :c)
  (format t "res1 is ~a, res2 is ~a, res2 is ~a~&" res1 res2 res3))
;; res1 is A, res2 is B, res2 is C
;; NIL

Its general form is

(multiple-value-bind (var-1 .. var-n) expr
  body)

The variables var-n are not available outside the scope of multiple-value-bind.

Last but not least: note that (values) with no values returns… no values at all.

Anonymous functions: `lambda`

Anonymous functions are created with lambda:

(lambda (x) (print x))

We can call a lambda with funcall or apply (see below).

If the first element of an unquoted list is a lambda expression, the lambda is called:

((lambda (x) (print x)) "hello")
;; hello

Calling functions programatically: `funcall` and `apply`

funcall is to be used with a known number of arguments, when apply can be used on a list, for example from &rest:

(funcall #'+ 1 2)
(apply #'+ '(1 2))

Higher order functions: functions that return functions

Writing functions that return functions is simple enough:

(defun adder (n)
  (lambda (x) (+ x n)))
;; ADDER

Here we have defined the function adder which returns an object of type function.

To call the resulting function, we must use funcall or apply:

(adder 5)
;; #<CLOSURE (LAMBDA (X) :IN ADDER) {100994ACDB}>
(funcall (adder 5) 3)
;; 8

Trying to call it right away leads to an illegal function call:

((adder 3) 5)
In: (ADDER 3) 5
    ((ADDER 3) 5)
Error: Illegal function call.

Indeed, CL has different namespaces for functions and variables, i.e. the same name can refer to different things depending on its position in a form that’s evaluated.

;; The symbol foo is bound to nothing:
CL-USER> (boundp 'foo)
NIL
CL-USER> (fboundp 'foo)
NIL
;; We create a variable:
CL-USER> (defparameter foo 42)
FOO
* foo
42
;; Now foo is "bound":
CL-USER> (boundp 'foo)
T
;; but still not as a function:
CL-USER> (fboundp 'foo)
NIL
;; So let's define a function:
CL-USER> (defun foo (x) (* x x))
FOO
;; Now the symbol foo is bound as a function too:
CL-USER> (fboundp 'foo)
T
;; Get the function:
CL-USER> (function foo)
#<FUNCTION FOO>
;; and the shorthand notation:
* #'foo
#<FUNCTION FOO>
;; We call it:
(funcall (function adder) 5)
#<CLOSURE (LAMBDA (X) :IN ADDER) {100991761B}>
;; and call the lambda:
(funcall (funcall (function adder) 5) 3)
8

To simplify a bit, you can think of each symbol in CL having (at least) two “cells” in which information is stored. One cell - sometimes referred to as its value cell - can hold a value that is bound to this symbol, and you can use boundp to test whether the symbol is bound to a value (in the global environment). You can access the value cell of a symbol with symbol-value.

The other cell - sometimes referred to as its function cell - can hold the definition of the symbol’s (global) function binding. In this case, the symbol is said to be fbound to this definition. You can use fboundp to test whether a symbol is fbound. You can access the function cell of a symbol (in the global environment) with symbol-function.

Now, if a symbol is evaluated, it is treated as a variable in that its value cell is returned (just foo). If a compound form, i.e. a cons, is evaluated and its car is a symbol, then the function cell of this symbol is used (as in (foo 3)).

In Common Lisp, as opposed to Scheme, it is not possible that the car of the compound form to be evaluated is an arbitrary form. If it is not a symbol, it must be a lambda expression, which looks like (lambdalambda-list form*).

This explains the error message we got above - (adder 3) is neither a symbol nor a lambda expression.

If we want to be able to use the symbol *my-fun* in the car of a compound form, we have to explicitely store something in its function cell (which is normally done for us by the macro defun):

;;; continued from above
CL-USER> (fboundp '*my-fun*)
NIL
CL-USER> (setf (symbol-function '*my-fun*) (adder 3))
#<CLOSURE (LAMBDA (X) :IN ADDER) {10099A5EFB}>
CL-USER> (fboundp '*my-fun*)
T
CL-USER> (*my-fun* 5)
8

Read the CLHS section about form evaluation for more.

Closures

Closures allow to capture lexical bindings:

(let ((limit 3)
      (counter -1))
    (defun my-counter ()
      (if (< counter limit)
          (incf counter)
          (setf counter 0))))

(my-counter)
0
(my-counter)
1
(my-counter)
2
(my-counter)
3
(my-counter)
0

Or similarly:

(defun repeater (n)
  (let ((counter -1))
     (lambda ()
       (if (< counter n)
         (incf counter)
         (setf counter 0)))))

(defparameter *my-repeater* (repeater 3))
;; *MY-REPEATER*
(funcall *my-repeater*)
0
(funcall *my-repeater*)
1
(funcall *my-repeater*)
2
(funcall *my-repeater*)
3
(funcall *my-repeater*)
0

See more on Practical Common Lisp.

`setf` functions

A function name can also be a list of two symbols with setf as the first one, and where the first argument is the new value:

(defun (setf <name>) (new-value <other arguments>)
  body)

This mechanism is particularly used for CLOS methods.

A silly example:

(defparameter *current-name* ""
  "A global name.")

(defun hello (name)
  (format t "hello ~a~&" name))

(defun (setf hello) (new-value)
  (hello new-value)
  (setf *current-name* new-value)
  (format t "current name is now ~a~&" new-value))

(setf (hello) "Alice")
;; hello Alice
;; current name is now Alice
;; NIL

Currying

Concept

A related concept is that of currying which you might be familiar with if you’re coming from a functional language. After we’ve read the last section that’s rather easy to implement:

CL-USER> (declaim (ftype (function (function &rest t) function) curry) (inline curry))
NIL
CL-USER> (defun curry (function &rest args)
           (lambda (&rest more-args)
	           (apply function (append args more-args))))
CURRY
CL-USER> (funcall (curry #'+ 3) 5)
8
CL-USER> (funcall (curry #'+ 3) 6)
9
CL-USER> (setf (symbol-function 'power-of-ten) (curry #'expt 10))
#<Interpreted Function "LAMBDA (FUNCTION &REST ARGS)" {482DB969}>
CL-USER> (power-of-ten 3)
1000

Note that the declaim statement above is just a hint for the compiler so it can produce more efficient code if it so wishes. Leaving it out won’t change the semantics of the function.

With the Alexandria library

Now that you know how to do it, you may appreciate using the implementation of the Alexandria library (in Quicklisp).

(ql:quickload :alexandria)

(defun adder (foo bar)
  "Add the two arguments."
  (+ foo bar))

(defvar add-one (alexandria:curry #'adder 1) "Add 1 to the argument.")

(funcall add-one 10)  ;; => 11

(setf (symbol-function 'add-one) add-one)
(add-one 10)  ;; => 11

Documentation

functions: http://www.lispworks.com/documentation/HyperSpec/Body/t_fn.htm#function
ordinary lambda lists: http://www.lispworks.com/documentation/HyperSpec/Body/03_da.htm
multiple-value-bind: http://clhs.lisp.se/Body/m_multip.htm

I Now Work Professionally in Common Lisp O_o

Mon, 20 May 2019 18:48:31 +0200

I’ve been paid to work on Common Lisp projects for a company for three months already. I didn’t expect it :) And we did hire !

My Github profile shows a good part of what my experience is. I am a regular “full stack developper”, with 7 years of professional experience. I worked on diverse Python and Javascript projects for huge to small companies, private and public. When I re-discovered Common Lisp, I saw it solved a lot of problems I had developing and deploying medium to large software, and it promised to solve all of them (we’re nearly there :D ). I started to write about my experience on this blog and I contributed to open source projects, of which a lot of documentation effort. It is this public activity that drew the attention of the guys at Atlas, who contacted me, without a job announce. We developed and we maintain a proprietary and successful web application that paid the bills, in Common Lisp of course, and we work hard on other projects, such as the Next browser.

So, I can only encourage you to start a Common Lisp project, to come enhance libraries and documentation and to write about it ! There are few official job announces, some are posted on reddit, and some jobs just won’t have a public announce. You’d better be ready.

For the curious, our web app is rather classic, it uses the Caveman web framework and is deployed on DigitalOcean. We do deploy with zero downtime, as CL permits, for trivial updates (we are more cautious otherwise). I’ll post its name in the comments if/when I know I can.

Happy lisping.

TIL how to interactively fix a failing test

Tue, 12 Mar 2019 13:10:27 +0200

I knew it was possible, but I got to try it recently.

Here I run a test with fiveam. It fails. I tell fiveam to enter the debugger on failures with

(setf 5am:*on-error* :debug)

so we have an immediate feedback and we can re-run the test from where it left off by choosing the appropriate restart.

Other test frameworks like Parachute allow that.

This is one of the things that make development in Common Lisp enjoyable and faster than with other workflows. Also, it’s built-in, there is no fancy editor plugin or configuration.

In the debugger:

<enter> on a backtrace shows more of it
v on a backtrace goes to the corresponding line or function.
more options with the menu.

These Years in Common Lisp 2018

Thu, 28 Feb 2019 14:42:46 +0100

It’s been already a little more than a year that I began my Lisp journey. I made quaterly news digests, mainly from reddit’s feed:

Q1 2018 - Q2 2018 - Q3 2018 - Q4 2018

Time has come for a yearly overview ! What happened in the Common Lisp world ? Are there (or groundbreaking promising useful fun) projects, articles, discussions, tutorials ?

No need to say, I won’t reference everything we find in the quaterly posts, which don’t list all new projects appearing on Quicklisp (we can find these in the monthly Quicklisp releases) or Github.

I hope this overview will sharpen your interest on what is in my opinion an under-sold and still very promising language and plateform, that I happen to like more and more (and sooo more than Python ;) ).

Happy discoveries.

Table of Contents

Documentation

Common Lisp’s online documentation could be more thorough and welcoming. Fortunately, a few of us revived some projects and work on it -my favourite project being the Common Lisp Coobook. This year, we got tutorials on:

Datastructures
Debugging, including how to interactively debug a spacecraft,
the Common Lisp Object System (CLOS)
Loop, iteration, mapping
Database access and persistence
Error and condition handling
Numbers and multidimensional arrays
Scripting and building self-contained executables
Working with types

along with many improvements on other pages, like on getting started and editor support.

Which brings me to it: the editors situation is much more open than you think:

The editor of choice is still Emacs with Slime (or Sly),
However, we can get started with Emacs and Lisp in 3 clicks with Portacle, a self-contained batteries-included sbcl-included portable Emacs tailored for CL,
For Vim and NeoVim we have SLIMV, VLIME, and plugins can be written for NeoVim using cl-neovim.
Or if we want an editor written in cl, there’s the self-contained Lem editor, which also works for Python, Go, Rust, Nim, Scheme, HTML, JSX, along with a directory mode, an experimental LSP mode, calc-mode, and more,
Not to forget that Mac Os X users can use the Clozure Common Lisp IDE
All editions of LispWorks (including the free) include the LW IDE
For users of Eclipse IDE, there is the Dandelion plugin
For popular editors, the experience is getting very good on Atom and the popular Visual Studio Code can be made to work with CL using cl-lsp.
We have an ipython-like REPL (cl-repl),
and for interactive notebooks, we have Jupyter kernels and yet another notebook (Darkmatter).

A very welcome improvement is the Common Lisp fundation’s website: https://common-lisp.net/ It got a massive update and is now attractive. We had http://lisp-lang.org/ (don’t miss its success stories section (did you know that pgloader was re-written from Python to CL ? :) )), but common-lisp.net was a googlers’ honey pot.

This website uses two “awesome” lists that were created or massively furnished last year:

the Awesome-CL list, updated with hundreds of commits, which hopefully makes for a more discoverable and appealing ecosystem, and
Awesome Lisp Companies: it was needed because Lispers didn’t know a lot of companies using CL appart from IRobot, Google’s ITA (powering Kayak, Orbitz and others), Grammatech, YCombinator, Siscog or other dead ones.

Other places to learn Common Lisp include:

cl-exercise: a Common Lisp Learning System running on browsers
coding schools, like Kattis
and competitive Programming websites like CodeForces, HackerEarth, HackerRank, and CodeChef.
lastly, Peter Norvig’s book Paradigms of Artificial Intelligence Programming is available on Github

We also regularly have new screencasts to enjoy:

a lot being from Baggers: he does the following and he streams live nearly weekly
- little bits of Lisp: short videos to learn Lisp basics
- lots of bits of Lisp: long videos to dive deep in advanced subjects (macros, CFFI,…)
- Pushing pixels with Lisp: mostly working with OpenGL
- and more !
Shinmera has lots of videos too, we can see him working on game engines, games, his libraries, Qt applications and more,
the CL study group (here, an introduction to Screamer, a non-deterministic programming library)

Implementations

Time is good for Common Lisp implementations. Most date back from decades and already proved what they can do (remember, SBCL is a descendant of the Lisp that went to space). Hence the lack of hype, IMO. Yet, many are in active development, and keep improving. As /u/defunkydrummer observed:

We are lucky to live in a time where Lisp development is still ongoing, many teams carrying the flag of open-source Lisp:

SBCL (new release today)

SICL (last commit 2 hours ago)

ECL (last commit, yesterday),

CLASP (last commit 2 days ago)

CCL (last commit 7 days ago),

CLISP (two weeks ago),

CMUCL (1 month ago)

ABCL (3 months ago)

SBCL has monthly releases. If you read the release notes, you might worry:

the amount of changes in each release is decreasing these years

but, as /u/baggers notes:

I think the commits tell a slightly different tale though. There is always a lot of background ‘making stuff better’ work than won’t appear as the explanation would either be ultra internal and specific or would be super vague and very similar each month (for example ‘stuff is slightly faster’).

For one that would be overly specific this one might make for a good example. It’s grand work, but doesn’t surface in any specific lisp feature, stuff is just better.

Furthermore, a maintainer:

Or the developers are too lazy to describe their changes.

which isn’t a good reason ;)

We got a new release of Corman Lisp, a high performance Windows/32bit specific implementation with a built in IDE,

we have CLASP, targetting C++ through LLVM (see “Lessons Learned Implementing Common Lisp with LLVM”), built with the Cleavir compiler, part of SICL, a very new implementation of Common Lisp with fresh ideas,

we have ABCL targetting the JVM, Embedable Common Lisp, without forgetting active commercial ones, like LispWorks and AllegroCL. While I’m at it, you might want to have a look at MOCL for IOs, Android and OSx.

We got a nice talk by Clozure Common Lisp’s maintainer: this Old Lisp (this one may be the second most used implementation, particularly good for development – super fast compilation times (I heard it compiles itself in seconds), advising, watched objects, its own IDE).

Last note, a SBCL maintainer started a RISC-V port: First RISCy Steps – Porting SBCL to the RISC-V

So: welcome to this new world. It’s bigger than I thought, for sure.

Projects

I only list some projects that can be of interest to anybody. For the full stuff see the quaterly posts !

New projects

Next browser 1.2.0 is out!: a browser exposing all its internals to CL. Be productive.
CANDO - A Computational Chemistry programming environment integrating Common Lisp and C++ based on the Jupyter notebook
Coalton, a dialect of ML embedded in Common Lisp (alpha)
Voxel game engine (Minecraft) - a Minecraft engine. Allows for interactive changes.
Emotiq - blockchain in Common Lisp
Temperance - logic programming (in development, reached v1.0.0)
MAGICL: Matrix Algebra proGrams In Common Lisp - Rigetti Computing (quantum computing)
SHCL: An Unholy Union of POSIX Shell and Common Lisp (reddit)
JSCL 0.7.0 now supports CLOS thanks to the work of vlad-km
cl-torrents 0.9 - readline interface and 1337x.to scraper - a simple tool to search for torrents on popular trackers. My first CL app. Web and GUI interfaces in the making.
Introducing Seed: An Interactive Software Environment in Common Lisp
Tovero is a 3D modeling system for Common Lisp
RMSBolt: See what your compiler is going inside of Emacs (has minimal support for Common Lisp)
pngload: A PNG (Portable Network Graphics) image format decoder
cl-vep: a video effects processor
algebraic-data-library
Petalisp: Elegant High Performance Computing
wiki-lang-detect: Text language identification using Wikipedia data
Dufy, a color library
ppath, a path manipulation library
cl-statistics.lisp
Powerlisp: A simple tool to automate your work with dmenu/rofi
json-mop: A metaclass for bridging CLOS and JSON objects
clsh: a set of Lispy bindings for running and composing *nix processes
filtered-functions - enables the use of arbitrary predicates for selecting and applying methods.

Web

Weblocks’ new quickstart - Weblocks is an isomorphic web frameworks that allows to write interactive web apps without writing Javascript (nor writing code that transpiles to JS). It is seeing a massive update right now. Being Lisp, we can build a self-contained executable of our web app, send it to the server, run it and see it from the outside.
three email libraries
reddit1.0 source code (comments), then Reddit’s code runs on SBCL. See also reddit.
Interactive Common Lisp code snippets in any web page
arboreta-wasm - Common Lisp tooling for WebAssembly

For web libraries, see https://github.com/CodyReichert/awesome-cl#network-and-internet

GUI

nodgui - yet another Tcl/Tk-based GUI package for Common Lisp (based on Ltk, with syntax sugar and more meta-widgets)
IUP bindings GUI stuff (in the works)
YstokWidgets Professional Edition
MIDGETS - A collection of CAPI widgets and utilities
subtext: A mostly-text-based UI bridges Common Lisp objects and runs of text. Minimal text-based user interface
ftw: Common Lisp Win32 GUI library
Cocoa interface code written in Lisp for use with Clozure Common Lisp
McCLIM 0.9.7 “Imbolc” release
Demo SBCL script using Gtk
Demo ABCL script using Java Swing

for GUI libraries: https://github.com/CodyReichert/awesome-cl#gui

Package management

Quicklisp is the de facto package manager, but new projects come to complement it and bypass its limitations:

the second version of Ultralisp is available - Ultralisp is an important project that fills a gap. It is a quicklisp distribution which updates every 5 minutes. It is also a Weblocks application!
quicksys - installs systems from multiple Quicklisp distributions.

For more options, see Qlot (install and pin libraries locally, like Python’s virtualenv) and Roswell.

Deployment

Apache Thrift gains CL support
s2i-lisp: Common Lisp + Quicklisp OpenShift Build Image
lisp-images: Docker images for common lisp development (with some others, see the awesome-list)
A docker container for CL development (also lisp-devel, CI on CL Cookbook)
Kubernetes Client Library for Common Lisp
Heroku buildpack for Common Lisp
cl-aws-custom-runtime - An example of using Common Lisp (SBCL) as a custom runtime on AWS lambda.
prometheus.cl - Prometheus.io client. Grafana dashboard for SBCL and Hunchentoot metrics (memory, threads, requests per second,…).

We can also deploy apps on Digital Ocean, and no need to say that deploying a self-contained executable is easy, connecting to a remote instance too.

Music

(re)Discoveries

Articles

and also

On games:

About Making Games in Lisp - Gamedev
Creating a (Non-Trivial) Lisp Game in 2018 (they just launched a Crowdfunding)
A Story of (defun games ())
Getting Started With trivial-gamekit

Other screencasts

Discussion

Learning Lisp

Common Lisp VS …

Enjoy the material, and see you soon !

thanks to /u/defunkydrummer for proofreading.

fix or improve this article on Gitlab.

Gray streams

Tue, 22 Jan 2019 07:51:49 +0100

This is a copy of http://www.nhplace.com/kent/CL/Issues/stream-definition-by-user.html with syntax highlighting.

FAILED Issue STREAM-DEFINITION-BY-USER (“Gray Streams”)

This is the writeup of failed issue STREAM-DEFINITION-BY-USER. Because it did not pass, it has no official standing other than as a historical document.

NOTES:

Several vendors have implemented this proposal anyway, so if you’d like to use this facility, you might check to see if it’s available in your implementation of choice in spite of not being part the “official” standard.
The facility described here is commonly referred to as “Gray Streams”, after David Gray, who wrote the proposal—please do not write this as “Grey Streams”!
Another facility of note that came later and may be available in some implementations is Franz’s “Simple Streams”. It is newer and addresses a broader range of issues, but its availability in this or that implementation may be different.

Click here to see my personal notes on this issue. –Kent Pitman (10-Mar-2001)

Issue: STREAM-DEFINITION-BY-USER

References: CLtL pages 329-332, 378-381, and 384-385.

Related issues: STREAM-INFO, CLOSED-STREAM-FUNCTIONS, STREAM-ACCESS, STREAM-CAPABILITIES

Category: ADDITION

Edit history: Version 1, 22-Mar-89 by David N. Gray

Status: For discussion and evaluation; not proposed for inclusion in the standard at this time.

Table of Contents

Problem description

Common Lisp does not provide a standard way for users to define their own streams for use by the standard I/O functions. This impedes the development of window systems for Common Lisp because, while there are standard Common Lisp I/O functions and there are beginning to be standard window systems, there is no portable way to connect them together to make a portable Common Lisp window system.

There are also many applications where users might want to define their own filter streams for doing things like printer device control, report formatting, character code translation, or encryption/decryption.

Proposal `stream-definition-by-user:generic-functions`

Overview

Define a set of generic functions for performing I/O. These functions will have methods that specialize on the stream argument; they would be used by the existing I/O functions. Users could write additional methods for them in order to support their own stream classes.

Define a set of classes to be used as the superclass of a stream class in order to provide some default methods.

## Classes

The following classes are to be used as super classes of user-defined stream classes. They are not intended to be directly instantiated; they just provide places to hang default methods.

fundamental-stream [Class]

This class is a subclass of `stream` and of `standard-object`.  `streamp`
will return true for an instance of any class that includes this.  (It
may return true for some other things also.)

fundamental-input-stream [Class]

A subclass of `fundamental-stream`.  Its inclusion causes `input-stream-p`
to return true.

fundamental-output-stream [Class]

A subclass of `fundamental-stream`.  Its inclusion causes `output-stream-p`
to return true.  Bi-direction streams may be formed by including both
`fundamental-output-stream` and `fundamental-input-stream`.

fundamental-character-stream [Class]

A subclass of `fundamental-stream`.  It provides a method for
`stream-element-type` which returns `character`.

fundamental-binary-stream [Class]

A subclass of `fundamental-stream`.  Any instantiable class that
includes this needs to define a method for `stream-element-type`.

fundamental-character-input-stream [Class]

Includes `fundamental-input-stream` and `fundamental-character-stream`.
It provides default methods for several generic functions used for
character input.

fundamental-character-output-stream [Class]

Includes `fundamental-output-stream` and `fundamental-character-stream`.
It provides default methods for several generic functions used for
character output.

fundamental-binary-input-stream [Class]

Includes `fundamental-input-stream` and `fundamental-binary-stream`.

fundamental-binary-output-stream [Class]

Includes `fundamental-output-stream` and `fundamental-binary-stream`.

Character input

A character input stream can be created by defining a class that includes fundamental-character-input-stream and defining methods for the generic functions below.

stream-read-char stream [Generic Function]

This reads one character from the stream.  It returns either a
character object, or the symbol :EOF if the stream is at end-of-file.
Every subclass of `fundamental-character-input-stream` must define a
method for this function.

Note that for all of these generic functions, the stream argument
must be a stream object, not T or NIL.

stream-unread-char stream character [Generic Function]

Un-does the last call to `stream-read-char`, as in `unread-char`.  Returns
NIL.  Every subclass of `fundamental-character-input-stream` must define
a method for this function.

stream-read-char-no-hang stream [Generic Function]

This is used to implement `read-char-no-hang`.  It returns either a
character, or NIL if no input is currently available, or :EOF if
end-of-file is reached.  The default method provided by
`fundamental-character-input-stream` simply calls `stream-read-char`; this
is sufficient for file streams, but interactive streams should define
their own method.

stream-peek-char stream [Generic Function]

Used to implement `peek-char`; this corresponds to peek-type of NIL.
It returns either a character or :EOF.  The default method
calls `stream-read-char` and `stream-unread-char`.

stream-listen stream [Generic Function]

Used by `listen`.  Returns true or false.  The default method uses
`stream-read-char-no-hang` and `stream-unread-char`.  Most streams should
define their own method since it will usually be trivial and will
always be more efficient than the default method.

stream-read-line stream [Generic Function]

Used by `read-line`.  A string is returned as the first value.  The
second value is true if the string was terminated by end-of-file
instead of the end of a line.  The default method uses repeated
calls to `stream-read-char`.

stream-clear-input stream [Generic Function]

Implements `clear-input` for the stream, returning NIL.  The default
method does nothing.

Character output

A character output stream can be created by defining a class that includes fundamental-character-output-stream and defining methods for the generic functions below.

stream-write-char stream character [Generic Function]

Writes character to the stream and returns the character.  Every
subclass of `fundamental-character-output-stream` must have a method
defined for this function.

stream-line-column stream [Generic Function]

This function returns the column number where the next character
will be written, or NIL if that is not meaningful for this stream.
The first column on a line is numbered 0.  This function is used in
the implementation of `pprint` and the FORMAT ~T directive.  For every
character output stream class that is defined, a method must be
defined for this function, although it is permissible for it to
always return NIL.

stream-start-line-p stream [Generic Function]

This is a predicate which returns T if the stream is positioned at the
beginning of a line, else NIL.  It is permissible to always return
NIL.  This is used in the implementation of `fresh-line`.  Note that
while a value of 0 from `stream-line-column` also indicates the
beginning of a line, there are cases where `stream-start-line-p` can be
meaningfully implemented although `stream-line-column` can't be.  For
example, for a window using variable-width characters, the column
number isn't very meaningful, but the beginning of the line does have
a clear meaning.  The default method for `stream-start-line-p` on class
`fundamental-character-output-stream` uses `stream-line-column`, so if
that is defined to return `nil`, then a method should be provided for
either `stream-start-line-p` or `stream-fresh-line`.

stream-write-string stream string &optional start end [Generic Function]

This is used by `write-string`.  It writes the string to the stream,
optionally delimited by start and end, which default to 0 and NIL.
The string argument is returned.  The default method provided by
`fundamental-character-output-stream` uses repeated calls to
`stream-write-char`.

stream-terpri stream [Generic Function]

Writes an end of line, as for `terpri`.  Returns NIL.  The default
method does (`stream-write-char` stream #\NEWLINE).

stream-fresh-line stream [Generic Function]

Used by `fresh-line`.  The default method uses `stream-start-line-p` and
`stream-terpri`.

stream-finish-output stream [Generic Function]

Implements `finish-output`.  The default method does nothing.

stream-force-output stream [Generic Function]

Implements `force-output`.  The default method does nothing.

stream-clear-output stream [Generic Function]

Implements `clear-output`.  The default method does nothing.

stream-advance-to-column stream column [Generic Function]

Writes enough blank space so that the next character will be written
at the specified column.  Returns true if the operation is
successful, or NIL if it is not supported for this stream.
This is intended for use by by `pprint` and FORMAT ~T.  The default
method uses `stream-line-column` and repeated calls to
`stream-write-char` with a #\SPACE character; it returns NIL if
`stream-line-column` returns NIL.

## Other functions

close stream &key abort [Generic Function]

The existing function `close` is redefined to be a generic function, but
otherwise behaves the same.  The default method provided by class
`fundamental-stream` sets a flag for `open-stream-p`.  The value returned
by `close` will be as specified by the issue `closed-stream-operations`.

open-stream-p stream [Generic Function]

This function [from proposal `stream-access]` is made generic.  A
default method is provided by class `fundamental-stream` which returns
true if `close` has not been called on the stream.

streamp object [Generic Function]

input-stream-p stream [Generic Function]

output-stream-p stream [Generic Function]

These three existing predicates may optionally be implemented as
generic functions for implementations that want to permit users to
define streams that are not `standard-object`s.  Normally, the default
methods provided by classes `fundamental-input-stream` and
`fundamental-output-stream` are sufficient.  Note that, for example,
(INPUT-STREAM-P x) is not equivalent to (TYPEP x
'FUNDAMENTAL-INPUT-STREAM) because implementations may have
additional ways of defining their own streams even if they don't
make that visible by making these predicates generic.

stream-element-type stream [Generic Function]

This existing function is made generic, but otherwise behaves the
same.  Class `fundamental-character-stream` provides a default method
which returns `character`.

pathname and truename are also permitted to be implemented as generic functions. There is no default method since these are not valid for all streams.

## Binary streams:

Binary streams can be created by defining a class that includes either
`fundamental-binary-input-stream` or `fundamental-binary-output-stream`
(or both) and defining a method for `stream-element-type` and for one or
both of the following generic functions.

stream-read-byte stream [Generic Function]

Used by `read-byte`; returns either an integer, or the symbol :EOF if the
stream is at end-of-file.

stream-write-byte stream integer [Generic Function]

Implements `write-byte`; writes the integer to the stream and returns
the integer as the result.

Rationale

The existing I/O functions cannot be made generic because, in nearly every case, the stream argument is optional, and therefore cannot be specialized. Therefore, it is necessary to define a lower-level generic function to be used by the existing function. It also isn’t appropriate to specialize on the second argument of print-object because it is a higher-level function – even when the first argument is a character or a string, it needs to format it in accordance with *PRINT-ESCAPE*.

In order to make the meaning as obvious as possible, the names of the generic functions have been formed by prefixing “stream-” to the corresponding non-generic function.

Having the generic input functions just return :EOF at end-of-file, with the higher-level functions handling the eof-error-p and eof-value arguments, simplifies the generic function interface and makes it more efficient by not needing to pass through those arguments. Note that the functions that use this convention can only return a character or integer as a stream element, so there is no possibility of ambiguity.

Functions stream-line-column, stream-start-line-p, and stream-advance-to-column may appear to be a reincarnation of the defeated proposal stream-info, but the motivation here is different. This interface needs to be defined if user-defined streams are to be able to be used by pprint and FORMAT ~T, which could be viewed as a separate question from whether the user can call then on system-defined streams.

Current practice

No one currently supports exactly this proposal, but this is very similar to the stream interface used in clue.

On descendants of the MIT Lisp Machine, streams can be implemented by users as either flavors, with methods to accept the various messages corresponding to the I/O operations, or as functions, which take a message keyword as their first argument.

Examples

  ;;;; Here is an example of how the default methods could be
  ;;;; implemented (omitting the most trivial ones):

  (defmethod STREAM-PEEK-CHAR ((stream fundamental-character-input-stream))
    (let ((character (stream-read-char stream)))
      (unless (eq character :eof)
	(stream-unread-char stream character))
      character))

  (defmethod STREAM-LISTEN ((stream fundamental-character-input-stream))
    (let ((char (stream-read-char-no-hang stream)))
      (and (not (null char))
	   (not (eq char :eof))
	   (progn (stream-unread-char stream char) t))))

  (defmethod STREAM-READ-LINE ((stream fundamental-character-input-stream))
    (let ((line (make-array 64 :element-type 'string-char
			    :fill-pointer 0 :adjustable t)))
      (loop (let ((character (stream-read-char stream)))
	      (if (eq character :eof)
		  (return (values line t))
		(if (eql character #\newline)
		    (return (values line nil))
		  (vector-push-extend character line)))))))

  (defmethod STREAM-START-LINE-P ((stream fundamental-character-output-stream))
    (equal (stream-line-column stream) 0))

  (defmethod STREAM-WRITE-STRING ((stream fundamental-character-output-stream)
				  string &optional (start 0)
				  (end (length string)))
    (do ((i start (1+ i)))
	((>= i end) string)
      (stream-write-char stream (char string i))))

  (defmethod STREAM-TERPRI ((stream fundamental-character-output-stream))
    (stream-write-char stream #\newline)
    nil)

  (defmethod STREAM-FRESH-LINE ((stream fundamental-character-output-stream))
    (if (stream-start-line-p stream)
	nil
      (progn (stream-terpri stream) t)))

  (defmethod STREAM-ADVANCE-TO-COLUMN ((stream fundamental-character-output-stream)
				       column)
    (let ((current (stream-line-column stream)))
      (unless (null current)
	(dotimes (i (- current column) t)
	  (stream-write-char stream #\space)))))

  (defmethod INPUT-STREAM-P ((stream fundamental-input-stream)) t)
  (defmethod INPUT-STREAM-P ((stream fundamental-output-stream))
    ;; allow the two classes to be mixed in either order
    (typep stream 'fundamental-input-stream))
  (defmethod OUTPUT-STREAM-P ((stream fundamental-output-stream)) t)
  (defmethod OUTPUT-STREAM-P ((stream fundamental-input-stream))
    (typep stream 'fundamental-output-stream))

  ;;;; Following is an example of how the existing I/O functions could
  ;;;; be implemented using standard Common Lisp and the generic
  ;;;; functions specified above.  The standard functions being defined
  ;;;; are in upper case.

  ;;  Internal helper functions

  (proclaim '(inline decode-read-arg decode-print-arg check-for-eof))
  (defun decode-read-arg (arg)
    (cond ((null arg) *standard-input*)
	  ((eq arg t) *terminal-io*)
	  (t arg)))

  (defun decode-print-arg (arg)
    (cond ((null arg) *standard-output*)
	  ((eq arg t) *terminal-io*)
	  (t arg)))

  (defun check-for-eof (value stream eof-errorp eof-value)
    (if (eq value :eof)
	(report-eof stream eof-errorp eof-value)
      value))

  (defun report-eof (stream eof-errorp eof-value)
    (if eof-errorp
	(error 'end-of-file :stream stream)
      eof-value))

  ;;;  Common Lisp input functions

  (defun READ-CHAR (&optional input-stream (eof-errorp t) eof-value recursive-p)
    (declare (ignore recursive-p)) ; a mistake in CLtL?
    (let ((stream (decode-read-arg input-stream)))
      (check-for-eof (stream-read-char stream) stream eof-errorp eof-value)))

  (defun PEEK-CHAR (&optional peek-type input-stream (eof-errorp t)
			eof-value recursive-p)
    (declare (ignore recursive-p))
    (let ((stream (decode-read-arg input-stream)))
      (if (null peek-type)
	  (check-for-eof (stream-peek-char stream) stream eof-errorp eof-value)
        (loop
	  (let ((value (stream-peek-char stream)))
	    (if (eq value :eof)
		(return (report-eof stream eof-errorp eof-value))
	      (if (if (eq peek-type t)
		      (not (member value '(#\space #\tab #\newline
					   #\page #\return #\linefeed)))
		    (char= peek-type value))
		  (return value)
		(stream-read-char stream))))))))

  (defun UNREAD-CHAR (character &optional input-stream)
    (stream-unread-char (decode-read-arg input-stream) character))

  (defun LISTEN (&optional input-stream)
    (stream-listen (decode-read-arg input-stream)))

  (defun READ-LINE (&optional input-stream (eof-error-p t)
			eof-value recursive-p)
    (declare (ignore recursive-p))
    (let ((stream (decode-read-arg input-stream)))
      (multiple-value-bind (string eofp)
	  (stream-read-line stream)
	(if eofp
	    (if (= (length string) 0)
		(report-eof stream eof-error-p eof-value)
	      (values string t))
	  (values string nil)))))

  (defun CLEAR-INPUT (&optional input-stream)
    (stream-clear-input (decode-read-arg input-stream)))

  (defun READ-CHAR-NO-HANG (&optional input-stream (eof-errorp t)
				eof-value recursive-p)
    (declare (ignore recursive-p))
    (let ((stream (decode-read-arg input-stream)))
      (check-for-eof (stream-read-char-no-hang stream)
		     stream eof-errorp eof-value)))

  ;;;  Common Lisp output functions

  (defun WRITE-CHAR (character &optional output-stream)
     (stream-write-char (decode-print-arg output-stream) character))

  (defun FRESH-LINE (&optional output-stream)
    (stream-fresh-line (decode-print-arg output-stream)))

  (defun TERPRI (&optional output-stream)
    (stream-terpri (decode-print-arg output-stream)))

  (defun WRITE-STRING (string &optional output-stream &key (start 0) end)
    (stream-write-string (decode-print-arg output-stream) string start end))

  (defun WRITE-LINE (string &optional output-stream &key (start 0) end)
    (let ((stream (decode-print-arg output-stream)))
      (stream-write-string stream string start end)
      (stream-terpri stream)
      string))

  (defun FORCE-OUTPUT (&optional stream)
    (stream-force-output (decode-print-arg stream)))

  (defun FINISH-OUTPUT (&optional stream)
    (stream-finish-output (decode-print-arg stream)))

  (defun CLEAR-OUTPUT (&optional stream)
    (stream-clear-output (decode-print-arg stream)))

  ;;;  Binary streams

  (defun READ-BYTE (binary-input-stream &optional (eof-errorp t) eof-value)
    (check-for-eof (stream-read-byte binary-input-stream)
		   binary-input-stream eof-errorp eof-value))

  (defun WRITE-BYTE (integer binary-output-stream)
    (stream-write-byte binary-output-stream integer))

  ;;;  String streams

  (defclass string-input-stream (fundamental-character-input-stream)
    ((string :initarg :string :type string)
     (index :initarg :start :type fixnum)
     (end :initarg :end :type fixnum)
     ))

  (defun MAKE-STRING-INPUT-STREAM (string &optional (start 0) end)
    (make-instance 'string-input-stream :string string
		   :start start :end (or end (length string))))

  (defmethod stream-read-char ((stream string-input-stream))
    (with-slots (index end string) stream
      (if (>= index end)
	  :eof
	(prog1 (char string index)
	       (incf index)))))

  (defmethod stream-unread-char ((stream string-input-stream) character)
    (with-slots (index end string) stream
      (decf index)
      (assert (eql (char string index) character))
      nil))

  (defmethod stream-read-line ((stream string-input-stream))
    (with-slots (index end string) stream
      (let* ((endline (position #\newline string :start index :end end))
	     (line (subseq string index endline)))
	(if endline
	    (progn (setq index (1+ endline))
		   (values line nil))
	  (progn (setq index end)
		 (values line t))))))

  (defclass string-output-stream (fundamental-character-output-stream)
    ((string :initform nil :initarg :string)))

  (defun MAKE-STRING-OUTPUT-STREAM ()
    (make-instance 'string-output-stream))

  (defun GET-OUTPUT-STREAM-STRING (stream)
    (with-slots (string) stream
      (if (null string)
	  ""
	(prog1 string (setq string nil)))))

  (defmethod stream-write-char ((stream string-output-stream) character)
    (with-slots (string) stream
      (when (null string)
	(setq string (make-array 64. :element-type 'string-char
				 :fill-pointer 0 :adjustable t)))
      (vector-push-extend character string)
      character))

  (defmethod stream-line-column ((stream string-output-stream))
    (with-slots (string) stream
      (if (null string)
	  0
	(let ((nx (position #\newline string :from-end t)))
	  (if (null nx)
	      (length string)
	    (- (length string) nx 1))
	  ))))

Cost to Implementors

Given that CLOS is supported, adding the above generic functions and methods is easy, since most of the code is included in the examples above. The hard part would be re-writing existing I/O functionality in terms of methods on these new generic functions. That could be simplified if methods can be defined to forward the operations to the old representation of streams. For a new implementation, the cost could be zero since an approach similar to this would likely be used anyway.

Cost to Users

None; this is an upward-compatible addition. Users won’t even need to know anything about this unless they actually need this feature.

Cost of non-adoption

Development of portable I/O extensions will be discouraged.

Performance impact

This shouldn’t affect performance of new implementations (assuming an efficient CLOS implementation), but it could slow down I/O if it were clumsily grafted on top of an existing implementation.

Benefits

A broader domain of programs that can be written portably.

Esthetics

This seems to be a simple, straight-forward approach.

Discussion

This proposal incorporates suggestions made by several people in response to an earlier outline. So far, no one has expressed opposition to the concept. There are some differences of opinion about whether certain operations should have default methods or required methods: stream-listen, stream-read-char-no-hang, stream-line-column, and stream-start-line-p.

An experimental prototype of this has been successfully implemented on the Explorer.

This proposal does not provide sufficient capability to implement forwarding streams such as for make-synonym-stream, make-broadcast-stream, make-concatenated-stream, make-two-way-stream, or make-echo-stream. The generic function approach does not lend itself as well to that as a message passing model where the intermediary does not need to know what all the possible messages are. A possible way of extending it for that would be to define a class

    (defclass stream-generic-function (standard-generic-function) ())

to be used as the :generic-function-class option for all of the I/O generic functions. This would then permit doing something like

  (defmethod no-applicable-method ((gfun stream-generic-function) &rest args)
    (if (streamp (first args))
	(apply #'stream-operation-not-handled (first args) gfun (rest args))
      (call-next-method)))

where stream-operation-not-handled is a generic function whose default method signals an error, but forwarding streams can define methods that will create a method to handle the unexpected operation. (Perhaps no-applicable-method should be changed to take two required arguments since all generic functions need at least one required argument, and that would make it unnecessary to define a new generic function class just to be able to write this one method.)

Another thing that is not addressed here is a way to cause an instance of a user-defined stream class to be created from a call to the open function. That should be part of a separate issue for generic functions on pathnames. If that capability were available, then pathname and truename should be required to be generic functions.

An earlier draft defined just two classes, fundamental-input-stream and fundamental-output-stream, that were used for both character and binary streams. It isn’t clear whether that simple approach is sufficient or whether the larger set of classes is really needed.

These Months in Common Lisp Q4 2018

Tue, 15 Jan 2019 20:01:30 +0200

I wanted to do this for a year and here we are ! I don’t think I’ll carry on, with this format at least.

If I missed anything crucial: you have comments and PRs: https://gitlab.com/lisp-journey/lisp-journey.gitlab.io/

Happy (re)discoveries !

Documentation

Announcements

various SBCL releases (from 1.4.13 to 1.4.15) (not many changes you think ?)
Release of Corman Lisp 3.1 (Windows), with personal notes (also this thread)
European Lisp Symposium 2019 - Call for Papers
Dandelion, Eclipse IDE plugin, updated for SBCL 1.4.10
December Quicklisp update, october
Common Lisp is now available to use at Kattis

Projects

GUI:

Developer utilities:

s2i-lisp: Common Lisp + Quicklisp OpenShift Build Image
lisp-images: Docker images for common lisp development (with some others, see the awesome-list)
Quicklisp.nvim - Common Lisp package management within Neovim

New releases:

(re)discoveries:

Articles

GUIs:

On games:

Discussion

Screencasts

Common Lisp VS …

How did the Common Lisp community survived without the equivalent of clojure.spec ?

Why Deftask Chose Common Lisp

Fri, 11 Jan 2019 12:21:30 +0100

We heard about Deftask, a task management app for teams, a few days ago, in an article about the internals of the Chronicity library. Deftask uses Common Lisp for its backend and its command-line app. This reasonates with the fact that Turtl doesn’t use CL anymore. So I asked Deftask’s author: why did you go with CL ?

More than anything else, I think its down to fun and productivity. I feel that I am at my most productive when I writing CL in Emacs+SLIME. It is probably down to the edit-compile-debug cycle which is much shorter in CL compared to other languages. I originally worked on CL way back in 2006-08 when I was with Cleartrip. Since then, I have worked on a number of platforms (frontend js, node, iOS, C, Java, etc.) but always wanted to go back to writing CL full time.

So when I left my last job a little over a year back, I had already made up my mind that the next thing I build would be in CL.

The lack of libraries (or rather, well supported libraries) is a problem, but honestly after over eight years of not working on Lisp, it doesn’t bother me much.

Did you already build software/services in CL, apart the libraries we can see on your github profile ?

As I mentioned I worked at Cleartrip for a little over two years. I was part of the team that managed the flight search engine, Unfortunately most of what we did there is gone forever. A small sliver of our work there resulted in https://lisper.in/restarts (backstory: https://www.reddit.com/r/lisp/comments/7k85sf/a_tutorial_on_conditions_and_restarts/).

Did you have enough libraries to build your service ? How’s deployment and maintenance going ?

Well you could say I had enough libraries. Although I still ended up writing a mini web framework on top of Hunchentoot. Another thing I wrote is my own Lisp-to-SQL converter. Hope to open source both of these one day. Apart from that I use drakma, postmodern, djula, plump, lparallel, cl-json to name a few (alongwith the usual suspects like alexandria and local-time).

Deployment and maintenance is extremely simple - I just update the relevant git branch and restart the service. At some point when the restarts become costly I might add the ability to reload changed code in the running service.

Thanks to him !

Below the backstory from reddit:

Hey that was written by me! (Aside: I redirected the page to point to the latest version of this post on my new blog)

Fun fact: I wrote this post way back in 2008 while working for an online travel portal. This was based on some actual work we’d done there. At that time, flight travel in India had started to boom. This I think went hand in hand with a bunch of online travel companies (including ours) gaining a lot of momentum.

To be more competitive, a couple of airlines decided that they wanted to introduce new discounted fares much more frequently than they were doing earlier. The only problem was that they were unable to upload their wonky fare rules in the GDS properly, so they started distributing excel sheets to travel agents with manual instructions on how to apply them.

So our business team starts sending these sheets over to us, and initially, the frequency was low so we just manually hard coded these rules. However then they started sending these sheets every week or so which made our life hell. So we asked asked our business team to “translate” the airline’s excel sheets and instructions into a csv, which was subsequently interpreted by a simple rules engine that we wrote.

The only problem? Well, as anyone who’s dealt with manually created CSVs will tell you, there were a lot of errors. This didn’t really help matters much. We then added a couple of restarts to our CSV parser which allowed us to correct these issues interactively. This made life much better for us – it was a lot easier than, say, getting a list of errors in a terminal and switching back and forth between the terminal window and the editor to correct them.

Later on we plonked the CSV parser behind hunchentoot and asked our bizdev guy to upload the file there. A handle-bind around the parser collected all the errors in one go and showed them in a nicely formatted way in the browser (see the last section of the post). And so it was no longer our problem :-)

Eventually these airlines decided they wanted to update fare rules daily. Thankfully our “business rules engine” was upto the task. Due to automatic feedback, our friend in bizdev became an expert at uploading the fare rules as soon as they came in. And for quite some time, we were the only ones who could show these cheap fares within minutes of them coming in (if I remember correctly, other portals would take hours to upload the same rules).

Ours was a small team, and we had to manage this in addition to a lot of other things. If it weren’t for CL’s condition system, I doubt we could have solved this as smoothly as we did. In particular, interactive restarts allowed us (devs) to correct CSV errors without wasting a lot of our own time, and without needing to build a full-fledged UI for a non-dev. And when the time did come for a UI, it was dead easy to write a web fontend on top of it.

Q: Did you ever work or need an optimal solution to the Tavelling Salesman Problem?

Nope. Domestic flight travel in India is simple… point to point or max over two legs. For international flights we used QPX.

(QPX was ITA software’s flight search engine. Probably among the biggest software systems written in Lisp. It now powers Google Flights (I think).)

Why Turtl Switched From CL to Js

Fri, 11 Jan 2019 10:21:18 +0100

Turtl is a very well done, secure collaborative notebook web app.

https://turtlapp.com

Its api backend is built in Common Lisp:

https://github.com/turtl/api/

It is based on many async libraries the developer wrote for Turtl, like the Wookie async HTTP server.

“is” ? No, was :/ Even though this repository is still maintained (latest commit: 2nd of december 2018), it is deprecated and the new server is written in NodeJS. I asked Andrew for the reasons behind this, here’s his answer.

(in a hurry to spread FUD ? Don’t miss this posts’s twin, Why Deftask chose Common Lisp ;) See also some companies using Common Lisp)

It was not an easy decision to replace CL with javascript. I find CL to be elegant, fast, stable, and above all else, easy to debug (having the ability to rewrite the program as it’s running is insanely useful). In fact, I still miss lisp and ask myself all the time if I made the right choice.

I think I did, though.

The path I went with CL was a hard one. I wanted to be able to use asynchronous programming because for the type of work I do (a lot of making APIs that talk to other services with very little CPU work) it’s hard to get much more performant. So I embarked on creating cl-async/cl-libuv (originally cl-libevent) and wookie, along with a few other drivers. Everything I built worked great (and still works great, as far as I can tell) however when things did go wrong, there was nobody to run ideas by and nobody to really help me…I had built all these things myself, and I also had to be responsible for fixing them when they broke. On top of having to maintain everything (and it did break from time to time) there is not much in the way of packages to help me out. For instance, there’s a package to upload files to S3, but it’s not async at all…I had to build this from scratch. There are more cases of this as well.

With CL, it felt like I was constantly fighting the tide. I was constantly battling just to get basic things working that are already solved problems in other languages (Node).

There was help and support from the community along the way, but I was mostly fighting it alone. I think the straw that broke the camel’s back was when a few people started making copycat projects that added no real value (other than benchmarking fast) but stole mindshare from all the work I had put in. It was the “well, that project is not exactly what I want so I’ll make my own from scratch” mindset that everyone always warned about when I was starting with CL (but I ignored). I had really hoped the community would have helped propel the async ecosystem I was building forward, but I just don’t think there’s enough people using CL for that to happen.

So between having to maintain everything myself and people putting out worthless copycat projects that ended up going nowhere, I didn’t have the energy anymore.

Honestly, it took me about a week of work, just nights and weekends, to reprogram the server in javascript. Granted, most of the “how should this work?” architecture stuff was already done so it was more of a rewrite than a build-from-scratch situation, but Node is fast to build APIs in. I’m decently fluent in javascript and the amount packages available is so immense that it just made sense.

On top of being fast to build in, it’s a well-traveled road. I don’t have people emailing me six times a day asking how to install the server like I did with CL. I don’t have to make weird custom loaders to run the app on any hosting providers…everyone supports Node. I don’t have to deal with weird FFI errors or libuv nuances. I don’t have to deal with quicklisp’s “all or nothing” packaging that doesn’t support version pinning. I don’t have to restart the server every 20 days because of some memory leak I have yet to track down somewhere between cl-libuv, cl-async, wookie, and turtl. There’s a whole set of bullshit I just don’t have to deal with anymore.

So I do miss lisp. I’d eventually like to build more things in it (like games). But I don’t think I’ll ever touch web stuff in CL again, and the whole journey left a bitter taste in my mouth. Sure I could have dropped the async thing and just done a threaded server in hunchentoot and cl-postgres. But once I decided I was going to reprogram everything anyway, it just made sense to go with Node.

I took on more work than I could realistically manage, and hoped that the community would help…but the CL community is small enough that it was a losing bet and I got burned out.

Hopefully none of this discourages you. CL is a great language. The community is a mix though. Some of the people in the community are smart and dedicated, and work on cool projects at a pace they can maintain. You won’t see articles about these projects, and many will only have a handful of stars on Github (don’t measure CL projects by stars). Seek these projects and these people out, and build things with them. There is a quiet corner of the internet, with a handful of people building amazing things in lisp.

Before commenting on this, I think we must realize what he achieved, and that he went the hard way.

Now don’t miss Why Deftask chose Common Lisp !

Introducing Replic: an executable and a library to build a readline app in no time

Wed, 09 Jan 2019 14:41:57 +0100

When I started dabbling in CL, I tried to build a readline application to see how it goes. I found cl-readline (I’m only the new maintainer) and it went smoothly. So I built a second and a third app, and found many things to refactor and provide out of the box: now comes replic.

It comes as a library (now in Quicklisp, since 2018-01) and as an executable. The library does the following for you:

it builds the repl loop, catches a C-c, a C-d, errors,
it asks confirmation to quit,
it asks depending on a .conf and a lispy config file,
it reads parameters from a config file,
it prints the help of all or one command (with optional highlighting),
and more importantly it handles the completion of commands and of their arguments.

For example, instead of this “repl” loop:

  (handler-case
      (do ((i 0 (1+ i))
           (text "")
           (verb "")
           (function nil)
           (variable nil)
           (args ""))
          ((string= "quit" (str:trim text)))

        (handler-case
            (setf text
                  (rl:readline :prompt (prompt)
                               :add-history t))
          (#+sbcl sb-sys:interactive-interrupt ()
                  (progn
                    (when (confirm)
                      (uiop:quit)))))

        (if (string= text "NIL")
            ;; that's a C-d, a blank input is just "".
            (when (confirm)
              (uiop:quit)))

        (unless (str:blank? text)
          (setf verb (first (str:words text)))
          (setf function (if (replic.completion:is-function verb)
                             ;; might do better than this or.
                             (replic.completion:get-function verb)))
          (setf variable (if (replic.completion:is-variable verb)
                             (replic.completion:get-variable verb)))
          (setf args (rest (str:words text)))


          (if (and verb function)
              (handler-case
                  ;; Call the function.
                  (apply function args)
                (#+sbcl sb-sys:interactive-interrupt (c)
                        (declare (ignore c))
                        (terpri))
                (error (c) (format t "Error: ~a~&" c)))

              (if variable
                  (format t "~a~&" (symbol-value variable))
                  (format t "No command or variable bound to ~a~&" verb)))

          (finish-output)

          (when (and *history*
                     *write-history*)
            (rl:write-history "/tmp/readline_history"))
          ))

    (error (c)
      (format t "~&Unknown error: ~&~a~&" c)))

you call:

(replic:repl)

To turn all exported functions of a package into commands, use

(replic:functions-to-commands :my-package)

and you can find them into the readline app.

Setting the completion of commands is easy, we use (replic.completion:add-completion "my-function" <list-or-lambda>. For example:

(in-package :replic.user)

(defparameter *names* '()
  "List of names (string) given to `hello`. Will be autocompleted by `goodbye`.")

(defun hello (name)
  "Takes only one argument. Adds the given name to the global
  `*names*` variable, used to complete arguments of `goodbye`.
  "
  (format t "hello ~a~&" name)
  (push name *names*))

(defun goodbye (name)
  "Says goodbye to name, where `name` should be completed from what was given to `hello`."
  (format t "goodbye ~a~&" name))

(replic.completion:add-completion "goodbye" (lambda () *names*))

(export '(hello goodbye))

This example can be used with the executable. What it does is read your code from a lisp file (~/.replic.lisp or an argument on the command line) and it turns the exported functions into commands, for which we can specify custom completion.

For more details, see the readme.

I use this currently in three apps of mine (like cl-torrents). It’s simple. It could be more: it could infer the arguments’ type, do fuzzy completion, maybe integrate a Lisp editor (Lem) or a lispy shell (shcl), separate the commands in apps, expose hooks, have a set of built-in shell related utilities, highlight the input line, it could be web-based,…

For now it’s going smoothly.

I’ll finish by recalling that it’s amazing to be able to ship self-contained executables to users !

Debugging in Common Lisp

Thu, 20 Dec 2018 12:26:02 +0100

You entered this new world of Lisp and now wonder: how can we debug what’s going on ? How is it more interactive than in other platforms ? What does bring the interactive debugger appart from stacktraces ?

note: this tutorial is available on the Common Lisp Cookbook and it will receive updates there.

If you want step-by-step examples of interactive debugging with nice screenshots and gifs, see the blog posts in the References section below.

Table of Contents

Print debugging

Well of course we can use the famous technique of “print debugging”. Let’s just recap a few print functions.

print works, it prints a READable representation of its argument, which means what is printed can be read back in by the Lisp reader.

princ focuses on an aesthetic representation.

format t "~a" …), with the aesthetic directive, prints a string (in t, the standard output stream) and returns nil, whereas format nil … doesn’t print anything and returns a string. With many format controls we can print several variables at once.

Logging

Logging is a good evolution from print debugging ;)

log4cl is the popular, de-facto logging library but it isn’t the only one. Download it:

(ql:quickload :log4cl)

and let’s have a dummy variable:

(defvar *foo* '(:a :b :c))

We can use log4cl with its log nickname, then it is as simple to use as:

(log:info *foo*)
;; <INFO> [13:36:49] cl-user () - *FOO*: (:A :B :C)

We can interleave strings and expressions, with or without format control strings:

(log:info "foo is " *foo*)
;; <INFO> [13:37:22] cl-user () - foo is *FOO*: (:A :B :C)
(log:info "foo is ~{~a~}" *foo*)
;; <INFO> [13:39:05] cl-user () - foo is ABC

With its companion library log4slime, we can interactively change the log level:

globally,
per package,
per function,
and by CLOS methods and CLOS hierarchy (before and after methods).

It is very handy, when we have a lot of output, to turn off the logging of functions or packages we know to work, and thus narrowing our search to the right area. We can even save this configuration and re-use it in another image, be it on another machine.

We can do all this through commands, keyboard shortcuts and also through a menu or mouse clicks.

We invite you to read log4cl’s readme.

Using the powerful REPL

Part of the joy of Lisp is the excellent REPL. Its existence usually delays the need to use other debugging tools, if it doesn’t annihilate them for the usual routine.

As soon as we define a function, we can try it in the REPL. In Slime, compile a function with C-c C-c (the whole buffer with C-c C-k), switch to the REPL with C-c C-z and try it. Eventually enter the package you are working on with (in-package :your-package).

The feedback is immediate. There is no need to recompile everything, nor to restart any process, nor to create a main function and define command line arguments for use in the shell (we can do this later on when needed).

We usually need to create some data to test our function(s). This is a subsequent art of the REPL existence and it may be a new discipline for newcomers. A trick is to write the test data alongside your functions but inside a #+nil declaration so that only you can manually compile them:

    #+nil
    (progn
       (defvar *test-data* nil)
       (setf *test-data* (make-instance 'foo …)))

When you load this file, *test-data* won’t exist, but you can manually create it with a C-c C-c away.

We can define tests functions like this.

Some do similarly inside #| … |# comments.

All that being said, keep in mind to write unit tests when time comes ;)

Inspect and describe

These two commands share the same goal, printing a description of an object, inspect being the interactive one.

(inspect *foo*)

The object is a proper list of length 3.
0. 0: :A
1. 1: :B

2. 2: :C
> q

We can also, in editors that support it, right-click on any object in the REPL and inspect them. We are presented a screen where we can dive deep inside the data structure and even change it.

Let’s have a quick look with a more interesting structure, an object:

(defclass foo ()
    ((a :accessor foo-a :initform '(:a :b :c))
     (b :accessor foo-b :initform :b)))
;; #<STANDARD-CLASS FOO>
(make-instance 'foo)
;; #<FOO {100F2B6183}>

We right-click on the #<FOO object and choose “inspect”. We are presented an interactive pane (in Slime):

#<FOO {100F2B6183}>
--------------------
Class: #<STANDARD-CLASS FOO>
--------------------
 Group slots by inheritance [ ]
 Sort slots alphabetically  [X]

All Slots:
[ ]  A = (:A :B :C)
[ ]  B = :B

[set value]  [make unbound]

When we click or press enter on the line of slot A, we inspect it further:

#<CONS {100F5E2A07}>
--------------------
A proper list:
0: :A
1: :B
2: :C

The interactive debugger

Whenever an exceptional situation happens (see error handling), the interactive debugger pops up.

It presents the error message, available actions (restarts), and the backtrace. A few remarks:

the restarts are programmable, we can create our owns,
in Slime, press v on a stacktrace to be redirected to the source file at the right line,
hit enter on a frame for more details,
we can explore the functionnality with the menu that should appear in our editor. See below in “break” section for a few more commands (eval in frame, etc).

Usually your compiler will optimize things out and this will reduce the amount of information available to the debugger. For example sometimes we can’t see intermediate variables of computations. You might want to print a function argument (with e to “eval in frame”, see below), but you keep getting a Variable XYZ is unbound error.

To fix this, we have to change the optimization choices with declaim, at the beginning of the file:

(declaim (optimize (speed 0) (space 0) (debug 3)))

or with declare, inside a defun:

(defun my-fun (xyz)
  (declare (optimize (debug 3)))
  …)

and recompile the code,

or in Emacs and Slime there’s a shortcut for this, just use C-u C-c C-c (universal argument, “slime compile defun” with max debug settings).

Now you should be able to see local variables such asxyz.

Trace

trace allows us to see when a function was called, what arguments it received, and the value it returned.

(defun factorial (n)
  (if (plusp n)
    (* n (factorial (1- n)))
    1))

(trace factorial)

(factorial 2)
  0: (FACTORIAL 3)
    1: (FACTORIAL 2)
      2: (FACTORIAL 1)
        3: (FACTORIAL 0)
        3: FACTORIAL returned 1
      2: FACTORIAL returned 1
    1: FACTORIAL returned 2
  0: FACTORIAL returned 6
6

(untrace factorial)

To untrace all functions, just evaluate (untrace).

In Slime we also have the shortcut C-c M-t to trace or untrace a function.

If you don’t see recursive calls, that may be because of the compiler’s optimizations. Try this before defining the function to be traced:

(declaim (optimize (debug 3)))

The output is printed to *trace-output* (see the CLHS).

In Slime, we also have an interactive trace dialog with M-x slime-trace-dialog bound to C-c T.

Tracing method invocation

In SBCL, we can use (trace foo :methods t) to trace the execution order of method combination (before, after, around methods). For example:

(trace foo :methods t)

(foo 2.0d0)
  0: (FOO 2.0d0)
    1: ((SB-PCL::COMBINED-METHOD FOO) 2.0d0)
      2: ((METHOD FOO (FLOAT)) 2.0d0)
        3: ((METHOD FOO (T)) 2.0d0)
        3: (METHOD FOO (T)) returned 3
      2: (METHOD FOO (FLOAT)) returned 9
      2: ((METHOD FOO :AFTER (DOUBLE-FLOAT)) 2.0d0)
      2: (METHOD FOO :AFTER (DOUBLE-FLOAT)) returned DOUBLE
    1: (SB-PCL::COMBINED-METHOD FOO) returned 9
  0: FOO returned 9
9

See the CLOS section for a tad more information.

Step

step is an interactive command with similar scope than trace. This:

(step (factorial 2))

gives an interactive pane with the available restarts:

Evaluating call:
  (FACTORIAL 2)
With arguments:
  2
   [Condition of type SB-EXT:STEP-FORM-CONDITION]

Restarts:
 0: [STEP-CONTINUE] Resume normal execution
 1: [STEP-OUT] Resume stepping after returning from this function
 2: [STEP-NEXT] Step over call
 3: [STEP-INTO] Step into call
 4: [RETRY] Retry SLIME REPL evaluation request.
 5: [*ABORT] Return to SLIME's top level.
 --more--

Backtrace:
  0: ((LAMBDA ()))
  1: (SB-INT:SIMPLE-EVAL-IN-LEXENV (LET ((SB-IMPL::*STEP-OUT* :MAYBE)) (UNWIND-PROTECT (SB-IMPL::WITH-STEPPING-ENABLED #))) #S(SB-KERNEL:LEXENV :FUNS NIL :VARS NIL :BLOCKS NIL :TAGS NIL :TYPE-RESTRICTIONS ..
  2: (SB-INT:SIMPLE-EVAL-IN-LEXENV (STEP (FACTORIAL 2)) #<NULL-LEXENV>)
  3: (EVAL (STEP (FACTORIAL 2)))

Stepping is useful, however it may be a sign that you need to simplify your function.

Break

A call to break makes the program enter the debugger, from which we can inspect the call stack.

Breakpoints in Slime

Look at the SLDB menu, it shows navigation keys and available actions. Of which:

e (sldb-eval-in-frame) prompts for an expression and evaluates it in the selected frame. This is how we can explore our intermediate variables.
d is similar with the addition of pretty printing the result.

Once we are in a frame and detect a suspicious behavior, we can even re-compile a function at runtime and resume the program execution from where it stopped (using the “step-continue” restart).

Advise and watch

advise and watch are available in some vendor implementations, like LispWorks. They are not available in SBCL. advise allows to modify a function without changing its source, or to do something before or after its execution, like CLOS’ method combination (befor, after around methods).

watch allows to specify variables to be displayed in some GUI during the program execution.

Unit tests

Last but not least, automatic testing of functions in isolation might be what you’re looking for ! See the testing section and a list of test frameworks and libraries.

References

“How to understand and use Common Lisp”, chap. 30, David Lamkins (book download from author’s site)
Malisper: debugging Lisp series
Two Wrongs: debugging Common Lisp in Slime

One liner, Git from Lisp: commit every file of this repository

Tue, 04 Dec 2018 21:49:06 +0100

Hey, pardon this very short post, it’s just for the pleasure of blogging, and to balance the usual lengthy ones.

I wanted to commit, one by one, every file of the current directory (it’s useless, don’t ask).

I use legit as the interface to Git, and this one-liner:

(dolist (file (uiop:directory-files "./"))
   (legit:git-add :paths (pathname file))
   (legit:git-commit :files (pathname file) :message (format nil "add ~a" (file-namestring file))))

I guessed the :paths and :files arguments with Slime’s command argument list which appears in the modline, I wanted a function to convert a /full/path/file.cl to a name file.cl and tried the completion for file-… and found the right thing without effort. I saw on the complete documentation that legit:commit wanted a repository object as first argument, which makes sense, but legit:git-commit doesn’t and I just iterate on the current working directory (btw change it in Slime with the ,cd command) so it was shorter for me.

Just a one liner.

Oh my god, I didn’t know we can do this in Lisp !

Of course we can :p

Overview of Documentation Generators

Wed, 07 Nov 2018 18:34:11 +0100

I have a simple need: I’d like to generate an html documentation from my code. What options do we have ?

I searched for “documentation tool” on Quickdocs: http://quickdocs.org/search?q=documentation%20tool, from which I remove old ones (clod, qbook, manifest).

I had two pure Lisp solutions working out of the box, two more are of interest, and there’s another non-Lisp of interest.

update: just found out that qbook (github mirror)is used for the documentation of Fiveam, which is pretty nice: https://common-lisp.net/project/fiveam/docs/index.html It can produce html and latex. It uses docstrings and comments that start with 4 commas to structure the page (exple in source that gives this). Running it has a lot of asdf deprecation warnings and it did not work out of the box for me (“The slot IT.BESE.QBOOK::GENERATOR is unbound in the object”).

Codex

Codex produces nice html but isn’t automatic.

https://github.com/CommonDoc/codex (by @eudoxia)
example: https://commondoc.github.io/codex/docs/tutorial.html
input fromat: scriba by default. No more format it seems.
output: multiple html files.
rendering: modern, colored, light, TOC on the side
granularity: good
link to CLHS: yes
totally automatic: no (one needs to create the documentation structure in manual.lisp, AFAIU).
used in the wild: yes

Getting started: write a simple docs/manifest.lisp and docs/manual.lisp.

Am I mistaking ? It’s exclusively manual. We must supply every function/generic function/method/class/macro to document in manual.lisp.

Coo

Coo is a new tool in town, it works out of the box and it is actively developed !

https://github.com/fisxoj/coo
https://fisxoj.github.io/coo/
input: docstrings in rst
output: multiple html
rendering: black & white, no TOC (active development)
links to CLHS: yes, since yesterday :)
granularity: doesn’t show class slots, doesn’t show generic functions.
used in the wild: coo no, more probably cl-docutils.

Based on cl-docutils.

Displays the functions’ documentation following their order in source (or not ? I saw exceptions).

It’s straightforward:

(coo:document-system :my-system)

and it produces html into docs/.

Staple (doesn’t work on Debian’s SBCL 1.2.4)

You may be familiar with Staple since it’s used by Shinmera in all his projects.

https://github.com/Shinmera/staple
output: html. The documentation string is plain text (no markup, rendered in a <pre> tag).
cross-references
can use a template.
more features than listed here.

It doesn’t support SBCL 1.2.4, so my tests fell short (upgrading isn’t 100% smooth here). If you’re on SBCL >= 1.4.8 Staple is a good option.

Declt

Declt has higher goals than “quick and easy” documentation generator.

https://github.com/didierverna/declt
https://www.lrde.epita.fr/~didier/software/lisp/declt/
output: a .texi file, that we can render into other formats (html, pdf).
cross-references: yes

It didn’t work out of the box (and had no explicit error information) and it’s also too involved for my use case.

Documentation-tool (not for general use)

It’s the template used for Edi Weitz software, like Hunchentoot.

https://edicl.github.io/documentation-template/
output: one html file

It works, but it thinks you publish an edicl software and has some hardcoded “http://weitz.de/files/…" urls.

Tinaa (unmaintained)

I liked the output, but it didn’t work (asdf-related error), and it’s unmaintained (authors’ words).

We now have comments. Thanks, Utterances !

Thu, 25 Oct 2018 18:45:20 +0200

We just installed a comment system, and it isn’t Disqus ! We just discovered https://utteranc.es/, a lightweight widget based on Github issues. If it doesn’t find an issue corresponding to the current article, it will create one and post your comment there. Simple :) You dreamed of it ? They did it.

These Months in Common Lisp: Q3 2018

Sat, 06 Oct 2018 23:47:06 +0200

Documentation

Announcements

Jobs

We are still looking for Lispers who want to work in AI (Toronto and around the world)

Projects

new releases:

(re)discoveries:

Articles

Discussion

Learning Lisp:

Screencasts

Common Lisp VS …

CLOS Tutorial

Fri, 05 Oct 2018 19:37:35 +0200

We just updated the CLOS page on the Common Lisp Cookbook. You should refer to it for updates.

CLOS is the “Common Lisp Object System”, arguably one of the most powerful object systems available in any language.

Some of its features include:

it is dynamic, making it a joy to work with in a Lisp REPL. For example, changing a class definition will update the existing objects, given certain rules which we have control upon.
it supports multiple dispatch and multiple inheritance,
it is different from most object systems in that class and method definitions are not tied together,
it has excellent introspection capabilities,
it is provided by a meta-object protocol, which provides a standard interface to the CLOS, and can be used to create new object systems.

The functionality belonging to this name was added to the Common Lisp language between the publication of Steele’s first edition of “Common Lisp, the Language” in 1984 and the formalization of the language as an ANSI standard ten years later.

This page aims to give a good understanding of how to use CLOS, but only a brief introduction to the MOP.

To learn the subjects in depth, you will need two books:

Object-Oriented Programming in Common Lisp: a Programmer’s Guide to CLOS, by Sonya Keene,
the Art of the Metaobject Protocol, by Gregor Kiczales, Jim des Rivières et al.

But see also

the introduction in Practical Common Lisp (online), by Peter Seibel.
Common Lisp, the Language
and for reference, the complete CLOS-MOP specifications.

Table of Contents

Classes and instances

Diving in

Let’s dive in with an example showing class definition, creation of objects, slot access, methods specialized for a given class, and inheritance.

(defclass person ()
  ((name
    :initarg :name
    :accessor name)
   (lisper
    :initform nil
    :accessor lisper)))

;; => #<STANDARD-CLASS PERSON>

(defvar p1 (make-instance 'person :name "me" ))
;;                                 ^^^^ initarg
;; => #<PERSON {1006234593}>

(name p1)
;;^^^ accessor
;; => "me"

(lisper p1)
;; => nil
;;    ^^ initform (slot unbound by default)

(setf (lisper p1) t)


(defclass child (person)
  ())

(defclass child (person)
  ((can-walk-p
     :accessor can-walk-p
     :initform t)))
;; #<STANDARD-CLASS CHILD>

(can-walk-p (make-instance 'child))
;; T

Defining classes (defclass)

The macro used for defining new data types in CLOS is defclass.

We used it like this:

(defclass person ()
  ((name
    :initarg :name
    :accessor name)
   (lisper
    :initform nil
    :accessor lisper)))

This gives us a CLOS type (or class) called person and two slots, named name and lisper.

(class-of p1)
#<STANDARD-CLASS PERSON>

(type-of p1)
PERSON

The general form of defclass is:

(defclass <class-name> (list of super classes)
  ((slot-1
     :slot-option slot-argument)
   (slot-2, etc))
  (:optional-class-option
   :another-optional-class-option))

So, our person class doesn’t explicitely inherit from another class (it gets the empty parentheses ()). However it still inherits by default from the class t and from standard-object. See below under “inheritance”.

We could write a minimal class definition without slots options like this:

(defclass point ()
  (x y z))

or even without slots specificiers: (defclass point () ()).

Creating objects (make-instance)

We create instances of a class with make-instance:

(defvar p1 (make-instance 'person :name "me" ))

It is generally good practice to define a constructor:

(defun make-person (name &key lisper)
  (make-instance 'person :name name :lisper lisper))

This has the direct advantage that you can control the required arguments. You should now export the constructor from your package and not the class itself.

Slots

A function that always works (slot-value)

The function to access any slot anytime is (slot-value <object> <slot-name>).

Given our point class above, which didn’t define any slot accessors:

(defvar pt (make-instance 'point))

(inspect pt)
The object is a STANDARD-OBJECT of type POINT.
0. X: "unbound"
1. Y: "unbound"
2. Z: "unbound"

We got an object of type POINT, but slots are unbound by default: trying to access them will raise an UNBOUND-SLOT condition:

(slot-value pt 'x) ;; => condition: the slot is unbound

slot-value is setf-able:

(setf (slot-value pt 'x) 1)
(slot-value pt 'x) ;; => 1

Initial and default values (initarg, initform)

:initarg :foo is the keyword we can pass to make-instance to give a value to this slot:

(make-instance 'person :name "me")

(again: slots are unbound by default)

:initform <val> is the default value in case we didn’t specify an initarg. This form is evaluated each time it’s needed, in the lexical environment of the defclass.

Sometimes we see the following trick to clearly require a slot:

(defclass foo ()
    ((a
      :initarg :a
      :initform (error "you didn't supply an initial value for slot a"))))
;; #<STANDARD-CLASS FOO>

(make-instance 'foo) ;; => enters the debugger.

Getters and setters (accessor, reader, writer)

:accessor foo: an accessor is both a getter and a setter. Its argument is a name that will become a generic function.

(name p1) ;; => "me"

(type-of #'name)
STANDARD-GENERIC-FUNCTION

:reader and :writer do what you expect. Only the :writer is setf-able.

If you don’t specify any of these, you can still use slot-value.

You can give a slot more than one :accessor, :reader or :initarg.

We introduce two macros to make the access to slots shorter in some situations:

1- with-slots allows to abbreviate several calls to slot-value. The first argument is a list of slot names. The second argument evaluates to a CLOS instance. This is followed by optional declarations and an implicit progn. Lexically during the evaluation of the body, an access to any of these names as a variable is equivalent to accessing the corresponding slot of the instance with slot-value.

(with-slots (name lisper)
    c1
  (format t "got ~a, ~a~&" name lisper))

(with-slots ((n name)
             (l lisper))
    c1
  (format t "got ~a, ~a~&" n l))

2- with-accessors is equivalent, but instead of a list of slots it takes a list of accessor functions. Any reference to the variable inside the macro is equivalent to a call to the accessor function.

(with-accessors ((name        name)
                  ^^variable  ^^accessor
                 (lisper lisper))
            p1
          (format t "name: ~a, lisper: ~a" name lisper))

Class VS instance slots

:allocation specifies whether this slot is local or shared.

a slot is local by default, that means it can be different for each instance of the class. In that case :allocation equals :instance.
a shared slot will always be equal for all instances of the class. We set it with :allocation :class.

In the following example, note how changing the value of the class slot species of p2 affects all instances of the class (whether or not those instances exist yet).

(defclass person ()
  ((name :initarg :name :accessor name)
   (species
      :initform 'homo-sapiens
      :accessor species
      :allocation :class)))

;; Note that the slot "lisper" was removed in existing instances.
(inspect p1)
;; The object is a STANDARD-OBJECT of type PERSON.
;; 0. NAME: "me"
;; 1. SPECIES: HOMO-SAPIENS
;; > q

(defvar p2 (make-instance 'person))

(species p1)
(species p2)
;; HOMO-SAPIENS

(setf (species p2) 'homo-numericus)
;; HOMO-NUMERICUS

(species p1)
;; HOMO-NUMERICUS

(species (make-instance 'person))
;; HOMO-NUMERICUS

(let ((temp (make-instance 'person)))
    (setf (species temp) 'homo-lisper))
;; HOMO-LISPER
(species (make-instance 'person))
;; HOMO-LISPER

Slot documentation

Each slot accepts one :documentation option.

Slot type

The :type slot option may not do the job you expect it does. If you are new to the CLOS, we suggest you skip this section and use your own constructors to manually check slot types.

Indeed, whether slot types are being checked or not is undefined. See the Hyperspec.

Few implementations will do it. Clozure CL does it, SBCL does it when safety is high ((declaim (optimize safety))).

To do it otherwise, see this Stack-Overflow answer, and see also quid-pro-quo, a contract programming library.

find-class, class-name, class-of

(find-class 'point)
;; #<STANDARD-CLASS POINT 275B78DC>

(class-name (find-class 'point))
;; POINT

(class-of my-point)
;; #<STANDARD-CLASS POINT 275B78DC>

(typep my-point (class-of my-point))
;; T

CLOS classes are also instances of a CLOS class, and we can find out what that class is, as in the example below:

(class-of (class-of my-point))
;; #<STANDARD-CLASS STANDARD-CLASS 20306534>

Note: this is your first introduction to the MOP. You don’t need that to get started !

The object my-point is an instance of the class named point, and the class named point is itself an instance of the class named standard-class. We say that the class named standard-class is the metaclass (i.e. the class of the class) of my-point. We can make good uses of metaclasses, as we’ll see later.

Subclasses and inheritance

As illustrated above, child is a subclass of person.

All objects inherit from the class standard-object and t.

Every child instance is also an instance of person.

(type-of c1)
;; CHILD

(subtypep (type-of c1) 'person)
;; T

(ql:quickload "closer-mop")
;; ...

(closer-mop:subclassp (class-of c1) 'person)
;; T

The closer-mop library is the portable way to do CLOS/MOP operations.

A subclass inherits all of its parents slots, and it can override any of their slot options. Common Lisp makes this process dynamic, great for REPL session, and we can even control parts of it (like, do something when a given slot is removed/updated/added, etc).

The class precedence list of a child is thus:

child <- person <-- standard-object <- t

Which we can get with:

(closer-mop:class-precedence-list (class-of c1))
;; (#<standard-class child>
;;  #<standard-class person>
;;  #<standard-class standard-object>
;;  #<sb-pcl::slot-class sb-pcl::slot-object>
;;  #<sb-pcl:system-class t>)

However, the direct superclass of a child is only:

(closer-mop:class-direct-superclasses (class-of c1))
;; (#<standard-class person>)

We can further inspect our classes with class-direct-[subclasses, slots, default-initargs] and many more functions.

How slots are combined follows some rules:

:accessor and :reader are combined by the union of accessors and readers from all the inherited slots.
:initarg: the union of initialization arguments from all the inherited slots.
:initform: we get the most specific default initial value form, i.e. the first :initform for that slot in the precedence list.
:allocation is not inherited. It is controlled solely by the class being defined and defaults to :instance.

Last but not least, be warned that inheritance is fairly easy to misuse, and multiple inheritance is multiply so, so please take a little care. Ask yourself whether foo really wants to inherit from bar, or whether instances of foo want a slot containing a bar. A good general guide is that if foo and bar are “same sort of thing” then it’s correct to mix them together by inheritance, but if they’re really separate concepts then you should use slots to keep them apart.

Multiple inheritance

CLOS supports multiple inheritance.

(defun baby (child person)
  ())

The first class on the list of parent classes is the most specific one, child’s slots will take precedence over the person’s

TODO (but remember how slots are merged).

Redefining and changing a class

This section briefly covers two topics:

redefinition of an existing class, which you might already have done by following our code snippets, and what we do naturally during development, and
changing an instance of one class into an instance of another, a powerful feature of CLOS that you’ll probably won’t use very often.

We’ll gloss over the details. Suffice it to say that everything’s configurable by implementing methods exposed by the MOP.

To redefine a class, simply evaluate a new defclass form. This then takes the place of the old definition, the existing class object is updated, and all instances of the class (and, recursively, its subclasses) are lazily updated to reflect the new definition. You don’t have to recompile anything other than the new defclass, nor to invalidate any of your objects. Think about it for a second: this is awesome !

For example, with our person class:

(defclass person ()
  ((name
    :initarg :name
    :accessor name)
   (lisper
    :initform nil
    :accessor lisper)))

(setf p1 (make-instance 'person :name "me" ))

Changing, adding, removing slots,…

(lisper p1)
;; NIL

(defclass person ()
  ((name
    :initarg :name
    :accessor name)
   (lisper
    :initform t        ;; <-- from nil to t
    :accessor lisper)))

(lisper p1)
;; NIL (of course!)

(lisper (make-instance 'person :name "You"))
;; T

(defclass person ()
  ((name
    :initarg :name
    :accessor name)
   (lisper
    :initform nil
    :accessor lisper)
   (age
    :initarg :arg
    :initform 18
    :accessor age)))

(age p1)
;; => slot unbound error. This is different from "slot missing":

(slot-value p1 'bwarf)
;; => "the slot bwarf is missing from the object #<person…>"

(setf (age p1) 30)
(age p1) ;; => 30

(defclass person ()
  ((name
    :initarg :name
    :accessor name)))

(slot-value p1 'lisper) ;; => slot lisper is missing.
(lisper p1) ;; => there is no applicable method for the generic function lisper when called with arguments #(lisper).

To change the class of an instance, use change-class:

(change-class p1 'child)
;; we can also set slots of the new class:
(change-class p1 'child :can-walk-p nil)

(class-of p1)
;; #<STANDARD-CLASS CHILD>

(can-walk-p p1)
;; T

In the above example, I became a child, and I inherited the can-walk-p slot, which is true by default.

Pretty printing

Everytime we printed an object so far we got an output like

#<PERSON {1006234593}>

which doesn’t say much.

What if we want to show more information ? Something like

#<PERSON me lisper: t>

Pretty printing is done by specializing the generic print-object method for this class:

(defmethod print-object ((obj person) stream)
      (print-unreadable-object (obj stream :type t)
        (with-accessors ((name name)
                         (lisper lisper))
            obj
          (format stream "~a, lisper: ~a" name lisper))))

It gives:

p1
;; #<PERSON me, lisper: T>

print-unreadable-object prints the #<...>, that says to the reader that this object can not be read back in. Its :type t argument asks to print the object-type prefix, that is, PERSON. Without it, we get #<me, lisper: T>.

We used the with-accessors macro, but of course for simple cases this is enough:

(defmethod print-object ((obj person) stream)
  (print-unreadable-object (obj stream :type t)
    (format stream "~a, lisper: ~a" (name obj) (lisper obj))))

Caution: trying to access a slot that is not bound by default will lead to an error. Use slot-boundp.

For reference, the following reproduces the default behaviour:

(defmethod print-object ((obj person) stream)
  (print-unreadable-object (obj stream :type t :identity t)))

Here, :identity to t prints the {1006234593} address.

Classes of traditional lisp types

Where we approach that we don’t need CLOS objects to use CLOS.

Generously, the functions introduced in the last section also work on lisp objects which are not CLOS instances:

(find-class 'symbol)
;; #<BUILT-IN-CLASS SYMBOL>
(class-name *)
;; SYMBOL
(eq ** (class-of 'symbol))
;; T
(class-of ***)
;; #<STANDARD-CLASS BUILT-IN-CLASS>

We see here that symbols are instances of the system class symbol. This is one of 75 cases in which the language requires a class to exist with the same name as the corresponding lisp type. Many of these cases are concerned with CLOS itself (for example, the correspondence between the type standard-class and the CLOS class of that name) or with the condition system (which might or might not be built using CLOS classes in any given implementation). However, 33 correspondences remain relating to “traditional” lisp types:

Note that not all “traditional” lisp types are included in this list. (Consider: atom, fixnum, short-float, and any type not denoted by a symbol.)

The presence of t is interesting. Just as every lisp object is of type t, every lisp object is also a member of the class named t. This is a simple example of membership of more then one class at a time, and it brings into question the issue of inheritance, which we will consider in some detail later.

(find-class t)
;; #<BUILT-IN-CLASS T 20305AEC>

In addition to classes corresponding to lisp types, there is also a CLOS class for every structure type you define:

(defstruct foo)
FOO

(class-of (make-foo))
;; #<STRUCTURE-CLASS FOO 21DE8714>

The metaclass of a structure-object is the class structure-class. It is implementation-dependent whether the metaclass of a “traditional” lisp object is standard-class, structure-class, or built-in-class. Restrictions:

|built-in-class| May not use make-instance, may not use slot-value, may not use defclass to modify, may not create subclasses.| |structure-class| May not use make-instance, might work with slot-value (implementation-dependent). Use defstruct to subclass application structure types. Consequences of modifying an existing structure-class are undefined: full recompilation may be necessary.| |standard-class|None of these restrictions.|

Introspection

we already saw some introspection functions.

Your best option is to discover the closer-mop libray and to keep the CLOS & MOP specifications at hand.

More functions:

closer-mop:class-default-initargs
closer-mop:class-direct-default-initargs
closer-mop:class-direct-slots
closer-mop:class-direct-subclasses
closer-mop:class-direct-superclasses
closer-mop:class-precedence-list
closer-mop:class-slots
closer-mop:classp
closer-mop:extract-lambda-list
closer-mop:extract-specializer-names
closer-mop:generic-function-argument-precedence-order
closer-mop:generic-function-declarations
closer-mop:generic-function-lambda-list
closer-mop:generic-function-method-class
closer-mop:generic-function-method-combination
closer-mop:generic-function-methods
closer-mop:generic-function-name
closer-mop:method-combination
closer-mop:method-function
closer-mop:method-generic-function
closer-mop:method-lambda-list
closer-mop:method-specializers
closer-mop:slot-definition
closer-mop:slot-definition-allocation
closer-mop:slot-definition-initargs
closer-mop:slot-definition-initform
closer-mop:slot-definition-initfunction
closer-mop:slot-definition-location
closer-mop:slot-definition-name
closer-mop:slot-definition-readers
closer-mop:slot-definition-type
closer-mop:slot-definition-writers
closer-mop:specializer-direct-generic-functions
closer-mop:specializer-direct-methods
closer-mop:standard-accessor-method

Methods

Diving in

Recalling our person and child classes from the beginning:

(defclass person ()
  ((name
    :initarg :name
    :accessor name)))
;; => #<STANDARD-CLASS PERSON>

(defclass child (person)
  ())
;; #<STANDARD-CLASS CHILD>

(setf p1 (make-instance 'person :name "me"))
(setf c1 (make-instance 'child :name "Alice"))

Below we create methods, we specialize them, we use method combination (before, after, around), and qualifiers.

(defmethod greet (obj)
  (format t "Are you a person ? You are a ~a.~&" (type-of obj)))
;; style-warning: Implicitly creating new generic function common-lisp-user::greet.
;; #<STANDARD-METHOD GREET (t) {1008EE4603}>

(greet :anything)
;; Are you a person ? You are a KEYWORD.
;; NIL
(greet p1)
;; Are you a person ? You are a PERSON.

(defgeneric greet (obj)
  (:documentation "say hello"))
;; STYLE-WARNING: redefining COMMON-LISP-USER::GREET in DEFGENERIC
;; #<STANDARD-GENERIC-FUNCTION GREET (2)>

(defmethod greet ((obj person))
  (format t "Hello ~a !~&" (name obj)))
;; #<STANDARD-METHOD GREET (PERSON) {1007C26743}>

(greet p1) ;; => "Hello me !"
(greet c1) ;; => "Hello Alice !"

(defmethod greet ((obj child))
  (format t "ur so cute~&"))
;; #<STANDARD-METHOD GREET (CHILD) {1008F3C1C3}>

(greet p1) ;; => "Hello me !"
(greet c1) ;; => "ur so cute"

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; Method combination: before, after, around.
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

(defmethod greet :before ((obj person))
  (format t "-- before person~&"))
#<STANDARD-METHOD GREET :BEFORE (PERSON) {100C94A013}>

(greet p1)
;; -- before person
;; Hello me

(defmethod greet :before ((obj child))
  (format t "-- before child~&"))
;; #<STANDARD-METHOD GREET :BEFORE (CHILD) {100AD32A43}>
(greet c1)
;; -- before child
;; -- before person
;; ur so cute

(defmethod greet :after ((obj person))
  (format t "-- after person~&"))
;; #<STANDARD-METHOD GREET :AFTER (PERSON) {100CA2E1A3}>
(greet p1)
;; -- before person
;; Hello me
;; -- after person

(defmethod greet :after ((obj child))
  (format t "-- after child~&"))
;; #<STANDARD-METHOD GREET :AFTER (CHILD) {10075B71F3}>
(greet c1)
;; -- before child
;; -- before person
;; ur so cute
;; -- after person
;; -- after child

(defmethod greet :around ((obj child))
  (format t "Hello my dear~&"))
;; #<STANDARD-METHOD GREET :AROUND (CHILD) {10076658E3}>
(greet c1) ;; Hello my dear


;; call-next-method

(defmethod greet :around ((obj child))
  (format t "Hello my dear~&")
  (when (next-method-p)
    (call-next-method)))
;; #<standard-method greet :around (child) {100AF76863}>

(greet c1)
;; Hello my dear
;; -- before child
;; -- before person
;; ur so cute
;; -- after person
;; -- after child

;;;;;;;;;;;;;;;;;
;; Adding in &key
;;;;;;;;;;;;;;;;;

;; In order to add "&key" to our generic method, we need to remove its definition first.
(fmakunbound 'greet)  ;; with Slime: C-c C-u (slime-undefine-function)
(defmethod greet ((obj person) &key talkative)
  (format t "Hello ~a~&" (name obj))
  (when talkative
    (format t "blah")))

(defgeneric greet (obj &key &allow-other-keys)
  (:documentation "say hi"))

(defmethod greet (obj &key &allow-other-keys)
  (format t "Are you a person ? You are a ~a.~&" (type-of obj)))

(defmethod greet ((obj person) &key talkative &allow-other-keys)
  (format t "Hello ~a !~&" (name obj))
  (when talkative
    (format t "blah")))

(greet p1 :talkative t) ;; ok
(greet p1 :foo t) ;; still ok


;;;;;;;;;;;;;;;;;;;;;;;

(defgeneric greet (obj)
  (:documentation "say hello")
  (:method (obj)
    (format t "Are you a person ? You are a ~a~&." (type-of obj)))
  (:method ((obj person))
    (format t "Hello ~a !~&" (name obj)))
  (:method ((obj child))
    (format t "ur so cute~&")))

;;;;;;;;;;;;;;;;
;;; Specializers
;;;;;;;;;;;;;;;;

(defgeneric feed (obj meal-type)
  (:method (obj meal-type)
    (declare (ignorable meal-type))
    (format t "eating~&")))

(defmethod feed (obj (meal-type (eql :dessert)))
    (declare (ignorable meal-type))
    (format t "mmh, dessert !~&"))

(feed c1 :dessert)
;; mmh, dessert !

(defmethod feed ((obj child) (meal-type (eql :soup)))
    (declare (ignorable meal-type))
    (format t "bwark~&"))

(feed p1 :soup)
;; eating
(feed c1 :soup)
;; bwark

Generic functions (defgeneric, defmethod)

A generic function is a lisp function which is associated with a set of methods and dispatches them when it’s invoked. All the methods with the same function name belong to the same generic function.

The defmethod form is similar to a defun. It associates a body of code with a function name, but that body may only be executed if the types of the arguments match the pattern declared by the lambda list.

They can have optional, keyword and &rest arguments.

The defgeneric form defines the generic function. If we write a defmethod without a corresponding defgeneric, a generic function is automatically created (see examples).

It is generally a good idea to write the defgenerics. We can add a default implementation and even some documentation.

(defgeneric greet (obj)
  (:documentation "says hi")
  (:method (obj)
    (format t "Hi")))

The required parameters in the method’s lambda list may take one of the following three forms:

1- a simple variable:

(defmethod greet (foo)
  ...)

This method can take any argument, it is always applicable.

The variable foo is bound to the corresponding argument value, as usual.

2- a variable and a specializer, as in:

(defmethod greet ((foo person))
  ...)

In this case, the variable foo is bound to the corresponding argument only if that argument is of specializer class person or a subclass, like child (indeed, a “child” is also a “person”).

If any argument fails to match its specializer then the method is not applicable and it cannot be executed with those arguments.We’ll get an error message like “there is no applicable method for the generic function xxx when called with arguments yyy”.

Only required parameters can be specialized. We can’t specialize on optional &key arguments.

3- a variable and an eql specializer

(defmethod feed ((obj child) (meal-type (eql :soup)))
    (declare (ignorable meal-type))
    (format t "bwark~&"))

(feed c1 :soup)
;; "bwark"

In place of a simple symbol (:soup), the eql specializer can be any lisp form. It is evaluated at the same time of the defmethod.

You can define any number of methods with the same function name but with different specializers, as long as the form of the lambda list is congruent with the shape of the generic function. The system chooses the most specific applicable method and executes its body. The most specific method is the one whose specializers are nearest to the head of the class-precedence-list of the argument (classes on the left of the lambda list are more specific). A method with specializers is more specific to one without any.

Notes:

It is an error to define a method with the same function name as an ordinary function. If you really want to do that, use the shadowing mechanism.
To add or remove keys or rest arguments to an existing generic method’s lambda list, you will need to delete its declaration with fmakunbound (or C-c C-u (slime-undefine-function) with the cursor on the function in Slime) and start again. Otherwise, you’ll see:

attempt to add the method
  #<STANDARD-METHOD NIL (#<STANDARD-CLASS CHILD>) {1009504233}>
to the generic function
  #<STANDARD-GENERIC-FUNCTION GREET (2)>;
but the method and generic function differ in whether they accept
&REST or &KEY arguments.

Methods can be redefined (exactly as for ordinary functions).
The order in which methods are defined is irrelevant, although any classes on which they specialize must already exist.
An unspecialized argument is more or less equivalent to being specialized on the class t. The only difference is that all specialized arguments are implicitly taken to be “referred to” (in the sense of declare ignore.)
Each defmethod form generates (and returns) a CLOS instance, of class standard-method.
An eql specializer won’t work as is with strings. Indeed, strings need equal or equalp to be compared. But, we can assign our string to a variable and use the variable both in the eql specializer and for the function call.
All the methods with the same function name belong to the same generic function.
All slot accessors and readers defined by defclass are methods. They can override or be overridden by other methods on the same generic function.

See more about defmethod on the CLHS.

Multimethods

Multimethods explicitly specialize more than one of the generic function’s required parameters.

They don’t belong to a particular class. Meaning, we don’t have to decide on the class that would be best to host this method, as we might have to in other languages.

(defgeneric hug (a b)
   (:documentation "Hug between two persons."))
;; #<STANDARD-GENERIC-FUNCTION HUG (0)>

(defmethod hug ((a person) (b person))
  :person-person-hug)

(defmethod hug ((a person) (b child))
  :person-child-hug)

Controlling setters (setf-ing methods)

In Lisp, we can define setf counterparts of functions or methods. We might want this to have more control on how to update an object.

(defmethod (setf name) (new-val (obj person))
  (if (equalp new-val "james bond")
    (format t "Dude that's not possible.~&")
    (setf (slot-value obj 'name) new-val)))

(setf (name p1) "james bond") ;; -> no rename

If you know Python, this behaviour is provided by the @property decorator.

Dispatch mechanism and next methods

When a generic function is invoked, the application cannot directly invoke a method. The dispatch mechanism proceeds as follows:

compute the list of applicable methods
if no method is applicable then signal an error
sort the applicable methods in order of specificity
invoke the most specific method.

Our greet generic function has three applicable methods:

(closer-mop:generic-function-methods #'greet)
(#<STANDARD-METHOD GREET (CHILD) {10098406A3}>
 #<STANDARD-METHOD GREET (PERSON) {1009008EC3}>
 #<STANDARD-METHOD GREET (T) {1008E6EBB3}>)

During the execution of a method, the remaining applicable methods are still accessible, via the local function call-next-method. This function has lexical scope within the body of a method but indefinite extent. It invokes the next most specific method, and returns whatever value that method returned. It can be called with either:

no arguments, in which case the next method will receive exactly the same arguments as this method did, or
explicit arguments, in which case it is required that the sorted set of methods applicable to the new arguments must be the same as that computed when the generic function was first called.

For example:

(defmethod greet ((obj child))
  (format t "ur so cute~&")
  (when (next-method-p)
    (call-next-method)))
;; STYLE-WARNING: REDEFINING GREET (#<STANDARD-CLASS CHILD>) in DEFMETHOD
;; #<STANDARD-METHOD GREET (child) {1003D3DB43}>

(greet c1)
;; ur so cute
;; Hello Alice !

Calling call-next-method when there is no next method signals an error. You can find out whether a next method exists by calling the local function next-method-p (which also has has lexical scope and indefinite extent).

Note finally that the body of every method establishes a block with the same name as the method’s generic function. If you return-from that name you are exiting the current method, not the call to the enclosing generic function.

Method qualifiers (before, after, around)

In our “Diving in” examples, we saw some use of the :before, :after and :around qualifiers:

(defmethod foo :before (obj) (...))
(defmethod foo :after (obj) (...))
(defmethod foo :around (obj) (...))

By default, in the standard method combination framework provided by CLOS, we can only use one of those three qualifiers, and the flow of control is as follows:

a before-method is called, well, before the applicable primary method. If they are many before-methods, all are called. The most specific before-method is called first (child before person).
the most specific applicable primary method (a method without qualifiers) is called (only one).
all applicable after-methods are called. The most specific one is called last (after-method of person, then after-method of child).

The generic function returns the value of the primary method. Any values of the before or after methods are ignored. They are used for their side effects.

And then we have around-methods. They are wrappers around the core mechanism we just described. They can be useful to catch return values or to set up an environment around the primary method (set up a catch, a lock, timing an execution,…).

If the dispatch mechanism finds an around-method, it calls it and returns its result. If the around-method has a call-next-method, it calls the next most applicable around-method. It is only when we reach the primary method that we start calling the before and after-methods.

Thus, the full dispatch mechanism for generic functions is as follows:

compute the applicable methods, and partition them into separate lists according to their qualifier;
if there is no applicable primary method then signal an error;
sort each of the lists into order of specificity;
execute the most specific :around method and return whatever that returns;
if an :around method invokes call-next-method, execute the next most specific :around method;
if there were no :around methods in the first place, or if an :around method invokes call-next-method but there are no further :around methods to call, then proceed as follows:

a. run all the :before methods, in order, ignoring any return values and not permitting calls to call-next-method or next-method-p;

b. execute the most specific primary method and return whatever that returns;

c. if a primary method invokes call-next-method, execute the next most specific primary method;

d. if a primary method invokes call-next-method but there are no further primary methods to call then signal an error;

e. after the primary method(s) have completed, run all the :after methods, in reverse order, ignoring any return values and not permitting calls to call-next-method or next-method-p.

Think of it as an onion, with all the :around methods in the outermost layer, :before and :after methods in the middle layer, and primary methods on the inside.

Other method combinations

The default method combination type we just saw is named standard, but other method combination types are available, and no need to say that you can define your own.

The built-in types are:

progn + list nconc and max or append min

You notice that these types are named after a lisp operator. Indeed, what they do is they define a framework that combines the applicable primary methods inside a call to the lisp operator of that name. For example, using the progn combination type is equivalent to calling all the primary methods one after the other:

(progn
  (method-1 args)
  (method-2 args)
  (method-3 args))

Here, unlike the standard mechanism, all the primary methods applicable for a given object are called, the most specific first.

To change the combination type, we set the :method-combination option of defgeneric and we use it as the methods’ qualifier:

(defgeneric foo (obj)
  (:method-combination progn))

(defmethod foo progn ((obj obj))
   (...))

An example with progn:

(defgeneric dishes (obj)
   (:method-combination progn)
   (:method progn (obj)
     (format t "- clean and dry.~&"))
   (:method progn ((obj person))
     (format t "- bring a person's dishes~&"))
   (:method progn ((obj child))
     (format t "- bring the baby dishes~&")))
;; #<STANDARD-GENERIC-FUNCTION DISHES (3)>

(dishes c1)
;; - bring the baby dishes
;; - bring a person's dishes
;; - clean and dry.

(greet c1)
;; ur so cute  --> only the most applicable method was called.

Similarly, using the list type is equivalent to returning the list of the values of the methods.

(list
  (method-1 args)
  (method-2 args)
  (method-3 args))

(defgeneric tidy (obj)
  (:method-combination list)
  (:method list (obj)
    :foo)
  (:method list ((obj person))
    :books)
  (:method list ((obj child))
    :toys))
;; #<STANDARD-GENERIC-FUNCTION TIDY (3)>

(tidy c1)
;; (:toys :books :foo)

Around methods are accepted:

(defmethod tidy :around (obj)
   (let ((res (call-next-method)))
     (format t "I'm going to clean up ~a~&" res)
     (when (> (length res)
              1)
       (format t "that's too much !~&"))))

(tidy c1)
;; I'm going to clean up (toys book foo)
;; that's too much !

Note that these operators don’t support before, after and around methods (indeed, there is no room for them anymore). They do support around methods, where call-next-method is allowed, but they don’t support calling call-next-method in the primary methods (it would indeed be redundant since all primary methods are called, or clunky to not call one).

CLOS allows us to define a new operator as a method combination type, be it a lisp function, macro or special form. We’ll let you refer to the books if you feel the need.

Debugging: tracing method combination

It is possible to trace the method combination, but this is implementation dependent.

In SBCL, we can use (trace foo :methods t). See this post by an SBCL core developer.

For example, given a generic:

(defgeneric foo (x)
  (:method (x) 3))
(defmethod foo :around ((x fixnum))
  (1+ (call-next-method)))
(defmethod foo ((x integer))
  (* 2 (call-next-method)))
(defmethod foo ((x float))
  (* 3 (call-next-method)))
(defmethod foo :before ((x single-float))
  'single)
(defmethod foo :after ((x double-float))
 'double)

Let’s trace it:

(trace foo :methods t)

(foo 2.0d0)
  0: (FOO 2.0d0)
    1: ((SB-PCL::COMBINED-METHOD FOO) 2.0d0)
      2: ((METHOD FOO (FLOAT)) 2.0d0)
        3: ((METHOD FOO (T)) 2.0d0)
        3: (METHOD FOO (T)) returned 3
      2: (METHOD FOO (FLOAT)) returned 9
      2: ((METHOD FOO :AFTER (DOUBLE-FLOAT)) 2.0d0)
      2: (METHOD FOO :AFTER (DOUBLE-FLOAT)) returned DOUBLE
    1: (SB-PCL::COMBINED-METHOD FOO) returned 9
  0: FOO returned 9
9

MOP

We gather here some examples that make use of the framework provided by the meta-object protocol, the configurable object system that rules Lisp’s object system. We touch advanced concepts so, new reader, don’t worry: you don’t need to understand this section to start using the Common Lisp Object System.

We won’t explain much about the MOP here, but hopefully sufficiently to make you see its possibilities or to help you understand how some CL libraries are built. We invite you to read the books referenced in the introduction.

Metaclasses

Metaclasses are needed to control the behaviour of other classes.

As announced, we won’t talk much. See also Wikipedia for metaclasses or CLOS.

The standard metaclass is standard-class:

(class-of p1) ;; #<STANDARD-CLASS PERSON>

But we’ll change it to one of our own, so that we’ll be able to count the creation of instances. This same mechanism could be used to auto increment the primary key of a database system (this is how the Postmodern or Mito libraries do), to log the creation of objects, etc.

Our metaclass inherits from standard-class:

(defclass counted-class (standard-class)
  ((counter :initform 0)))
#<STANDARD-CLASS COUNTED-CLASS>

(unintern 'person)
;; this is necessary to change the metaclass of person.
;; or (setf (find-class 'person) nil)
;; https://stackoverflow.com/questions/38811931/how-to-change-classs-metaclass#38812140

(defclass person ()
  ((name
    :initarg :name
    :accessor name)
  (:metaclass counted-class))) ;; <- metaclass
;; #<COUNTED-CLASS PERSON>
;;   ^^^ not standard-class anymore.

The :metaclass class option can appear only once.

Actually you should have gotten a message asking to implement validate-superclass. So, still with the closer-mop library:

(defmethod closer-mop:validate-superclass ((class counted-class)
                                           (superclass standard-class))
  t)

Now we can control the creation of new person instances:

(defmethod make-instance :after ((class counted-class) &key)
  (incf (slot-value class 'counter)))
;; #<STANDARD-METHOD MAKE-INSTANCE :AFTER (COUNTED-CLASS) {1007718473}>

See that an :after qualifier is the safest choice, we let the standard method run as usual and return a new instance.

The &key is necessary, remember that make-instance is given initargs.

Now testing:

(defvar p3 (make-instance 'person :name "adam"))
#<PERSON {1007A8F5B3}>

(slot-value p3 'counter)
;; => error. No, our new slot isn't on the person class.
(slot-value (find-class 'person) 'counter)
;; 1

(make-instance 'person :name "eve")
;; #<PERSON {1007AD5773}>
(slot-value (find-class 'person) 'counter)
;; 2

It’s working.

Controlling the initialization of instances (initialize-instance)

To further customize the creation of instances by specializing initialize-instance, which is called by make-instance, just after it has created a new instance but didn’t initialize it yet with the default initargs and initforms.

It is recommended (Keene) to create an after method, since creating a primary method would prevent slots’ initialization.

(defmethod initialize-instance :after ((obj person) &key)
  (do something with obj))

Another rational. The CLOS implementation of make-instance is in two stages: allocate the new object, and then pass it along with all the make-instance keyword arguments, to the generic function initialize-instance. Implementors and application writers define :after methods on initialize-instance, to initialize the slots of the instance. The system-supplied primary method does this with regard to (a) :initform and :initarg values supplied with the class was defined and (b) the keywords passed through from make-instance. Other methods can extend this behaviour as they see fit. For example, they might accept an additional keyword which invokes a database access to fill certain slots. The lambda list for initialize-instance is:

initialize-instance instance &rest initargs &key &allow-other-keys

See more in the books !

Shuffletron, a Common Lisp Music Player for the terminal

Tue, 11 Sep 2018 16:36:23 +0200

Shuffletron is a nice music player for the terminal written in Common Lisp, “based on search and tagging”, that seduced me with its attention to details. Moreover, its author was very responsive to fix a couple issues.

The first time you launch it, it will ask for a music repository and will propose to scan it for id3 tags with the scanid3 command. It is optional, but it allows to print colored information:

The basic commands to know are the following:

search with / followed by your search terms. You’ll notice that your prompt changed from library to xy matches. You can refine the results by searching again. To enter a new query we have to go back to the library, with as many successive “enters” as needed.
to play songs: play. We can select which songs to play, using their index:
- comma-separated indexes of songs: 1,3,10
- a selection with a dash and an optional end: 1-10, 0-
- a combination of the two: 1,3-10.
there are the obvious pause, shuffle, skip, next, seek, repeat,…
now to show the currently playing song.

There is also a queue, id3 tags management, profiles to use an alternate library (./shuffletron --help), and even an alarm clock feature which allows to program music with something like:

  alarm at 7:45 am      # \"at\" is optional and doesn't change the meaning
  alarm 7:45 am
  alarm 9 pm
  alarm 7               # If AM/PM not specified, assumes AM
  alarm in 5 minutes    # Relative alarm times, in minutes or hours
  alarm in 10m          # minutes, minute, mins, min, , m are synonyms
  alarm in 7 hours      # hours, hour, hr, h are synonyms
  alarm in 8h
  alarm in 7:29         # h:mm format - seven hours, twenty-nine minutes
  alarm reset           # off/never/delete/disable/cancel/clear/reset

I can see a use for a pomodoro-like technic :)

I’ll list the complete set of commands below (available on the sources), but first a note on installation.

Installation

Shuffletron doesn’t provide executables (yet ?). The procedure is now documented in the readme so you just have to

make shuffletron-bin  # sbcl
sudo make install
./shuffletron

This last line calls a script and it is actually important to use it, to link dependencies and to use rlwrap. There is room for improvement here.

To read Flac and Ogg files, you need those system dependencies:

apt install libflac-dev
apt install libvorbis-dev

Finally, scanning my library failed for the first time, because of badly manually encoded ogg files coming from youtube. The mixalot library prefered to fail instead of showing error messages. If you encounter a similar problem, see this PR.

All commands

In the application, type help, and help commands to get this list:

Command list:

  /[query]       Search library for [query].
  show           Print search matches, highlighting songs in queue.
  back           Undo last search.
  [songs]        Play list of songs.
  all            Play all songs in selection (equivalent to \"0-\")
  +[songs]       Append list of songs to queue.
  pre[songs]     Prepend list of songs to queue.
  random         Play a random song from the current selection.
  random QUERY   Play a random song matching QUERY
  shuffle SONGS  Play songs in random order.

  queue          Print queue contents and current song playing.
  shuffle        Randomize order of songs in queue.
  clear          Clear the queue (current song continues playing)
  loop           Toggle loop mode (loop through songs in queue)
  qdrop          Remove last song from queue
  qdrop RANGES   Remove songs from queue
  qtag TAGS      Apply tags to all songs in queue
  fromqueue      Transfer queue to selection
  toqueue        Replace queue with selection

  now            Print name of song currently playing.
  play           Resume playing
  stop           Stop playing (current song pushed to head of queue)
  pause          Toggle paused/unpaused.
  skip           Skip currently playing song. If looping is enabled, this
                 song won't played again.
  next           Advance to next song. If looping is enabled, the current
                 song will be enqueued.
  repeat N       Add N repetitions of currently playing song to head of queue.
  seek TIME      Seek to time (in [h:]m:ss format, or a number in seconds)
  seek +TIME     Seek forward
  seek -TIME     Seek backward
  startat TIME   Always start playback at a given time (to skip long intros)

  tag            List tags of currently playing song.
  tag TAGS       Add one or more textual tags to the current song.
  untag TAGS     Remove the given tags from the currently playing song.
  tagged TAGS    Search for files having any of specified tags.
  tags           List all tags (and # occurrences) within current query.
  killtag TAGS   Remove all occurances of the given tags
  tagall TAGS    Apply tags to all selected songs
  untagall TAGS  Remove given tags from all selected songs

  time           Print current time
  alarm          Set alarm (see \"help alarms\")

  scanid3        Scan new files for ID3 tags
  prescan        Toggle file prescanning (useful if file IO is slow)
  exit           Exit the program.

  help [topic]   Help

These Months in Common Lisp: Q2 2018

Mon, 02 Jul 2018 14:59:46 +0200

Q1 2018

Documentation

Announcements

Jobs

Projects

Quicklisp dist update, march 2018

SHCL: An Unholy Union of POSIX Shell and Common Lisp (reddit)
cl-notebook
Heroku buildpack for Common Lisp
Kubernetes Client Library for Common Lisp
Interactive Common Lisp code snippets in any web page
rove - small testing framework (Fukamachi’s successor to Prove)
can - a role-based access right control library
house - custom asynchronous HTTP server for the Deal project.
oxenfurt - A Common Lisp client library for the Oxford dictionary API.
algebraic-data-library
Lisp Interface to Federal Reserve Economic Data (FRED®)
reddit1.0 source code (comments), then Reddit’s code runs on SBCL. See also reddit.
Petalisp: Elegant High Performance Computing
Code Golf Site with Common Lisp Support!
vseloved/wiki-lang-detect: Text language identification using Wikipedia data
ppath, a path manipulation library
CRAM: Cognitive Robot Abstract Machine - a toolbox for designing, implementing and deploying software on autonomous robots
cl-statistics.lisp
Pseudoscheme - An implementation of Scheme embedded in Common Lisp (“with minor changes it runs in ABCL, CCL, ECL and LispWorks. But not in SBCL…”)
Powerlisp: A simple tool to automate your work with dmenu/rofi
curry-compose-reader-macros - concise function partial application and composition
json-mop: A metaclass for bridging CLOS and JSON objects
clsh: a set of Lispy bindings for running and composing *nix processes
snakes - Python style generators for Common Lisp. (Includes a port of itertools.)

new releases:

Portacle: Release Linux Portability Fixes, Included SLY, Usability Improvements

(re)discoveries:

From the Lisp Game Jam 2018:

all results

Articles

Discussion

Learning Lisp:

Screencasts

Common Lisp VS …

New Weblocks tutorial: widgets

Mon, 25 Jun 2018 18:51:49 +0100

Weblocks is a web framework, created circa 2007, that allows to write dynamic web applications in full Lisp, without a line of Javascript. It is based on so called widgets, that are rendered server-side and updated on the client, and it was also based on continuations (they were removed in this fork, at least for now). It was quietly being forgotten but it is being fixed, refactored, documented and simplified by Alexander “svetlyak40wt” since a year or so.

I rewrote the quickstart to show the use of widgets, the heart of Weblocks, Alexander proof-read and here we are:

http://40ants.com/weblocks/quickstart.html (and reddit comments)

I copy it below, of course check the official website for updates.

The old Weblocks website is at https://common-lisp.net/project/cl-weblocks/

Btw, other isomorphic web frameworks I know are Haskell’s Haste, Nim’s Karax, Ocaml’s Heliom, Python’s Nagare, of course Smalltalk’s Seaside and a couple more.

Weblocks is now the easiest to get started with, the one “really isomorphic” (bluring the line between server and client code), the most elegant (to me) and one of the two, with Seaside, that allow for REPL-driven development.

For what I tested, Heliom is a hell to install, it seems a hell to deploy, and it is bloated with specific Ocaml syntax that is always upcoming in new compiler versions.

Karax seems promising, it was recently used to rewrite the Nim forum. It has currently zero docs (not even a quickstart ;) ) and Nim’s ecosystem isn’t near as large as CL’s. (and, well, no REPL, no Lisp)

Nagare is actually based on Stackless Python. Looking at the code of the successfull Kansha Trello clone, it actually includes inline Javascript. And personnally, I’m running away from Python so…

Quickstart

warning

This version of Weblocks is not in Quicklisp yet. To

install it you need to clone the repository somewhere where ASDF will find it, for example, to the ~/common-lisp/ directory.

Load weblocks and create a package for a sandbox:

CL-USER> (ql:quickload '(:weblocks :weblocks-ui :find-port))
CL-USER> (defpackage todo
           (:use #:cl
                 #:weblocks-ui/form
                 #:weblocks/html)
           (:import-from #:weblocks/widget
                    #:render
                    #:update
                    #:defwidget)
           (:import-from #:weblocks/actions
                    #:make-js-action)
           (:import-from #:weblocks/app
                    #:defapp))
#<PACKAGE "TODO">
CL-USER> (in-package todo)
#<PACKAGE "TODO">

Now, create an application:

TODO> (defapp tasks)
TODO> (weblocks/debug:on)
TODO> (defvar *port* (find-port:find-port))
TODO> (weblocks/server:start :port *port*)
 <INFO> [19:41:00] weblocks/server server.lisp (start) -
  Starting weblocks WEBLOCKS/SERVER::PORT: 40000
  WEBLOCKS/SERVER::SERVER-TYPE: :HUNCHENTOOT DEBUG: T
 <INFO> [19:41:00] weblocks/server server.lisp (start-server) -
  Starting webserver on WEBLOCKS/SERVER::INTERFACE: "localhost"
  WEBLOCKS/SERVER::PORT: 40000 DEBUG: T
 #<SERVER port=40000 running>
 (NIL)

Open http://localhost:40000/tasks/ in your browser (double check the port) and you’ll see a text like that:

No weblocks/session:init method defined.
Please define a method weblocks.session:init to initialize a session.

It could be something simple, like this one:

(defmethod weblocks/session:init ((app tasks))
            "Hello world!")

Read more in the documentaion.

It means that you didn’t write any code for your application. Let’s do it now and make an application which outputs a list of tasks.

In the end, we’ll build the mandatory TODO-list app:

TODO> (defwidget task ()
        ((title
          :initarg :title
          :accessor title)
         (done
          :initarg :done
          :initform nil
          :accessor done)))

This code defines a task widget, the building block of our application. defwidget is similar to Common Lisp’s defclass, in fact it is only a wrapper around it. It takes a name, a list of super-classes (here ()) and a list of slot definitions.

We can create a task with make-instance:

TODO> (defvar *task-1* (make-instance 'task :title "Make my first Weblocks app"))
TODO> *task-1*
#<TASK {1005406F33}>

Above, :title is the initarg, and since we didn’t give a :done argument, it will be instanciated to its :initform, which is nil.

We defined accessors for both slots, so we can read and set them easily:

TODO> (title *task-1*)
"Make my first Weblocks app"
TODO> (done *TASK-1*)
NIL
TODO> (setf (done *TASK-1*) t)
T

We define a constructor for our task:

TODO> (defun make-task (&key title done)
        (make-instance 'task :title title :done done))

It isn’t mandatory, but it is good practice to do so.

If you are not familiar with the Common Lisp Object System (CLOS), you can have a look at Practical Common Lisp and the Common Lisp Cookbook.

Now let’s carry on with our application.

Below we define a more general widget that contains a list of tasks, and we tell Weblocks how to display them by specializing the render method for our newly defined classes:

TODO> (defwidget task-list ()
        ((tasks
          :initarg :tasks
          :accessor tasks)))

TODO> (defmethod render ((task task))
        "Render a task."
        (with-html
              (:span (if (done task)
                         (with-html
                               (:s (title task)))
                       (title task)))))

TODO> (defmethod render ((widget task-list))
        "Render a list of tasks."
        (with-html
              (:h1 "Tasks")
              (:ul
                (loop for task in (tasks widget) do
                      (:li (render task))))))

The with-html macro uses Spinneret under the hood, but you can use anything that outputs html.

We can check how the generated html looks like by calling render in the REPL:

TODO> (render *task-1*)
<div class="widget task"><span>Make my first Weblocks app</span>
</div>
NIL

But we still don’t get anything in the browser.

TODO> (defun make-task-list (&rest rest)
             "Create some tasks from titles."
             (loop for title in rest collect
                   (make-task :title title)))

TODO> (defmethod weblocks/session:init ((app tasks))
         (declare (ignorable app))
         (let ((tasks (make-task-list "Make my first Weblocks app"
                                      "Deploy it somewhere"
                                      "Have a profit")))
           (make-instance 'task-list :tasks tasks)))

This defines a list of tasks (for simplicity, they are defined as a list in memory) and returns what will be our session’s root widget..

Restart the application:

TODO> (weblocks/debug:reset-latest-session)

Right now it should look like this:

Adding tasks

Now, we’ll add some ability to interact with a list – to add some tasks into it, like so:

Import a new module, weblocks-ui to help in creating forms and other UI elements:

TODO> (ql:quickload "weblocks-ui")
TODO> (use-package :weblocks-ui/form)

Write a new add-task function and modify the render method of a task-list:

TODO> (defmethod render ((widget task-list))
          (flet ((add-task (&key title &allow-other-keys)
                   (push (make-task :title title)
                         (tasks (weblocks/widgets/root:get)))
                   (update (weblocks/widgets/root:get))))
            (with-html
              (:h1 "Tasks")
              (loop for task in (tasks widget) do
                   (render task))
              (with-html-form (:POST #'add-task)
                (:input :type "text"
                        :name "title"
                        :placeholder "Task's title")
                (:input :type "submit"
                        :value "Add")))))

TODO> (weblocks/debug:reset-latest-session)

The function add-task does only two simple things:

it adds a task into a list;
it tells Weblocks that our root widget should be redrawn.

This second point is really important because it allows Weblocks to render necessary parts of the page on the server and to inject it into the HTML DOM in the browser. Here it rerenders the root widget, but we can as well update a specific task widget, as we’ll do soon.

We also took care of defining add-task inline, as a closure, for it to be thread safe.

Another block in our new version of render of a task-list is the form:

(with-html-form (:POST #'add-task)
   (:input :type "text"
    :name "task"
    :placeholder "Task's title")
   (:input :type "submit"
    :value "Add"))

It defines a text field, a submit button and an action to perform on form submit. The add-task function will receive the text input as argument.

note

This is really amazing!

With Weblocks, you can handle all the business logic server-side, because an action can be any lisp function, even an anonymous lambda, closuring all necessary variables.

Restart the application and reload the page. Test your form now and see in a Webinspector how Weblocks sends requests to the server and receives HTML code with rendered HTML block.

Now we’ll make our application really useful – we’ll add code to toggle the tasks’ status.

Toggle tasks

TODO> (defmethod toggle ((task task))
        (setf (done task)
              (if (done task)
                  nil
                  t))
        (update task))

TODO> (defmethod render ((task task))
        (with-html
          (:p (:input :type "checkbox"
            :checked (done task)
            :onclick (make-js-action
                      (lambda (&rest rest)
                        (declare (ignore rest))
                      (toggle task))))
              (:span (if (done task)
                   (with-html
                         ;; strike
                         (:s (title task)))
                 (title task))))))

We defined a small helper to toggle the done attribute, and we’ve modified our task rendering function by adding a code to render a checkbox with an anonymous lisp function, attached to its onclick attribute.

The function make-js-action returns a Javascript code, which calls back a lisp lambda function when evaluated in the browser. And because toggle updates a Task widget, Weblocks returns on this callback a new prerendered HTML for this one task only.

What is next?

As a homework:

Play with lambdas and add a “Delete” button next after each task.
Add the ability to sort tasks by name or by completion flag.
Save tasks in a database (the Cookbook might help).
Read the rest of the documentation and make a real application, using the full power of Common Lisp.

Models and databases with the Mito ORM and SxQL

Tue, 29 May 2018 07:51:49 +0100

Following is a tutorial on how to use the Mito ORM.

As usual, this is best read on the Common Lisp Cookbook. It will be updated there.

The Database section on the Awesome-cl list is a resource listing popular libraries to work with different kind of databases. We can group them roughly in four categories:

wrappers to one database engine (cl-sqlite, postmodern, cl-redis,…),
interfaces to several DB engines (clsql, sxql,…),
persistent object databases (bknr.datastore (see chap. 21 of “Common Lisp Recipes”), ubiquitous,…),
Object Relational Mappers (Mito),

and other DB-related tools (pgloader).

We’ll begin with an overview of Mito. If you must work with an existing DB, you might want to have a look at cl-dbi and clsql. If you don’t need a SQL database and want automatic persistence of Lisp objects, you also have a choice of libraries.

The Mito ORM and SxQL

Mito is in Quicklisp:

(ql:quickload :mito)

Overview

Mito is “an ORM for Common Lisp with migrations, relationships and PostgreSQL support”.

it supports MySQL, PostgreSQL and SQLite3,
when defining a model, it adds an id (serial primary key), created_at and updated_at fields by default like Ruby’s ActiveRecord or Django,
handles DB migrations for the supported backends,
permits DB schema versioning,
is tested under SBCL and CCL.

As an ORM, it allows to write class definitions, to specify relationships, and provides functions to query the database. For custom queries, it relies on SxQL, an SQL generator that provides the same interface for several backends.

Working with Mito generally involves these steps:

connecting to the DB
writing CLOS classes to define models
running migrations to create or alter tables
creating objects, saving same in the DB,

and iterating.

Connecting to a DB

Mito provides the function connect-toplevel to establish a connection to RDBMs:

(mito:connect-toplevel :mysql :database-name "myapp" :username "fukamachi" :password "c0mon-1isp")

The driver type can be of :mysql, :sqlite3 and :postgres.

With sqlite you don’t need the username and password:

(connect-toplevel :sqlite3 :database-name "myapp")

As usual, you need to create the MySQL or Postgre database beforehand. Refer to their documentation.

Connecting sets mito:*connection* to the new connection and returns it.

Disconnect with disconnect-toplevel.

=> you might make good use of a wrapper function:

(defun connect ()
  "Connect to the DB."
  (connect-toplevel :sqlite3 :database-name "myapp"))

Models

Defining models

In Mito, you can define a class which corresponds to a database table by specifying (:metaclass mito:dao-table-class):

(defclass user ()
  ((name :col-type (:varchar 64)
         :initarg :name
         :accessor user-name)
   (email :col-type (or (:varchar 128) :null)
          :initarg :email
          :accessor user-email))
  (:metaclass mito:dao-table-class))

Note that the class automatically adds some slots: a primary key named id if there’s no primary keys, created_at and updated_at for recording timestamps. To disable these behaviors, specify :auto-pk nil or :record-timestamps nil to the defclass forms.

You can inspect the new class:

(mito.class:table-column-slots (find-class 'user))
;=> (#<MITO.DAO.COLUMN:DAO-TABLE-COLUMN-CLASS MITO.DAO.MIXIN::ID>
;    #<MITO.DAO.COLUMN:DAO-TABLE-COLUMN-CLASS COMMON-LISP-USER::NAME>
;    #<MITO.DAO.COLUMN:DAO-TABLE-COLUMN-CLASS COMMON-LISP-USER::EMAIL>
;    #<MITO.DAO.COLUMN:DAO-TABLE-COLUMN-CLASS MITO.DAO.MIXIN::CREATED-AT>
;    #<MITO.DAO.COLUMN:DAO-TABLE-COLUMN-CLASS MITO.DAO.MIXIN::UPDATED-AT>)

The class inherits mito:dao-class implicitly.

(find-class 'user)
;=> #<MITO.DAO.TABLE:DAO-TABLE-CLASS COMMON-LISP-USER::USER>

(c2mop:class-direct-superclasses *)
;=> (#<STANDARD-CLASS MITO.DAO.TABLE:DAO-CLASS>)

This may be useful when you define methods which can be applied for all table classes.

For more information on using the Common Lisp Object System, see the clos page.

Creating the tables

After defining the models, you must create the tables:

(mito:ensure-table-exists 'user)

So a helper function:

(defun ensure-tables ()
  (mapcar #'mito:ensure-table-exists '(user foo bar)))

See Mito’s documentation for a couple more ways.

When you alter the model you’ll need to run a DB migration, see the next section.

Fields

Fields types

Field types are:

(:varchar <integer>) ,

:serial, :bigserial, :integer, :bigint, :unsigned,

:timestamp, :timestamptz,

:bytea,

Optional fields

Use (or <real type> :null):

   (email :col-type (or (:varchar 128) :null)
          :initarg :email
          :accessor user-email))

Field constraints

:unique-keys can be used like so:

(defclass user ()
  ((name :col-type (:varchar 64)
         :initarg :name
         :accessor user-name)
   (email :col-type (:varchar 128)
          :initarg :email
          :accessor user-email))
  (:metaclass mito:dao-table-class)
  (:unique-keys email))

We already saw :primary-key.

You can change the table name with :table-name.

Relationships

You can define a relationship by specifying a foreign class with :col-type:

(defclass tweet ()
  ((status :col-type :text
           :initarg :status
           :accessor tweet-status)
   ;; This slot refers to USER class
   (user :col-type user
         :initarg :user
         :accessor tweet-user))
  (:metaclass mito:dao-table-class))

(table-definition (find-class 'tweet))
;=> (#<SXQL-STATEMENT: CREATE TABLE tweet (
;        id BIGSERIAL NOT NULL PRIMARY KEY,
;        status TEXT NOT NULL,
;        user_id BIGINT NOT NULL,
;        created_at TIMESTAMP,
;        updated_at TIMESTAMP
;    )>)

Now you can create or retrieve a TWEET by a USER object, not a USER-ID.

(defvar *user* (mito:create-dao 'user :name "Eitaro Fukamachi"))
(mito:create-dao 'tweet :user *user*)

(mito:find-dao 'tweet :user *user*)

Mito doesn’t add foreign key constraints for refering tables.

One-to-one

A one-to-one relationship is simply represented with a simple foreign key on a slot (as :col-type user in the tweet class). Besides, we can add a unicity constraint, as with (:unique-keys email).

One-to-many, many-to-one

The tweet example above shows a one-to-many relationship between a user and his tweets: a user can write many tweets, and a tweet belongs to only one user.

The relationship is defined with a foreign key on the “many” side linking back to the “one” side. Here the tweet class defines a user foreign key, so a tweet can only have one user. You didn’t need to edit the user class.

A many-to-one relationship is actually the contraty of a one-to-many. You have to put the foreign key on the approriate side.

Many-to-many

A many-to-many relationship needs an intermediate table, which will be the “many” side for the two tables it is the intermediary of.

And, thanks to the join table, we can store more information about the relationship.

Let’s define a book class:

(defclass book ()
    ((title
       :col-type (:varchar 128)
       :initarg :title
       :accessor title)
     (ean
       :col-type (or (:varchar 128) :null)
       :initarg :ean
       :accessor ean))
    (:metaclass mito:dao-table-class))

A user can have many books, and a book (as the title, not the physical copy) is likely to be in many people’s library. Here’s the intermediate class:

(defclass user-books ()
    ((user
      :col-type user
      :initarg :user)
    (book
      :col-type book
      :initarg :book))
    (:metaclass mito:dao-table-class))

Each time we want to add a book to a user’s collection (say in a add-book function), we create a new user-books object.

But someone may very well own many copies of one book. This is an information we can store in the join table:

(defclass user-books ()
    ((user
      :col-type user
      :initarg :user)
    (book
      :col-type book
      :initarg :book)
    ;; Set the quantity, 1 by default:
    (quantity
      :col-type :integer
      :initarg :quantity
      :initform 1
      :accessor quantity))
    (:metaclass mito:dao-table-class))

Inheritance and mixin

A subclass of DAO-CLASS is allowed to be inherited. This may be useful when you need classes which have similar columns:

(defclass user ()
  ((name :col-type (:varchar 64)
         :initarg :name
         :accessor user-name)
   (email :col-type (:varchar 128)
          :initarg :email
          :accessor user-email))
  (:metaclass mito:dao-table-class)
  (:unique-keys email))

(defclass temporary-user (user)
  ((registered-at :col-type :timestamp
                  :initarg :registered-at
                  :accessor temporary-user-registered-at))
  (:metaclass mito:dao-table-class))

(mito:table-definition 'temporary-user)
;=> (#<SXQL-STATEMENT: CREATE TABLE temporary_user (
;        id BIGSERIAL NOT NULL PRIMARY KEY,
;        name VARCHAR(64) NOT NULL,
;        email VARCHAR(128) NOT NULL,
;        registered_at TIMESTAMP NOT NULL,
;        created_at TIMESTAMP,
;        updated_at TIMESTAMP,
;        UNIQUE (email)
;    )>)

If you need a ‘template’ for tables which doesn’t related to any database tables, you can use DAO-TABLE-MIXIN. Below the has-email class will not create a table.

(defclass has-email ()
  ((email :col-type (:varchar 128)
          :initarg :email
          :accessor object-email))
  (:metaclass mito:dao-table-mixin)
  (:unique-keys email))
;=> #<MITO.DAO.MIXIN:DAO-TABLE-MIXIN COMMON-LISP-USER::HAS-EMAIL>

(defclass user (has-email)
  ((name :col-type (:varchar 64)
         :initarg :name
         :accessor user-name))
  (:metaclass mito:dao-table-class))
;=> #<MITO.DAO.TABLE:DAO-TABLE-CLASS COMMON-LISP-USER::USER>

(mito:table-definition 'user)
;=> (#<SXQL-STATEMENT: CREATE TABLE user (
;       id BIGSERIAL NOT NULL PRIMARY KEY,
;       name VARCHAR(64) NOT NULL,
;       email VARCHAR(128) NOT NULL,
;       created_at TIMESTAMP,
;       updated_at TIMESTAMP,
;       UNIQUE (email)
;   )>)

See more examples of use in mito-auth.

Troubleshooting

“Cannot CHANGE-CLASS objects into CLASS metaobjects.”

If you get the following error message:

Cannot CHANGE-CLASS objects into CLASS metaobjects.
   [Condition of type SB-PCL::METAOBJECT-INITIALIZATION-VIOLATION]
See also:
  The Art of the Metaobject Protocol, CLASS [:initialization]

it is certainly because you first wrote a class definition and then added the Mito metaclass and tried to evaluate the class definition again.

If this happens, you must remove the class definition from the current package:

(setf (find-class 'foo) nil)

or, with the Slime inspector, click on the class and find the “remove” button.

More info here.

Migrations

First create the tables if needed:

(ensure-table-exists 'user)

then alter the tables, if needed:

(mito:migrate-table 'user)

You can check the SQL generated code with migration-expressions 'class. For example, we create the user table:

(ensure-table-exists 'user)
;-> ;; CREATE TABLE IF NOT EXISTS "user" (
;       "id" BIGSERIAL NOT NULL PRIMARY KEY,
;       "name" VARCHAR(64) NOT NULL,
;       "email" VARCHAR(128),
;       "created_at" TIMESTAMP,
;       "updated_at" TIMESTAMP
;   ) () [0 rows] | MITO.DAO:ENSURE-TABLE-EXISTS

There are no changes from the previous user definition:

(mito:migration-expressions 'user)
;=> NIL

Now let’s add a unique email field:

(defclass user ()
  ((name :col-type (:varchar 64)
         :initarg :name
         :accessor user-name)
   (email :col-type (:varchar 128)
          :initarg :email
          :accessor user-email))
  (:metaclass mito:dao-table-class)
  (:unique-keys email))

The migration will run the following code:

(mito:migration-expressions 'user)
;=> (#<SXQL-STATEMENT: ALTER TABLE user ALTER COLUMN email TYPE character varying(128), ALTER COLUMN email SET NOT NULL>
;    #<SXQL-STATEMENT: CREATE UNIQUE INDEX unique_user_email ON user (email)>)

so let’s apply it:

(mito:migrate-table 'user)
;-> ;; ALTER TABLE "user" ALTER COLUMN "email" TYPE character varying(128), ALTER COLUMN "email" SET NOT NULL () [0 rows] | MITO.MIGRATION.TABLE:MIGRATE-TABLE
;   ;; CREATE UNIQUE INDEX "unique_user_email" ON "user" ("email") () [0 rows] | MITO.MIGRATION.TABLE:MIGRATE-TABLE
;-> (#<SXQL-STATEMENT: ALTER TABLE user ALTER COLUMN email TYPE character varying(128), ALTER COLUMN email SET NOT NULL>
;    #<SXQL-STATEMENT: CREATE UNIQUE INDEX unique_user_email ON user (email)>)

Queries

Creating objects

We can create user objects with the regular make-instance:

(defvar me
  (make-instance 'user :name "Eitaro Fukamachi" :email "e.arrows@gmail.com"))
;=> USER

To save it in DB, use insert-dao:

(mito:insert-dao me)
;-> ;; INSERT INTO `user` (`name`, `email`, `created_at`, `updated_at`) VALUES (?, ?, ?, ?) ("Eitaro Fukamachi", "e.arrows@gmail.com", "2016-02-04T19:55:16.365543Z", "2016-02-04T19:55:16.365543Z") [0 rows] | MITO.DAO:INSERT-DAO
;=> #<USER {10053C4453}>

Do the two steps above at once:

(mito:create-dao 'user :name "Eitaro Fukamachi" :email "e.arrows@gmail.com")

You should not export the user class and create objects outside of its package (it is good practice anyway to keep all database-related operations in say a models package and file). You should instead use a helper function:

(defun make-user (&key name)
  (make-instance 'user :name name))

Updating fields

(setf (slot-value me 'name) "nitro_idiot")
;=> "nitro_idiot"

and save it:

(mito:save-dao me)

Deleting

(mito:delete-dao me)
;-> ;; DELETE FROM `user` WHERE (`id` = ?) (1) [0 rows] | MITO.DAO:DELETE-DAO

;; or:
(mito:delete-by-values 'user :id 1)
;-> ;; DELETE FROM `user` WHERE (`id` = ?) (1) [0 rows] | MITO.DAO:DELETE-DAO

Get the primary key value

(mito:object-id me)
;=> 1

Count

(mito:count-dao 'user)
;=> 1

Find one

(mito:find-dao 'user :id 1)
;-> ;; SELECT * FROM `user` WHERE (`id` = ?) LIMIT 1 (1) [1 row] | MITO.DB:RETRIEVE-BY-SQL
;=> #<USER {10077C6073}>

So here’s a possibility of generic helpers to find an object by a given key:

(defgeneric find-user (key-name key-value)
  (:documentation "Retrieves an user from the data base by one of the unique
keys."))

(defmethod find-user ((key-name (eql :id)) (key-value integer))
  (mito:find-dao 'user key-value))

(defmethod find-user ((key-name (eql :name)) (key-value string))
  (first (mito:select-dao 'user
                          (sxql:where (:= :name key-value)))))

Find all

Use select-dao.

Get a list of all users:

(mito:select-dao 'user)
;=> (#<USER {10077C6073}>)

Find by relationship

As seen above:

(mito:find-dao 'tweet :user *user*)

Custom queries

It is with select-dao that you can write more precise queries by giving it SxQL statements.

Example:

(select-dao 'tweet
    (where (:like :status "%Japan%")))

Clauses

See the SxQL documentation.

Examples:

(select-dao 'foo
  (where (:and (:> :age 20) (:<= :age 65))))

(order-by :age (:desc :id))

(group-by :sex)

(having (:>= (:sum :hoge) 88))

(limit 0 10)

and joins, etc.

Operators

:not
:is-null, :not-null
:asc, :desc
:distinct
:=, :!=
:<, :>, :<= :>=
:a<, :a>
:as
:in, :not-in
:like
:and, :or
:+, :-, :* :/ :%
:raw

Triggers

Since insert-dao, update-dao and delete-dao are defined as generic functions, you can define :before, :after or :around methods to those, like regular method combination.

(defmethod mito:insert-dao :before ((object user))
  (format t "~&Adding ~S...~%" (user-name object)))

(mito:create-dao 'user :name "Eitaro Fukamachi" :email "e.arrows@gmail.com")
;-> Adding "Eitaro Fukamachi"...
;   ;; INSERT INTO "user" ("name", "email", "created_at", "updated_at") VALUES (?, ?, ?, ?) ("Eitaro Fukamachi", "e.arrows@gmail.com", "2016-02-16 21:13:47", "2016-02-16 21:13:47") [0 rows] | MITO.DAO:INSERT-DAO
;=> #<USER {100835FB33}>

Inflation/Deflation

Inflation/Deflation is a function to convert values between Mito and RDBMS.

(defclass user-report ()
  ((title :col-type (:varchar 100)
          :initarg :title
          :accessor report-title)
   (body :col-type :text
         :initarg :body
         :initform ""
         :accessor report-body)
   (reported-at :col-type :timestamp
                :initarg :reported-at
                :initform (local-time:now)
                :accessor report-reported-at
                :inflate #'local-time:universal-to-timestamp
                :deflate #'local-time:timestamp-to-universal))
  (:metaclass mito:dao-table-class))

Eager loading

One of the pains in the neck to use ORMs is the “N+1 query” problem.

;; BAD EXAMPLE

(use-package '(:mito :sxql))

(defvar *tweets-contain-japan*
  (select-dao 'tweet
    (where (:like :status "%Japan%"))))

;; Getting names of tweeted users.
(mapcar (lambda (tweet)
          (user-name (tweet-user tweet)))
        *tweets-contain-japan*)

This example sends a query to retrieve a user like “SELECT * FROM user WHERE id = ?” at each iteration.

To prevent this performance issue, add includes to the above query which only sends a single WHERE IN query instead of N queries:

;; GOOD EXAMPLE with eager loading

(use-package '(:mito :sxql))

(defvar *tweets-contain-japan*
  (select-dao 'tweet
    (includes 'user)
    (where (:like :status "%Japan%"))))
;-> ;; SELECT * FROM `tweet` WHERE (`status` LIKE ?) ("%Japan%") [3 row] | MITO.DB:RETRIEVE-BY-SQL
;-> ;; SELECT * FROM `user` WHERE (`id` IN (?, ?, ?)) (1, 3, 12) [3 row] | MITO.DB:RETRIEVE-BY-SQL
;=> (#<TWEET {1003513EC3}> #<TWEET {1007BABEF3}> #<TWEET {1007BB9D63}>)

;; No additional SQLs will be executed.
(tweet-user (first *))
;=> #<USER {100361E813}>

Schema versioning

$ ros install mito
$ mito
Usage: mito command [option...]

Commands:
    generate-migrations
    migrate

Options:
    -t, --type DRIVER-TYPE          DBI driver type (one of "mysql", "postgres" or "sqlite3")
    -d, --database DATABASE-NAME    Database name to use
    -u, --username USERNAME         Username for RDBMS
    -p, --password PASSWORD         Password for RDBMS
    -s, --system SYSTEM             ASDF system to load (several -s's allowed)
    -D, --directory DIRECTORY       Directory path to keep migration SQL files (default: "/Users/nitro_idiot/Programs/lib/mito/db/")
    --dry-run                       List SQL expressions to migrate

Introspection

Mito provides some functions for introspection.

We can access the information of columns with the functions in (mito.class.column:...):

table-column-[class, name, info, not-null-p,...]
primary-key-p

and likewise for tables with (mito.class.table:...).

Given we get a list of slots of our class:

(ql:quickload "closer-mop")

(closer-mop:class-direct-slots (find-class 'user))
;; (#<MITO.DAO.COLUMN:DAO-TABLE-COLUMN-CLASS NAME>
;;  #<MITO.DAO.COLUMN:DAO-TABLE-COLUMN-CLASS EMAIL>)

(defparameter user-slots *)

We can answer the following questions:

What is the type of this column ?

(mito.class.column:table-column-type (first user-slots))
;; (:VARCHAR 64)

Is this column nullable ?

(mito.class.column:table-column-not-null-p
  (first user-slots))
;; T
(mito.class.column:table-column-not-null-p
  (second user-slots))
;; NIL

Testing

We don’t want to test DB operations against the production one. We need to create a temporary DB before each test.

The macro below creates a temporary DB with a random name, creates the tables, runs the code and connects back to the original DB connection.

(defun random-string (length)
  ;; thanks 40ants/hacrm.
  (let ((chars "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"))
    (coerce (loop repeat length
                  collect (aref chars (random (length chars))))
            'string)))

(defmacro with-empty-db (&body body)
  "Run `body` with a new temporary DB.
  "
  `(let* ((*random-state* (make-random-state t))
          (prefix (concatenate 'string
                               (random-string 8)
                               "/"))
          (connection mito:*connection*))
     (uiop:with-temporary-file (:pathname name :prefix prefix)
       (let* ((*db-name* name)
              (*db* (connect)))
         (ensure-tables-exist)
         (migrate-all)
         ,@body
         (setf mito:*connection* connection)
         (connect)))))

Use it like this:

(prove:subtest "Creation in a temporary DB."
  (with-empty-db
    (let ((user (make-user :name "Cookbook")))
      (save-user user)

      (prove:is (name user)
                "Cookbook"
                "Test username in a temp DB."))))
;; Creation in a temporary DB
;;  CREATE TABLE "user" (
;;       id BIGSERIAL NOT NULL PRIMARY KEY,
;;       name VARCHAR(64) NOT NULL,
;;       email VARCHAR(128) NOT NULL,
;;       created_at TIMESTAMP,
;;       updated_at TIMESTAMP,
;;       UNIQUE (email)
;; ) () [0 rows] | MITO.DB:EXECUTE-SQL
;; ✓ Test username in a temp DB.

These months in Common Lisp: Q1 2018

Sun, 01 Apr 2018 07:51:49 +0100

Documentation

Announcements

Projects

Next web browser
cl-repl, the Common Lisp ipython-like REPL
Emotiq - blockchain in Common Lisp
original reddit code from 2005. reddit.
Temperance - logic programming (in development, reached v1.0.0)
QR code 2005 encoder in Common Lisp
print-licences: print licenses used by the given project and its dependencies (from sjl and dk_jackdaniel’s utils)
Dufy, color library
cl-flow - Data-flowish computation tree library for non-blocking concurrent Common Lisp.
A docker container for CL development (also lisp-devel, CI on CL Cookbook)
Quickfork - Quicklisp for teams
Apache Thrift gains CL support
JSCL playground
MAGICL: Matrix Algebra proGrams In Common Lisp - Rigetti Computing
Eturia has been open-sourced. An Open Computer for Education for teaching #Lisp written in #Javascript.

New releases:

Discoveries:

Articles

Discussion

Common Lisp VS …

Screencasts

Jobs

If we missed anything: https://gitlab.com/lisp-journey/lisp-journey.gitlab.io

Tip: capture standard and error output

Tue, 06 Mar 2018 07:51:49 +0100

What if we want to capture standard (and/or error) output in order to ignore it or post-process it ? It’s very simple, a little search and we’re good:

(let ((*standard-output* (make-string-output-stream))
      (*error-output* (make-string-output-stream)))
  (apply function args) ;; anything
  (setf standard-output (get-output-stream-string *standard-output*)))
(print-results standard-output))

and now in print-results we can print to standard output without being intercepted (and in our case, we’ll highlight some user-defined keywords).

Above, just don’t forget to get the output content with (get-output-stream-string *standard-output*).

A thing to note is that if your app printed stuff on error output and standard output consecutively, now it will print all standard output as a single block.

(edit) Of course, with-output-to-string is simpler to capture one stream:

(setf standard-output (with-output-to-string (*standard-output*)
                        (apply function args)))

edit 2, thanks to redditers:

Don’t bind standard-output directly; bind the string stream to a lexical, then bind *standard-output* to that:

(with-output-to-string (s)
  (let ((*standard-output* s)) (write-string "abc")))
-> "abc"

Now, let’s bind both *standard-output* and *standard-error* to s:

(with-output-to-string (s)
  (let ((*standard-output* s)
        (*standard-error* s))
    (write-string "abc")
    (write-string "def" *standard-error*)))
-> "abcdef"

Eliminate s and just bind *standard-output* and then tie *standard-error* to the same stream:

(with-output-to-string (*standard-output*)
  (let ((*standard-error* *standard-output*))
    (write-string "abc")
    (write-string "def" *standard-error*)))
--> "abcdef"

The conclusion stays: it’s handy and easy :)

Fixing a CL21 error message in an unrelated library after a quicklisp update (it's about cache)

Mon, 12 Feb 2018 07:51:49 +0100

I just updated my Quicklisp dist and suddenly couldn’t load some libraries any more. I got an error related to cl21 looking like the one below (I didn’t note the exact message sorry), even though the library was unrelate to cl21 (it was about osicat and cffi.grovel):

couldn’t find adjustable-vectors from CL21.core.arrays

If you skip through the restarts you’ll see mentions of a cache in ~/.cache/common-lisp/sbclxx-xx/quicklisp/…. It contains the compiled .fasl files.

I deleted this cache and I’m good to go.

(bookmark) Get a list of all the dependencies of a lisp system

Tue, 23 Jan 2018 07:51:49 +0100

I’ll save here a reddit discussion, which I find interesting but that will be burried quickly down reddit’s history. The goal is to get all the dependencies of a system.

You’d better read the OP’s question and the discussion (where the OP is the experimented svetlyak40wt/40ants, at the moment doing a god’s work on Weblocks).

His solution is https://gist.github.com/svetlyak40wt/03bc68c820bb3e45bc7871870379c42e

(ql:quickload :fset)

(defun get-dependencies (system)
  "Returns a set with all dependencies of a given system.
   System should be loaded first."
  (labels ((normalize (name)
             (etypecase name
               (string (string-downcase name))
               (symbol (normalize (symbol-name name)))
               (list
                (let ((dep-type (first name))
                      (supported-dep-types (list :version :feature :require)))
                  (unless (member dep-type
                                  supported-dep-types)
                    (error "This component \"~A\" should have first element from this list: ~A."
                           name
                           supported-dep-types))

                  (normalize
                   (case dep-type
                     (:version (second name))
                     (:feature (third name))
                     (:require (second name)))))))))

    (let ((processed (fset:set))
          (queue (fset:set (normalize system))))

      (do ((current-name (fset:arb queue)
                         (fset:arb queue)))
          ((null current-name)
           ;; return result
           processed)

        ;; Remove current name from the queue
        (setf queue
              (fset:less queue current-name))
        ;; And put it into the "processed" pool
        (setf processed
              (fset:with processed current-name))

        ;; And add it's dependencies which aren't processed or in the queue already
        ;; Sometimes system can't be found because itself depends on some feature,
        ;; for example, you can specify dependency as a list:
        ;; (:FEATURE :SBCL (:REQUIRE :SB-INTROSPECT))
        ;; and it will be loaded only on SBCL.
        ;; When we are collecting dependencies on another implementation,
        ;; we don't want to fail with an error because ASDF is unable to find
        ;; such dependencies
        (let* ((system (ignore-errors
                        (asdf:find-system current-name)))
               (deps (when system
                       (asdf:component-sideway-dependencies system))))
          (dolist (dep deps)
            (let ((normalized-dep (normalize dep)))
              (unless (or (fset:lookup processed normalized-dep)
                          (fset:lookup queue normalized-dep))
                (setf queue
                      (fset:with queue normalized-dep)))))))

      (values processed))))

#|
DEPENDENCIES> (ql:quickload :clinch)
DEPENDENCIES> (get-dependencies :clinch)
#{
  "cffi"
  "sdl2"
  "uiop"
  "babel"
  "swank"
  "clinch"
  "cl-glut"
  "cl-json"
  "cl-ppcre"
  "rtg-math"
  "cl-opengl"
  "cl-plus-c"
  "alexandria"
  "cl-autowrap"
  "glsl-symbols"
  "defpackage-plus"
  "trivial-garbage"
  "trivial-timeout"
  "bordeaux-threads"
  "trivial-channels"
  "trivial-features" }
|#

There’s also (ql-dist:dependency-tree "mgl") which has limitations though, it’s only for Quicklisp projects and doesn’t work with everything (see the thread).

That’s all folks !

snippets - functional style, sequences, debugging and more utilities

Tue, 23 Jan 2018 07:51:49 +0100

From sjl’s utilities (thanks so much for the nice docstrings). The goal here is to read some code and learn about (hidden) gems.

The following snippets should be copy-pastable. They are the ones I find most interesting, I left some behind.

To reduce the dependency load, Alexandria or Quickutil functions can be imported one by one with Quickutil.

Table of Contents

Higher order functions

(defun juxt (&rest functions)
  "Return a function that will juxtapose the results of `functions`.
  This is like Clojure's `juxt`.  Given functions `(f0 f1 ... fn)`, this will
  return a new function which, when called with some arguments, will return
  `(list (f0 ...args...) (f1 ...args...) ... (fn ...args...))`.
  Example:
    (funcall (juxt #'list #'+ #'- #'*) 1 2)
    => ((1 2) 3 -1 2)
  "
  (lambda (&rest args)
    (mapcar (alexandria:rcurry #'apply args) functions)))

(defun nullary (function &optional result)
  "Return a new function that acts as a nullary-patched version of `function`.
  The new function will return `result` when called with zero arguments, and
  delegate to `function` otherwise.
  Examples:
    (max 1 10 2) ; => 10
    (max)        ; => invalid number of arguments
    (funcall (nullary #'max))          ; => nil
    (funcall (nullary #'max 0))        ; => 0
    (funcall (nullary #'max 0) 1 10 2) ; => 10
    (reduce #'max nil)                  ; => invalid number of arguments
    (reduce (nullary #'max) nil)        ; => nil
    (reduce (nullary #'max :empty) nil) ; => :empty
    (reduce (nullary #'max) '(1 10 2))  ; => 10
  "
  (lambda (&rest args)
    (if (null args) result (apply function args))))

(defmacro gathering (&body body)
  ;; https://github.com/sjl/cl-losh/blob/master/losh.lisp#L515
  "Run `body` to gather some things and return a fresh list of them.
  `body` will be executed with the symbol `gather` bound to a function of one
  argument.  Once `body` has finished, a list of everything `gather` was called
  on will be returned.
  It's handy for pulling results out of code that executes procedurally and
  doesn't return anything, like `maphash` or Alexandria's `map-permutations`.
  The `gather` function can be passed to other functions, but should not be
  retained once the `gathering` form has returned (it would be useless to do so
  anyway).
  Examples:
    (gathering
      (dotimes (i 5)
        (gather i))
    =>
    (0 1 2 3 4)
    (gathering
      (mapc #'gather '(1 2 3))
      (mapc #'gather '(a b)))
    =>
    (1 2 3 a b)
  "
  (with-gensyms (result)
    `(let ((,result (make-queue)))
      (flet ((gather (item)
               (enqueue item ,result)))
        (declare (dynamic-extent #'gather))
        ,@body)
      (queue-contents ,result))))

Here we need the queue struct.

(defstruct (queue (:constructor make-queue))
  (contents nil :type list)
  (last nil :type list)
  (size 0 :type fixnum))

;; real code is richer, with inline and inlinable function declarations.

(defun make-queue ()
  "Allocate and return a fresh queue."
  (make-queue%))

(defun queue-empty-p (queue)
  "Return whether `queue` is empty."
  (zerop (queue-size queue)))

(defun enqueue (item queue)
  "Enqueue `item` in `queue`, returning the new size of the queue."
  (let ((cell (cons item nil)))
    (if (queue-empty-p queue)
      (setf (queue-contents queue) cell)
      (setf (cdr (queue-last queue)) cell))
    (setf (queue-last queue) cell))
  (incf (queue-size queue)))

(defun dequeue (queue)
  "Dequeue an item from `queue` and return it."
  (when (zerop (decf (queue-size queue)))
    (setf (queue-last queue) nil))
  (pop (queue-contents queue)))

(defun queue-append (queue list)
  "Enqueue each element of `list` in `queue` and return the queue's final size."
  (loop :for item :in list
        :for size = (enqueue item queue)
        :finally (return size)))

Sequences

(defun frequencies (sequence &key (test 'eql))
  ;; https://github.com/sjl/cl-losh/blob/master/losh.lisp#L1910
  "Return a hash table containing the frequencies of the items in `sequence`.
  Uses `test` for the `:test` of the hash table.
  Example:
    (frequencies '(foo foo bar))
    => {foo 2
        bar 1}
  "
  (iterate
    (with result = (make-hash-table :test test))
    (for i :in-whatever sequence)
    (incf (gethash i result 0))
    (finally (return result))))

(defun proportions (sequence &key (test 'eql) (float t))
  "Return a hash table containing the proportions of the items in `sequence`.
  Uses `test` for the `:test` of the hash table.
  If `float` is `t` the hash table values will be coerced to floats, otherwise
  they will be left as rationals.
  Example:
    (proportions '(foo foo bar))
    => {foo 0.66666
        bar 0.33333}
    (proportions '(foo foo bar) :float nil)
    => {foo 2/3
        bar 1/3}
  "
  (let* ((freqs (frequencies sequence :test test))
         (total (reduce #'+ (hash-table-values freqs)
                        :initial-value (if float 1.0 1))))
    (mutate-hash-values (lambda (v) (/ v total))
                        freqs)))

(defun group-by (function sequence &key (test #'eql) (key #'identity))
  "Return a hash table of the elements of `sequence` grouped by `function`.
  This function groups the elements of `sequence` into buckets.  The bucket for
  an element is determined by calling `function` on it.
  The result is a hash table (with test `test`) whose keys are the bucket
  identifiers and whose values are lists of the elements in each bucket.  The
  order of these lists is unspecified.
  If `key` is given it will be called on each element before passing it to
  `function` to produce the bucket identifier.  This does not effect what is
  stored in the lists.
  Examples:
    (defparameter *items* '((1 foo) (1 bar) (2 cats) (3 cats)))
    (group-by #'first *items*)
    ; => { 1 ((1 foo) (1 bar))
    ;      2 ((2 cats))
    ;      3 ((3 cats)) }
    (group-by #'second *items*)
    ; => { foo  ((1 foo))
    ;      bar  ((1 bar))
    ;      cats ((2 cats) (3 cats)) }
    (group-by #'evenp *items* :key #'first)
    ; => { t   ((2 cats))
    ;      nil ((1 foo) (1 bar) (3 cats)) }
  "
  (iterate
    (with result = (make-hash-table :test test))
    (for i :in-whatever sequence)
    (push i (gethash (funcall function (funcall key i)) result))
    (finally (return result))))


(defun take-list (n list)
  (iterate (declare (iterate:declare-variables))
           (repeat n)
           (for item :in list)
           (collect item)))

(defun take-seq (n seq)
  (subseq seq 0 (min n (length seq))))

(defmacro do-repeat (n &body body)
  "Perform `body` `n` times."
  `(dotimes (,(gensym) ,n)
     ,@body))

(defmacro do-range (ranges &body body)
  "Perform `body` on the given `ranges`.

  Each range in `ranges` should be of the form `(variable from below)`.  During
  iteration `body` will be executed with `variable` bound to successive values
  in the range [`from`, `below`).

  If multiple ranges are given they will be iterated in a nested fashion.

  Example:

    (do-range ((x  0  3)
               (y 10 12))
      (pr x y))
    ; =>
    ; 0 10
    ; 0 11
    ; 1 10
    ; 1 11
    ; 2 10
    ; 2 11

  "
  (if (null ranges)
    `(progn ,@body)
    (destructuring-bind (var from below) (first ranges)
      `(loop :for ,var :from ,from :below ,below
             :do (do-range ,(rest ranges) ,@body)))))

(defun enumerate (sequence &key (start 0) (step 1) key)
  "Return an alist of `(n . element)` for each element of `sequence`.
  `start` and `step` control the values generated for `n`, NOT which elements of
  the sequence are enumerated.
  Examples:
    (enumerate '(a b c))
    ; => ((0 . A) (1 . B) (2 . C))
    (enumerate '(a b c) :start 1)
    ; => ((1 . A) (2 . B) (3 . C))
    (enumerate '(a b c) :key #'ensure-keyword)
    ; => ((0 . :A) (1 . :B) (2 . :C))
  "
  (iterate (for el :in-whatever sequence)
           (for n :from start :by step)
           (collect (cons n (if key
                              (funcall key el)
                              el)))))

uses iterate, on Quicklisp (see also Shinmera’s For).

The followingtake is taken from Serapeum (also available in CL21).

The original helpers (take-list, etc) are originally inlined for optimal performance with a custom “defun-inline”.

(defun take (n seq)
  "Return a fresh sequence of the first `n` elements of `seq`.
  The result will be of the same type as `seq`.
  If `seq` is shorter than `n` a shorter result will be returned.
  Example:
    (take 2 '(a b c))
    => (a b)
    (take 4 #(1))
    => #(1)
  From Serapeum.
  "
  (check-type n array-index)
  (ctypecase seq
    (list (take-list n seq))
    (sequence (take-seq n seq))))

(defun take-list (n list)
  (iterate (declare (iterate:declare-variables))
           (repeat n)
           (for item :in list)
           (collect item)))

(defun take-seq (n seq)
  (subseq seq 0 (min n (length seq))))

(defun take-while-list (predicate list)
  (iterate (for item :in list)
           (while (funcall predicate item))
           (collect item)))

(defun take-while-seq (predicate seq)
  (subseq seq 0 (position-if-not predicate seq)))

(defun take-while (predicate seq)
  "Take elements from `seq` as long as `predicate` remains true.
  The result will be a fresh sequence of the same type as `seq`.
  Example:
    (take-while #'evenp '(2 4 5 6 7 8))
    ; => (2 4)
    (take-while #'evenp #(1))
    ; => #()
  "
  (ctypecase seq
    (list (take-while-list predicate seq))
    (sequence (take-while-seq predicate seq))))

(defun drop-list (n list)
  (copy-list (nthcdr n list)))

(defun drop-seq (n seq)
  (subseq seq (min n (length seq))))

(defun drop (n seq)
  "Return a fresh copy of the `seq` without the first `n` elements.
  The result will be of the same type as `seq`.
  If `seq` is shorter than `n` an empty sequence will be returned.
  Example:
    (drop 2 '(a b c))
    => (c)
    (drop 4 #(1))
    => #()
  From Serapeum.
  "
  (check-type n array-index)
  (ctypecase seq
    (list (drop-list n seq))
    (sequence (drop-seq n seq))))

(defun drop-while-list (predicate list)
  (iterate (for tail :on list)
           (while (funcall predicate (first tail)))
           (finally (return (copy-list tail)))))

(defun drop-while-seq (predicate seq)
  (let ((start (position-if-not predicate seq)))
    (if start
      (subseq seq start)
      (subseq seq 0 0))))

(defun drop-while (predicate seq)
  "Drop elements from `seq` as long as `predicate` remains true.
  The result will be a fresh sequence of the same type as `seq`.
  Example:
    (drop-while #'evenp '(2 4 5 6 7 8))
    ; => (5 6 7 8)
    (drop-while #'evenp #(2))
    ; => #(2)
  "
  (ctypecase seq
    (list (drop-while-list predicate seq))
    (sequence (drop-while-seq predicate seq))))

(defun extrema (predicate sequence)
  "Return the smallest and largest elements of `sequence` according to `predicate`.
  `predicate` should be a strict ordering predicate (e.g. `<`).
  Returns the smallest and largest elements in the sequence as two values,
  respectively.
  "
  (iterate (with min = (elt sequence 0))
           (with max = (elt sequence 0))
           (for el :in-whatever sequence)
           (when (funcall predicate el min) (setf min el))
           (when (funcall predicate max el) (setf max el))
           (finally (return (values min max)))))

(defun summation (sequence &key key)
  "Return the sum of all elements of `sequence`.
  If `key` is given, it will be called on each element to compute the addend.
  This function's ugly name was chosen so it wouldn't clash with iterate's `sum`
  symbol.  Sorry.
  Examples:
    (sum #(1 2 3))
    ; => 6
    (sum '(\"1\" \"2\" \"3\") :key #'parse-integer)
    ; => 6
    (sum '(\"1\" \"2\" \"3\") :key #'length)
    ; => 3
  "
  (if key
    (iterate (for n :in-whatever sequence)
             (sum (funcall key n)))
    (iterate (for n :in-whatever sequence)
             (sum n))))

(defun product (sequence &key key)
  ;; https://github.com/sjl/cl-losh/blob/master/losh.lisp#L2181
  "Return the product of all elements of `sequence`.
  If `key` is given, it will be called on each element to compute the
  multiplicand.
  Examples:
    (product #(1 2 3))
    ; => 6
    (product '(\"1\" \"2\" \"3\") :key #'parse-integer)
    ; => 6
    (product '(\"1\" \"2\" \"3\") :key #'length)
    ; => 1
  "
  (if key
    (iterate (for n :in-whatever sequence)
             (multiplying (funcall key n)))
    (iterate (for n :in-whatever sequence)
             (multiplying n))))

Debugging and logging

(defun pr (&rest args)
  "Print `args` readably, separated by spaces and followed by a newline.
  Returns the first argument, so you can just wrap it around a form without
  interfering with the rest of the program.
  This is what `print` should have been.
  "
  (format t "~{~S~^ ~}~%" args)
  (finish-output)
  (first args))

(defmacro prl (&rest args)
  "Print `args` labeled and readably.
  Each argument form will be printed, then evaluated and the result printed.
  One final newline will be printed after everything.
  Returns the last result.
  Examples:
    (let ((i 1)
          (l (list 1 2 3)))
      (prl i (second l)))
    ; =>
    i 1
    (second l) 2
  "
  `(prog1
    (progn ,@(mapcar (lambda (arg) `(pr ',arg ,arg)) args))
    (terpri)
    (finish-output)))

(defmacro shut-up (&body body)
  "Run `body` with stdout and stderr redirected to the void."
  `(let ((*standard-output* (make-broadcast-stream))
         (*error-output* (make-broadcast-stream)))
    ,@body))

(defmacro comment (&body body)
  "Do nothing with a bunch of forms.
  Handy for block-commenting multiple expressions.
  "
  (declare (ignore body))
  nil)

Pretty-print a table.

Didn’t test.

(defun print-table (rows)
  ;; https://github.com/sjl/cl-losh/blob/master/losh.lisp#L2334
  "Print `rows` as a nicely-formatted table.
  Each row should have the same number of colums.
  Columns will be justified properly to fit the longest item in each one.
  Example:
    (print-table '((1 :red something)
                   (2 :green more)))
    =>
    1 | RED   | SOMETHING
    2 | GREEN | MORE
  "
  (when rows
    (iterate
      (with column-sizes =
            (reduce (alexandria:curry #'mapcar #'max)
                    (mapcar (alexandria:curry #'mapcar (compose #'length #'aesthetic-string))
                            rows))) ; lol
      (for row :in rows)
      (format t "~{~vA~^ | ~}~%" (weave column-sizes row))))
  (values))


;; from Quickutil.
(defun ensure-function (function-designator)
    "Returns the function designated by `function-designator`:
if `function-designator` is a function, it is returned, otherwise
it must be a function name and its `fdefinition` is returned."
    (if (functionp function-designator)
        function-designator
        (fdefinition function-designator)))

;; from Quickutil.
(defun compose (function &rest more-functions)
    "Returns a function composed of `function` and `more-functions` that applies its ;
arguments to to each in turn, starting from the rightmost of `more-functions`,
and then calling the next one with the primary value of the last."
    (declare (optimize (speed 3) (safety 1) (debug 1)))
    (reduce (lambda (f g)
              (let ((f (ensure-function f))
                    (g (ensure-function g)))
                (lambda (&rest arguments)
                  (declare (dynamic-extent arguments))
                  (funcall f (apply g arguments)))))
            more-functions
            :initial-value function))

(defun make-gensym-list (length &optional (x "G"))
    "Returns a list of `length` gensyms, each generated as if with a call to `make-gensym`,
using the second (optional, defaulting to `\"G\"`) argument."
    (let ((g (if (typep x '(integer 0)) x (string x))))
      (loop repeat length
            collect (gensym g))))

  (define-compiler-macro compose (function &rest more-functions)
    (labels ((compose-1 (funs)
               (if (cdr funs)
                   `(funcall ,(car funs) ,(compose-1 (cdr funs)))
                   `(apply ,(car funs) arguments))))
      (let* ((args (cons function more-functions))
             (funs (make-gensym-list (length args) "COMPOSE")))
        `(let ,(loop for f in funs for arg in args
                     collect `(,f (ensure-function ,arg)))
           (declare (optimize (speed 3) (safety 1) (debug 1)))
           (lambda (&rest arguments)
             (declare (dynamic-extent arguments))
             ,(compose-1 funs))))))


;; from Quickutil.
(defun weave (&rest lists)
    "Return a list whose elements alternate between each of the lists
`lists`. Weaving stops when any of the lists has been exhausted."
    (apply #'mapcan #'list lists))

(defun aesthetic-string (thing)
  "Return the string used to represent `thing` when printing aesthetically."
  (format nil "~A" thing))

Pretty print a hash-table:

(defun print-hash-table (hash-table &optional (stream t))
  "Print a pretty representation of `hash-table` to `stream.`
  Respects `*print-length*` when printing the elements.
  "
  (let* ((keys (alexandria:hash-table-keys hash-table))
         (vals (alexandria:hash-table-values hash-table))
         (count (hash-table-count hash-table))
         (key-width (-<> keys
                      (mapcar (alexandria:compose #'length #'prin1-to-string) <>)
                      (reduce #'max <> :initial-value 0)
                      (clamp 0 20 <>))))
    (print-unreadable-object (hash-table stream :type t)
      (princ
        ;; Something shits the bed and output gets jumbled (in SBCL at least) if
        ;; we try to print to `stream` directly in the format statement inside
        ;; `print-unreadable-object`, so instead we can just render to a string
        ;; and `princ` that.
        (format nil ":test ~A :count ~D {~%~{~{  ~vs ~s~}~%~}}"
                (hash-table-test hash-table)
                count
                (loop
                  :with limit = (or *print-length* 40)
                  :for key :in keys
                  :for val :in vals
                  :for i :from 0 :to limit
                  :collect
                  (if (= i limit)
                    (list key-width :too-many-items (list (- count i) :more))
                    (list key-width key val))))
        stream)))
  (terpri stream)
  (values))

(defun pht (hash-table &optional (stream t))
  "Synonym for `print-hash-table` for less typing at the REPL."
  (print-hash-table hash-table stream))

(defun print-hash-table-concisely (hash-table &optional (stream t))
  "Print a concise representation of `hash-table` to `stream.`
  Should respect `*print-length*` when printing the elements.
  "
  (print-unreadable-object (hash-table stream :type t)
    (prin1 (hash-table-test hash-table))
    (write-char #\space stream)
    (prin1 (hash-table-contents hash-table) stream)))

;; needed:
(defun clamp (from to value)
  "Clamp `value` between `from` and `to`."
  (let ((max (max from to))
        (min (min from to)))
    (cond
      ((> value max) max)
      ((< value min) min)
      (t value))))

;; see
(defmacro -<> (expr &rest forms)
  "Thread the given forms, with `<>` as a placeholder."
  ;; I am going to lose my fucking mind if I have to program lisp without
  ;; a threading macro, but I don't want to add another dep to this library, so
  ;; here we are.
  `(let* ((<> ,expr)
          ,@(mapcar (lambda (form)
                      (if (symbolp form)
                        `(<> (,form <>))
                        `(<> ,form)))
                    forms))
     <>))

For the -<> threading macro, see cl-arrows and arrow-macros.

Profiling (with SBCL)

#+sbcl
(defun dump-profile (filename)
  (with-open-file (*standard-output* filename
                                     :direction :output
                                     :if-exists :supersede)
    (sb-sprof:report :type :graph
                     :sort-by :cumulative-samples
                     :sort-order :ascending)
    (sb-sprof:report :type :flat
                     :min-percent 0.5)))

#+sbcl
(defun start-profiling (&key call-count-packages (mode :cpu))
  "Start profiling performance.  SBCL only.
  `call-count-packages` should be a list of package designators.  Functions in
  these packages will have their call counts recorded via
  `sb-sprof::profile-call-counts`.
  "
  (sb-sprof::reset)
  (-<> call-count-packages
    (mapcar #'mkstr <>)
    (mapcar #'string-upcase <>)
    (mapc #'sb-sprof::profile-call-counts <>))
  (sb-sprof::start-profiling :max-samples 50000
                             :mode mode
                             ; :mode :time
                             :sample-interval 0.01
                             :threads :all))

#+sbcl
(defun stop-profiling (&optional (filename "lisp.prof"))
  "Stop profiling performance and dump a report to `filename`.  SBCL only."
  (sb-sprof::stop-profiling)
  (dump-profile filename))

#+sbcl
(defmacro profile (&body body)
  "Profile `body` and dump the report to `lisp.prof`."
  `(progn
     (start-profiling)
     (unwind-protect
         (time (progn ,@body))
       (stop-profiling))))

Print licences used by a given project and its dependencies

Mon, 22 Jan 2018 07:51:49 +0100

print-licenses is a little utility found in Steve Losh’s gigantic utilities and ported to a stand alone project.

Example usage:

  (print-licenses 'fast-io)
  =>
  alexandria           | Public Domain / 0-clause MIT
  babel                | MIT
  cffi                 | MIT
  cffi-grovel          | MIT
  cffi-toolchain       | MIT
  fast-io              | NewBSD
  static-vectors       | MIT
  trivial-features     | MIT
  trivial-gray-streams | MIT
  uiop                 | Unspecified

It may be available on february, 2018 Quicklisp update (request).

One potential source of caution (feedback on reddit):

what many authors put as the license in their asd file is not the license file that is actually included in the source code.

Let’s read the source, there are many useful bits. The core of the job is:

(defun print-licenses (quicklisp-project-designator)
  (print-table (sort (license-list quicklisp-project-designator)
                     #'string<
                     :key #'car)))

(defun license-list (quicklisp-project-designator)
  (remove-duplicates
    (mapcar (alexandria:rcurry #'coerce 'list)
            (alexandria:flatten (license-tree quicklisp-project-designator)))
    :key #'car :test #'string=))

(defun license-tree (quicklisp-project-designator)
  (let ((sys (ql-dist:dependency-tree quicklisp-project-designator)))
    (assert (not (null sys)) ()
      "Cannot find Quicklisp project for designator ~S"
      quicklisp-project-designator)
    (shut-up
      (ql:quickload quicklisp-project-designator))
    (map-tree
      (lambda (s)
        (vector (slot-value s 'ql-dist:name)
                (or (asdf:system-license
                      (asdf:find-system
                        (ql-dist:system-file-name s)))
                    "Unspecified")))
      sys)))

and those are the remaining building blocks, with a useful print-table function, and three of them taken from Quickutil. See their website and how sjl does to include them (and only them, to keep lightweight dependencies) in a project without copy-pasting.

(defmacro shut-up (&body body)
  "Run `body` with stdout and stderr redirected to the void."
  `(let ((*standard-output* (make-broadcast-stream))
         (*error-output* (make-broadcast-stream)))
     ,@body))

;; from Quickutil.
(defun map-tree (function tree)
    "Map `function` to each of the leave of `tree`."
    (check-type tree cons)
    (labels ((rec (tree)
               (cond
                 ((null tree) nil)
                 ((atom tree) (funcall function tree))
                 ((consp tree)
                  (cons (rec (car tree))
                        (rec (cdr tree)))))))
      (rec tree)))

;; from Quickutil
(defun aesthetic-string (thing)
  "Return the string used to represent `thing` when printing aesthetically."
  (format nil "~A" thing))

;; from Quickutil
(defun weave (&rest lists)
  "Return a list whose elements alternate between each of the lists
`lists`. Weaving stops when any of the lists has been exhausted."
  (apply #'mapcan #'list lists))

(defun print-table (rows)
  "Print `rows` as a nicely-formatted table.
  Each row should have the same number of colums.
  Columns will be justified properly to fit the longest item in each one.
  Example:
    (print-table '((1 :red something)
                   (2 :green more)))
    =>
    1 | RED   | SOMETHING
    2 | GREEN | MORE
  "
  (when rows
    (iterate
      (with column-sizes =
            (reduce (curry #'mapcar #'max)
                    (mapcar (curry #'mapcar (compose #'length #'aesthetic-string))
                            rows))) ; lol
      (for row :in rows)
      (format t "~{~vA~^ | ~}~%" (weave column-sizes row))))
  (values))

Generic, consistent and dotted access of data structures with Access

Fri, 12 Jan 2018 07:51:49 +0100

A common frustration for (impatient) beginners is to see different function names to access common data structures (alists, plists, hash-tables) and their inconsistencies (the order of arguments).

Now they are well documented in the… Common Lisp Coobook of course: https://lispcookbook.github.io/cl-cookbook/data-structures.html, but still;

and it is annoying to try things out with a data structure and refactor the code to use another one.

The library Access solves those problems, it’s always

(access my-var elt)

(if you’re into this, note that CL21 also does this with a generic and extensible getf).

edit: also rutils with generic-elt or ? in the rutilsx contrib package.

Access also solves another usecase.

Sometimes we deal with nested data structures (alist inside alist inside alist, or mixed data structures, happens when working with an API) and, as in other languages, we’d like a shortcut to access a nested element. In Python, we can use addict to write foo.one.2.three instead of foo['one'][2]['three'], with Access we have two possibilities, see below.

Oh, and we can be confident it is a battle-tested library, since it is the one that powers Djula’s template variables interplolation (doc is here), where we can write

{{ var.foo }}

à la Django for the supported data structures, and Djula is in the top 100 of the most downloaded Quicklisp libraries (december 2017 stats).

Let’s install it:

(ql:quickload "access")

import its symbols in Slime:

(use-package :access)

Generic and consistent access accross alists, plists, hash-tables, CLOS slots

Let’s create our test variables first:

(defparameter my-alist '((:foo . "foo") (:bar . "bar")))
MY-ALIST

(defparameter my-plist (list :foo "foo" :bar "bar"))
MY-PLIST

(defparameter my-hashtable (make-hash-table))
MY-HASHTABLE
(setf (gethash :foo my-hashtable) "foo")
"foo"

(defclass obj-test ()
  ((foo :accessor foo :initarg :foo :initform :foo)
   (bar :accessor bar :initarg :bar :initform :bar)))
;; #<STANDARD-CLASS OBJ-TEST>
(defparameter my-obj (make-instance 'obj-test))
;; MY-OBJ

Now, let’s access the :foo slot.

With alists:

(access my-alist :foo)
"foo"
T

instead of (cdr (assoc :foo my-alist)) (with :foo first argument) or alexandria’s (assoc-value my-alist :foo) (:foo second argument).

plists:

(access my-plist :foo)
"foo"
T

instead of (getf my-plist :foo) (unlike alists, with :foo as last argument).

hash-tables:

(access my-hashtable :foo)
"foo"
T

instead of (gethash :foo my-hashtable) (:foo first argument).

objects:

(access my-obj :foo) ;; <= accessor, not slot name
;; :FOO
;; T

instead of… it depends. Here we named the accessor foo, so we would have used simply (foo my-obj).

Also note that access returns two values, the value and a boolean, t if the slot exists, nil otherwise.

And access is setfable:

(setf (access my-alist :foo) "oof")

with-access

Below, we can bind temporary variables inside with-access:

(with-access (foo bar (other-name plist))
             my-obj
           (format t "Got: ~a~a~a~&" foo bar other-name)
           ;; we can change variables
           (setf other-name "hello plist")
           (format t "other-name: ~a~&" other-name)
           ;; it changed the object too
           (format t "object slot: ~a~&" (plist my-obj)))
Got: FOOBAR(FOO foo BAR bar)
other-name: hello plist
object slot: hello plist
NIL

Nested access

For this example we add a plist slot to our object, which copies our my-plist by default.

(defclass obj-test ()
  ((foo :accessor foo :initarg :foo :initform :foo)
   (bar :accessor bar :initarg :bar :initform :bar)
   (plist :accessor plist :initarg plist :initform (copy-list MY-PLIST))))
#<STANDARD-CLASS OBJ-TEST>

(as being a CLOS object, my-obj is automatically updated with the new slot).

We can access the nested plist element :foo inside the object in one go with accesses (plurial):

(accesses MY-OBJ 'plist :foo)
;; "foo"

instead of (getf (plist my-obj) :foo).

Dotted access

with-dot or #D

We can rewrite the previous examples with a dot:

(with-dot ()
    my-alist.foo)
"foo"

or again

(with-dot ()
    my-obj.foo)
"hello plist"

but even shorter, with the #D reader macro that we enable with

(enable-dot-syntax)

(also works in Slime/Sly, for what I am not sure if I enabled a special feature)

#Dmy-alist.foo
"foo"

and so, a nested dotted access through an object and a plist:

;; back to initial case
(setf my-obj (make-instance 'obj-test))
;; #<OBJ-TEST {1005AA3B13}>
#Dmy-obj.plist.foo
;; "foo"

It will return nil instead of an error if someone in the middle doesn’t have the requested field.

Usage will tell how it is useful, and I hope it will be to fellow newcomers.

Structures: lightweight records, a step before classes

Wed, 03 Jan 2018 07:51:49 +0100

Structures offer a way to store data in named slots. They support single inheritance.

Classes provided by the Common Lisp Object System (CLOS) are more flexible however structures may offer better performance (see for example the SBCL manual).

As usual, this is best read in the Common Lisp Cookbook.

Structures

Creation

defstruct

(defstruct person
   id name age)

At creation slots are optional and default to nil.

To set a default value:

(defstruct person
   id
   (name "john doe")
   age)

Also specify the type after the default value:

(defstruct person
  id
  (name "john doe" :type string)
  age)

We create an instance with the generated constructor make- + <structure-name>, so make-person:

(defparameter *me* (make-person))
*me*
#S(PERSON :ID NIL :NAME "john doe" :AGE NIL)

note that printed representations can be read back by the reader.

With a bad name type:

(defparameter *bad-name* (make-person :name 123))

Invalid initialization argument:
  :NAME
in call for class #<STRUCTURE-CLASS PERSON>.
   [Condition of type SB-PCL::INITARG-ERROR]

We can set the structure’s constructor so as to create the structure without using keyword arguments, which can be more convenient sometimes. We give it a name and the order of the arguments:

(defstruct (person (:constructor create-person (id name age)))
     id
     name
     age)

Our new constructor is create-person:

(create-person 1 "me" 7)
#S(PERSON :ID 1 :NAME "me" :AGE 7)

However, the default make-person does not work any more:

(make-person :name "me")
;; debugger:
obsolete structure error for a structure of type PERSON
[Condition of type SB-PCL::OBSOLETE-STRUCTURE]

Slot access

We access the slots with accessors created by <name-of-the-struct>- + slot-name:

(person-name *me*)
;; "john doe"

we then also have person-age and person-id.

Setting

Slots are setf-able:

(setf (person-name *me*) "Cookbook author")
(person-name *me*)
;; "Cookbook author"

Predicate

(person-p *me*)
T

Single inheritance

With the :include <struct> argument:

(defstruct (female (:include person))
     (gender "female" :type string))
(make-female :name "Lilie")
;; #S(FEMALE :ID NIL :NAME "Lilie" :AGE NIL :GENDER "female")

Limitations

After a change, instances are not updated.

If we try to add a slot (email below), we have the choice to lose all instances, or to continue using the new definition of person. But the effects of redefining a structure are undefined by the standard, so it is best to re-compile and re-run the changed code.

(defstruct person
       id
       (name "john doe" :type string)
       age
       email)

attempt to redefine the STRUCTURE-OBJECT class PERSON
incompatibly with the current definition
   [Condition of type SIMPLE-ERROR]

Restarts:
 0: [CONTINUE] Use the new definition of PERSON, invalidating already-loaded code and instances.
 1: [RECKLESSLY-CONTINUE] Use the new definition of PERSON as if it were compatible, allowing old accessors to use new instances and allowing new accessors to use old instances.
 2: [CLOBBER-IT] (deprecated synonym for RECKLESSLY-CONTINUE)
 3: [RETRY] Retry SLIME REPL evaluation request.
 4: [*ABORT] Return to SLIME's top level.
 5: [ABORT] abort thread (#<THREAD "repl-thread" RUNNING {1002A0FFA3}>)

If we choose restart 0, to use the new definition, we lose access to *me*:

*me*
obsolete structure error for a structure of type PERSON
   [Condition of type SB-PCL::OBSOLETE-STRUCTURE]

There is also very little introspection. Portable Common Lisp does not define ways of finding out defined super/sub-structures nor what slots a structure has.

The Common Lisp Object System (which came after into the language) doesn’t have such limitations. See the CLOS section.

structures on the hyperspec
David B. Lamkins, “Successful Lisp, How to Undertsand and Use Common Lisp”.

Error and condition handling

Tue, 02 Jan 2018 15:51:49 +0100

Common Lisp has mechanisms for error and condition handling as found in other languages, and can do more.

What is a condition ?

Just like in languages that support exception handling (Java, C++, Python, etc.), a condition represents, for the most part, an “exceptional” situation. However, even more so that those languages, a condition in Common Lisp can represent a general situation where some branching in program logic needs to take place, not necessarily due to some error condition. Due to the highly interactive nature of Lisp development (the Lisp image in conjunction with the REPL), this makes perfect sense in a language like Lisp rather than say, a language like Java or even Python, which has a very primitive REPL. In most cases, however, we may not need (or even allow) the interactivity that this system offers us. Thankfully, the same system works just as well even in non-interactive mode.

z0ltan

Let’s dive into it step by step. More resources are given afterwards.

Now best read in the Common Lisp Cookbook.

Credit: our cl-torrents tutorial.

Note: you can contribute any fix or addition to the Cookbook or this page via git ;)

Table of Contents

Ignore all errors (and return nil)

Sometimes you know that a function can fail and you just want to ignore it: use ignore-errors:

(ignore-errors
  (/ 3 0))
; in: IGNORE-ERRORS (/ 3 0)
;     (/ 3 0)
;
; caught STYLE-WARNING:
;   Lisp error during constant folding:
;   arithmetic error DIVISION-BY-ZERO signalled
;   Operation was (/ 3 0).
;
; compilation unit finished
;   caught 1 STYLE-WARNING condition
NIL
#<DIVISION-BY-ZERO {1008FF5F13}>

We get a welcome division-by-zero warning but the code runs well and it returns two things: nil and the condition that was signaled. We could not choose what to return.

Rembember that we can inspect the condition with a right click in Slime.

Catching any condition - handler-case

ignore-error is built from handler-case. We can write the previous example by catching the general error but now we can return whatever we want:

(handler-case (/ 3 0)
  (error (c)
    (format t "We caught a condition.~&")
    (values 0 c)))
; in: HANDLER-CASE (/ 3 0)
;     (/ 3 0)
;
; caught STYLE-WARNING:
;   Lisp error during constant folding:
;   Condition DIVISION-BY-ZERO was signalled.
;
; compilation unit finished
;   caught 1 STYLE-WARNING condition
We caught a condition.
0
#<DIVISION-BY-ZERO {1004846AE3}>

We also returned two values, 0 and the signaled condition.

The general form of handler-case is

(handler-case (code that errors out)
   (condition-type (the-condition) ;; <-- optional argument
      (code))
   (another-condition (the-condition)
       ...))

We can also catch all conditions by matching t, like in a cond:

(handler-case
    (progn
      (format t "This won't work…~%")
      (/ 3 0))
  (t (c)
    (format t "Got an exception: ~a~%" c)
    (values 0 c)))
;; …
;; This won't work…
;; Got an exception: arithmetic error DIVISION-BY-ZERO signalled
;; Operation was (/ 3 0).
;; 0
;; #<DIVISION-BY-ZERO {100608F0F3}>

Catching a specific condition

We can specify what condition to handle:

(handler-case (/ 3 0)
  (division-by-zero (c)
    (format t "Caught division by zero: ~a~%" c)))
;; …
;; Caught division by zero: arithmetic error DIVISION-BY-ZERO signalled
;; Operation was (/ 3 0).
;; NIL

This workflow is similar to a try/catch as found in other languages, but we can do more.

Ignoring the condition argument

If you don’t access the condition object in your handlers, but you still keep it has an argument for good practice, you’ll see this compiler warning often:

; caught STYLE-WARNING:
;   The variable C is defined but never used.

To remove it, use a declare call as in:

(handler-case (/ 3 0)
  (division-by-zero (c)
   (declare (ignore c))
   (format t "Caught division by zero~%"))) ;; we don't print "c" here and don't get the warning.

handler-case VS handler-bind

handler-case is similar to the try/catch forms that we find in other languages.

handler-bind (see the next examples), is what to use when we need absolute control over what happens when a signal is raised. It allows us to use the debugger and restarts, either interactively or programmatically.

If some library doesn’t catch all conditions and lets some bubble out to us, we can see the restarts (established by restart-case) anywhere deep in the stack, including restarts established by other libraries whose this library called. And we can see the stack trace, with every frame that was called and, in some lisps, even see local variables and such. Once we handler-case, we “forget” about this, everything is unwound. handler-bind does not rewind the stack.

Handling conditions - handler-bind

Here we use handler-bind.

Its general form is:

(handler-bind ((a-condition #'function-to-handle-it)
               (another-one #'another-function))
    (code that can...)
    (...error out))

So, our simple example:

(handler-bind
             ((division-by-zero #'(lambda (c) (format t "hello condition~&"))))
           (/ 3 0))

This prints some warnings, then it prints our “hello” and still enters the debugger. If we don’t want to enter the debugger, we have to define a restart and invoke it.

A real example with the unix-opts library, that parses command line arguments. It defined some conditions: unknown-option, missing-arg and arg-parser-failed, and it is up to use to write what to do in these cases.

(handler-bind ((opts:unknown-option #'unknown-option)
               (opts:missing-arg #'missing-arg)
               (opts:arg-parser-failed #'arg-parser-failed))
  (opts:get-opts))

Our unknown-option function is simple and looks like this:

(defun unknown-option (condition)
  (format t "~s option is unknown.~%" (opts:option condition))
  (opts:describe)
  (exit)) ;; <-- we return to the command line, no debugger.

it takes the condition as parameter, so we can read information from it if needed. Here we get the name of the erronous option with the defined reader (opts:option condition) (see below).

Creating conditions

With define-condition, and we can inherit from error or simple-error:

(define-condition my-division-by-zero (error) ())

It is a regular class, so we can add information into slots. Here, we add a custom message:

(define-condition my-division-by-zero (error)
  ((dividend :initarg :dividend
            :reader dividend)) ;; <= so we'll get the dividend with (dividend condition), as soon as on the next line.
  ;; the :report is the message into the debugger:
  (:report (lambda (condition stream) (format stream "You were going to divide ~a by zero.~&" (dividend condition)))))

The general form looks like a regular class definition:

(define-condition my-condition (condition-it-inherits-from)
  ;; list of arguments, can be "()".
  ((message :initarg :message
            :reader my-condition-message)
   (second ...))
  ;; class arguments
  (:report (lambda (condition stream) (...))) ;; what is printed in the REPL.
  (:documentation "a string")) ;; good practice ;)

Now when we throw this condition we must pass it an error message (it is a required argument), and read it with error-message (the :reader).

What’s in :report will be printed in the REPL.

Let’s try our condition. We define a simple function that checks our divisor, and signals our condition if it is equal to zero:

(defun my-division (x y)
    (if (= y 0)
        (error 'MY-DIVISION-BY-ZERO :dividend x))
    (/ x y))

When we use it, we enter the debugger:

(my-division 3 0)
;;
;; into the debugger:
;;
You were going to divide 3 by zero.
   [Condition of type MY-DIVISION-BY-ZERO]

Restarts:
 0: [RETRY] Retry SLIME REPL evaluation request.
 1: [*ABORT] Return to SLIME's top level.
 2: [ABORT] abort thread (#<THREAD "repl-thread" RUNNING {1002957FA3}>)

Backtrace:
  0: (MY-DIVISION 3 0)

We can inspect the backtrace, go to the source (v in Slime), etc.

Here is how unix-opts defines its unknown-option condition:

(define-condition troublesome-option (simple-error)
  ((option
    :initarg :option
    :reader option))
  (:report (lambda (c s) (format s "troublesome option: ~s" (option c))))
  (:documentation "Generalization over conditions that have to do with some
particular option."))

(define-condition unknown-option (troublesome-option)
  ()
  (:report (lambda (c s) (format s "unknown option: ~s" (option c))))
  (:documentation "This condition is thrown when parser encounters
unknown (not previously defined with `define-opts') option."))

Signaling (throwing) conditions

We can use error, like we did above. Two ways:

(error "some text"): signals a condition of type simple-error
(error 'my-error :message "We did this and this and it didn't work.")

Throwing these conditions will enter the interactive debugger, where a few options will be presented by default. We can give more options with restarts, and we can prevent from entering the debugger by handling the condition and invoking a restart.

Simple example from unix-opts: it adds information into the option slot:

(error 'unknown-option
        :option opt)

Restarts, interactive choices in the debugger

Defining restarts

Restarts are the choices we get in the debugger, which always has the RETRY and ABORT ones. We can add choices to the top of the list:

(defun division-restarter ()
  (restart-case (/ 3 0)
    (return-zero () 0)
    (divide-by-one () (/ 3 1))))

By calling this stupid function we get two new choices at the top of the debugger:

Note: read in lisper.in’s blogpost on csv parsing (see Resources) how this system was used effectively in production.

But that’s not all, by handling restarts we can start over the operation as if the error didn’t occur (as seen in the stack).

Calling restarts programmatically

With invoke-restart.

(defun division-and-bind ()
           (handler-bind
               ((error (lambda (c)
                         (format t "Got error: ~a~%" c) ;; error-message
                         (format t "and will divide by 1~&")
                         (invoke-restart 'divide-by-one))))
             (division-restarter)))
;; (DIVISION-AND-BIND)
;; Got error: arithmetic error DIVISION-BY-ZERO signalled
;; and will divide by 1
;; Operation was (/ 3 0).
;; 3

Note that we called the form that contains our restarts (division-restarter) and not the function that throws the error.

Using other restarts

find-restart 'name-of-restart will return the most recent bound restart with the given name, or nil. We can invoke it with invoke-restart.

Prompting the user to enter a new value

Let’s add a restart in our division-restarter to offer the user to enter a new dividend, and run the division again.

(defun division-restarter ()
  (restart-case (/ 3 0)
    (return-nil () nil)
    (divide-by-one () (/ 3 1))
    (choose-another-dividend (new-dividend)
      :report "Please choose another dividend"
      :interactive (lambda ()
                     (format t "Enter a new dividend: ")
                     (list (read))) ;; <-- must return a list.
      (format t "New division: 3/~a = ~a~&" new-dividend (/ 3 new-dividend)))))

We get prompted in the debugger:

arithmetic error DIVISION-BY-ZERO signalled
Operation was (/ 3 0).
  [Condition of type DIVISION-BY-ZERO]

Restarts:
 0: [RETURN-NIL] RETURN-NIL
 1: [DIVIDE-BY-ONE] DIVIDE-BY-ONE
 2: [CHOOSE-ANOTHER-DIVIDEND] Please choose another dividend <-- new
 3: [RETRY] Retry SLIME REPL evaluation request.
 4: [*ABORT] Return to SLIME's top level.
 5: [ABORT] abort thread (#<THREAD "repl-thread" RUNNING {1002A47FA3}>)

The new choose-another-dividend restart takes an argument for the new dividend, that will be fed by the :interactive lambda, which reads for user input and must return a list.

We use it like this:

(division-restarter)
;;
;; Entered debugger, chose the 2nd restart.
;;
Enter a new dividend: 10  <-- got prompted to enter a new value.
New division: 3/10 = 3/10
NIL

In a real situation we might want to call our “restarter” recursively, to get into the debugger again if we enter a bad value.

Hide and show restarts

Restarts can be hidden. In restart-case, in addition to :report and :interactive, they also accept a :test key:

(restart-case
   (return-zero ()
     :test (lambda ()
             (some-test))
    ...

Run some code, condition or not (“finally”)

The “finally” part of others try/catch/finally forms is done with unwind-protect.

It is the construct used in “with-” macros, like with-open-file, which always closes the file after it.

You’re now more than ready to write some code and to dive into other resources !

Resources

Practical Common Lisp: “Beyond Exception Handling: Conditions and Restarts” - the go-to tutorial, more explanations and primitives.
Common Lisp Recipes, chap. 12, by E. Weitz
language reference
lisper.in - example with parsing a csv file and using restarts with success, in a flight travel company.
Condition Handling in the Lisp family of languages
z0ltan.wordpress.com
https://github.com/svetlyak40wt/python-cl-conditions - implementation of the CL conditions system in Python.

Scripting. Parsing command line arguments, building self-contained executables.

Tue, 02 Jan 2018 07:51:49 +0100

Using a program from a REPL is fine and well, but if we want to distribute our program easily, we’ll want to build an executable.

Lisp implementations differ in their processes, but they all create self-contained executables, for the architecture they are built on. The final user doesn’t need to install a Lisp implementation, he can run the software right away.

Start-up times are near to zero, specially with SBCL and CCL.

Binaries size are large-ish. They include the whole Lisp including its libraries, the names of all symbols, information about argument lists to functions, the compiler, the debugger, source code location information, and more.

Note that we can similarly build self-contained executables for web apps.

Now best read in the Common Lisp Cookbook.

Table of Contents

Building a self-contained executable

With SBCL

How to build (self-contained) executables is implementation-specific (see below Buildapp and Rowsell). With SBCL, as says its documentation, it is a matter of:

(sb-ext:save-lisp-and-die #P"path/name-of-executable" :toplevel #'my-app:main-function :executable t)

sb-ext is an SBCL extension to run external processes. See other SBCL extensions (many of them are made implementation-portable in other libraries).

:executable t tells to build an executable instead of an image. We could build an image to save the state of our current Lisp image, to come back working with it later. Specially useful if we made a lot of work that is computing intensive.

If you try to run this in Slime, you’ll get an error about threads running:

Cannot save core with multiple threads running.

Run the command from a simple SBCL repl.

I suppose your project has Quicklisp dependencies. You must then:

ensure Quicklisp is installed and loaded at Lisp startup (you completed Quicklisp installation)
load the project’s .asd
install dependencies
build the executable.

That gives:

(load "my-app.asd")
(ql:quickload :my-app)
(sb-ext:save-lisp-and-die #p"my-app-binary" :toplevel #'my-app:main :executable t)

From the command line, or from a Makefile, use --load and --eval:

build:
	sbcl --load my-app.asd \
	     --eval '(ql:quickload :my-app)' \
         --eval "(sb-ext:save-lisp-and-die #p\"my-app\" :toplevel #my-app:main :executable t)"

With ASDF [updated]

Now that we’seen the basics, we need a portable method. Since its version 3.1, ASDF allows to do that. It introduces the make command, that reads parameters from the .asd. Add this to your .asd declaration:

:build-operation "program-op" ;; leave as is
:build-pathname "<binary-name>"
:entry-point "<my-package:main-function>"

and call asdf:make :my-package.

So, in a Makefile:

LISP ?= sbcl

build:
    $(LISP) --load torrents.asd \
    	--eval '(ql:quickload :my-app)' \
		--eval '(asdf:make :my-app)' \
		--eval '(quit)'

With Buildapp or Roswell

We might like a more shell-friendly way to build our executable, and while we’re at it a portable one, so we would have the same command to work with various implementations.

Buildapp is a battle-tested “application for SBCL or CCL that configures and saves an executable Common Lisp image”.

Example usage:

buildapp --output myapp \
         --asdf-path . \
         --asdf-tree ~/quicklisp/dists \
         --load-system my-app \
         --entry my-app:main

Many applications use it (for example, pgloader). It is available on Debian: apt install buildapp.

Roswell, an implementation manager and much more, also has the ros build command, that should work for more implementations than Buildapp.

We can also make our app installable with Roswell by a ros install my-app. See its documentation

For web apps

We can similarly build a self-contained executable for our web-app. It would thus contain a web server and would be able to run on the command line:

$ ./my-web-app
Hunchentoot server is started.
Listening on localhost:9003.

Note that this runs the production webserver, not a development one, so we can run the binary on our VPS right away and access the app from outside.

We have one thing to take care of, it is to find and put the thread of the running web server on the foreground. In our main function, we can do something like this:

(defun main ()
  (start-app :port 9003) ;; our start-app, for example clack:clack-up
  ;; let the webserver run.
  ;; warning: hardcoded "hunchentoot".
  (handler-case (bt:join-thread (find-if (lambda (th)
                                            (search "hunchentoot" (bt:thread-name th)))
                                         (bt:all-threads)))
    ;; Catch a user's C-c
    (#+sbcl sb-sys:interactive-interrupt
      #+ccl  ccl:interrupt-signal-condition
      #+clisp system::simple-interrupt-condition
      #+ecl ext:interactive-interrupt
      #+allegro excl:interrupt-signal
      () (progn
           (format *error-output* "Aborting.~&")
           (clack:stop *server*)
           (uiop:quit)))
    (error (c) (format t "Woops, an unknown error occured:~&~a~&" c))))

We used the bordeaux-threads library ((ql:quickload "bordeaux-threads"), alias bt) and uiop, which is part of ASDF so already loaded, in order to exit in a portable way (uiop:quit, with an optional return code, instead of sb-ext:quit).

Size and startup times of executables per implementation

SBCL isn’t the only Lisp implementation. ECL, Embeddable Common Lisp, transpiles Lisp programs to C. That creates a smaller executable.

According to this reddit source, ECL produces indeed the smallest executables of all, an order of magnituted smaller than SBCL, but with a longer startup time.

CCL’s binaries seem to be as fast as SBCL and nearly half the size.

| program size | implementation |  CPU | startup time |
|--------------+----------------+------+--------------|
|           28 | /bin/true      |  15% |        .0004 |
|         1005 | ecl            | 115% |        .5093 |
|        48151 | sbcl           |  91% |        .0064 |
|        27054 | ccl            |  93% |        .0060 |
|        10162 | clisp          |  96% |        .0170 |
|         4901 | ecl.big        | 113% |        .8223 |
|        70413 | sbcl.big       |  93% |        .0073 |
|        41713 | ccl.big        |  95% |        .0094 |
|        19948 | clisp.big      |  97% |        .0259 |

Parsing command line arguments

SBCL stores the command line arguments into sb-ext:*posix-argv*.

But that variable name differs from implementations, so we may want a library to handle the differences for us.

We also want to parse the arguments.

A quick look at the awesome-cl#scripting list and we’ll do that with the unix-opts library.

(ql:quickload "unix-opts")

We can call it with its opts alias (nickname).

As often work happens in two phases:

declaring the options our app accepts, their optional argument, defining their type (string, integer,…), long and short names, and the required ones,
parsing them (and handling missing or malformed parameters).

Declaring arguments

We define the arguments with opts:define-opts:

(opts:define-opts
    (:name :help
           :description "print this help text"
           :short #\h
           :long "help")
    (:name :nb
           :description "here we want a number argument"
           :short #\n
           :long "nb"
           :arg-parser #'parse-integer) ;; <- takes an argument
    (:name :info
           :description "info"
           :short #\i
           :long "info"))

Here parse-integer is a built-in CL function.

Example output on the command line (auto-generated help text):

$ my-app -h
my-app. Usage:

Available options:
  -h, --help               print this help text
  -n, --nb ARG             here we want a number argument
  -i, --info               info

Parsing

We parse and get the arguments with opts:get-opts, which returns two values: the list of valid options and the remaining free arguments. We then must use multiple-value-bind to assign both into variables:

  (multiple-value-bind (options free-args)
      ;; There is no error handling yet.
      (opts:get-opts)
      ...

We can test this by giving a list of strings to get-opts:

(multiple-value-bind (options free-args)
                   (opts:get-opts '("hello" "-h" "-n" "1"))
                 (format t "Options: ~a~&" options)
                 (format t "free args: ~a~&" free-args))
Options: (HELP T NB-RESULTS 1)
free args: (hello)
NIL

If we put an unknown option, we get into the debugger. We’ll see error handling in a moment.

So options is a property list. We use getf and setf with plists, so that’s how we do our logic. Below we print the help with opts:describe and then exit (in a portable way).

  (multiple-value-bind (options free-args)
      (opts:get-opts)

    (if (getf options :help)
        (progn
          (opts:describe
           :prefix "You're in my-app. Usage:"
           :args "[keywords]") ;; to replace "ARG" in "--nb ARG"
          (opts:exit))) ;; <= optional return status.
    (if (getf options :nb)
       ...)

For a full example, see its official example and cl-torrents’ tutorial.

The example in the unix-opts repository suggests a macro to do slightly better. Now to error handling.

Handling malformed or missing arguments

There are 4 situations that unix-opts doesn’t handle, but signals conditions for us to take care of:

for an unknown argument: an unknown-option condition is signaled
also missing-arg
arg-parser-failed when, for example, it expected an integer but got text
missing-required-option

So, we must create simple functions to handle those conditions, and surround the parsing of the options with an handler-bind:

  (multiple-value-bind (options free-args)
      (handler-bind ((opts:unknown-option #'unknown-option) ;; the condition / our function
                     (opts:missing-arg #'missing-arg)
                     (opts:arg-parser-failed #'arg-parser-failed)
                     (opts:missing-required-option))
         (opts:get-opts))
    …
    ;; use "options" and "free-args"

Here we suppose we want one function to handle each case, but it could be a simple one. They take the condition as argument.

(defun handle-arg-parser-condition (condition)
  (format t "Problem while parsing option ~s: ~a .~%" (opts:option condition) ;; reader to get the option from the condition.
                                                       condition)
  (opts:describe) ;; print help
  (opts:exit)) ;; portable exit

For more about condition handling, see error and condition handling.

Catching a C-c termination signal

Let’s build a simple binary, run it, try a C-c and read the stacktrace:

$ ./my-app
sleep…
^C
debugger invoked on a SB-SYS:INTERACTIVE-INTERRUPT in thread   <== condition name
#<THREAD "main thread" RUNNING {1003156A03}>:
  Interactive interrupt at #x7FFFF6C6C170.

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [CONTINUE     ] Return from SB-UNIX:SIGINT.               <== it was a SIGINT indeed
  1: [RETRY-REQUEST] Retry the same request.

The signaled condition is named after our implementation: sb-sys:interactive-interrupt. We just have to surround our application code with a handler-case:

(handler-case
    (run-my-app free-args)
  (sb-sys:interactive-interrupt () (progn
                                     (format *error-output* "Abort.~&")
                                     (opts:exit))))

This code only for SBCL though. We know about trivial-signal, but we were not satisfied with our test yet. So we can use something like this:

(handler-case
    (run-my-app free-args)
  (#+sbcl sb-sys:interactive-interrupt
   #+ccl  ccl:interrupt-signal-condition
   #+clisp system::simple-interrupt-condition
   #+ecl ext:interactive-interrupt
   #+allegro excl:interrupt-signal
   ()
   (opts:exit)))

here #+ includes the line at compile time depending on the implementation. There’s also #-. What #+ does is to look for symbols in the *features* list. We can also combine symbols with and, or and not.

Continuous delivery of executables

We can make a Continuous Integration system (Travis CI, Gitlab CI,…) build binaries for us at every commit, or at every tag pushed or at wichever other policy.

See Continuous Integration.

Credit

cl-torrents, app and (extensive) tutorial: web scraping and building executables

Wed, 20 Dec 2017 14:21:20 +0100

Lately we exercised our Lisp skills by writing cl-torrents, an app that searches for torrents on several sources (the Pirate Bay through piratebay.to, Kickass torrents and torrent.cd), and we wrote an extensive tutorial in the making (that was actually our primary goal). It comes as a library to use from the REPL and as a self-contained executable (download and run, nothing more to install). You’ll find the following topics in the tutorial:

how to create and load a new project,
common pitfalls, basic data structures, useful libraries, where to find documentation,
(async) web scraping,
unit testing, with mocks,
continuous integration and delivery of executables (Gitlab CI, Docker),
parsing command line arguments,
building self-contained executables,
basics of error handling,
…

Some topics have been ported to the Cookbook, some not (yet).

The next iteration will be about a self-contained web app.

Continuous Integration and delivery on Gitlab CI, testing locally with Docker

Fri, 01 Dec 2017 14:21:20 +0100

Best read in the Cookbook ! also Travis CI, code coverage, testing with Prove.

Gitlab CI is part of Gitlab and is available on Gitlab.com, for public and private repositories. Let’s see straight away a simple .gitlab-ci.yml:

image: daewok/lisp-devel

before_script:
  - apt-get update -qy
  - apt-get install -y git-core
  - git clone https://github.com/foo/bar ~/quicklisp/local-projects/

test:
  script:
    - make test

Gitlab CI is based on Docker. With image we tell it to use the daewok/lisp-devel one. It includes SBCL, ECL, CCL and ABCL, and Quicklisp is installed in the home (/home/lisp/), so we can quickload packages right away. If you’re interested it also has a more bare bones option. Gitlab will load the image, clone our project and put us at the project root with administrative rights to run the rest of the commands.

test is a “job” we define, script is a recognized keywords that takes a list of commands to run.

Suppose we must install dependencies before running our tests: before_script will run before each job. Here we clone a library where Quicklisp can find it, and for doing so we must install git (Docker images are usually pretty bare bones).

We can try locally ourselves. If we already installed Docker and started its daemon (sudo service docker start), we can do:

docker run --rm -it -v /path/to/local/code:/usr/local/share/common-lisp/source daewok/lisp-devel:latest bash

This will download the lisp image (±400Mo), mount some local code in the image where indicated, and drop us in bash. Now we can try a make test.

To show you a more complete example:

image: daewok/lisp-devel

stages:
  - test
  - build

before_script:
  - apt-get update -qy
  - apt-get install -y git-core
  - git clone https://github.com/foo/bar ~/quicklisp/local-projects/

test:
  stage: test
  script:
    - make test

build:
  stage: build
  only:
    - tags
  script:
    - make build
  artifacts:
    paths:
      - some-file-name

Here we defined two stages (see environments), “test” and “build”, defined to run one after another. A “build” stage will start only if the “test” one succeesds.

“build” is asked to run only when a new tag is pushed, not at every commit. When it succeeds, it will make the files listed in artifacts’s paths available for download. We can download them from Gitlab’s Pipelines UI, or with an url. This one will download the file “some-file-name” from the latest “build” job:

https://gitlab.com/username/project-name/-/jobs/artifacts/master/raw/some-file-name?job=build

When the pipelines pass, you will see:

You now have a ready to use Gitlab CI.

Installing libraries, dependencies management

Fri, 27 Oct 2017 11:01:31 +0200

Common Lisp may have more libraries than you think. See:

Quickdocs - the library documentation hosting for CL.
the Awesome-cl list, a curated list of libraries.
lisp-lang.org’s recommended libraries (from State of the CL ecosystem, 2015)

Quicklisp is the de-facto package manager, but not the only tool.

Some terminology first

In the Common Lisp world, a package is a way of grouping symbols together and of providing encapsulation. It is similar to a C++ namespace, a Python module or a Java package.
A system is a collection of CL source files bundled with an .asd file which tells how to compile and load them. There is often a one-to-one relationship between systems and packages, but this is in no way mandatory. A system may declare a dependency on other systems. Systems are managed by ASDF (Another System Definition Facility), which offers functionalities similar to those of make and ld.so, and has become a de facto standard.
A Common Lisp library or project typically consists of one or several ASDF systems (and is distributed as one Quicklisp project).

Install Quicklisp

Quicklisp is more than a package manager, it is also a central repository (a dist) that ensures that all libraries build together. This involves some manual work (like reporting errors to package authors), so this is why Quicklisp releases its dist updates once a month (but fear not, we have other tools).

It provides its own dist but it is also possible to build our own.

To install it, we can either:

1- run this command, anywhere:

curl -O https://beta.quicklisp.org/quicklisp.lisp

and enter a Lisp REPL and load this file:

sbcl --load quicklisp.lisp

2- install the Debian package:

apt-get install cl-quicklisp

and load it, from a REPL:

(load "/usr/share/cl-quicklisp/quicklisp.lisp")

Then, in both cases, still from the REPL:

(quicklisp-quickstart:install)

This will create the ~/quicklisp/ directory, where Quicklisp will maintain its state and downloaded projects.

If you want Quicklisp to always be loaded in your Lisp sessions, run (ql:add-to-init-file): this adds the right stuff to the init file of your CL implementation. Otherwise, you have to run (load "~/quicklisp/setup.lisp") in every session if you want to use Quicklisp or any of the libraries installed through it.

It adds the following in your (for example) ~/.sbclrc:

#-quicklisp
  (let ((quicklisp-init (merge-pathnames "quicklisp/setup.lisp"
                                         (user-homedir-pathname))))
    (when (probe-file quicklisp-init)
      (load quicklisp-init)))

Install libraries

In the REPL:

(ql:quickload "package-name")

and voilà. See Quicklisp’s documentation for more commands.

Note also that dozens of Common Lisp libraries are packaged in Debian. The package names usually begin with the cl- prefix (use apt-cache search --names-only "^cl-.*" to list them all).

For example, in order to use the CL-PPCRE library (for regular expressions), one should first install the cl-ppcre package.

Then, in SBCL and ECL, it can be used with:

(require "asdf")
(require "cl-ppcre")
(cl-ppcre:regex-replace "fo+" "foo bar" "frob")

See more: https://wiki.debian.org/CommonLisp

Advanced dependencies management

Quicklisp installs the libraries into ~/quicklisp/local-projects/. A library installed here is automatically available for every project.

Providing our own version of a library. Cloning projects.

Given the property above, we can clone any library into the local-projects directory and it will be found by quicklisp and available right-away:

(ql:quickload "package")

And also given the M-. “go to this symbol definition” feature in Slime (and M-, to go back), it’s really easy to not only explore but start tweaking and extending other libraries.

How to work with local versions of libraries

If we need libraries to be installed locally, for only one project, or in order to easily ship a list of dependencies with an application, we can use Qlot. This is like Python’s virtual environments.

Quicklisp also provides Quicklisp bundles. They are self-contained sets of systems that are exported from Quicklisp and loadable without involving Quicklisp.

At last, there’s Quicklisp controller to help us build dists. Some projects use this, like CL21.

Source code organization, libraries and packages: https://lispmethods.com/libraries.html
https://wiki.debian.org/CommonLisp

Lisp software

Thu, 19 Oct 2017 07:51:49 +0100

There’s a fantastic showcase of succesfull Common Lisp software on lisp-lang.org: http://lisp-lang.org/success/ so go there first.

However all are not open source and it doesn’t list new or not so awesome but very cool or interesting software. That’s what we’ll do here to see that yeah Lisp is used today and to have code bases to look at.

If you are looking for CL libraries, you of course had a look to the Awesome CL list.

Awesome Lisp software

pgloader - re-written from Python with a 20 to 30x speed gain and more O_o - blog post By Dimitri Fontaine working at Postgres in 2014. He also maintains around 50 Debian packages of Lisp libraries for these needs. He gave a lightning talk at the 7th European Lisp Symposium. Uses Postmodern and lparallel for asynchronous IO. Also,

the new code base and feature set seems to attract way more users than the previous implementation ever did, despite using a less popular programming language.

pgchart - a self-contained web application that takes as input an SQL query text and outputs its data as a chart. By the same Dimitri Fontaine.
potato - a Slack-like conversation platform. Many features. CL in the backend, ClojureScript to the frontend. Apache Solr, RabbitMQ, email updates,… Web, Emacs and Android clients. web coding video.
turtl - a security focused online note taking app. Deployed at scale at Framanotes. Backend in CL.

Internet

nEXT browser - Qt based web browser.
cl-torrents - (my) library, cli and readline app to search for torrents on popular trackers, with an extensive tutorial.

Publishing software

Coleslaw - a static site generator similar to Jekyll. With nifty features (build on git push,…). Example blog: http://40ants.com/
Radiance - publishing software, between CMS and framework.
Reader - a simple blogging platform for Radiance.
Purplish - an imageboard app for Radiance.

Editors

Darkmatter - Common Lisp notebook (also exists cl-jupyter). Built on Clack.
Lem - an Emacs clone tailored for Common Lisp development, for the terminal or Electron. Screencast.

GUI apps

Halftone - a multiplatform and portable image viewer. Simple app to demo building Qt GUIs with Qtools.

Terminal apps

todo: enhance !

Pml - cli tool to parse my nginx logs and print statistics (and Tankfeeder for apache)
shuffletron - a terminal music player. [staling]
shtookovina - a program to help learn natural languages, based on audio recording by the Shtooka project, with an advanced readline interface and fully hackable in Common Lisp. Deprecated since 2015.

Why do we have to wait one month for Quicklisp updates ?

Thu, 31 Aug 2017 18:14:37 +0200

If you didn’t know that, now you do. Quicklisp releases software updates once a month (see Quicklisp’s blog). I didn’t know why, it isn’t explained on its website, so I asked (issue #148). I found the discussion very insightful, everybody being constructive, existing solutions being discussed and architectural choices explained. But it ended up brutally with one more Common Lisp oddity.

My first impression was that this fact is annoying, because it already prevented me a couple of times to use my own library and its most recent updates into other projects. They would pull the lib from Quicklisp but wouldn’t benefit from its latest features.

This way of doing was also not the package management model I was most used to (pip, npm,…), but by wording things differently it makes more sense to me. As an user says:

I think it is a very unique to quicklisp, making sure that everything compiles together. I can’t think of any other libraries/frameworks system on other languages/platforms that would go as far. Really great work.

So Quicklisp is more than a package manager, it is a “dist” builder ensuring everything works together, closer to apt than to pip.

The situation and shortcomings is well described by axity on his blog post (update: now available on mfiano.net):

Zach Beane has done an excellent job with Quicklisp, and it is far better than what we previously had available, but still has a few problems. Zach puts a lot of effort into curating a list of compatible software together into the form of a “Quicklisp dist” rolled out approximately every month. While this is great and puts near-zero maintenance on developers, it poses a few problems.

Firstly, Zach is a single point of failure. Yes, anyone can maintain their very own Quicklisp dist, but it isn’t going to see the masses, and the Quicklisp internals are not very well understood by anyone other than Zach.

Also the fact that the official dists are rolled out so far apart (a month or longer in the software world is an eternity), means developers cannot push hot-fixes or address user-reported bugs in a timely manner, without pushing maintenance onto the users by having them checkout upstream sources.

Modern languages such as Julia and Racket offer central repositories where a developer can register their software projects, and will be automatically indexed periodically, so that users can continue to install and update software without any maintenance, and still receive updates quickly when needed. Additionally, they push managing version dependencies onto the developer, which I do not believe to be a bad thing. In contrast, Common Lisp libraries are rarely versioned, and all of that maintenance is forced upon the Quicklisp dist curator.

I also like /u/ruricolist’s explanations (on reddit):

Quicklisp provides more assurance than you might expect. The criterion for inclusion, and maintenance, of a project in Quicklisp is that the project successfully load in SBCL alongside all the libraries in that Quicklisp distribution, and load without any errors or warnings. SBCL has extensive type inference and can catch and warn about many potential issues at compile time. And because of the pervasive use of macros in CL, successfully loading a library usually exercises a lot of code paths in its dependencies. To a surprising extent, “if it loads, it runs.”

Qlot isn’t something a library would use. You use it to set up the dependencies for an application. My approach (with TBRSS) is this. Obviously, every time I upgrade to a new Quicklisp dist, I run a test suite to make sure everything is working together. On the rare occasion there’s a problem, I either pin the offending library to an older version (with Qlot) or I fork it and fix it, again using Qlot to pull from the fork until the fix makes it into the next Quicklisp dist. And of course I also use Qlot for dependencies that are not, for whatever reason, available in Quicklisp.

So Quicklisp’s author answers:

The work relying on me is that I build everything in Quicklisp to make sure they build together before making a release. This covers a useful class of bugs. I’d love to incorporate more tests in the process to catch release problems that don’t manifest at build time.

I hope to make it easier for people to learn how to make their own dists. Then people will have the opportunity to make new software sets following the policies that are most important to them. I think there’s also plenty of room for other package managers for Common Lisp - maybe something styled more like clbuild would suit people who want instant access to updates.

Why a month ?

A month between updates is a compromise between chaos and stability.

I don’t think shorter release cycles are an unqualified good. In my experience, short release cycles can lead to instability and unpredictability, and I chose one month as a balance between getting timely updates and having a reliable, stable base.

The most labor-intensive part of making releases is monitoring daily failures and reporting bugs to the right people. The daily failure report is automated, but reporting bugs (and following up) can be a slog.

Then the discussion began. When voices support more frequent releases, Zach advertises again quicklisp-controller; I read one (and only one) small but direct criticism towards the maintainer:

@Hexstream I don’t think a fork will be required. I think this is less of an issue with @xach not willing to make changes required by the community and more of lack of manpower that’s willing and able to take up the burden of building and maintaining an automated Quicklisp dist/repository.

A one month update has its supporters, of course:

I have benefited multiple times from the 1 month release cycle and the due diligence that Xach and others put in every month. Even if they had a team of 50 I’d still vote to keep it as it is now. Making a separate dist is far more sensible if you need extra control over delivery times and it will still play well with quicklisp and your other projects.

I asked whether it would be possible to specify a git version, and I had an explanation:

Regarding pulling from git in the client, Quicklisp doesn’t work like that. I did not want to rely on external processes in order to be portable to all Common Lisp implementations and platforms. This was a real issue in clbuild and asdf-install.

If you want to update a bugfix of your library, fix it, and wait a month.

and the reminder to try Qlot, which allows exactly that (and to set dependencies locally).

Qlot will not help if you are writing a library and want to push fixes to its users quickly. It only helps for end-user application development. It works similar to virtualenv + pip (requirements.txt) from Python’s world, for example.

Phoe summed up the situation:

This thread is turning into a discussion about “why X approach is better than Quicklisp approach” which leads to a fruitful, but dead point.

If someone is not satisfied with the way the current Quicklisp dist works and would rather have a more automated solution, then they are free to extend the quicklisp-controller to their liking and implement the required functionality for:

creating a centralized service that acts as a QL repository and dist manager,

allowing the authors to upload and/or update their projects on that service,

setting up a CI test loop on that service for verifying that the packages build,

automatically updating the service’s dists with new releases,

maintaining all of the above.

Until such a person or group of people appears, nothing is going to be be achieved and nothing in Quicklisp is going to change.

Talk is cheap - @xach has at least built something that works and can act as a foundation.

And now, after only 17 messages to the thread, Zach Beane closes the thread:

I can appreciate there are other approaches that have advantages over how Quicklisp works. I hope this thread has helped shed some light on why it works the way it does, and my hopes for the future. I’m not against requests for changes, but not all of them can or will be accommodated. I’m also fully in favour of people doing their own thing if they have other priorities, experiences, and preferences - I think it would be great if there were even more options for Common Lisp project management.

Closing this for now - thanks for the discussion.

Hexstream had just the time to disagree

Generally agree with your last comment, but I just wanted to express my discontent at the premature closing of this thread, it seemed pretty fruitful to me and I don’t think it had yet reached a point of serious diminishing returns.

(Your project, your rules, though.)

and this is it.

I think, and I’m not the only one, that the thread was fruitful.

Indeed, we had explanations (that don’t appear in the doc), we had presentation of means to the resolution (in no doc), we had the presentation of how to fix the mentioned problem for library developers (Qlot, comprehensibly not referenced in Quicklisp doc, and little talked about over here), we had questions regarding the lack of documentation that could be tracked from this thread (Zach too said he wanted to write more doc). And I think the discussion was professional, with no animosity. So we could have tracked some progress, maybe we would have received more tips, but more importantly we’d have been done with this question.

But the thread is closed O_o Preventing people from communicating means preventing people from learning from each other, and thus makes the CL world evolving slower. Or people quitting, or simply be very surprised and not staying here. Indeed, it’s a negative feeling to see that. Why did Zach close the thread ? Is he bored of this discussion ? He can ignore it. Bored of being asked that ? It’s a recurrent question in reddit and in blogs. But this issue is the only mention of the subject on Quicklisp’s website and repository. Closing it makes it much less visible.

Thus it is more likely newcomers will ask again. It’s sure newcomers won’t learn why Quicklisp works like it does, which appears in good light in that issue (and below in reddit). What happened is again a thing that makes the CL world impenetrable (or with great effort) and subject to rants. It doesn’t need more reasons, seriously.

I wouldn’t bother if this subject was documented, but it is not. The issue about documenting Quicklisp, which Zach wanted to fix “soon”, stalls since 2014. It’s still open, at least.

Final links, with a glimpse of light:

the reddit thread
Qi, a Common Lisp package manager in the making - more traditional, no surprises - didn’t try.

How to do Functional Programming in Common Lisp ?

Fri, 28 Jul 2017 17:38:24 +0200

or “Common Lisp is not very functional-programming oriented”. What are the options ?

I mean, map is uncommon and there is no short words like take etc for functional composition. Right ?

edit see those snippets.

edit january, 2019: see this SO answer and the modf library.

Map and filter

Indeed, there are 8 or so map functions. The one we’re used to is mapcar. The simple map needs a second argument to specify its return type: (map 'list (lambda…. “filter” is named remove-if-not.

mapcan can also be useful. It concatenates the items into one list. So with mapcar we get a list of lists with our items, with mapcan simply a list of our items.

=> improved in cl21. It defines the usual map and keep-if and more functional verbs: take, drop, take-while, drop-while, butlast, sum,… in addition to fill, last, find-if[-not], remove-if[-not], delete[-if[-not]], reverse, reduce, sort, remove-duplicates, every, some,…

Note: remember that we still have access to cl symbols in CL21.

Functional composition

Lack of functional composition ? There is the Series library since 1989 that seems great, it lets us “write our program in a functional style without any runtime penalty at all !” [malisper on his blog post]

But yes again, it has not the modern vocabulary we expect and it seems abandonware (and its documentation is an old pdf paper but now hopefully this wiki is better).

edit 2021: GTWIWTG is a “generators” library similar in scope from Series, with modern idioms, but probably not as efficient. BTW, there are now lots of Series example snippets in the Cookbook/iteration page.

CL21 has an operator to compose functions: https://github.com/cl21/cl21/wiki/Language-Difference-between-CL21-and-Common-Lisp#function

Threading macros (pipes)

We have two packages in Quicklisp:

cl-arrows defines ->, ->> and the generalized ones -<> and -<>>. At the time of writing it has two unanswered PRs to add more, like the “when-guarded”-“nil shortcuting diamond wand” some->.
arrow-macros is a bit more complete but has more dependencies (it needs a code walker) which are not portable (failed on ECL, Allegro, ABCL, Clisp). It has the some->, cond-> and as-> ones.

Data structures

We have the nice FSet, the functional collection for Common Lisp.

Anaphoric macros (for shorter lambdas)

I like anaphoric macros but I didn’t find one ready to use in a library to write shorter lambdas. The first macro I wrote was directly to mimick elisp’s dash.el excellent library:

(defmacro --map (form list)
  `(mapcar (lambda (it) ,form) ,list))

so than we can write

(--map (* it 2) '(2 3))

instead of (mapcar (lambda (it) (* it 2) '(2 3))). This macro is very simple. It should be in a library, I don’t want to copy-paste it in every project of mine. => in CL21 ? In Anaphora ?

=> Personnally I’m very happy with cl21’s short lambdas (that were not documented…):

(map ^(foo-bar %) items)

or with (lm (x)…. Unused arguments will be ignored automatically.

I also quite like the Arc way of doing, that I found in the Clamp project. Arc is another language and Clamp is more of a POC I guess.

For shorter lambdas we also have f-underscore, that I see used in the wild. It defines some macros to write shorter lambdas:

f is a synonym for lambda
f0 is a lambda that takes 0 arguments
f_ takes one, accessible in _ (underscore)
f_n takes one &rest argument
f_% ignores its rest argument
m is a lambda “that has a macro-lambda-list instead of an ordinary lambda-list”.

You might like Trivia for pattern matching too.

Hope this helps !

Data Structures

Thu, 27 Jul 2017 16:48:42 +0200

What the heck are alists and plists exactly, how do we manipulate data structures ? It seems tedious sometimes, are there helpers ?

Best read in the Cookbook.

We hope to give here a clear reference of the common data structures. To really learn the language, you should take the time to read other resources. The following ones, which we relied upon, have many more details:

Practical CL, by Peter Seibel
CL Recipes, by E. Weitz, full of explanations and tips,
the CL standard with a nice TOC, functions reference, extensive descriptions, more examples and warnings (i.e: everything).
a Common Lisp quick reference

Table of Contents

Lists

Building lists. Cons cells, lists.

A list is also a sequence, so we can use the functions shown below.

The list basic element is the cons cell. We build lists by assembling cons cells.

(cons 1 2)
;; => (1 . 2) ;; representation with a point, a dotted pair.

It looks like this:

[o|o]--- 2
 |
 1

If the cdr of the first cell is another cons cell, and if the cdr of this last one is nil, we build a list:

(cons 1 (cons 2 nil))
;; => (1 2)

It looks like this:

[o|o]---[o|/]
 |       |
 1       2

(ascii art by draw-cons-tree).

See that the representation is not a dotted pair ? The Lisp printer understands the convention.

Finally we can simply build a literal list with list:

(list 1 2)
;; => (1 2)

or by calling quote:

'(1 2)
;; => (1 2)

which is shorthand notation for the function call (quote (1 2)).

car/cdr or first/rest (and second… to tenth)

(car (cons 1 2)) ;; => 1
(cdr (cons 1 2)) ;; => 2
(first (cons 1 2)) ;; => 1
(first '(1 2 3)) ;; => 1
(rest '(1 2 3)) ;; => (2 3)

We can assign any new value with setf.

last, butlast, nbutlast (&optional n)

return the last cons cell in a list (or the nth last cons cells).

(last '(1 2 3))
;; => (3)
(car (last '(1 2 3)) )
;; => 3
(butlast '(1 2 3))
;; => (1 2)

reverse, nreverse

reverse and nreverse return a new sequence.

nreverse is destructive. The N stands for non-consing, meaning it doesn’t need to allocate any new cons cells. It might (but in practice, does) reuse and modify the original sequence:

(defparameter mylist '(1 2 3))
;; => (1 2 3)
(reverse mylist)
;; => (3 2 1)
mylist
;; => (1 2 3)
(nreverse mylist)
;; => (3 2 1)
mylist
;; => (1) in SBCL but implementation dependant.

append

append takes any number of list arguments and returns a new list containing the elements of all its arguments:

(append (list 1 2) (list 3 4))
;; => (1 2 3 4)

The new list shares some cons cells with the (3 4):

http://gigamonkeys.com/book/figures/after-append.png

Note: cl21’s append is generic (for strings, lists, vectors and its abstract-sequence).

nconc is the recycling equivalent.

push (item, place)

push prepends item to the list that is stored in place, stores the resulting list in place, and returns the list.

(defparameter mylist '(1 2 3))
(push 0 mylist)
;; => (0 1 2 3)

(defparameter x ’(a (b c) d))
;; => (A (B C) D)
(push 5 (cadr x))
;; => (5 B C)
x
;; => (A (5 B C) D)

push is equivalent to (setf place (cons item place )) except that the subforms of place are evaluated only once, and item is evaluated before place.

There is no built-in function to add to the end of a list. It is a more costly operation (have to traverse the whole list). So if you need to do this: either consider using another data structure, either just reverse your list when needed.

pop

a destructive operation.

nthcdr (index, list)

Use this if first, second and the rest up to tenth are not enough.

car/cdr and composites (cadr, caadr…) - accessing lists inside lists

They make sense when applied to lists containing other lists.

(caar (list 1 2 3))                  ==> error
(caar (list (list 1 2) 3))           ==> 1
(cadr (list (list 1 2) (list 3 4)))  ==> (3 4)
(caadr (list (list 1 2) (list 3 4))) ==> 3

destructuring-bind (parameter*, list)

It binds the parameter values to the list elements. We can destructure trees, plists and even provide defaults.

Simple matching:

(destructuring-bind (x y z) (list 1 2 3)
  (list :x x :y y :z z))
;; => (:X 1 :Y 2 :Z 3)

Matching inside sublists:

(destructuring-bind (x (y1 y2) z) (list 1 (list 2 20) 3)
  (list :x x :y1 y1 :y2 y2 :z z))
;; => (:X 1 :Y1 2 :Y2 20 :Z 3)

The parameter list can use the usual &optional, &rest and &key parameters.

(destructuring-bind (x (y1 &optional y2) z) (list 1 (list 2) 3)
  (list :x x :y1 y1 :y2 y2 :z z))
;; => (:X 1 :Y1 2 :Y2 NIL :Z 3)

(destructuring-bind (&key x y z) (list :z 1 :y 2 :x 3)
  (list :x x :y y :z z))
;; => (:X 3 :Y 2 :Z 1)

The &whole parameter is bound to the whole list. It must be the first one and others can follow.

(destructuring-bind (&whole whole-list &key x y z) (list :z 1 :y 2 :x 3)
  (list :x x :y y :z z :whole whole-list))
;; => (:X 3 :Y 2 :Z 1 :WHOLE-LIST (:Z 1 :Y 2 :X 3))

Destructuring a plist, giving defaults:

(example from Common Lisp Recipes, by E. Weitz, Apress, 2016)

(destructuring-bind (&key a (b :not-found) c
                     &allow-other-keys)
    ’(:c 23 :d "D" :a #\A :foo :whatever)
  (list a b c))
;; => (#\A :NOT-FOUND 23)

If this gives you the will to do pattern matching, see pattern matching.

Predicates: null, listp

null is equivalent to not, but considered better style.

listp tests wether an object is a cons cell or nil.

and sequences’ predicates.

ldiff, tailp, list*, make-list, fill, revappend, nreconc, consp, atom

(make-list 3 :initial-element "ta")
;; => ("ta" "ta" "ta")

(make-list 3)
;; => (NIL NIL NIL)
(fill * "hello")
;; => ("hello" "hello" "hello")

Sequences

lists and vectors (and thus strings) are sequences.

Note: see also the strings page.

Many of the sequence functions take keyword arguments. All keyword arguments are optional and, if specified, may appear in any order.

Pay attention to the :test argument. It defaults to eql (for strings, use :equal).

The :key argument should be passed either nil, or a function of one argument. This key function is used as a filter through which the elements of the sequence are seen. For instance, this:

(find x y :key 'car)

is similar to (assoc* x y): It searches for an element of the list whose car equals x, rather than for an element which equals x itself. If :key is omitted or nil, the filter is effectively the identity function.

Example with an alist (see definition below):

(defparameter my-alist (list (cons 'foo "foo")
                             (cons 'bar "bar")))
;; => ((FOO . "foo") (BAR . "bar"))
(find 'bar my-alist)
;; => NIL
(find 'bar my-alist :key 'car)
;; => (BAR . "bar")

For more, use a lambda that takes one parameter.

(find 'bar my-alist :key (lambda (it) (car it)))

Note: and cl21 has short lambdas:

(find 'bar my-alist :key ^(car %))
(find 'bar my-alist :key (lm (it) (car it)))

Predicates: every, some,…

every, notevery (test, sequence): return nil or t, respectively, as soon as one test on any set of the corresponding elements of sequences returns nil.

(defparameter foo '(1 2 3))
(every #'evenp foo)
;; => NIL
(some #'evenp foo)
;; => T

with a list of strings:

(defparameter str '("foo" "bar" "team"))
(every #'stringp str)
;; => T
(some #'(lambda (it) (= 3 (length it))) str)
;; => T
(some ^(= 3 (length %)) str) ;; in CL21
;; => T

some, notany (test, sequence): return either the value of the test, or nil.

mismatch (sequence-a, sequence-b): Return position in sequence-a where sequence-a and sequence-b begin to mismatch. Return NIL if they match entirely. Other parameters: :from-end bool, :start1, :start2 and their :end[1,2].

Functions

See also sequence functions defined in Alexandria: starts-with, ends-with, ends-with-subseq, length=, emptyp,…

length (sequence)

member (elt, sequence)

elt (sequence, index)

beware, here the sequence comes first.

count (foo sequence)

Return the number of elements in sequence that match foo.

Additional paramaters: :from-end, :start, :end.

See also count-if, count-not (test-function sequence).

subseq (sequence start, [end])

It is “setf”able, but only works if the new sequence has the same length of the one to replace.

sort, stable-sort (sequence, test [, key function])

find, position (foo, sequence)

also find-if, find-if-not, position-if, position-if-not (test sequence). See :key and :test parameters.

search (sequence-a, sequence-b)

Search sequence-b for a subsequence matching sequence-a. Return position in sequence-b, or NIL. Has the from-end, end1/2 and others parameters.

substitute, nsubstitute[if,if-not]

sort, stable-sort, merge

replace (sequence-a, sequence-b)

Replace elements of sequence-a with elements of sequence-b.

remove, delete (foo sequence)

Make a copy of sequence without elements matching foo. Has :start/end, :key and :count parameters.

delete is the recycling version of remove.

(remove "foo" '("foo" "bar" "foo") :test 'equal)
;; => ("bar")

mapping (map, mapcar, remove-if[-not],…)

If you’re used to map and filter in other languages, you probably want mapcar. But it only works on lists, so to iterate on vectors (and produce either a vector or a list, use (map 'list function vector).

mapcar also accepts multiple lists with &rest more-seqs. The mapping stops as soon as the shortest sequence runs out.

Note: cl21’s map is a generic mapcar for lists and vectors.

map takes the output-type as first argument ('list, 'vector or 'string):

(defparameter foo '(1 2 3))
(map 'list (lambda (it) (* 10 it)) foo)

reduce (function, sequence). Special parameter: :initial-value.

(reduce '- '(1 2 3 4))
;; => -8
(reduce '- '(1 2 3 4) :initial-value 100)
;; => 90

Filter is here called remove-if-not.

Flatten a list (Alexandria)

With Alexandria, we have the flatten function.

Creating lists with variables

That’s one use of the backquote:

(defparameter *var* "bar")
;; First try:
'("foo" *var* "baz") ;; no backquote
;; => ("foo" *VAR* "baz") ;; nope

Second try, with backquote interpolation:

`("foo" ,*var* "baz")     ;; backquote, comma
;; => ("foo" "bar" "baz") ;; good

The backquote first warns we’ll do interpolation, the comma introduces the value of the variable.

If our variable is a list:

(defparameter *var* '("bar" "baz"))
;; First try:
`("foo" ,*var*)
;; => ("foo" ("bar" "baz")) ;; nested list
`("foo" ,@*var*)            ;; backquote, comma-@ to
;; => ("foo" "bar" "baz")

E. Weitz warns that “objects generated this way will very likely share structure (see Recipe 2-7)“.

Comparing lists

We can use sets functions.

Set

intersection

What elements are both in list-a and list-b ?

(defparameter list-a '(0 1 2 3))
(defparameter list-b '(0 2 4))
(intersection list-a list-b)
;; => (2 0)

set-difference

Remove the elements of list-b from list-a:

(set-difference list-a list-b)
;; => (3 1)
(set-difference list-b list-a)
;; => (4)

union

join the two lists:

(union list-a list-b)
;; => (3 1 0 2 4) ;; order can be different in your lisp

set-exclusive-or

Remove the elements that are in both lists:

(set-exclusive-or list-a list-b)
;; => (4 3 1)

and their recycling “n” counterpart (nintersection,…).

See also functions in Alexandria: setp, set-equal,…

Fset - immutable data structure

You may want to have a look at the FSet library (in Quicklisp).

Arrays and vectors

Arrays have constant-time access characteristics.

They can be fixed or adjustable. A simple array is neither displaced (using :displaced-to, to point to another array) nor adjustable (:adjust-array), nor does it have a fill pointer (fill-pointer, that moves when we add or remove elements).

A vector is an array with rank 1 (of one dimension). It is also a sequence (see above).

A simple vector is a simple array that is also not specialized (it doesn’t use :element-type to set the types of the elements).

Create an array, one or many dimensions

make-array (sizes-list :adjustable bool)

adjust-array (array, sizes-list, :element-type, :initial-element)

Access: aref (array i [j …])

aref (array i j k …) or row-major-aref (array i) equivalent to (aref i i i …).

The result is setfable.

(defparameter myarray (make-array '(2 2 2) :initial-element 1))
myarray
;; => #3A(((1 1) (1 1)) ((1 1) (1 1)))
(aref myarray 0 0 0)
;; => 1
(setf (aref myarray 0 0 0) 9)
;; => 9
(row-major-aref myarray 0)
;; => 9

Sizes

array-total-size (array): how many elements will fit in the array ?

array-dimensions (array): list containing the length of the array’s dimensions.

array-dimension (array i): length of the *i*th dimension.

array-rank number of dimensions of the array.

(defparameter myarray (make-array '(2 2 2)))
;; => MYARRAY
myarray
;; => #3A(((0 0) (0 0)) ((0 0) (0 0)))
(array-rank myarray)
;; => 3
(array-dimensions myarray)
;; => (2 2 2)
(array-dimension myarray 0)
;; => 2
(array-total-size myarray)
;; => 8

Vectors

Create with vector or the reader macro #(). It returns a simple vector.

(vector 1 2 3)
;; => #(1 2 3)
#(1 2 3)
;; => #(1 2 3)

vector-push (foo vector): replace the vector element pointed to by the fill pointer by foo. Can be destructive.

vector-push-extend *(foo vector [extension-num])*t

vector-pop (vector): return the element of vector its fill pointer points to.

fill-pointer (vector). setfable.

and see also the sequence functions.

Transforming a vector to a list.

If you’re mapping over it, see the map function whose first parameter is the result type.

Or use (coerce vector 'list).

Hash Table

Hash Tables are a powerful data structure, associating keys with values in a very efficient way. Hash Tables are often preferred over association lists whenever performance is an issue, but they introduce a little overhead that makes assoc lists better if there are only a few key-value pairs to maintain.

Alists can be used sometimes differently though:

they can be ordered
we can push cons cells that have the same key, remove the one in front and we have a stack
they have a human-readable printed representation
they can be easily (de)serialized
because of RASSOC, keys and values in alists are essentially interchangeable; whereas in hash tables, keys and values play very different roles (as usual, see CL Recipes for more).

Creating a Hash Table

Hash Tables are created using the function make-hash-table. It has no required argument. Its most used optional keyword argument is :test, specifying the function used to test the equality of keys.

If we are using the cl21 extension library, we can create a hash table and add elements in the same time with the new #H reader syntax:

(defparameter *my-hash* #H(:name "Eitaro Fukamachi"))

then we access an element with

(getf *my-hash* :name)

Getting a value from a Hash Table

The function gethash takes two required arguments: a key and a hash table. It returns two values: the value corresponding to the key in the hash table (or nil if not found), and a boolean indicating whether the key was found in the table. That second value is necessary since nil is a valid value in a key-value pair, so getting nil as first value from gethash does not necessarily mean that the key was not found in the table.

Getting a key that does not exist with a default value

gethash has an optional third argument:

(gethash 'bar *my-hash* "default-bar")
;; => "default-bar"
;;     NIL

Getting all keys or all values of a hash table

The Alexandria library (in Quicklisp) has the functions hash-table-keys and hash-table-values for that.

(ql:quickload :alexandria)
;; […]
(alexandria:hash-table-keys *my-hash*)
;; => (BAR)

Adding an Element to a Hash Table

If you want to add an element to a hash table, you can use gethash, the function to retrieve elements from the hash table, in conjunction with setf.

CL-USER> (defparameter *my-hash* (make-hash-table))
*MY-HASH*
CL-USER> (setf (gethash 'one-entry *my-hash*) "one")
"one"
CL-USER> (setf (gethash 'another-entry *my-hash*) 2/4)
1/2
CL-USER> (gethash 'one-entry *my-hash*)
"one"
T
CL-USER> (gethash 'another-entry *my-hash*)
1/2
T

Testing for the Presence of a Key in a Hash Table

The first value returned by gethash is the object in the hash table that’s associated with the key you provided as an argument to gethash or nil if no value exists for this key. This value can act as a generalized boolean if you want to test for the presence of keys.

CL-USER> (defparameter *my-hash* (make-hash-table))
*MY-HASH*
CL-USER> (setf (gethash 'one-entry *my-hash*) "one")
"one"
CL-USER> (if (gethash 'one-entry *my-hash*)
           "Key exists"
           "Key does not exist")
"Key exists"
CL-USER> (if (gethash 'another-entry *my-hash*)
           "Key exists"
           "Key does not exist")
"Key does not exist"

But note that this does not work if nil is amongst the values that you want to store in the hash.

CL-USER> (setf (gethash 'another-entry *my-hash*) nil)
NIL
CL-USER> (if (gethash 'another-entry *my-hash*)
           "Key exists"
           "Key does not exist")
"Key does not exist"

In this case you’ll have to check the second return value of gethash which will always return nil if no value is found and T otherwise.

CL-USER> (if (nth-value 1 (gethash 'another-entry *my-hash*))
           "Key exists"
           "Key does not exist")
"Key exists"
CL-USER> (if (nth-value 1 (gethash 'no-entry *my-hash*))
           "Key exists"
           "Key does not exist")
"Key does not exist"

Deleting from a Hash Table

Use remhash to delete a hash entry. Both the key and its associated value will be removed from the hash table. remhash returns T if there was such an entry, nil otherwise.

CL-USER> (defparameter *my-hash* (make-hash-table))
*MY-HASH*
CL-USER> (setf (gethash 'first-key *my-hash*) 'one)
ONE
CL-USER> (gethash 'first-key *my-hash*)
ONE
T
CL-USER> (remhash 'first-key *my-hash*)
T
CL-USER> (gethash 'first-key *my-hash*)
NIL
NIL
CL-USER> (gethash 'no-entry *my-hash*)
NIL
NIL
CL-USER> (remhash 'no-entry *my-hash*)
NIL
CL-USER> (gethash 'no-entry *my-hash*)
NIL
NIL

Traversing a Hash Table

If you want to perform an action on each entry (i.e., each key-value pair) in a hash table, you have several options:

You can use maphash which iterates over all entries in the hash table. Its first argument must be a function which accepts two arguments, the key and the value of each entry. Note that due to the nature of hash tables you can’t control the order in which the entries are provided by maphash (or other traversing constructs). maphash always returns nil.

CL-USER> (defparameter *my-hash* (make-hash-table))
*MY-HASH*
CL-USER> (setf (gethash 'first-key *my-hash*) 'one)
ONE
CL-USER> (setf (gethash 'second-key *my-hash*) 'two)
TWO
CL-USER> (setf (gethash 'third-key *my-hash*) nil)
NIL
CL-USER> (setf (gethash nil *my-hash*) 'nil-value)
NIL-VALUE
CL-USER> (defun print-hash-entry (key value)
    (format t "The value associated with the key ~S is ~S~%" key value))
PRINT-HASH-ENTRY
CL-USER> (maphash #'print-hash-entry *my-hash*)
The value associated with the key FIRST-KEY is ONE
The value associated with the key SECOND-KEY is TWO
The value associated with the key THIRD-KEY is NIL
The value associated with the key NIL is NIL-VALUE

You can also use with-hash-table-iterator, a macro which turns (via macrolet) its first argument into an iterator that on each invocation returns three values per hash table entry - a generalized boolean that’s true if an entry is returned, the key of the entry, and the value of the entry. If there are no more entries, only one value is returned - nil.

;;; same hash-table as above
CL-USER> (with-hash-table-iterator (my-iterator *my-hash*)
           (loop
              (multiple-value-bind (entry-p key value)
                  (my-iterator)
                (if entry-p
                    (print-hash-entry key value)
                    (return)))))
The value associated with the key FIRST-KEY is ONE
The value associated with the key SECOND-KEY is TWO
The value associated with the key THIRD-KEY is NIL
The value associated with the key NIL is NIL-VALUE
NIL

Note the following caveat from the HyperSpec: “It is unspecified what happens if any of the implicit interior state of an iteration is returned outside the dynamic extent of the with-hash-table-iterator form such as by returning some closure over the invocation form.”

And there’s always loop:

;;; same hash-table as above
CL-USER> (loop for key being the hash-keys of *my-hash*
           do (print key))
FIRST-KEY
SECOND-KEY
THIRD-KEY
NIL
NIL
CL-USER> (loop for key being the hash-keys of *my-hash*
           using (hash-value value)
           do (format t "The value associated with the key ~S is ~S~%" key value))
The value associated with the key FIRST-KEY is ONE
The value associated with the key SECOND-KEY is TWO
The value associated with the key THIRD-KEY is NIL
The value associated with the key NIL is NIL-VALUE
NIL
CL-USER> (loop for value being the hash-values of *my-hash*
           do (print value))
ONE
TWO
NIL
NIL-VALUE
NIL
CL-USER> (loop for value being the hash-values of *my-hash*
           using (hash-key key)
           do (format t "~&~A -> ~A" key value))
FIRST-KEY -> ONE
SECOND-KEY -> TWO
THIRD-KEY -> NIL
NIL -> NIL-VALUE
NIL

Last, we also have cl21’s (doeach ((key val) *hash*) …).

Traversign keys or values

To map over keys or values we can again rely on Alexandria with maphash-keys and maphash-values.

Counting the Entries in a Hash Table

No need to use your fingers - Common Lisp has a built-in function to do it for you: hash-table-count.

CL-USER> (defparameter *my-hash* (make-hash-table))
*MY-HASH*
CL-USER> (hash-table-count *my-hash*)
0
CL-USER> (setf (gethash 'first *my-hash*) 1)
1
CL-USER> (setf (gethash 'second *my-hash*) 2)
2
CL-USER> (setf (gethash 'third *my-hash*) 3)
3
CL-USER> (hash-table-count *my-hash*)
3
CL-USER> (setf (gethash 'second *my-hash*) 'two)
TWO
CL-USER> (hash-table-count *my-hash*)
3
CL-USER> (clrhash *my-hash*)
#<EQL hash table, 0 entries {48205F35}>
CL-USER> (hash-table-count *my-hash*)
0

Performance Issues: The Size of your Hash Table

The make-hash-table function has a couple of optional parameters which control the initial size of your hash table and how it’ll grow if it needs to grow. This can be an important performance issue if you’re working with large hash tables. Here’s an (admittedly not very scientific) example with CMUCL pre-18d on Linux:

CL-USER> (defparameter *my-hash* (make-hash-table))
*MY-HASH*
CL-USER> (hash-table-size *my-hash*)
65
CL-USER> (hash-table-rehash-size *my-hash*)
1.5
CL-USER> (time (dotimes (n 100000) (setf (gethash n *my-hash*) n)))
Compiling LAMBDA NIL:
Compiling Top-Level Form:

Evaluation took:
  0.27 seconds of real time
  0.25 seconds of user run time
  0.02 seconds of system run time
  0 page faults and
  8754768 bytes consed.
NIL
CL-USER> (time (dotimes (n 100000) (setf (gethash n *my-hash*) n)))
Compiling LAMBDA NIL:
Compiling Top-Level Form:

Evaluation took:
  0.05 seconds of real time
  0.05 seconds of user run time
  0.0 seconds of system run time
  0 page faults and
  0 bytes consed.
NIL

The values for hash-table-size and hash-table-rehash-size are implementation-dependent. In our case, CMUCL chooses and initial size of 65, and it will increase the size of the hash by 50 percent whenever it needs to grow. Let’s see how often we have to re-size the hash until we reach the final size…

CL-USER> (log (/ 100000 65) 1.5)
18.099062
CL-USER> (let ((size 65)) (dotimes (n 20) (print (list n size)) (setq size (* 1.5 size))))
(0 65)
(1 97.5)
(2 146.25)
(3 219.375)
(4 329.0625)
(5 493.59375)
(6 740.3906)
(7 1110.5859)
(8 1665.8789)
(9 2498.8184)
(10 3748.2275)
(11 5622.3413)
(12 8433.512)
(13 12650.268)
(14 18975.402)
(15 28463.104)
(16 42694.656)
(17 64041.984)
(18 96062.98)
(19 144094.47)
NIL

The hash has to be re-sized 19 times until it’s big enough to hold 100,000 entries. That explains why we saw a lot of consing and why it took rather long to fill the hash table. It also explains why the second run was much faster - the hash table already had the correct size.

Here’s a faster way to do it: If we know in advance how big our hash will be, we can start with the right size:

CL-USER> (defparameter *my-hash* (make-hash-table :size 100000))
*MY-HASH*
CL-USER> (hash-table-size *my-hash*)
100000
CL-USER> (time (dotimes (n 100000) (setf (gethash n *my-hash*) n)))
Compiling LAMBDA NIL:
Compiling Top-Level Form:

Evaluation took:
  0.04 seconds of real time
  0.04 seconds of user run time
  0.0 seconds of system run time
  0 page faults and
  0 bytes consed.
NIL

That’s obviously much faster. And there was no consing involved because we didn’t have to re-size at all. If we don’t know the final size in advance but can guess the growth behaviour of our hash table we can also provide this value to make-hash-table. We can provide an integer to specify absolute growth or a float to specify relative growth.

CL-USER> (defparameter *my-hash* (make-hash-table :rehash-size 100000))
*MY-HASH*
CL-USER> (hash-table-size *my-hash*)
65
CL-USER> (hash-table-rehash-size *my-hash*)
100000
CL-USER> (time (dotimes (n 100000) (setf (gethash n *my-hash*) n)))
Compiling LAMBDA NIL:
Compiling Top-Level Form:

Evaluation took:
  0.07 seconds of real time
  0.05 seconds of user run time
  0.01 seconds of system run time
  0 page faults and
  2001360 bytes consed.
NIL

Also rather fast (we only needed one re-size) but much more consing because almost the whole hash table (minus 65 initial elements) had to be built during the loop.

Note that you can also specify the rehash-threshold while creating a new hash table. One final remark: Your implementation is allowed to completely ignore the values provided for rehash-size and rehash-threshold…

Alist

An association list is a list of cons cells.

This simple example:

(defparameter my-alist (list (cons 'foo "foo")
                             (cons 'bar "bar")))
;; => ((FOO . "foo") (BAR . "bar"))

looks like this:

[o|o]---[o|/]
 |       |
 |      [o|o]---"bar"
 |       |
 |      BAR
 |
[o|o]---"foo"
 |
FOO

The constructor pairlis associates a list of keys and a list of values:

(pairlis '(:foo :bar)
         '("foo" "bar"))
;; => ((:BAR . "bar") (:FOO . "foo"))

To get a key, we have assoc (use :test 'equal when your keys are strings, as usual). It returns the whole cons cell, so you may want to use cdr or second to get the value. There is assoc-if, and rassoc to get a cons cell by its value.

To add a key, we push another cons cell:

(push (cons 'team "team") my-alist)
;; => ((TEAM . "team") (FOO . "foo") (BAR . "bar"))

We can use pop and other functions that operate on lists, like remove:

(remove :team my-alist)
;; => ((:TEAM . "team") (FOO . "foo") (BAR . "bar")) ;; didn't remove anything
(remove :team my-alist :key 'car)
;; => ((FOO . "foo") (BAR . "bar")) ;; returns a copy

Remove only one element with :count:

(push (cons 'bar "bar2") my-alist)
;; => ((BAR . "bar2") (TEAM . "team") (FOO . "foo") (BAR . "bar")) ;; twice the 'bar key
(remove 'bar my-alist :key 'car :count 1)
;; => ((TEAM . "team") (FOO . "foo") (BAR . "bar"))
;; because otherwise:
(remove 'bar my-alist :key 'car)
;; => ((TEAM . "team") (FOO . "foo")) ;; no more 'bar

In the Alexandria library, see some functions like remove-from-plist, alist-plist,…

Plist

A property list is simply a list that alternates a key, a value, and so on, where its keys are symbols (we can not set its :test). More precisely, it first has a cons cell whose car is the key, whose cdr points to the following cons cell whose car is the value.

For example this plist:

(defparameter my-plist (list 'foo "foo" 'bar "bar"))

looks like this:

[o|o]---[o|o]---[o|o]---[o|/]
 |       |       |       |
FOO     "foo"   BAR     "bar"

We access an element with getf (list elt) (it returns the value) (the list comes as first element),

we remove an element with remf.

(defparameter my-plist (list 'foo "foo" 'bar "bar"))
;; => (FOO "foo" BAR "bar")
(setf (getf my-plist 'foo) "foo!!!")
;; => "foo!!!"

Tree

tree-equal, copy-tree. They descend recursively into the car and the cdr of the cons cells they visit.

Sycamore - purely functional weight-balanced binary trees

https://github.com/ndantam/sycamore

Features:

Fast, purely functional weight-balanced binary trees.
- Leaf nodes are simple-vectors, greatly reducing tree height.
Interfaces for tree Sets and Maps (dictionaries).
Ropes
Purely functional pairing heaps
Purely functional amortized queue.

See more in other resources !

Common Lisp Async Web Scraping

Thu, 29 Jun 2017 17:47:41 +0100

The set of tools to do web scraping in Common Lisp is pretty complete and pleasant. In this short tutorial we’ll see how to make http requests, parse html, extract content and do asynchronous requests.

Our simple task will be to extract the list of links on the CL Cookbook’s index page and check if they are reachable.

Best read in the Cookbook !

We’ll use the following libraries:

Dexador - an HTTP client (that aims at replacing the venerable Drakma),
Plump - a markup parser, that works on malformed HTML,
Lquery - a DOM manipulation library, to extract content from our Plump result,
lparallel - a library for parallel programming (read more in the process section).

Before starting let’s install those libraries with Quicklisp:

(ql:quickload '(:dexador :plump :lquery :lparallel))

Table of Contents

HTTP Requests

Easy things first. Install Dexador. Then we use the get function:

(defvar *url* "https://lispcookbook.github.io/cl-cookbook/")
(defvar *request* (dex:get *url*))

This returns a list of values: the whole page content, the return code (200), the response headers, the uri and the stream.

"<!DOCTYPE html>
 <html lang=\"en\">
  <head>
    <title>Home &ndash; the Common Lisp Cookbook</title>
    […]
    "
200
#<HASH-TABLE :TEST EQUAL :COUNT 19 {1008BF3043}>
#<QURI.URI.HTTP:URI-HTTPS https://lispcookbook.github.io/cl-cookbook/>
#<CL+SSL::SSL-STREAM for #<FD-STREAM for "socket 192.168.0.23:34897, peer: 151.101.120.133:443" {100781C133}>>

Remember, in Slime we can inspect the objects with a right-click on them.

Parsing and extracting content with CSS selectors

We’ll use Plump to parse the html and Lquery to extract content. They have nice documentation:

(defvar *parsed-content* (plump:parse *request*))
;; => *PARSED-CONTENT**
*parsed-content**
;; => #<PLUMP-DOM:ROOT {1009EE5FE3}>

Now we’ll extract the links with CSS selectors.

Note: to find out what should be the CSS selector of the element I’m interested in, I right click on an element in the browser and I choose “Inspect element”. This opens up the inspector of my browser’s web dev tool and I can study the page structure.

So the links I want to extract are in a page with an id of value “content”, and they are in regular list elements (li).

Let’s try something:

(lquery:$  *parsed-content* "#content li")
;; => #(#<PLUMP-DOM:ELEMENT li {100B3263A3}> #<PLUMP-DOM:ELEMENT li {100B3263E3}>
;;  #<PLUMP-DOM:ELEMENT li {100B326423}> #<PLUMP-DOM:ELEMENT li {100B326463}>
;;  #<PLUMP-DOM:ELEMENT li {100B3264A3}> #<PLUMP-DOM:ELEMENT li {100B3264E3}>
;;  #<PLUMP-DOM:ELEMENT li {100B326523}> #<PLUMP-DOM:ELEMENT li {100B326563}>
;;  #<PLUMP-DOM:ELEMENT li {100B3265A3}> #<PLUMP-DOM:ELEMENT li {100B3265E3}>
;;  #<PLUMP-DOM:ELEMENT li {100B326623}> #<PLUMP-DOM:ELEMENT li {100B326663}>
;;  […]

Wow it works ! We get here a vector of plump elements.

Since it is a vector we could map over them with (map 'vector (lambda (elt) (…)) *).

But I’d like to easily check what those elements are. To see their textual content we can append (text) to our lquery form:

(lquery:$  *parsed-content* "#content" (text))
#("License" "Editor support" "Strings" "Dates and Times" "Hash Tables"
  "Pattern Matching / Regular Expressions" "Functions" "Loop" "Input/Output"
  "Files and Directories" "Packages" "Macros and Backquote"
  "CLOS (the Common Lisp Object System)" "Sockets" "Interfacing with your OS"
  "Foreign Function Interfaces" "Threads" "Defining Systems"
  "Using the Win32 API" "Testing" "Miscellaneous" "License" "Marco Antoniotti"
  "Zach Beane" "Pierpaolo Bernardi" "Christopher Brown" "Frederic Brunel"
  "Jeff Caldwell" "Bill Clementson" "Martin Cracauer" "Gerald Doussot"
  "Paul Foley" "Jörg-Cyril
  […]
  "Edi Weitz" "Fernando Borretti" "lisp-lang.org" "The Awesome-cl list"
  "The Common Lisp HyperSpec by Kent M. Pitman" "The Common Lisp UltraSpec"
  "Practical Common Lisp by Peter Seibel"
  "Common Lisp Recipes by Edmund Weitz, published in 2016,"
  […]
  "A Tutorial on Good Lisp Style by Peter Norvig and Kent Pitman"
  "Lisp and Elements of Style by Nick Levine"
  "Pascal Costanza’s Highly Opinionated Guide to Lisp"
  "Loving Lisp - the Savy Programmer’s Secret Weapon by Mark Watson"
  "FranzInc, a company selling Common Lisp and Graph Database solutions.")

Allright, so we see we are manipulating what we want. Now to get their href, a quick look at lquery’s doc and:

(lquery:$  *parsed-content* "#content li a" (attr :href))
;; => #("license.html" "editor-support.html" "strings.html" "dates_and_times.html"
;;  "hashes.html" "pattern_matching.html" "functions.html" "loop.html" "io.html"
;;  "files.html" "packages.html" "macros.html"
;;  "/cl-cookbook/clos-tutorial/index.html" "sockets.html" "os.html" "ffi.html"
;;  "process.html" "systems.html" "win32.html" "testing.html" "misc.html"
;;  "license.html" "mailto:xach@xach.com" "mailto:skeptomai@mac.com"
;;  "mailto:brunel@mail.dotcom.fr" "mailto:jdcal@yahoo.com"
;;  "mailto:bill_clementson@yahoo.com" "mailto:gdoussot@yahoo.com"
;;  […]
;;  "mailto:matthieu@matthieu-villeneuve.net" "mailto:edi@agharta.de"
;;  "http://lisp-lang.org/" "https://github.com/CodyReichert/awesome-cl"
;;  "http://www.lispworks.com/documentation/HyperSpec/Front/index.htm"
;;  "http://phoe.tymoon.eu/clus/doku.php" "http://www.gigamonkeys.com/book/"
;;  […]
;;  "http://www.nicklevine.org/declarative/lectures/"
;;  "http://www.p-cos.net/lisp/guide.html" "https://leanpub.com/lovinglisp/"
;;  "https://franz.com/")

Nice, we now have the list (well, a vector) of links of the page. We’ll now write an async program to check and validate they are reachable.

External resources:

CSS selectors by example

Async requests

In this example we’ll take the list of url from above and we’ll check if they are reachable. We want to do this asynchronously, but to see the benefits we’ll first do it synchronously !

We need a bit of filtering first to exclude the email adresses (maybe that was doable in the CSS selector ?).

We put the vector of urls in a variable:

(defvar *urls* (lquery:$  *parsed-content* "#content li a" (attr :href)))

We remove the elements that start with “mailto:”: (a quick look at the strings page will help)

(remove-if (lambda (it) (string= it "mailto:" :start1 0 :end1 (length "mailto:"))) *urls*)
;; => #("license.html" "editor-support.html" "strings.html" "dates_and_times.html"
;;  […]
;;  "process.html" "systems.html" "win32.html" "testing.html" "misc.html"
;;  "license.html" "http://lisp-lang.org/"
;;  "https://github.com/CodyReichert/awesome-cl"
;;  "http://www.lispworks.com/documentation/HyperSpec/Front/index.htm"
;;  […]
;;  "https://franz.com/")

Actually before writting the remove-if (which works on any sequence, including vectors) I tested with a (map 'vector …) to see that the results where indeed nil or t.

As a side note, there is a handy starts-with-p function in cl-strings (disclainer: that’s our lib), available in Quicklisp. So we could do:

(map 'vector (lambda (it) (str:starts-with-p "mailto:" it)) *urls*)

it also has an option to ignore or respect the case.

While we’re at it, we’ll only consider links starting with “http”, in order not to write too much stuff irrelevant to web scraping:

(remove-if-not (lambda (it) (string= it "http" :start1 0 :end1 (length "http"))) *) ;; note the remove-if-NOT

Allright, we put this result in another variable:

(defvar *filtered-urls* *)

and now to the real work. For every url, we want to request it and check that its return code is 200. We have to ignore certain errors. Indeed, a request can timeout, be redirected (we don’t want that) or return an error code.

To be in real conditions we’ll add a link that times out in our list:

(setf (aref *filtered-urls* 0) "http://lisp.org")  ;; too bad indeed

We’ll take the simple approach to ignore errors and return nil in that case. If all goes well, we return the return code, that should be 200.

As we saw at the beginning, dex:get returns many values, including the return code. We’ll catch only this one with nth-value (instead of all of them with multiple-value-bind) and we’ll use ignore-errors, that returns nil in case of an error. We could also use handler-case and catch specific error types (see examples in dexador’s documentation) or (better yet ?) use handler-bind to catch any condition.

(ignore-errors has the caveat that when there’s an error, we can not return the element it comes from. We’ll get to our ends though.)

(map 'vector (lambda (it)
  (ignore-errors
    (nth-value 1 (dex:get it))))
  *filtered-urls*)

we get:

#(NIL 200 200 200 200 200 200 200 200 200 200 NIL 200 200 200 200 200 200 200
  200 200 200 200)

update: we could write something like the following with handler-case to be more flexible:

 (handler-case ( <code> )
   (error (c)
     ( <return sthg> )))

it works, but it took a very long time. How much time precisely ? with (time …):

Evaluation took:
  21.554 seconds of real time
  0.188000 seconds of total run time (0.172000 user, 0.016000 system)
  0.87% CPU
  55,912,081,589 processor cycles
  9,279,664 bytes consed

21 seconds ! Obviously this synchronous method isn’t efficient. We wait 10 seconds for links that time out. It’s time to write and measure and async version.

After installing lparallel and looking at its documentation, we see that the parallel map pmap seems to be what we want. And it’s only a one word edit. Let’s try:

(time (lparallel:pmap 'vector
  (lambda (it)
    (ignore-errors (let ((status (nth-value 1 (dex:get it)))) status)))
  *filtered-urls*)
;;  Evaluation took:
;;  11.584 seconds of real time
;;  0.156000 seconds of total run time (0.136000 user, 0.020000 system)
;;  1.35% CPU
;;  30,050,475,879 processor cycles
;;  7,241,616 bytes consed
;;
;;#(NIL 200 200 200 200 200 200 200 200 200 200 NIL 200 200 200 200 200 200 200
;;  200 200 200 200)

Bingo. It still takes more than 10 seconds because we wait 10 seconds for one request that times out. But otherwise it proceeds all the http requests in parallel and so it is much faster.

Shall we get the urls that aren’t reachable, remove them from our list and measure the execution time in the sync and async cases ?

What we do is: instead of returning only the return code, we check it is valid and we return the url:

... (if (and status (= 200 status)) it) ...
(defvar *valid-urls* *)

we get a vector of urls with a couple of nils: indeed, I thought I would have only one unreachable url but I discovered another one. Hopefully I have pushed a fix before you try this tutorial.

But what are they ? We saw the status codes but not the urls :S We have a vector with all the urls and another with the valid ones. We’ll simply treat them as sets and compute their difference. This will show us the bad ones. We must transform our vectors to lists for that.

(set-difference (coerce *filtered-urls* 'list)
                (coerce *valid-urls* 'list))
;; => ("http://lisp-lang.org/" "http://www.psg.com/~dlamkins/sl/cover.html")

Gotcha !

BTW it takes 8.280 seconds of real time to me to check the list of valid urls synchronously, and 2.857 seconds async.

Have fun doing web scraping in CL !

More helpful libraries:

we could use VCR, a store and replay utility to set up repeatable tests or to speed up a bit our experiments in the REPL.
cl-async, carrier and others network, parallelism and concurrency libraries to see on the awesome-cl list, Cliki or Quickdocs.
Mockingbird is nice to mock network requests in unit tests.

How to Get a File Size (and others Posix Attributes like its mtime) in Common Lisp

Fri, 16 Jun 2017 16:26:00 +0200

There is nothing built-in since CL predates the posix standard.

After a look at Awesome CL, the Osicat library was my go-to package to look for such functionnality. There is its osicat-posix package indeed, even though it is undocumented (issue)…

Now a look at the Cookbook is ok.

osicat, osicat-posix

osicat-posix is included in osicat.

(ql:quickload :osicat)

(describe (osicat-posix:stat #P"/tmp/file"))

#<OSICAT-POSIX:STAT {1004F20C93}>
  [standard-object]

Slots with :INSTANCE allocation:
  DEV      = 2065
  INO      = 7349974
  MODE     = 33204
  NLINK    = 1
  UID      = 1000
  GID      = 1000
  RDEV     = 0
  SIZE     = 4304
  BLKSIZE  = 4096
  BLOCKS   = 16
  ATIME    = 1497626097
  MTIME    = 1497347216
  CTIME    = 1497347216
; No value

and so we can access the slots with their related functions:

osicat-posix:stat-dev
osicat-posix:stat-gid
osicat-posix:stat-ino
osicat-posix:stat-uid
osicat-posix:stat-mode
osicat-posix:stat-rdev
osicat-posix:stat-size
osicat-posix:stat-atime
osicat-posix:stat-ctime
osicat-posix:stat-mtime
osicat-posix:stat-nlink
osicat-posix:stat-blocks
osicat-posix:stat-blksize

so for example:

(let ((stat (osicat-posix:stat #P"./files.md")))
  (osicat-posix:stat-size stat))  ;; => 10629

Trivial-file-size

Now for the size there’s also the lightweight (and portable) trivial-file-size.

This library exports a single function, file-size-in-octets. It returns the size of a file in bytes, using system calls when possible.

The canonical way to determine the size of a file in bytes, using Common Lisp, is to open the file with an element type of (unsigned-byte 8) and then calculate the length of the stream. This is less than ideal. In most cases it would be better to get the size of the file from its metadata, using a system call.

The author new about osicat-posix.

How to access url query parameters in Clack, Lucerne or Caveman

Thu, 04 May 2017 12:59:30 +0200

edit: I found Snooze (by Sly and Yasnippet’s author) easier and cleaner in this regard. It also has built-in settings to choose where to catch errors: with Slime’s debugger, with a full stacktrace in the browser or displaying a custom error page.

If you’re using Lucerne don’t search more like I did, its with-params macro works with url query parameters (as well as POST parameters).

If you’re accessing the url hello?search=kw, this works:

@route app "/hello"
(defview index (name)
  (with-params (search)
    (render-template (+index+)
                     :search search)))

An illustration with a POST parameter from the “utweet” example:

@route app (:post "/tweet")
(defview tweet ()
  (if (lucerne-auth:logged-in-p)
      (let ((user (current-user)))
        (with-params (tweet)
          (utweet.models:tweet user tweet))
        (redirect "/"))
      (render-template (+index+)
                       :error "You are not logged in.")))

The macro is implemented like this:

;; https://github.com/eudoxia0/lucerne/blob/master/src/http.lisp
(defmacro with-params (params &body body)
  "Extract the parameters in @cl:param(param) from the @c(*request*), and bind
them for use in @cl:param(body)."
  `(let ,(loop for param in params collecting
               `(,param (let ((str (parameter *request*
                                              ,(intern (string-downcase
                                                        (symbol-name param))
                                                       :keyword))))
                          (if (equal str "")
                              nil
                              str))))
     ,@body))

For Caveman it is possible but a bit awkward and inconsistent.

There’s an example for Ningle on the related StackOverflow question.

And in Clack generally ?

It is only scarcely documented on Clack’s api documentation.

We can access the parameters with (clack.request:query-parameter lucerne:*request*). So to get the value of a given param:

(assoc "a-param" (clack.request:query-parameter lucerne:*request*) :test 'string=)

and this returns the key and the value, so we need another cdr to get the value…

(defun query-param (param)
  (cdr (assoc param (clack.request:query-parameter lucerne:*request*) :test #'string=)))

See also:

https://jasom.github.io/clack-tutorial/pages/getting-started-with-clack/

Is There Something Like Clojure's Figwheel for interactive web dev with the browser in Common Lisp ?

Thu, 04 May 2017 12:32:01 +0200

Looks like there is: trident-mode, an “Emacs minor mode for live Parenscript interaction”, based on skewer but: trident-mode doesn’t seem to be used in the wild (while skewer-mode is) and I don’t know Figwheel so all I can say is that it seems a bit different: instead of letting us selectively evaluate and send code to the browser, Figwheels seems to rebuild the entire project and send the result when we write a file.

I tried trident-mode quickly, it works and the author was responsive. It offers commands and shortcuts to see the Javascript code produced by Parenscript forms and (optionally) send them to the browser.

An example use:

((@ document write)
  (ps-html ((:a :href "foobar") "blorg")))

to evaluate with trident-eval-dwim, which generates

document.write("<A HREF=\"foobar\">blorg</A>")

so it uses js to insert html into the DOM. It doesn’t leverage Skewer’s capacity to send only html.

I’ll update this post if/when I can.

Figwheel

builds your ClojureScript code and hot loads it into the browser as you are coding!

Skewer

Provides live interaction with JavaScript, CSS, and HTML in a web browser. Expressions are sent on-the-fly from an editing buffer to be evaluated in the browser, just like Emacs does with an inferior Lisp process in Lisp modes.

and we can also connect to sites on servers we don’t control.

They have demo videos.

Getting started: how to install a Common Lisp development environment

Wed, 03 May 2017 10:41:33 +0200

or “could not find recent and easy installation steps [fixed]“.

When I started I was a bit confused by old instructions (google is not good at CL), so hopefully this post will help show up recent and easy steps and most of all, help every CL enthousiast discover Portacle.

(and this post is editable through its Gitlab repository)

Portable, a multiplatform development environment

The productive Shinmera was waiting for the last details to be fixed before showing Portacle but it was already great. On GNU/Linux, MacOs or Windows, just download an archive and click an icon to open Emacs ready to use for CL development. It is that easy.

It ships: Emacs (customized), the SBCL implementation, Slime (Emacs IDE), Quicklisp (package manager) and Git. Emacs comes with a nice theme, autocompletion in drop-downs (company-mode) and Magit.

Manual install

Lisp implementation

Install a CL implementation:

apt-get install sbcl

Now you can run sbcl and write lisp at the prompt:

(print "hello lisp!")
(quit) ;; or C-d

More are packaged for Debian and probably for your distro, notably ECL, and note that you can install more easily with Roswell.

If you find the prompt horribly unfriendly (no history, no navigation…) use rlwrap:

apt-get install rlwrap

and now this will be slightly better:

rwrap sbcl

Even better, a slight wrapper around the SBCL REPL with readline support (Emacs and Vim modes, history, etc): sbcli, straightforward to use.

But still, we really need an editor.

Editors support

You’re not bound to Emacs, there’s good support for Vim, Sublime Text (via the SublimeREPL package) and Atom.

See the Cookbook#editors.

For Emacs, Slime is the de-facto solution (there’s also the Sly fork). It is in the GNU Elpa default Emacs package repository, so:

M-x package-install RET slime RET

(you may need a M-x package-refresh-content).

Now start Slime with M-x slime and wait a few seconds that it starts its backend (Swank server).

Might help:

using Emacs (and other instructions): https://www.darkchestnut.com/2017/getting-started-with-common-lisp/#using-emacs
http://wikemacs.org/wiki/Common_Lisp
Slime manual: https://common-lisp.net/project/slime/doc/html/ (see the Emacs menu). In very short: compile a file with C-c C-k, compile one function with C-c C-c and use it at the REPL.

Quicklisp package manager

To install Quicklisp:

from anywhere, download this file:

 wget https://beta.quicklisp.org/quicklisp.lisp

start a Lisp and load this file:

sbcl --load quicklisp.lisp

we get in the sbcl prompt. We have one Quicklisp command to type to install it:

(quicklisp-quickstart:install)

it will install itself in ~/quicklisp/quicklisp/.

it should output something like this, showing the basic commands:

==================================================
2,846 bytes in 0.001 seconds (2779.30KB/sec)
Upgrading ASDF package from version 2.004 to version 2.009
; Fetching #<URL "http://beta.quicklisp.org/dist/quicklisp.txt">
; 0.40KB
==================================================
408 bytes in 0.003 seconds (132.81KB/sec)

  ==== quicklisp installed ====

    To load a system, use: (ql:quickload "system-name")

    To find systems, use: (ql:system-apropos "term")

    To load Quicklisp every time you start Lisp, use: (ql:add-to-init-file)

    For more information, see http://www.quicklisp.org/beta/

NIL

Does it work ? Let’s try to install something:

(ql:quickload "dexador")

It is installed but we want to have Quicklisp available everytime we start sbcl. Otherwise we’d have to load the file located at ~/quicklisp/quicklisp/setup.lisp.

Each implementation uses a startup file, like our shells, so we can add this into our ~/.sbclrc:

;;; The following lines added by ql:add-to-init-file:
  #-quicklisp
  (let ((quicklisp-init (merge-pathnames "quicklisp/setup.lisp"
                                         (user-homedir-pathname))))
    (when (probe-file quicklisp-init)
      (load quicklisp-init)))

To quit sbcl, (quit) or C-d.

Quicklisp is a bit different than others package managers and it is not the only solution. That’s for another post.

Starting a project

I advise cl-project which, unlike others (quickproject) also sets up tests.

Now we can C-c C-k the .asd file and (ql:quickload "my-app") our app in the Slime REPL. But this is for another post.

Managing implementations and installing libraries in the command line: Roswell

This is done together with Roswell.

Roswell is in brew for MacOS, in linuxbrew, and it has a Debian package.

It allows to install pinned versions of SBCL or of other implementations (Embedable CL, Clozure CL,…) easily:

ros install sbcl/1.2.14
ros install sbcl  # the latest
ros install ccl-bin

what’s available ?

ros list versions

change the current lisp:

ros use sbcl/1.2.14

Install scripts:

ros install qlot

Install packages:

ros install dexador  # http client

and it does more to help scripting and distributing software. See its wiki !

String manipulation is frustrating [fixed]

Tue, 02 May 2017 11:07:01 +0200

One of the first things I wanted to do in the REPL was some string manipulation. But it was tedious.

To trim whitespace, and I mean all whitespaces, we had to define #\Space #\Newline #\Backspace #\Tab #\Linefeed #\Page #\Return #\Rubout.

To concatenate two strings: either giving an unusual 'string argument to concatenate, like this:

(concatenate 'string "fo" "o")

either we had to use a format construct, which is another source of frustration for (impatient) beginners, and sure isn’t straightforward and self-explanatory.

Many common stuff was split in various external libraries (cl-ppcre), and many common stuff was made more difficult than necessary (weird format construct again, entering a regexp, thus esaping what’s necessary, when all you want to do is simple search and replace, dealing with strings’ lengths and corner cases, lack of verbs,… see below).

And all of that with many inconsistencies (the string as first argument, then as the last, etc).

So I just joined everything in a little library, which has now more features. Let’s see its code and its tests to learn at the canonical way to do stuff, their shortcomings, and the library api at the same time.

I just don’t know how come this lib didn’t exist yet.

`str`

You can install it with

(ql:quickload "str")

See on https://github.com/vindarel/cl-str.

Package definition

(in-package #:asdf-user)

(defsystem :str
  :source-control (:git "git@github.com:vindarel/cl-s.git")
  :description "Modern, consistent and terse Common Lisp string manipulation library."
  :depends-on (:prove :cl-ppcre)  ;; <= depends only on cl-ppcre.
  :components ((:file "str"))
  )

Trim

(defvar *whitespaces* '(#\Space #\Newline #\Backspace #\Tab
                        #\Linefeed #\Page #\Return #\Rubout))

(defun trim-left (s)
  "Remove whitespaces at the beginning of s. "
  (string-left-trim *whitespaces* s))

(defun trim-right (s)
  "Remove whitespaces at the end of s."
  (string-right-trim *whitespaces* s))


(defun trim (s)
  (string-trim *whitespaces* s))

Concat

(defun concat (&rest strings)
  "Join all the string arguments into one string."
  (apply #'concatenate 'string strings))

Join

Snippets on the old cookbook or stackoverflow advised to use a format construct. Which is weird, and causes problems if your separator contains the ~ symbol.

(defun join (separator strings)
  (let ((separator (replace-all "~" "~~" separator)))
    (format nil
            (concatenate 'string "~{~a~^" separator "~}")
            strings)))

Now:

(is "foo~bar"
    (join "~" '("foo" "bar")))

Split

cl-ppcre takes a regexp, but we don’t need this for the basic cases of split. And disabling this regexp was not straightforward:

(defun split (separator s &key omit-nulls)
  "Split s into substring by separator (cl-ppcre takes a regex, we do not)."
  ;; cl-ppcre:split doesn't return a null string if the separator appears at the end of s.
  (let* ((val (concat s
                      (string separator)
                      ;; so we need an extra character, but not the user's.
                      (if (string-equal separator #\x) "y" "x")))
         (res (butlast (cl-ppcre:split (cl-ppcre:quote-meta-chars (string separator)) val))))
    (if omit-nulls
        (remove-if (lambda (it) (empty? it)) res)
        res)))

Now: (split "." "foo.bar") just works.

Repeat

(defun repeat (count s)
  "Make a string of S repeated COUNT times."
  (let ((result nil))
    (dotimes (i count)
      (setf result (cons s result)))
    (apply #'concat result)))

Replace-all

This required to use cl-ppcre and one switch of it to avoid regexps.

(defun replace-all (old new s)
  "Replace `old` by `new` in `s`. Arguments are not regexs."
  (let* ((cl-ppcre:*allow-quoting* t)
         (old (concatenate 'string  "\\Q" old))) ;; treat metacharacters as normal.
    (cl-ppcre:regex-replace-all old s new)))

starts-with? start string

The Lisp way was to check if the beginning of “string” contains “start”, taking its length, dealing with corner cases,…

(defun starts-with? (start s &key (ignore-case nil))
  "Return t if s starts with the substring 'start', nil otherwise."
  (when (>= (length s) (length start))
    (let ((fn (if ignore-case #'string-equal #'string=)))
      (funcall fn s start :start1 0 :end1 (length start)))))

;; An alias:
;; Serapeum defines a "defalias".
(setf (fdefinition 'starts-with-p) #'starts-with?)

(defun ends-with? (end s &key (ignore-case nil))
  "Return t if s ends with the substring 'end', nil otherwise."
  (when (>= (length s) (length end))
    (let ((fn (if ignore-case #'string-equal #'string=)))
      (funcall fn s end :start1 (- (length s) (length end))))))

(setf (fdefinition 'ends-with-p) #'ends-with?)

Usage illustrated by the tests:

(subtest "starts-with?"
  (ok (starts-with? "foo" "foobar") "default case")
  (ok (starts-with? "" "foo") "with blank start")
  (ok (not (starts-with? "rs" "")) "with blank s")
  (ok (not (starts-with? "foobar" "foo")) "with shorter s")
  (ok (starts-with? "" "") "with everything blank")
  (ok (not (starts-with? "FOO" "foobar")) "don't ignore case")
  (ok (starts-with-p "f" "foo") "starts-with-p alias")
  (ok (starts-with? "FOO" "foobar" :ignore-case t) "ignore case"))

Predicates: empty? blank?

There was no built-in to make those differences.

(defun empty? (s)
  "Is s nil or the empty string ?"
  (or (null s) (string-equal "" s)))

(defun emptyp (s)
  "Is s nil or the empty string ?"
  (empty? s))

(defun blank? (s)
  "Is s nil or only contains whitespaces ?"
  (or (null s) (string-equal "" (trim s))))

(defun blankp (s)
  "Is s nil or only contains whitespaces ?"
  (blank? s))

words, unwords, lines, unlines

Classic stuff:

(defun words (s &key (limit 0))
  "Return list of words, which were delimited by white space. If the optional limit is 0 (the default), trailing empty strings are removed from the result list (see cl-ppcre)."
  (if (not s)
      nil
      (cl-ppcre:split "\\s+" (trim-left s) :limit limit)))

(defun unwords (strings)
  "Join the list of strings with a whitespace."
  (join " " strings))

(defun lines (s &key omit-nulls)
  "Split the string by newline characters and return a list of lines."
  (split #\NewLine s :omit-nulls omit-nulls))

(defun unlines (strings)
  "Join the list of strings with a newline character."
  (join (make-string 1 :initial-element #\Newline) strings))

Substring

The builtin subseq is much poorer compared to what we have in other languages.

Take Python, we can do:

"foo"[:-1] # negative index and starting from the end
"foo"[0:100] # end is too large, thus it returns the entire array.

This was not possible with subseq, it throws a condition. Nothing found in Alexandria or other helper libraries.

(defun substring (start end s)
  "Return the substring of `s' from `start' to `end'.

It uses `subseq' with differences:
- argument order, s at the end
- `start' and `end' can be lower than 0 or bigger than the length of s.

- for convenience `end' can be nil or t to denote the end of the string.
"
  (let* ((s-length (length s))
         (end (cond
                ((null end) s-length)
                ((eq end t) s-length)
                (t end))))
    (setf start (max 0 start))
    (if (> start s-length)
        ""
        (progn
          (setf end (min end s-length))
          (when (< end (- s-length))
            (setf end 0))
          (when (< end 0)
            (setf end (+ s-length end)))
          (if (< end start)
              ""
              (subseq s start end))))))

Usage:

(subtest "substring"
  (is "abcd" (substring 0 4 "abcd") "normal case")
  (is "ab" (substring 0 2 "abcd") "normal case substing")
  (is "bc" (substring 1 3 "abcd") "normal case substing middle")
  (is "" (substring 4 4 "abcd") "normal case")
  (is "" (substring 0 0 "abcd") "normal case")
  (is "d" (substring 3 4 "abcd") "normal case")
  (is "abcd" (substring 0 t "abcd") "end is t")
  (is "abcd" (substring 0 nil "abcd") "end is nil")
  (is "abcd" (substring 0 100 "abcd") "end is too large")
  (is "abc" (substring 0 -1 "abcd") "end is negative")
  (is "b" (substring 1 -2 "abcd") "end is negative")
  (is "" (substring 2 1 "abcd") "start is bigger than end")
  (is "" (substring 0 -100 "abcd") "end is too low")
  (is "" (substring 100 1 "abcd") "start is too big")
  (is "abcd" (substring -100 4 "abcd") "start is too low")
  (is "abcd" (substring -100 100 "abcd") "start and end are too low and big")
  (is "" (substring 100 -100 "abcd") "start and end are too big and low")
  )

Why is there no generic operators ?

Fri, 14 Apr 2017 16:27:44 +0200

TLDR; because the object system came afterwards (and it was not the intention to make CL entirely object oriented).

As a CL enthousiast coming from Python, I feel the pain not to have generic or polymorphic operators but having to learn about many specialized operators instead. Why is it so and are there solutions ?

I asked on SO.

In CL, there are many operators to check for equality that depend on the data type: =, string-equal, char=, then equal, eql and whatnot, so on for other data types, and the same for comparison operators. There are no generic and extendible operators. For our own types, we define its own functions.

As a reminder for those equality operators: equal does work on integers, strings and characters and equalp also works for lists, vectors and hash tables an other Common Lisp types but objects. See the SO answers for precisions and experienced lispers debatting of the inner subtilities and traps of those functions.

The language has mechanisms to create generics though, see generics (defgeneric, defmethod) as described in Practical Common Lisp.

There have been work in that direction (https://common-lisp.net/project/cdr/document/8/cleqcmp.html) but no library as of today.

It’s a recurrent concern, also this blog post (“Not a monad tutorial”, great serie) points to this. The guy moved to Clojure, for other reasons too of course, where there are only one (or two?) equality operators.

So it seems that the reason is mostly historical, the object system (CLOS) appearing afterwards. Of course the generics would be slower. But how much slower ? I really don’t care, as a beginner and for web stuff.

Generic CLOS functions were added several years after CL was originally designed (82-84). The variant with CLOS was widely published with CLtL2 (1990) and then ANSI CL. The language was only slightly updated. It was not the intention to make Common Lisp fully object oriented. Also performance of CLOS for relatively low-level functions is kind of problematic. Dylan, which is something like Scheme + CLOS - s-expression syntax, did this: it defines more of the language in terms of generic functions. [Rainer Joswig on SO]

The CL21 way

Fortunately CL21 introduces (more) generic operators, particularly for sequences it defines length, append, setf, getf, first, rest, subseq, replace, take, drop, fill, take-while, drop-while, last, butlast, find-if, search, remove-if, delete-if, reverse, reduce, sort, split, join, remove-duplicates, every, some, map, sum (and some more). Those should work at least for strings, lists, vectors and extend the new abstract-sequence type.

Now CL21 is something worth presenting and debatting in another post.

Example of interactively changing the current executed Lisp code in the debugger

Fri, 14 Apr 2017 13:06:27 +0200

The awesome example we will read comes from a comment by user lispm inside a discussion on this reddit thread: https://www.reddit.com/r/programming/comments/65ct5j/a_pythonist_finds_a_new_home_at_clojure_land/.

The article it discusses is a “Not a monad tutorial” post, where the interviewee is experienced in C++, Java, Javascript and Python and turns into Clojure. He wrote about his first impressions with Common Lisp here, where he raises usual concerns that I agree with but IMO that stay supercifial (“not readable” because of stuff like (format t "~{~{~a:~10t~a~%~}~%~}" *db*), “huge operators set”, “macros look promising”…).

Here starts the discussion.

dzecniv On Common Lisp, I agree with the criticisms except

the code was very difficult to read

I find it very easy, always well expressed, with concise functions. And I find Clojure’s harder, with more [, { and the same number of other symbols (#, *).

Anyway, I’m in the process of trying to go from python to CL. The CL ecosystem is quite good nowadays (equivalents of pip, venvs, pyenv, implementations (even for the JVM or iOS), CI, sphinx, readthedocs, wsgi, setup.py,,…), it’s moving, we can do quite a lot (awesome list), it has unique features but yeah, the ecosystem is tiny compared to clojure’s…

ps: interested ? http://lisp-lang.org/ !

lispm

tiny compared to clojure

In many ways is it much broader than Clojure, since there is much more choice. Interpreters, compilers, native code compilers, batch compilers, compilers targeting C/LLVM/JVM/ARM/ARM64/x86/x86-64/SPARC64/POWER/…

Clojure on the JVM uses a relatively simple and not very user-friendly compiler to the JVM. No Interpreter. No mixed use of interpreted and compiled code. Functions need to be declared before used. Error messages are exposing the underlying JVM. No TCO. No images. Slow startup.

The Roomba cleans your home with a CL program.

MagicMurderBagYT

No Interpreter.

Hol up. What about the REPL?

lispm

That’s not an interpreter. A REPL is not the same as a Lisp interpreter. REPL means read eval print loop. EVAL can be implemented by a compiler or an interpreter. Common Lisp has both and mixed implementations with both compiler and interpreter.

A Lisp interpreter is executing Lisp code directly. Clojure does not have an Interpreter.

https://clojure.org/reference/evaluation

Clojure has no interpreter.

Example in LispWorks, which uses the Interpreter in the REPL:

CL-USER 29 > (let ((f (lambda (a b)
                         (+ (prog1 2 (break))  ; we have a break here
                            (* a b)))))
              (funcall f 2 3))

Break.
  1 (continue) Return from break.
  2 (abort) Return to level 0.
  3 Return to top loop level 0.

Type :b for backtrace or :c <option number> to proceed.
Type :bug-form "<subject>" for a bug report template or :? for other options.

As you see Lisp comes with a sub-repl in the break. The sub-repl is just another repl, but in the context of the break. The break could be done by the debugger when it sees an error or by user code - as above.

Now we ask the interpreter for the current lambda expression:

CL-USER 30 : 1 > :lambda
(LAMBDA (A B) (+ (PROG1 2 (BREAK)) (* A B)))

Above is actually Lisp data. Code as data.

Now I’m changing the + function in the code to be expt, exponentiation. To be clear: I’m changing in the debugger the current executed Lisp function on the Lisp level. We take the third element of the list, and then the first one of that. This is the + symbol. We change it to be expt. * holds the last evaluation result of the REPL.

CL-USER 31 : 1 > (setf (first (third *)) 'expt)
EXPT

Then I’m restarting the current stack frame:

CL-USER 32 : 1 > :res

We get another break, which we just continue from:

Break.
  1 (continue) Return from break.
  2 (abort) Return to level 0.
  3 Return to top loop level 0.

Type :b for backtrace or :c <option number> to proceed.
Type :bug-form "<subject>" for a bug report template or :? for other options.

CL-USER 33 : 1 > :c 1
64                                   ; we computed 2^(2*3)  instead of 2+(2*3)

What did we see? We saw that the interpreter uses actual Lisp code. Lisp code we can change with Lisp code in the debugger.

A second example.

What can we do with that for debugging? Well, we can for example write our own evaluation tracer. The Evaluator prints each expression and its result nicely indented, while walking the expression tree and evaluating subexpressions. Remember: this is now user-level code. The example is from CLtL2. You will also see that LispWorks can freely mix compiled and interpreted functions. The function COMPILE takes a function name and compiles its Lisp code to machine code.

CL-USER 1 > (defvar *hooklevel* 0)
*HOOKLEVEL*

CL-USER 2 > (defun hook (x)
              (let ((*evalhook* 'eval-hook-function))
                (eval x)))
HOOK

CL-USER 3 > (compile 'hook)
HOOK
NIL
NIL

CL-USER 4 > (defun eval-hook-function (form &rest env)
              (let ((*hooklevel* (+ *hooklevel* 1)))
                (format *trace-output* "~%~V@TForm:  ~S"
                        (* *hooklevel* 2) form)
                (let ((values (multiple-value-list
                               (evalhook form
                                         #'eval-hook-function
                                         nil
                                         env))))
                  (format *trace-output* "~%~V@TValue:~{ ~S~}"
                          (* *hooklevel* 2) values)
                  (values-list values))))
EVAL-HOOK-FUNCTION

CL-USER 5 > (compile 'eval-hook-function)
EVAL-HOOK-FUNCTION
NIL
NIL

Now we can trace the evaluation of expressions on the Lisp level:

CL-USER 6 > (hook '(cons (floor *print-base* 2) 'b))

  Form:  (CONS (FLOOR *PRINT-BASE* 2) (QUOTE B))
    Form:  (FLOOR *PRINT-BASE* 2)
      Form:  *PRINT-BASE*
      Value: 10
      Form:  2
      Value: 2
    Value: 5 0
    Form:  (QUOTE B)
    Value: B
  Value: (5 . B)
(5 . B)

dzecniv

that’s an awesome example and tutorial that I’d love to see on a blog post or just a gist or something for further reference and better archiving, this will be buried too quickly on reddit !

So here it is.

Epilogue: the Roomba robot vacuums.

About me

Sun, 05 Feb 2017 07:51:49 +0100

Hi, it’s Vincent. I write about my Common Lisp journey here. I started the blog when I was discovering the language and the ecosystem, wishing more people wrote about CL. Because Common Lisp is was the most hidden world I know.

I now wrote tools, libraries and software, I run a web app in production©, and I am creating videos on Youtube and a Common Lisp course on Udemy

I write intensively about Common Lisp on collaborative resources. My hidden plan is to make Common Lisp popular again. For this, I contribute to the Common Lisp Cookbook (I am the main contributor, by far). I wrote about: CLOS, data structures, building executables, scripting, web scraping, debugging, error handling, testing, databases, GUI programming, web development, the LispWorks IDE etc, and I added a theme, a sidebar and syntax highlighting. I also take time to maintain the awesome-cl list, an important resource in my eyes. I dig up, reference and sort Common Lisp libraries (and I still discover hidden gems ~~three~~ five years after). I do community stuff for Lisp Advocates (not the creator).

But I still want Common Lisp to be easier to learn. So I am creating this Lisp course in videos:

🎥 Common Lisp programming: from novice to effective programmer (on GitHub)

I truly think it is the most effective way to learn CL! I sum up in 5-15 minutes videos what took me a long time to learn or simply discover (a longer time to admit).

So now a newcomer has far more practical information for getting started than a few years ago. But there’s still a lot to do. Besides purchasing or sharing my course, you can thank and encourage me by donations on the following platforms. As I currently don’t have a fixed (nor big) income, that helps. Thanks!

GitHub sponsors: https://github.com/sponsors/vindarel/
ko-fi: https://ko-fi.com/vindarel
liberapay: https://liberapay.com/vindarel/

I also write and maintain tools, libraries, software and project skeletons. Among others:

libraries:

cl-str, that fixed my first frustration with CL
CIEL - a fast startup scripting environment for Common Lisp, with batteries included. Usable, still in development.

and also:

replic, to help create a readline application in no time
- see how I use it in cl-torrents or in lyrics-cli, there is one commit that adds replic support.
fuzzy-match
cl-sendgrid, to send emails easily with the Sendgrid API.
progressons - a progress bar.
cl-readline (maintainer)
cl-ansi-term (maintainer)

software:

ABStock, a catalogue of books (and other products)
OpenBookStore, a personal book manager, aiming to replace Abelujo
cl-torrents, with an experimental Weblocks front-end

tools:

colisper, an interface to Comby, for syntactic code checking and refactoring of Lisp code.
indent-tools (emacs package)
print-licences
a simple tool to fetch the status of your Gitlab pipelines: pipelines-viewer (just a demo on how to access third-party APIs and how to build a binary, really)
format-colors, a set of simple format directives to print colored text

project skeletons and demos:

cl-cookieproject
- check out this video: 🎥 How to create, run, build and load a new Common Lisp project
- the companion skeleton for web apps: cl-cookieweb
lisp-web-template-productlist: Hunchentoot + easy-routes + Djula templates + Bulma CSS + a Makefile to build the project
demo-web-live-reload: an example of how image-based development is useful. Learn to interact with a running website, including with a remote one with Swank.
demo-ISSR-djula - a demo on how to build a dynamic web app without JavaScript, but with ISSR, Interactive Server-Side Rendering (and Djula templates).
Weblocks-todomvc

and others:

the lisp-maintainers list.
the funny Hacker Typer in Lisp.
the list of languages implemented in Lisp.

I contribute to awesome projects such as:

Weblocks, an isomorphic web framework. I helped write the quickstart, fixed HTML generation in tables, wrote more documentation, raised issues.
the Nyxt browser: I was part of the team in 2019, I am the second contributor of that year.

and I fix bugs when I see them (Mito (my contributions!), Djula…).

You can reach me by email at vindarel, mailz dot org. I am /u/(reverse (str:concat “vince” “zd”)) on reddit.

This website’s sources are on Gitlab: https://gitlab.com/lisp-journey/lisp-journey.gitlab.io/issues

Lisp journey

🖌️ Lisp screenshots: today's Common Lisp applications in action

lisp-screenshots.org

Practice for Advent Of Code in Common Lisp

Prerequisites

Exercise 1 - lists of lists

Exercise 2 - prepare to parse a grid as a hash-table

Harder puzzle - hash-tables, grid, coordinates

Closing words

🎥 ⭐ Learn Common Lisp data structures: 9 videos, 90 minutes of video tutorials to write efficient Lisp

What is this course anyways?

Course outcome

Chapter content

Closing words

Lisp tutorial: variables. defparameter vs defvar, let/let* and lexical scope, unbound variables, style guidelines

defparameter: top-level variables

redefining a defparameter

defvar: no redefinition

The “*earmuff*” convention

Global variables are created in the “dynamic scope”

setf: change values

let, let*: create lexical scopes

let vs let*

setf inside let

When you don’t use defined variables

Unbound variables

Global variables are thread safe

Addendum: defconstant

Guidelines and best practices

Lisp error handling (advanced): how handler-bind doesn't unwind the stack

Absolute control over conditions and restarts: handler-bind

handler-bind doesn’t unwind the stack

When to use which?

Invoking the debugger manually

Closing words

Hacker News now runs on top of Common Lisp

These years in Common Lisp: 2023-2024 in review

Community

Documentation

Implementations

SBCL

ABCL

CCL

Allegro

LispWorks

ECL

CLASP

SICL - the new, portable and modular implementation

New implementations

Historical: Medley Interlisp

Companies and jobs

Projects

Editors

About Emacs

About VSCode

About Lem and Rooms pair programming environment

About LispWorks

About the Jetbrains plugin

About Jupyter

Other tools

Coalton

Package managers

Gamedev

GUI

Web

Nyxt 4.0 pre-realease - now on Electron

More libraries

Scripting

Software releases

Other articles

Videos

New resource specialized on web development in Common Lisp

Towards a Django-like database admin dashboard for Common Lisp

Models

Select tables to show

Start the admin

Configure the admin

Form handling

Form validation

One-to-many relations

`defparameter`: top-level variables

`defvar`: no redefinition

The “earmuff” convention

`setf`: change values

`let`, `let*`: create lexical scopes

`let` vs `let*`

Addendum: `defconstant`

Absolute control over conditions and restarts: `handler-bind`

1. `defclass`, `make-instance`, `slots`… aka CLOS crash course, part 1. This one is free to watch 🆓

4. `defclass` review

6. `initialize-instance`: control if and how any objects are created

8. `defgeneric` vs `defmethod`: when to use which, which is better?

`=` is for numbers (beware of `NIL`)

`eq` is low-level. Think pointers, position in memory.

`eql` is a better `eq` also for numbers of same types and characters.

`equal` is also for strings (for objects whose printed representation is similar).

`equalp` is case-insensitive for strings and for numerical value of numbers.

`null`

`eql` is used by default by many CL built-ins

`char-equal`

strings and `string-equal`

Compare trees with `tree-equal`