From 7a1ccaa997e76d0404a962c0f8c89aab4ce61882 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kasper=20Ga=C5=82kowski?= Date: Wed, 1 Mar 2023 19:40:28 +0100 Subject: [PATCH] lisp-modules: Add manual section --- doc/languages-frameworks/index.xml | 1 + doc/languages-frameworks/lisp.section.md | 328 +++++++++++++++++++++++ 2 files changed, 329 insertions(+) create mode 100644 doc/languages-frameworks/lisp.section.md diff --git a/doc/languages-frameworks/index.xml b/doc/languages-frameworks/index.xml index f089b99a045..e1e35564e32 100644 --- a/doc/languages-frameworks/index.xml +++ b/doc/languages-frameworks/index.xml @@ -25,6 +25,7 @@ + diff --git a/doc/languages-frameworks/lisp.section.md b/doc/languages-frameworks/lisp.section.md new file mode 100644 index 00000000000..ef50da05c4d --- /dev/null +++ b/doc/languages-frameworks/lisp.section.md @@ -0,0 +1,328 @@ +# lisp-modules {#lisp} + +This document describes the Nixpkgs infrastructure for building Common Lisp +libraries that use ASDF (Another System Definition Facility). It lives in +`pkgs/development/lisp-modules`. + +## Overview + +The main entry point of the API are the Common Lisp implementation packages +(e.g. `abcl`, `ccl`, `clasp-common-lisp`, `clisp` `ecl`, `sbcl`) +themselves. They have the `pkgs` and `withPackages` attributes, which can be +used to discover available packages and to build wrappers, respectively. + +The `pkgs` attribute set contains packages that were automatically imported from +Quicklisp, and any other manually defined ones. Not every package works for all +the CL implementations (e.g. `nyxt` only makes sense for `sbcl`). + +The `withPackages` function is of primary utility. It is used to build runnable +wrappers, with a pinned and pre-built ASDF FASL available in the `ASDF` +environment variable, and `CL_SOURCE_REGISTRY`/`ASDF_OUTPUT_TRANSLATIONS` +configured to find the desired systems on runtime. + +With a few exceptions, the primary thing that the infrastructure does is to run +`asdf:load-system` for each system specified in the `systems` argument to +`build-asdf-system`, and save the FASLs to the Nix store. Then, it makes these +FASLs available to wrappers. Any other use-cases, such as producing SBCL +executables with `sb-ext:save-lisp-and-die`, are achieved via overriding the +`buildPhase` etc. + +In addition, Lisps have the `packageOverrides` argument, which can be used +(through `override`) to substitute any package in the scope of their +`pkgs`. This will be useful together with `overrideLispAttrs` when dealing with +slashy ASDF systems, because they should stay in the main package and be build +by specifying the `systems` argument to `build-asdf-system`. + +## The 90% use case example + +The most common way to use the library is to run ad-hoc wrappers like this: + +`nix-shell -p 'sbcl.withPackages (ps: with ps; [ alexandria ])'` + +Then, in a shell: + +``` +$ result/bin/sbcl +* (load (sb-ext:posix-getenv "ASDF")) +* (asdf:load-system 'alexandria) +``` + +Also one can create a `pkgs.mkShell` environment in `shell.nix`/`flake.nix`: + +``` +let + sbcl' = sbcl.withPackages (ps: [ ps.alexandria ]); +in mkShell { + buildInputs = [ sbcl' ]; +} +``` + +Such a Lisp can be now used e.g. to compile your sources: + +``` +buildPhase = '' + ${sbcl'}/bin/sbcl --load my-build-file.lisp +'' +``` + +## Importing packages from Quicklisp + +The library is able to very quickly import all the packages distributed by +Quicklisp by parsing its `releases.txt` and `systems.txt` files. These files are +available from [http://beta.quicklisp.org/dist/quicklisp.txt]. + +The import process is implemented in the `import` directory as Common Lisp +functions in the `org.lispbuilds.nix` ASDF system. To run the script, one can +execute `ql-import.lisp`: + +``` +nix-shell --run 'sbcl --script ql-import.lisp' +``` + +The script will: + +1. Download the Quicklisp `systems.txt` and `releases.txt` files +2. Generate an SQLite database of all QL systems in `packages.sqlite` +3. Generate an `imported.nix` file from the database + +The maintainer's job there is to: + +1. Update the `import/main.lisp` file for new QL releases +2. Re-run the `ql-import.lisp` script +3. Add missing native dependencies in `ql.nix` +4. For packages that still don't build, package them manually in `packages.nix` + +Also, the `imported.nix` file **must not be edited manually**! It should only be +generated as described in this section. + +### Adding native dependencies + +The Quicklisp files contain ASDF dependency data, but don't include native +library (CFFI) dependencies, and, in the case of ABCL, Java dependencies. + +The `ql.nix` file contains a long list of overrides, where these dependencies +can be added. + +Packages defined in `packages.nix` contain these dependencies naturally. + +### Trusting `systems.txt` and `releases.txt` + +The previous implementation of `lisp-modules` didn't fully trust the Quicklisp +data, because there were times where the dependencies specified were not +complete, and caused broken builds. It instead used a `nix-shell` environment to +discover real dependencies by using the ASDF APIs. + +The current implementation has chosen to trust this data, because it's faster to +parse a text file than to build each system to generate its Nix file, and +because that way packages can be mass-imported. Because of that, there may come +a day where some packages will break, due to bugs in Quicklisp. In that case, +the fix could be a manual override in `packages.nix` and `ql.nix`. + +A known fact is that Quicklisp doesn't include dependencies on slashy systems in +its data. This is an example of a situation where such fixes were used, e.g. to +replace the `systems` attribute of the affected packages. (See the definition of +`iolib`). + +### Quirks + +During Quicklisp import: + +- `+` in names are converted to `_plus{_,}`: `cl+ssl`->`cl_plus_ssl`, `alexandria+`->`alexandria_plus` +- `.` to `_dot_`: `iolib.base`->`iolib_dot_base` +- names starting with a number have a `_` prepended (`3d-vectors`->`_3d-vectors`) + + +## Defining packages manually inside Nixpkgs + +New packages, that for some reason are not in Quicklisp, and so cannot be +auto-imported, can be written in the `packages.nix` file. + +In that file, use the `build-asdf-system` function, which is a wrapper around +`mkDerivation` for building ASDF systems. Various other hacks are present, such +as `build-with-compile-into-pwd` for systems which create files during +compilation. + +The `build-asdf-system` function is documented with comments in +`nix-cl.nix`. Also, `packages.nix` is full of examples of how to use it. + +## Defining packages manually outside Nixpkgs + +Lisp derivations (`abcl`, `sbcl` etc.) also export the `buildASDFSystem` +function, which is the same as `build-asdf-system`, except for the `lisp` +argument which is set to the given CL implementation. + +It can be used to define packages outside Nixpkgs, and, for example, add them +into the package scope with `override` and `packageOverrides`, both of which +will be discussed later on. + +### Including an external package in scope + +A package defined outside Nixpkgs using `buildASDFSystem` can be woven into the +Nixpkgs-provided scope like this: + +``` +let + alexandria = sbcl.buildASDFSystem rec { + pname = "alexandria"; + version = "1.4"; + src = fetchFromGitLab { + domain = "gitlab.common-lisp.net"; + owner = "alexandria"; + repo = "alexandria"; + rev = "v${version}"; + hash = "sha256-1Hzxt65dZvgOFIljjjlSGgKYkj+YBLwJCACi5DZsKmQ="; + }; + }; + sbcl' = sbcl.override { + packageOverrides = self: super: { + inherit alexandria; + }; + }; +in sbcl'.pkgs.alexandria +``` + +## Overriding package attributes + +Packages export the `overrideLispAttrs` function, which can be used to build a +new package with different parameters. + +Example of overriding `alexandria`: + +``` +sbcl.pkgs.alexandria.overrideLispAttrs (oldAttrs: rec { + version = "1.4"; + src = fetchFromGitLab { + domain = "gitlab.common-lisp.net"; + owner = "alexandria"; + repo = "alexandria"; + rev = "v${version}"; + hash = "sha256-1Hzxt65dZvgOFIljjjlSGgKYkj+YBLwJCACi5DZsKmQ="; + }; +}) +``` + +## Overriding packages in scope + +Packages can be woven into a new scope by using `override` with +`packageOverrides`: + +``` +let + sbcl' = sbcl.override { + packageOverrides = self: super: { + alexandria = super.alexandria.overrideLispAttrs (oldAttrs: rec { + pname = "alexandria"; + version = "1.4"; + src = fetchFromGitLab { + domain = "gitlab.common-lisp.net"; + owner = "alexandria"; + repo = "alexandria"; + rev = "v${version}"; + hash = "sha256-1Hzxt65dZvgOFIljjjlSGgKYkj+YBLwJCACi5DZsKmQ="; + }; + }); + }; + }; +in builtins.elemAt sbcl'.pkgs.bordeaux-threads.lispLibs 0 +``` + +### Dealing with slashy systems + +Slashy (secondary) systems should not exist in their own packages! Instead, they +should be included in the parent package as an extra entry in the `systems` +argument to the `build-asdf-system`/`buildASDFSystem` functions. + +The reason is that ASDF searches for a secondary system in the `.asd` of the +parent package. Thus, having them separate would cause either one of them not to +load cleanly, because one will contains FASLs of itself but not the other, and +vice versa. + +To package slashy systems, use `overrideLispAttrs`, like so: + +``` +ecl.pkgs.alexandria.overrideLispAttrs (oldAttrs: { + systems = oldAttrs.systems ++ [ "alexandria/tests" ]; + lispLibs = oldAttrs.lispLibs ++ [ ecl.pkgs.rt ]; +}) +``` + +See the respective section for how to weave it back into `ecl.pkgs`. + +Note that sometimes the slashy systems might not only have more dependencies +than the main one, but create a circular dependency between `.asd` +files. Unfortunately, in this case an adhoc solution becomes necessary. + +## Building Wrappers + +Wrappers can be built using the `withPackages` function of Common Lisp +implementations (`abcl`, `ecl`, `sbcl` etc.): + +``` +sbcl.withPackages (ps: [ ps.alexandria ps.bordeaux-threads ]) +``` + +Such a wrapper can then be executed like this: + +``` +result/bin/sbcl +``` + +### Loading ASDF + +For best results, avoid calling `(require 'asdf)` When using the +library-generated wrappers. + +Use `(load (ext:getenv "ASDF"))` instead, supplying your implementation's way of +getting an environment variable for `ext:getenv`. This will load the +(pre-compiled to FASL) Nixpkgs-provided version of ASDF. + +### Loading systems + +There, you can simply use `asdf:load-system`. This works by setting the right +values for the `CL_SOURCE_REGISTRY`/`ASDF_OUTPUT_TRANSLATIONS` environment +variables, so that systems are found in the Nix store and pre-compiled FASLs are +loaded. + +## Adding a new Lisp + +Three additional functions are exposed, and are meant for wrapping Common Lisp +derivations: `commonLispPackagesFor`, `lispWithPackages` and`build-asdf-system`. + +`commonLispPackagesFor` returns a package set for the provided Lisp "spec". Such +a spec is an attribute set of the following keys: + +- `pkg`: the Lisp package derivation +- `program`: The name of executable file in `${pkg}/bin/` +- `flags`: A list of flags to always pass to `program` +- `faslExt`: Implementation-specific extension for FASL files +- `asdf`: The ASDF version to use + +The `spec` is an argument to every Lisp, and can be customized via `override`: + +``` +sbcl.override { + spec = { + pkg = pkgs.sbcl_2_1_1; + flags = [ "--dynamic-space-size" "4096" ]; + faslExt = "fasl"; + asdf = pkgs.asdf_3_1; + }; +} +``` + +`lispWithPackages` returns a function to create wrappers. + +`build-asdf-system` is the wrapper around `stdenv.mkDerivation`. + +To wrap a new Lisp, include the following in its `passthru`: + +``` +passthru = let + spec' = spec // { pkg = sbcl; }; + pkgs = (commonLispPackagesFor spec').overrideScope' packageOverrides; +in { + inherit pkgs; + withPackages = lispWithPackages pkgs; + buildASDFSystem = args: build-asdf-system (args // spec'); +} +```