3.1 Loading a pre-installed ASDF

The recommended way to load ASDF is via:

(require "asdf")

All actively maintained Lisp implementations now include a copy of ASDF 3 that you can load this way using Common Lisp’s require function.¹

If the implementation you are using doesn’t provide a recent ASDF 3, we recommend you upgrade it. If for some reason you would rather not upgrade it, we recommend you replace your implementation’s ASDF. See Replacing your implementation’s ASDF. If all else fails, see see Loading ASDF from source below.

If you use an actively maintained implementation that fails to provide an up-to-date enough stable release of ASDF, you may also send a bug report to your Lisp vendor and complain about it — or you may fix the issue yourself if it’s free software.

As of the writing of this manual, the following implementations provide ASDF 3 this way: ABCL, Allegro CL, CLASP, Clozure CL, CMUCL, ECL, GNU CLISP, LispWorks, MKCL, SBCL. The following implementations only provide ASDF 2: MOCL, XCL. The following implementations don’t provide ASDF: Corman CL, GCL, Genera, MCL, SCL. The latter implementations are not actively maintained (except maybe GCL); if some of them are ever released again, they probably will include ASDF 3.

For maximum convenience you might want to have ASDF loaded whenever you start your Lisp implementation, for example by loading it from the startup script or dumping a custom core — check your Lisp implementation’s manual for details. SLIME notably sports a slime-asdf contrib that makes life easier with ASDF.

3.2 Checking whether ASDF is loaded

To check that ASDF is properly loaded, you can run this form:

(asdf:asdf-version)

If it returns a string, that is the version of ASDF that is currently installed. If that version is suitably recent (say, 3.1.2 or later), then you can skip directly to next chapter: See Configuring ASDF.

If it raises an error, then either ASDF is not loaded, or you are using a very old version of ASDF, and need to install ASDF 3.

For more precision in detecting versions old and new, see How do I detect the ASDF version?.

If you are experiencing problems with ASDF, please try upgrading to the latest released version, using the method below, before you contact us and raise an issue.

3.3 Upgrading ASDF

If your implementation already provides ASDF 3 or later (and it should), but you want a more recent ASDF version than your implementation provides, then you just need to ensure the more recent ASDF is installed in a configured path, like any other system. We recommend you download an official tarball or checkout a release from git into ~/common-lisp/asdf/. (see Configuring ASDF to find your systems).

Once the source code for ASDF is installed, you don’t need any extra step to load it beyond the usual (require "asdf"): ASDF 3 will automatically look whether an updated version of itself is available amongst the regularly configured systems, before it compiles anything else.

If your implementation fails to provide ASDF 3 or later, see Replacing your implementation’s ASDF.

3.4 Replacing your implementation’s ASDF

All maintained implementations now provide ASDF 3 in their latest release. If yours doesn’t, we recommend you upgrade it.

Now, if you insist on using an old implementation that didn’t provide ASDF or provided an old version, we recommend installing a recent ASDF, as explained below, into your implementation’s installation directory. Thus your modified implementation will now provide ASDF 3. This requires proper write permissions and may necessitate execution as a system administrator.

The ASDF source repository contains a tool to help you upgrade your implementation’s ASDF. You can invoke it from the shell command-line as tools/asdf-tools install-asdf lispworks (where you can replace lispworks by the name of the relevant implementation), or you can (load "tools/install-asdf.lisp") from your Lisp REPL.

This script works on Allegro CL, Clozure CL, CMU CL, ECL, GCL, GNU CLISP, LispWorks, MKCL, SBCL, SCL, XCL. It doesn’t work on ABCL, Corman CL, Genera, MCL, MOCL. Happily, ABCL is usually pretty up to date and shouldn’t need that script. GCL requires a very recent version, and hasn’t been tested much. Corman CL, Genera, MCL are obsolete anyway. MOCL is incomplete.

3.5 Loading ASDF from source

If you write build scripts that must remain portable to old machines with old implementations that you cannot ensure have been upgraded or modified to provide a recent ASDF, you may have to install the file asdf.lisp somewhere and load it with:

(load "/path/to/your/installed/asdf.lisp")

The single file asdf.lisp is all you normally need to use ASDF.

You can extract this file from latest release tarball on the ASDF website. If you are daring and willing to report bugs, you can get the latest and greatest version of ASDF from its git repository. See Getting the latest version.

For scripts that try to use ASDF simply via require at first, and make heroic attempts to load it the hard way if at first they don’t succeed, see tools/load-asdf.lisp distributed with the ASDF source repository, or the code of cl-launch.

4.1 Configuring ASDF to find your systems

In order to compile and load your systems, ASDF must be configured to find the .asd files that contain system definitions.

There are a number of different techniques for setting yourself up with ASDF, starting from easiest to the most complex:

• Put all of your systems in one of the standard locations, subdirectories of
  • ~/common-lisp/ or
  • ~/.local/share/common-lisp/source/.

  If you install software there, you don’t need further configuration.² You can then skip to the next section. See Loading a system.

• If you’re using some tool to install software (e.g. Quicklisp), the authors of that tool should already have configured ASDF.
• If you have more specific desires about how to lay out your software on disk, the preferred way to configure where ASDF finds your systems is the source-registry facility, fully described in its own chapter of this manual. See Controlling where ASDF searches for systems. Here is a quick recipe for getting started.

  First create the directory ~/.config/common-lisp/source-registry.conf.d/³; there create a file with any name of your choice but with the type conf⁴, for instance 50-luser-lisp.conf; in this file, add the following line to tell ASDF to recursively scan all the subdirectories under /home/luser/lisp/ for .asd files: (:tree "/home/luser/lisp/")

  That’s enough. You may replace /home/luser/lisp/ by wherever you want to install your source code. You don’t actually need to specify anything if you use the default ~/common-lisp/ as above and your implementation provides ASDF 3.1.2 or later. If your implementation provides an earlier variant of ASDF 3, you might want to specify (:tree (:home "common-lisp/")) for bootstrap purposes, then install a recent source tree of ASDF under ~/common-lisp/asdf/.

  If you prefer to use a “link farm”, which is faster to use but costlier to manage than a recursive traversal, say at /home/luser/.asd-link-farm/, then you may instead (or additionally) create a file 42-asd-link-farm.conf, containing the line: (:directory "/home/luser/.asd-link-farm/")

  ASDF will automatically read your configuration the first time you try to find a system. If necessary, you can reset the source-registry configuration with:

  (asdf:clear-source-registry)
  

• In earlier versions of ASDF, the system source registry was configured using a global variable, asdf:*central-registry*. For more details about this, see the following section, Configuring ASDF to find your systems — old style. Unless you need to understand this, skip directly to Configuring where ASDF stores object files.

Note that your Operating System distribution or your system administrator may already have configured system-managed libraries for you.

4.2 Configuring ASDF to find your systems — old style

Novices may skip this section. Please do not use the central-registry if you are a novice, and do not instruct novices to use the central-registry.

The old way to configure ASDF to find your systems is by pushing directory pathnames onto the variable asdf:*central-registry*.

You must configure this variable after you load ASDF 3 or later, yet before the first time you try to use it. This loading and configuring of ASDF must happen as part of some initialization script: typically, either a script you maintain that builds your project, or your implementation’s initialization script (e.g. ~/.sbclrc for SBCL).

Also, if you are using an ancient ASDF 2 or earlier to load ASDF 3 or later, then after it loads the ancient ASDF, your script must configure the central-registry a first time to tell ASDF 1 or 2 where to find ASDF 3, then load ASDF 3 with e.g. (asdf:operate 'asdf:load-op "asdf"), then configure the central-registry again, because ASDF 3 will not preserve the central-registry from ASDF 2 when upgrading. You should probably be using the source-registry instead, which will be preserved (unless you manually called asdf:initialize-source-registry with an argument, in which case you will have to do it again indeed). However, if you are using an ancient ASDF 2 or earlier, we strongly recommend that you should instead upgrade your implementation, or overwrite the ancient ASDF installation with a more recent one: See Replacing your implementation’s ASDF.

The asdf:*central-registry* is empty by default in ASDF 2 or ASDF 3, but is still supported for compatibility with ASDF 1. When used, it takes precedence over the above source-registry.⁵

For example, let’s say you want ASDF to find the .asd file /home/me/src/foo/foo.asd. In your Lisp initialization file, you could have the following:

(require "asdf")
(push "/home/me/src/foo/" asdf:*central-registry*)

Note the trailing slash: when searching for a system, ASDF will evaluate each entry of the central registry and coerce the result to a pathname.⁶ The trailing directory name separator is necessary to tell Lisp that you’re discussing a directory rather than a file. If you leave it out, ASDF is likely to look in /home/me/src/ instead of /home/me/src/foo/ as you intended, and fail to find your system definition. Modern versions of ASDF will issue an error and offer you to remove such entries from the central-registry.

Typically there are a lot of .asd files, and a common idiom was to put symbolic links to all of one’s .asd files in a common directory and push that directory (the “link farm”) onto asdf:*central-registry*, instead of pushing each individual system directory.

ASDF knows to follow symlinks to the actual location of the systems.⁷

For example, if #p"/home/me/cl/systems/" is an element of *central-registry*, you could set up the system foo as follows:

$ cd /home/me/cl/systems/
$ ln -s ~/src/foo/foo.asd .

This old style for configuring ASDF is not recommended for new users, but it is supported for old users, and for users who want a simple way to programmatically control what directories are added to the ASDF search path.

4.3 Configuring where ASDF stores object files

ASDF lets you configure where object files will be stored. Sensible defaults are provided and you shouldn’t normally have to worry about it.

This allows the same source code repository to be shared between several versions of several Common Lisp implementations, between several users using different compilation options, with users who lack write privileges on shared source directories, etc. This also keeps source directories from being cluttered with object/fasl files.

Starting with ASDF 2, the asdf-output-translations facility was added to ASDF itself. This facility controls where object files will be stored. This facility is fully described in a chapter of this manual, Controlling where ASDF saves compiled files.

Note that before ASDF 2, other ASDF add-ons offered the same functionality, each in subtly different and incompatible ways: ASDF-Binary-Locations, cl-launch, common-lisp-controller. ASDF-Binary-Locations is now not needed anymore and should not be used. cl-launch 3.000 and common-lisp-controller 7.2 have been updated to delegate object file placement to ASDF.

4.4 Resetting the ASDF configuration

When you dump and restore an image, or when you tweak your configuration, you may want to reset the ASDF configuration. For that you may use the following function:

Function: clear-configuration ¶

Undoes any ASDF configuration regarding source-registry or output-translations.

This function is pushed onto the uiop:*image-dump-hook* by default, which means that if you save an image using uiop:dump-image, or via asdf:image-op and asdf:program-op, it will be automatically called to clear your configuration. If for some reason you prefer to call your implementation’s underlying functionality, be sure to call clear-configuration manually, or push it into your implementation’s equivalent of uiop:*image-dump-hook*, e.g. sb-ext:*save-hooks* on SBCL, or ext:*before-save-initializations* on CMUCL and SCL, etc.

5.1 Loading a system

The system foo is loaded (and compiled, if necessary) by evaluating the following Lisp form:

(asdf:load-system :foo)

On some implementations (see Convenience Functions), ASDF hooks into the cl:require facility and you can just use:

(require :foo)

Note that the canonical name of a system is a string, conventionally lowercase. A system name can also be specified as a symbol (including a keyword), in which case its symbol-name is taken and lowercased. The name must be a suitable value for the :name initarg to make-pathname in whatever filesystem the system is to be found.

The lower-casing-symbols behaviour is unconventional, but was selected after some consideration. The type of systems we want to support either have lowercase as customary case (Unix, Mac, Windows) or silently convert lowercase to uppercase (lpns).

5.2 Convenience Functions

ASDF provides three commands for the most common system operations: load-system, compile-system, and test-system.

ASDF also provides require-system, a variant of load-system that skips loading systems that are already loaded. This is sometimes useful, for example, in order to avoid re-loading libraries that come pre-loaded into your lisp implementation.

ASDF also provides make, a way of allowing system developers to choose a default operation for their systems. For example, a developer who has created a system intended to format a specific document, might make document-formatting the default operation invoked by make, instead of loading. If the system developer doesn’t specify in the system definition, the default operation will be loading.

Because ASDF is an extensible system for defining operations on components, it also provides a generic function operate, so you may arbitrarily operate on your systems beyond the default operations. (At the interactive REPL, users often use its shorter alias oos, which stands for operate-on-system, a name inherited from mk-defsystem.) You’ll use operate whenever you want to do something beyond compiling, loading and testing.

Function: load-system system &rest keys &key force force-not verbose version &allow-other-keys ¶

Apply operate with the operation from *load-system-operation* the system, and any provided keyword arguments. *load-system-operation* by default is load-op; it would be load-bundle-op by default on ECL, if only an implementation bug were fixed. Calling load-system is the regular, recommended way to load a system into the current image.

Function: compile-system system &rest keys &key force force-not verbose version &allow-other-keys ¶

Apply operate with the operation compile-op, the system, and any provided keyword arguments. This will make sure all the files in the system are compiled, but not necessarily load any of them in the current image; on most systems, it will not load all compiled files in the current image. This function exists for symmetry with load-system but is not recommended unless you are writing build scripts and know what you’re doing. But then, you might be interested in program-op rather than compile-op.

Function: test-system system &rest keys &key force force-not verbose version &allow-other-keys ¶

Apply operate with the operation test-op, the system, and any provided keyword arguments. See test-op.

Function: make system &rest keys &key &allow-other-keys ¶

Do “The Right Thing” with your system. Starting with ASDF 3.1, this function make is also available. The default behaviour is to load the system as if by load-system; but system authors can override this default in their system definition they may specify an alternate operation as the intended use of their system, with a :build-operation option in the defsystem form (see build-operation), and an intended output pathname for that operation with :build-pathname. This function is experimental and largely untested. Use at your own risk.

Function: require-system system &rest keys &key &allow-other-keys ¶

require-system skips any update to systems that have already been loaded, in the spirit of cl:require. It does it by calling load-system with a keyword option excluding already loaded systems.⁸. On actively maintained free software implementations (namely recent versions of ABCL, Clozure CL, CMUCL, ECL, GNU CLISP, MKCL and SBCL), once ASDF itself is loaded, cl:require too can load ASDF systems, by falling back on require-system for module names not recognized by the implementation. (Note however that require-system does not fall back on cl:require; that would introduce an “interesting” potential infinite loop to break somehow.)

cl:require and require-system are appropriate to load code that is not being modified during the current programming session. cl:require will notably load the implementation-provided extension modules; require-system won’t, unless they are also defined as systems somehow, which SBCL and MKCL do. require-system may also be used to load any number of ASDF systems that the user isn’t either developing or debugging, for which a previously installed version is deemed to be satisfactory; cl:require on the above-mentioned implementations will delegate to require-system and may load them as well. But for code that you are actively developing, debugging, or otherwise modifying, you should use load-system, so ASDF will pick on your modifications and transitively re-build the modified files and everything that depends on them (that the requested system itself depends on — ASDF itself never builds anything unless it’s an explicitly requested system or the dependencies thereof).

5.3 Moving on

That’s all you need to know to use ASDF to load systems written by others. The rest of this manual deals with writing system definitions for Common Lisp software you write yourself, including how to extend ASDF to define new operation and component types.

6.1 The defsystem form

This section begins with an example of a system definition, then gives the full grammar of defsystem.

Let’s look at a simple system. This is a complete file that should be saved as hello-lisp.asd (in order that ASDF can find it when ordered to operate on the system named "hello-lisp").

;; Usual Lisp comments are allowed here

(defsystem "hello-lisp"
  :description "hello-lisp: a sample Lisp system."
  :version "0.0.1"
  :author "Joe User <joe@example.com>"
  :licence "Public Domain"
  :depends-on ("optima.ppcre" "command-line-arguments")
  :components ((:file "packages")
               (:file "macros" :depends-on ("packages"))
               (:file "hello" :depends-on ("macros"))))

Some notes about this example:

• The defsystem form defines a system named hello-lisp that contains three source files: packages.lisp, macros.lisp and hello.lisp.
• The .lisp suffix is implicit for Lisp source files. The source files are located in the same directory as the .asd file with the system definition.
• The file macros depends on packages (presumably because the package it’s in is defined in packages), and the file hello depends on macros (and hence, transitively on packages). This means that ASDF will compile and load packages then macros before starting the compilation of file hello.
• This example system has external dependencies on two other systems, optima.ppcre (that provides a friendly interface to matching regular expressions), and command-line-arguments (that provides a way to parse arguments passed from the shell command line). To use this system, ASDF must be configured to find installed copies of these systems; it will load them before it tries to compile and load hello-lisp.
• This system also defines a bunch of metadata. While it is optional to define these fields (and other fields like :bug-tracker, :mailto, :long-name, :long-description, :source-control), it is strongly recommended to define the fields :description, :version, :author, and :licence, especially if you intend your software to be eventually included in Quicklisp.
• Make sure you know how the :version numbers will be parsed! Only period-separated non-negative integers are accepted at present. See below Version specifiers in The defsystem grammar.
• This file contains a single form, the defsystem declaration. No in-package form, no asdf: package prefix, no nothing. Just the one naked defsystem form. This is what we recommend. More complex system definition files are possible with arbitrary Lisp code, but we recommend that you keep it simple if you can. This will make your system definitions more robust and more future-proof.

This is all you need to know to define simple systems. The next example is much more involved, to give you a glimpse of how you can do more complex things. However, since it’s ultimately arbitrary Lisp code, there is no bottom to the rabbit hole.

6.2 A more involved example

Let’s illustrate some more involved uses of defsystem via a slightly convoluted example:

(in-package :asdf-user)

(defsystem "foo"
  :version (:read-file-form "variables" :at (3 2))
  :components
  ((:file "package")
   (:file "variables" :depends-on ("package"))
   (:module "mod"
     :depends-on ("package")
     :serial t
     :components ((:file "utils")
                  (:file "reader")
                  (:file "cooker")
                  (:static-file "data.raw"))
     :output-files (compile-op (o c) (list "data.cooked"))
     :perform (compile-op :after (o c)
        (cook-data
         :in (component-pathname (find-component c "data.raw"))
         :out (first (output-files o c)))))
   (:file "foo" :depends-on ("mod"))))

(defmethod action-description
    ((o compile-op) (c (eql (find-component "foo" "mod"))))
  "cooking data")

Here are some notes about this example:

• The main thing this file does is define a system foo. It also contains other Lisp forms, which we’ll examine below.
• Besides Lisp source files, this system contains a :module component named "mod", which is a collection of three Lisp source files utils.lisp, reader.lisp, cooker.lisp and data.raw
• Note that the :static-file does not have an implicit file type, unlike the Lisp source files.
• This files will be located in a subdirectory of the main code directory named mod/ (this location could have been overridden to be in the same directory, or in a different subdirectory; see the discussion of the :pathname option in The defsystem grammar).
• The :serial t says that each sub-component of mod depends on the previous components, so that cooker.lisp depends-on utils.lisp, which depends-on reader.lisp. Also data.raw depends on all of them, but that doesn’t matter since it’s a static file; on the other hand, if it appeared first, then all the Lisp files would be recompiled when the data is modified, which is probably not what is desired in this case.
• The method-form tokens provide a shorthand for defining methods on particular components. This part

       :output-files (compile-op (o c) (list "data.cooked"))
       :perform (compile-op :after (o c)
          (cook-data
           :in (component-pathname (find-component c "data.raw"))
           :out (first (output-files o c))))
  

  has the effect of

  (defmethod output-files ((o compile-op) (c (eql ...)))
    (list "data.cooked"))
  (defmethod perform :after ((o compile-op) (c (eql ...)))
    (cook-data
     :in (component-pathname (find-component c "data.raw"))
     :out (first (output-files o c))))
  

  where ... is the component in question. In this case ... would expand to something like

  (find-component "foo" "mod")
  

  For more details on the syntax of such forms, see The defsystem grammar. For more details on what these methods do, see Operations in The Object model of ASDF.

• There is an additional defmethod with a similar effect, because ASDF (as of ASDF 3.1.5) fails to accept inline-methods as above for action-description, instead only supporting the deprecated explain interface.
• In this case, these methods describe how this module defines code that it then uses to cook some data.
• Importantly, ASDF is told about the input and output files used by the data cooker, and to make sure everyone agrees, the cooking function explicitly uses ASDF to access pathnames to the input and output data.
• The file starts with a form (in-package :asdf-user), but it is actually redundant, not necessary and not recommended. But yet more complex cases (also not recommended) may usefully use an in-package form.
• Indeed, ASDF does not load .asd files simply with cl:load, and neither should you. You should let ASDF find and load them when you operate on systems. If you somehow must load a .asd file, use the same function asdf:load-asd that ASDF uses. Among other things, it already binds the *package* to asdf-user. Recent versions of SLIME (2013-02 and later) know to do that when you C-c C-k when you use the slime-asdf contrib.
• You shouldn’t use an in-package form if you’re keeping things simple. You should only use in-package (and before it, a defpackage) when you’re going to define new classes, functions, variables, macros, etc., in the .asd file, and want to thereby avoid name clashes. Manuals for old versions of ASDF recommended use of such an idiom in .asd files, but as of ASDF 3, we recommend that you don’t do that anymore, and instead define any ASDF extensions in their own system, on which you can then declare a dependency using :defsystem-depends-on. See The defsystem grammar.
• More generally, you can always rely on symbols from packages asdf, common-lisp and uiop being available in .asd files — most importantly including defsystem. It is therefore redundant and in bad taste to use a package-prefixed asdf:defsystem symbol in a .asd file. Just use (defsystem ...). Only package-prefix it when somehow dynamically generating system definitions from a package that doesn’t already use the ASDF package.
• asdf-user is actually only available starting since ASDF 3, but then again, ASDF 1 and 2 did crazy things with packages that ASDF 3 has stopped doing⁹, and since all implementations provide ASDF 3, you shouldn’t care about compatibility with ASDF 2. We do not support ASDF 2 anymore, and we recommend that neither should you.
• Starting with ASDF 3.1, asdf-user uses uiop, whereas in earlier variants of ASDF 3 it only used uiop/package. We recommend you either prefix use of UIOP functions with the package prefix uiop:, or make sure your system :depends-on ((:version "asdf" "3.1.2")) or has a #-asdf3.1 (error "MY-SYSTEM requires ASDF 3.1.2").
• Finally, we elided most metadata, but showed how you can have ASDF automatically extract the system’s version from a source file. In this case, the 3rd subform of the 4th form (note that Lisp uses 0-based indexing, English uses 1-based indexing). Presumably, the 4th form looks like (defparameter *foo-version* "5.6.7").

6.3 The defsystem grammar

system-definition := ( defsystem system-designator system-option* )

system-option := :defsystem-depends-on system-list
                 | :weakly-depends-on system-list
                 | :class class-name (see discussion below)
                 | :build-operation operation-name
                 | system-option
                 | module-option
                 | option

# These are only available since ASDF 3 (actually its alpha release 2.27)
system-option := :homepage string
                 | :bug-tracker string
                 | :mailto string
                 | :long-name string
                 | :source-control source-control
                 | :version version-specifier

source-control := (keyword string)

module-option := :components component-list
                 | :serial [ t | nil ]

option :=
        | :pathname pathname-specifier
        | :default-component-class class-name
        | :perform method-form
        | :explain method-form
        | :output-files method-form
        | :operation-done-p method-form
        | :if-feature feature-expression
        | :depends-on ( dependency-def* )
        | :in-order-to ( dependency+ )


system-list := ( simple-component-name* )

component-list := ( component-def* )

component-def  := ( component-type simple-component-name option* )

component-type := :module | :file | :static-file | other-component-type

other-component-type := symbol-by-name
                        (see Component types)

# This is used in :depends-on, as opposed to ``dependency,''
# which is used in :in-order-to
dependency-def := simple-component-name
               | ( :feature feature-expression dependency-def )
                 # (see Feature dependencies)
               | ( :version simple-component-name version-specifier )
               | ( :require module-name )

# ``dependency'' is used in :in-order-to, as opposed to
# ``dependency-def''
dependency := (dependent-op requirement+)
requirement := (required-op required-component+)
dependent-op := operation-name
required-op := operation-name

simple-component-name := string
                      |  symbol

pathname-specifier := pathname | string | symbol

version-specifier := string
                     | (:read-file-form <pathname-specifier> <form-specifier>?)
                     | (:read-file-line <pathname-specifier> <line-specifier>?)
line-specifier := :at integer # base zero
form-specifier := :at [ integer | ( integer+ )]

method-form := (operation-name qual lambda-list &rest body)
qual := method qualifier?

feature-expression := keyword
                      | (:and feature-expression*)
                      | (:or feature-expression*)
                      | (:not feature-expression)

• Component names
• Component types
• System class names
• Defsystem depends on
• Build-operation
• Weakly depends on
• Pathname specifiers
• Version specifiers
• Require
• Feature dependencies
• Using logical pathnames
• Serial dependencies
• Source location (:pathname)
• if-feature option
• feature requirement

:class :MY-NEW-SYSTEM-SUBCLASS

:serial t
:components ((:file "a") (:file "b") (:file "c"))

:components ((:file "a")
             (:file "b" :depends-on ("a"))
             (:file "c" :depends-on ("a" "b")))

6.4 Other code in .asd files

Files containing defsystem forms are regular Lisp files that are executed by load. Consequently, you can put whatever Lisp code you like into these files. However, it is recommended to keep such forms to a minimal, and to instead define defsystem extensions that you use with :defsystem-depends-on.

If however, you might insist on including code in the .asd file itself, e.g., to examine and adjust the compile-time environment, possibly adding appropriate features to *features*. If so, here are some conventions we recommend you follow, so that users can control certain details of execution of the Lisp in .asd files:

• Any informative output (other than warnings and errors, which are the condition system’s to dispose of) should be sent to the standard CL stream *standard-output*, so that users can easily control the disposition of output from ASDF operations.

6.5 The package-inferred-system extension

Starting with release 3.1.2, ASDF supports a one-package-per-file style of programming, whereby each file is its own system, and dependencies are deduced from the defpackage form (or its variant uiop:define-package).

In this style, packages refer to a system with the same name (downcased); and if a system is defined with :class package-inferred-system, then system names that start with that name (using the slash / separator) refer to files under the filesystem hierarchy where the system is defined. For instance, if system my-lib is defined in /foo/bar/my-lib/my-lib.asd, then system my-lib/src/utility will be found in file /foo/bar/my-lib/src/utility.lisp.

This style was made popular by faslpath and quick-build before, and at the cost of a stricter package discipline, seems to make for more maintainable code. It is used by ASDF itself (starting with ASDF 3), by lisp-interface-library, and a few other libraries.

To use this style, choose a toplevel system name, e.g. my-lib, and create a file my-lib.asd with the :class :package-inferred-system option in its defsystem. For instance:

#-asdf3.1 (error "my-lib requires ASDF 3.1")
(defsystem "my-lib"
  :class :package-inferred-system
  :depends-on ("my-lib/interface/all"
               "my-lib/src/all"
               "my-lib/extras/all")
  :in-order-to ((test-op (load-op "my-lib/test/all")))
  :perform (test-op (o c) (symbol-call :my-lib/test/all :test-suite)))

(defsystem "my-lib/test" :depends-on ("my-lib/test/all"))

(register-system-packages "my-lib/interface/all" '(:my-lib-interface))
(register-system-packages "my-lib/src/all" '(:my-lib-implementation))
(register-system-packages "my-lib/test/all" '(:my-lib-test))

(register-system-packages
 "closer-mop"
 '(:c2mop :closer-common-lisp :c2cl :closer-common-lisp-user :c2cl-user))

In the code above, the first line checks that we are using ASDF 3.1, which provides package-inferred-system.

The function register-system-packages has to be called to register packages used or provided by your system and its components where the name of the system that provides the package is not the downcase of the package name.

Then, file interface/order.lisp under the lil hierarchy, that defines abstract interfaces for order comparisons, starts with the following form, dependencies being trivially computed from the :use and :mix clauses:

(uiop:define-package :lil/interface/order
  (:use :closer-common-lisp
   :lil/interface/definition
   :lil/interface/base
   :lil/interface/eq :lil/interface/group)
  (:mix :fare-utils :uiop :alexandria)
  (:export ...))

ASDF can tell that this file depends on system closer-mop (registered above), lil/interface/definition, lil/interface/base, lil/interface/eq, and lil/interface/group (package and system names match, and they will be looked up hierarchically).

ASDF also detects dependencies from :import-from clauses. You may thus import a well-defined set of symbols from an existing package as loaded from suitably named system; or if you prefer to use any such symbol fully qualified by a package prefix, you may declare a dependency on such a package and its corresponding system via an :import-from clause with an empty list of symbols, as in:

(defpackage :foo/bar
  (:use :cl)
  (:import-from :foo/baz #:sym1 #:sym2)
  (:import-from :foo/quux)
  (:export ...))

The form uiop:define-package is supported as well as defpackage, and has many options that prove useful in this context, such as :use-reexport and :mix-reexport that allow for “inheritance” of symbols being exported.

Note that starting with ASDF 3.1.5.6 only, ASDF will look for source files under the component-pathname as specified via the :pathname option, whereas earlier versions ignore this option and use the system-source-directory where the .asd file resides.

7.1 Operations

An operation object of the appropriate type is instantiated whenever the user wants to do something with a system like

• compile all its files
• load the files into a running lisp environment
• copy its source files somewhere else

Operations can be invoked directly, or examined to see what their effects would be without performing them. There are a bunch of methods specialised on operation and component type that actually do the grunt work. Operations are invoked on systems via operate (see operate).

ASDF contains a number of pre-defined operation classes for common, and even fairly uncommon tasks that you might want to do with it. In addition, ASDF contains “abstract” operation classes that programmers can use as building blocks to define ASDF extensions. We discuss these in turn below.

Operations are invoked on systems via operate.

Generic function: operate operation component &rest initargs &key force force-not verbose &allow-other-keys ¶

Generic function: oos operation component &rest initargs &key &allow-other-keys ¶

operate invokes operation on system. oos is a synonym for operate (it stands for operate-on-system).

operation is an operation designator: it can be an operation object itself, or, typically, a symbol that is passed to make-operation (which will call make-instance), to create the operation object. component is a component designator: it can be a component object itself, or, typically, a string or symbol (to be string-downcased) that names a system, more rarely a list of strings or symbols that designate a subcomponent of a system.

The ability to pass initargs to make-operation is now deprecated, and will be removed. For more details, see make-operation. Note that dependencies may cause the operation to invoke other operations on the system or its components: the new operations may or may not be created with the same initargs as the original one (for the moment).

If force is :all, then all systems are forced to be recompiled even if not modified since last compilation. If force is t, then only the system being loaded is forced to be recompiled even if not modified since last compilation, but other systems are not affected. If force is a list, then it specifies a list of systems that are forced to be recompiled even if not modified since last compilation. If force-not is :all, then all systems are forced not to be recompiled even if modified since last compilation. If force-not is t, then all systems but the system being loaded are forced not to be recompiled even if modified since last compilation (note: this was changed in ASDF 3.1.2). If force-not is a list, then it specifies a list of systems that are forced not to be recompiled even if modified since last compilation.

Both force and force-not apply to systems that are dependencies and were already compiled. force-not takes precedences over force, as it should, really, but unhappily only since ASDF 3.1.2. Moreover, systems which have been registered as immutable by register-immutable-system (since ASDF 3.1.5) are always considered forced-not, and even their .asd are not refreshed from the filesystem. See Miscellaneous Functions.

To see what operate would do, you can use:

(asdf:traverse operation-class system-name)

Function: make-operation operation-class &rest initargs ¶

The initargs are passed to make-instance call when creating the operation object.

Note:initargs for operations are now deprecated, and will be removed from ASDF in the near future.

Note: operation instances must never be created using make-instance directly: only through make-operation. Attempts to directly make operation instances will cause a run-time error.

• Predefined operations of ASDF
• Creating new operations

(asdf:operate 'asdf:operation-name :system-name {operation-options ...})

7.2 Components

A component represents an individual source file or a group of source files, and the things that get transformed into. A system is a component at the top level of the component hierarchy, that can be found via find-system. A source-file is a component representing a single source-file and the successive output files into which it is transformed. A module is an intermediate component itself grouping several other components, themselves source-files or further modules.

A system designator is a system itself, or a string or symbol that behaves just like any other component name (including with regard to the case conversion rules for component names).

A component designator, relative to a base component, is either a component itself, or a string or symbol, or a list of designators.

Function: find-system system-designator &optional (error-p t) ¶

Given a system designator, find-system finds and returns a system. If no system is found, an error of type missing-component is thrown, or nil is returned if error-p is false.

To find and update systems, find-system funcalls each element in the *system-definition-search-functions* list, expecting a pathname to be returned, or a system object, from which a pathname may be extracted, and that will be registered. The resulting pathname (if any) is loaded if one of the following conditions is true:

• there is no system of that name in memory
• the pathname is different from that which was previously loaded
• the file’s last-modified time exceeds the last-modified time of the system in memory

When system definitions are loaded from .asd files, they are implicitly loaded into the ASDF-USER package, which uses ASDF, UIOP and UIOP/COMMON-LISP¹² Programmers who do anything non-trivial in a .asd file, such as defining new variables, functions or classes, should include defpackage and in-package forms in this file, so they will not overwrite each others’ extensions. Such forms might also help the files behave identically if loaded manually with cl:load for development or debugging, though we recommend you use the function asdf::load-asd instead, which the slime-asdf contrib knows about.

The default value of *system-definition-search-functions* is a list of three functions. The first function looks in each of the directories given by evaluating members of *central-registry* for a file whose name is the name of the system and whose type is asd; the first such file is returned, whether or not it turns out to actually define the appropriate system. The second function does something similar, for the directories specified in the source-registry, but searches the filesystem only once and caches its results. The third function makes the package-inferred-system extension work, see The package-inferred-system extension.

Because of the way these search functions are defined, you should put the definition for a system foo in a file named foo.asd, in a directory that is in the central registry or which can be found using the source registry configuration.

It is often useful to define multiple systems in a same file, but ASDF can only locate a system’s definition file based on the system name. For this reason, ASDF 3’s system search algorithm has been extended to allow a file foo.asd to contain secondary systems named foo/bar, foo/baz, foo/quux, etc., in addition to the primary system named foo. The first component of a system name, separated by the slash character, /, is called the primary name of a system. The primary name may be extracted by function asdf::primary-system-name; when ASDF 3 is told to find a system whose name has a slash, it will first attempt to load the corresponding primary system, and will thus see any such definitions, and/or any definition of a package-inferred-system.¹³ If your file foo.asd also defines systems that do not follow this convention, e.g., a system named foo-test, ASDF will not be able to automatically locate a definition for these systems, and will only see their definition if you explicitly find or load the primary system using e.g. (asdf:find-system "foo") before you try to use them. We strongly recommend against this practice, though it is currently supported for backward compatibility.

Function: primary-system-name name ¶

Internal (not exported) function, asdf::primary-system-name. Returns the primary system name (the portion before the slash, /, in a secondary system name) from name.

Function: locate-system name ¶

This function should typically not be invoked directly. It is exported as part of the API only for programmers who wish to provide their own *system-definition-search-functions*.

Given a system name designator, try to locate where to load the system definition from. Returns five values: foundp, found-system, pathname, previous, and previous-time. foundp is true when a system was found, either a new as yet unregistered one, or a previously registered one. The found-system return value will be a system object, if a system definition is found in your source registry. The system definition will not be loaded if it hasn’t been loaded already. pathname when not null is a path from which to load the system, either associated with found-system, or with the previous system. If previous is not null, it will be a previously loaded system object of the same name (note that the system definition is previously-loaded: the system itself may or may not be). previous-time when not null is the timestamp of the previous system definition file, at the time when the previous system definition was loaded.

For example, if your current registry has foo.asd in /current/path/to/foo.asd, but system foo was previously loaded from /previous/path/to/foo.asd then locate-system will return the following values:

1. foundp will be t,
2. found-system will be nil,
3. pathname will be #p"/current/path/to/foo.asd",
4. previous will be an object of type SYSTEM with system-source-file slot value of #p"/previous/path/to/foo.asd"
5. previous-time will be the timestamp of #p"/previous/path/to/foo.asd" at the time it was loaded.

Function: find-component base path ¶

Given a base component (or designator for such), and a path, find the component designated by the path starting from the base.

If path is a component object, it designates itself, independently from the base.

If path is a string, or symbol denoting a string via coerce-name, then base is resolved to a component object, which must be a system or module, and the designated component is the child named by the path.

If path is a cons cell, find-component with the base and the car of the path, and the resulting object is used as the base for a tail call to find-component with the car of the path.

If base is a component object, it designates itself.

If base is null, then path is used as the base, with nil as the path.

If base is a string, or symbol denoting a string via coerce-name, it designates a system as per find-system.

If base is a cons cell, it designates the component found by find-component with its car as base and cdr as path.

• Common attributes of components
• Pre-defined subclasses of component
• Creating new component types

     #+sbcl       (:file "cffi-sbcl")

:in-order-to ((compile-op (load-op "a" "b") (compile-op "c"))
              (load-op (load-op "foo")))

(this-op @{(other-op required-components)@}+)

simple-component-name := string
                      |  symbol

required-components := simple-component-name
                     | (required-components required-components)

component-name := simple-component-name
                | (:version simple-component-name minimum-version-object)

  for all source file x, (load x) depends on (compile x)

  (compile b) depends on (load a)

((load-op "a") (load-op "b") (compile-op "c") ...)

(defclass unportable-cl-source-file (cl-source-file)
  ())

(defmethod component-pathname ((component unportable-cl-source-file))
  (merge-pathnames*
   (parse-unix-namestring (format nil "~(~A~)/" (asdf:implementation-type)))
   (call-next-method)))

(defsystem :foo
    :components
    ((:file "packages")
     ...
     (:unportable-cl-source-file "threads"
      :depends-on ("packages" ...))
     ...
    )

7.3 Dependencies

To be successfully build-able, this graph of actions must be acyclic. If, as a user, extender or implementer of ASDF, you introduce a cycle into the dependency graph, ASDF will fail loudly. To clearly distinguish the direction of dependencies, ASDF 3 uses the words requiring and required as applied to an action depending on the other: the requiring action depends-on the completion of all required actions before it may itself be performed.

Using the defsystem syntax, users may easily express direct dependencies along the graph of the object hierarchy: between a component and its parent, its children, and its siblings. By defining custom CLOS methods, you can express more elaborate dependencies as you wish. Most common operations, such as load-op, compile-op or load-source-op are automatically propagate “downward” the component hierarchy and are “covariant” with it: to act the operation on the parent module, you must first act it on all the children components, with the action on the parent being parent of the action on each child. Other operations, such as prepare-op and prepare-source-op (introduced in ASDF 3) are automatically propagated “upward” the component hierarchy and are “contravariant” with it: to perform the operation of preparing for compilation of a child component, you must perform the operation of preparing for compilation of its parent component, and so on, ensuring that all the parent’s dependencies are (compiled and) loaded before the child component may be compiled and loaded. Yet other operations, such as test-op or load-bundle-op remain at the system level, and are not propagated along the hierarchy, but instead do something global on the system.

7.4 Functions

Function: version-satisfies version version-spec ¶

Does version satisfy the version-spec. A generic function. ASDF provides built-in methods for version being a component or string. version-spec should be a string. If it’s a component, its version is extracted as a string before further processing.

A version string satisfies the version-spec if after parsing, the former is no older than the latter. Therefore "1.9.1", "1.9.2" and "1.10" all satisfy "1.9.1", but "1.8.4" or "1.9" do not. For more information about how version-satisfies parses and interprets version strings and specifications, see The defsystem grammar (version specifiers) and Common attributes of components.

Note that in versions of ASDF prior to 3.0.1, including the entire ASDF 1 and ASDF 2 series, version-satisfies would also require that the version and the version-spec have the same major version number (the first integer in the list); if the major version differed, the version would be considered as not matching the spec. But that feature was not documented, therefore presumably not relied upon, whereas it was a nuisance to several users. Starting with ASDF 3.0.1, version-satisfies does not treat the major version number specially, and returns T simply if the first argument designates a version that isn’t older than the one specified as a second argument. If needs be, the (:version ...) syntax for specifying dependencies could be in the future extended to specify an exclusive upper bound for compatible versions as well as an inclusive lower bound.

8.1 Configurations

Configurations specify paths where to find system files.

1. The search registry may use some hardcoded wrapping registry specification. This allows some implementations (notably SBCL) to specify where to find some special implementation-provided systems that need to precisely match the version of the implementation itself.
2. An application may explicitly initialize the source-registry configuration using the configuration API (see Configuration API, below) in which case this takes precedence. It may itself compute this configuration from the command-line, from a script, from its own configuration file, etc.
3. The source registry will be configured from the environment variable CL_SOURCE_REGISTRY if it exists.
4. The source registry will be configured from user configuration file $XDG_CONFIG_DIRS/common-lisp/source-registry.conf (which defaults to ~/.config/common-lisp/source-registry.conf) if it exists.
5. The source registry will be configured from user configuration directory $XDG_CONFIG_DIRS/common-lisp/source-registry.conf.d/ (which defaults to ~/.config/common-lisp/source-registry.conf.d/) if it exists.
6. The source registry will be configured from default user configuration trees ~/common-lisp/ (since ASDF 3.1.2 only), ~/.sbcl/systems/ (on SBCL only), $XDG_DATA_HOME/common-lisp/systems/ (no recursion, link farm) $XDG_DATA_HOME/common-lisp/source/. The XDG_DATA_HOME directory defaults to ~/.local/share/. On Windows, the local-appdata and appdata directories are used instead.
7. The source registry will be configured from system configuration file /etc/common-lisp/source-registry.conf if it exists.
8. The source registry will be configured from system configuration directory /etc/common-lisp/source-registry.conf.d/ if it exists.
9. The source registry will be configured from a default configuration. This configuration may allow for implementation-specific systems to be found, for systems to be found the current directory (at the time that the configuration is initialized) as well as :directory entries for $XDG_DATA_DIRS/common-lisp/systems/ and :tree entries for $XDG_DATA_DIRS/common-lisp/source/, where XDG_DATA_DIRS defaults to /usr/local/share and /usr/share on Unix, and the common-appdata directory on Windows.
10. The source registry may include implementation-dependent directories that correspond to implementation-provided extensions.

Each of these configurations is specified as an s-expression in a trivial domain-specific language (defined below). Additionally, a more shell-friendly syntax is available for the environment variable (defined yet below).

Each of these configurations is only used if the previous configuration explicitly or implicitly specifies that it includes its inherited configuration.

Additionally, some implementation-specific directories may be automatically prepended to whatever directories are specified in configuration files, no matter if the last one inherits or not.

8.2 Truenames and other dangers

One great innovation of the original ASDF was its ability to leverage CL:TRUENAME to locate where your source code was and where to build it, allowing for symlink farms as a simple but effective configuration mechanism that is easy to control programmatically. ASDF 3 still supports this configuration style, and it is enabled by default; however we recommend you instead use our source-registry configuration mechanism described below, because it is easier to setup in a portable way across users and implementations.

Additionally, some people dislike truename, either because it is very slow on their system, or because they are using content-addressed storage where the truename of a file is related to a digest of its individual contents, and not to other files in the same intended project. For these people, ASDF 3 allows to eschew the TRUENAME mechanism, by setting the variable asdf:*resolve-symlinks* to nil.

PS: Yes, if you haven’t read Vernor Vinge’s short but great classic “True Names... and Other Dangers” then you’re in for a treat.

8.3 XDG base directory

Note that we purport to respect the XDG base directory specification as to where configuration files are located, where data files are located, where output file caches are located. Mentions of XDG variables refer to that document.

http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html

This specification allows the user to specify some environment variables to customize how applications behave to his preferences.

On Windows platforms, even when not using Cygwin, and starting with ASDF 3.1.5, we still do a best effort at following the XDG base directory specification, even though it doesn’t exactly fit common practice for Windows applications. However, we replace the fixed Unix paths ~/.local, /usr/local and /usr with their rough Windows equivalent Local AppData, AppData, Common AppData, etc. Since support for querying the Windows registry is not possible to do in reasonable amounts of portable Common Lisp code, ASDF 3 relies on the environment variables that Windows usually exports, and are hopefully in synch with the Windows registry. If you care about the details, see uiop/configuration.lisp and don’t hesitate to suggest improvements.

8.4 Backward Compatibility

For backward compatibility as well as to provide a practical backdoor for hackers, ASDF will first search for .asd files in the directories specified in asdf:*central-registry* before it searches in the source registry above.

See Configuring ASDF to find your systems — old style.

By default, asdf:*central-registry* will be empty.

This old mechanism will therefore not affect you if you don’t use it, but will take precedence over the new mechanism if you do use it.

8.5 Configuration DSL

Here is the grammar of the s-expression (SEXP) DSL for source-registry configuration:

;; A configuration is a single SEXP starting with the keyword
;; :source-registry followed by a list of directives.
CONFIGURATION := (:source-registry DIRECTIVE ...)

;; A directive is one of the following:
DIRECTIVE :=
    ;; INHERITANCE DIRECTIVE:
    ;; Your configuration expression MUST contain
    ;; exactly one of the following:
    :inherit-configuration |
    ;; splices inherited configuration (often specified last) or
    :ignore-inherited-configuration |
    ;; drop inherited configuration (specified anywhere)

    ;; forward compatibility directive (since ASDF 2.011.4), useful when
    ;; you want to use new configuration features but have to bootstrap
    ;; the newer required ASDF from an older release that doesn't
    ;; support said features:
    :ignore-invalid-entries |

    ;; add a single directory to be scanned (no recursion)
    (:directory DIRECTORY-PATHNAME-DESIGNATOR) |

    ;; add a directory hierarchy, recursing but
    ;; excluding specified patterns
    (:tree DIRECTORY-PATHNAME-DESIGNATOR) |

    ;; override the defaults for exclusion patterns
    (:exclude EXCLUSION-PATTERN ...) |
    ;; augment the defaults for exclusion patterns
    (:also-exclude EXCLUSION-PATTERN ...) |
    ;; Note that the scope of a an exclude pattern specification is
    ;; the rest of the current configuration expression or file.

    ;; splice the parsed contents of another config file
    (:include REGULAR-FILE-PATHNAME-DESIGNATOR) |

    ;; This directive specifies that some default must be spliced.
    :default-registry

REGULAR-FILE-PATHNAME-DESIGNATOR
    := PATHNAME-DESIGNATOR ; interpreted as a file
DIRECTORY-PATHNAME-DESIGNATOR
    := PATHNAME-DESIGNATOR ; interpreted as a directory

PATHNAME-DESIGNATOR :=
    NIL | ;; Special: skip this entry.
    ABSOLUTE-COMPONENT-DESIGNATOR ;; see pathname DSL

EXCLUSION-PATTERN := a string without wildcards, that will be matched
    exactly against the name of a any subdirectory in the directory
    component of a path. e.g. "_darcs" will match
    #p"/foo/bar/_darcs/src/bar.asd"

Pathnames are designated using another DSL, shared with the output-translations configuration DSL below. The DSL is resolved by the function asdf::resolve-location, to be documented and exported at some point in the future.

ABSOLUTE-COMPONENT-DESIGNATOR :=
    (ABSOLUTE-COMPONENT-DESIGNATOR RELATIVE-COMPONENT-DESIGNATOR ...) |
    STRING |
    ;; namestring (better be absolute or bust, directory assumed where
    ;; applicable).  In output-translations, directory is assumed and
    ;; **/*.*.* added if it's last.  On MCL, a MacOSX-style POSIX
    ;; namestring (for MacOS9 style, use #p"..."); Note that none of the
    ;; above applies to strings used in *central-registry*, which
    ;; doesn't use this DSL: they are processed as normal namestrings.
    ;; however, you can compute what you put in the *central-registry*
    ;; based on the results of say
    ;; (asdf::resolve-location "/Users/fare/cl/cl-foo/")
    PATHNAME |
    ;; pathname (better be an absolute path, or bust)
    ;; In output-translations, unless followed by relative components,
    ;; it better have appropriate wildcards, as in **/*.*.*
    :HOME | ; designates the user-homedir-pathname ~/
    :USER-CACHE | ; designates the default location for the user cache
    :HERE |
    ;; designates the location of the configuration file
    ;; (or *default-pathname-defaults*, if invoked interactively)
    :ROOT
    ;; magic, for output-translations source only: paths that are relative
    ;; to the root of the source host and device

They keyword :SYSTEM-CACHE is not accepted in ASDF 3.1 and beyond: it
was a security hazard.

RELATIVE-COMPONENT-DESIGNATOR :=
    (RELATIVE-COMPONENT-DESIGNATOR RELATIVE-COMPONENT-DESIGNATOR ...) |
    STRING |
      ;; relative directory pathname as interpreted by
      ;; parse-unix-namestring.
      ;; In output translations, if last component, **/*.*.* is added
    PATHNAME | ; pathname; unless last component, directory is assumed.
    :IMPLEMENTATION |
       ;; directory based on implementation, e.g. sbcl-1.0.45-linux-x64
    :IMPLEMENTATION-TYPE |
       ;; a directory based on lisp-implementation-type only, e.g. sbcl
    :DEFAULT-DIRECTORY |
       ;; a relativized version of the default directory
    :*/ | ;; any direct subdirectory (since ASDF 2.011.4)
    :**/ | ;; any recursively inferior subdirectory (since ASDF 2.011.4)
    :*.*.* | ;; any file (since ASDF 2.011.4)

The keywords :UID and :USERNAME are no longer supported.

For instance, as a simple case, my ~/.config/common-lisp/source-registry.conf, which is the default place ASDF looks for this configuration, once contained:

(:source-registry
  (:tree (:home "cl")) ;; will expand to e.g. "/home/joeluser/cl/"
  :inherit-configuration)

8.6 Configuration Directories

Configuration directories consist in files each containing a list of directives without any enclosing (:source-registry ...) form. The files will be sorted by namestring as if by string< and the lists of directives of these files with be concatenated in order. An implicit :inherit-configuration will be included at the end of the list.

System-wide or per-user Common Lisp software distributions such as Debian packages or some future version of clbuild may then include files such as /etc/common-lisp/source-registry.conf.d/10-foo.conf or ~/.config/common-lisp/source-registry.conf.d/10-foo.conf to easily and modularly register configuration information about software being distributed.

The convention is that, for sorting purposes, the names of files in such a directory begin with two digits that determine the order in which these entries will be read. Also, the type of these files must be .conf, which not only simplifies the implementation by allowing for more portable techniques in finding those files, but also makes it trivial to disable a file, by renaming it to a different file type.

Directories may be included by specifying a directory pathname or namestring in an :include directive, e.g.:

        (:include "/foo/bar/")

Hence, to achieve the same effect as my example ~/.config/common-lisp/source-registry.conf above, I could simply create a file ~/.config/common-lisp/source-registry.conf.d/33-home-fare-cl.conf alone in its directory with the following contents:

(:tree "/home/fare/cl/")

• The :here directive

   (:tree "path/to/dir")

(:source-registry
   (:tree (:here "src/lisp/"))
   (:tree (:here "extlib/lisp"))
   (:directory (:here "outlier/")))

(:include "/path/to/my/checkout/directory/asdf.conf")

8.7 Shell-friendly syntax for configuration

When considering environment variable CL_SOURCE_REGISTRY ASDF will skip to next configuration if it’s an empty string. It will READ the string as a SEXP in the DSL if it begins with a paren (, otherwise it will be interpreted much like TEXINPUTS, as a list of paths, where

* paths are separated by a : (colon) on Unix platforms (including cygwin), by a ; (semicolon) on other platforms (mainly, Windows).

* each entry is a directory to add to the search path.

* if the entry ends with a double slash // then it instead indicates a tree in the subdirectories of which to recurse.

* if the entry is the empty string (which may only appear once), then it indicates that the inherited configuration should be spliced there.

8.8 Search Algorithm

In case that isn’t clear, the semantics of the configuration is that when searching for a system of a given name, directives are processed in order.

When looking in a directory, if the system is found, the search succeeds, otherwise it continues.

When looking in a tree, if one system is found, the search succeeds. If multiple systems are found, the consequences are unspecified: the search may succeed with any of the found systems, or an error may be raised. ASDF 3.2.1 or later returns the pathname whose normalized directory component has the shortest length (as a list), and breaks ties by choosing the system with the smallest unix-namestring when compared with string<. Earlier versions of ASDF return ASDF return the first system found, which is implementation-dependent, and may or may not be the pathname with the smallest unix-namestring when compared with string<. XCVB raises an error. If none is found, the search continues.

Exclude statements specify patterns of subdirectories the systems from which to ignore. Typically you don’t want to use copies of files kept by such version control systems as Darcs. Exclude statements are not propagated to further included or inherited configuration files or expressions; instead the defaults are reset around every configuration statement to the default defaults from asdf::*default-source-registry-exclusions*.

Include statements cause the search to recurse with the path specifications from the file specified.

An inherit-configuration statement cause the search to recurse with the path specifications from the next configuration (see Configurations above).

8.9 Caching Results

The implementation is allowed to either eagerly compute the information from the configurations and file system, or to lazily re-compute it every time, or to cache any part of it as it goes. In practice, the recommended source-registry eagerly collects and caches results and you need to explicitly flush the cache for change to be taken into account, whereas the old-style *central-registry* mechanism queries the filesystem every time.

To explicitly flush any information cached by the system after a change was made in the filesystem, See Configuration API, and e.g. call asdf:clear-source-registry.

Starting with ASDF 3.1.4, you can also explicitly build a persistent cache of the .asd files found under a tree: when recursing into a directory declared by :tree and its transitive subdirectories, if a file .cl-source-registry.cache exists containing a form that is a list starting with :source-registry-cache followed by a list of strings, as in (:source-registry-cache "foo/bar.asd" "path/to/more.asd" ...), then the strings are assumed to be unix-namestrings designating the available asd files under that tree, and the recursion otherwise stops. The list can also be empty, allowing to stop a costly recursion in a huge directory tree.

To update such a cache after you install, update or remove source repositories, you can run a script distributed with ASDF: tools/cl-source-registry-cache.lisp /path/to/directory. To wholly invalidate the cache, you can delete the file .cl-source-registry.cache in that directory. In either case, for an existing Lisp process to see this change, it needs to clear its own cache with e.g. (asdf:clear-source-registry).

Developers may safely create a cache in their development tree, and we recommend they do it at the top of their source tree if it contains more than a small number of files and directories; they only need update it when they create, remove or move .asd files. Software distribution managers may also safely create such a cache, but they must be careful to update it every time they install, update or remove a software source repository or installation package. Finally, advanced developers who juggle with a lot of code in their source-registry may manually manage such a cache, to allow for faster startup of Lisp programs.

This persistence cache can help you reduce startup latency. For instance, on one machine with hundreds of source repositories, such a cache shaves half a second at the startup of every #!/usr/bin/cl script using SBCL, more on other implementations; this makes a notable difference as to their subjective interactivity and usability. The speedup will only happen if the implementation-provided ASDF is recent enough (3.1.3.7 or later); it is not enough for a recent ASDF upgrade to be present, since the upgrade will itself be found but after the old version has scanned the directories without heeding such a cache. To upgrade the implementation-provided ASDF, see Replacing your implementation’s ASDF.

8.10 Configuration API

The specified functions are exported from your build system’s package. Thus for ASDF the corresponding functions are in package ASDF, and for XCVB the corresponding functions are in package XCVB.

Function: initialize-source-registry &optional PARAMETER ¶

will read the configuration and initialize all internal variables. You may extend or override configuration from the environment and configuration files with the given PARAMETER, which can be nil (no configuration override), or a SEXP (in the SEXP DSL), a string (as in the string DSL), a pathname (of a file or directory with configuration), or a symbol (fbound to function that when called returns one of the above).

Function: clear-source-registry ¶

undoes any source registry configuration and clears any cache for the search algorithm. You might want to call this function (or better, clear-configuration) before you dump an image that would be resumed with a different configuration, and return an empty configuration. Note that this does not include clearing information about systems defined in the current image, only about where to look for systems not yet defined.

Function: ensure-source-registry &optional PARAMETER ¶

checks whether a source registry has been initialized. If not, initialize it with the given PARAMETER.

Every time you use ASDF’s find-system, or anything that uses it (such as operate, load-system, etc.), ensure-source-registry is called with parameter nil, which the first time around causes your configuration to be read. If you change a configuration file, you need to explicitly initialize-source-registry again, or maybe simply to clear-source-registry (or clear-configuration) which will cause the initialization to happen next time around.

8.11 Introspection

• *source-registry-parameter* variable
• Information about system dependencies

8.12 Status

This mechanism is vastly successful, and we have declared that asdf:*central-registry* is not recommended anymore, though we will continue to support it. All hooks into implementation-specific search mechanisms have been integrated in the wrapping-source-registry that everyone uses implicitly.

8.13 Rejected ideas

Alternatives I (FRR) considered and rejected while developing ASDF 2 included:

1. Keep asdf:*central-registry* as the master with its current semantics, and somehow the configuration parser expands the new configuration language into a expanded series of directories of subdirectories to lookup, pre-recursing through specified hierarchies. This is kludgy, and leaves little space of future cleanups and extensions.
2. Keep asdf:*central-registry* as the master but extend its semantics in completely new ways, so that new kinds of entries may be implemented as a recursive search, etc. This seems somewhat backwards.
3. Completely remove asdf:*central-registry* and break backwards compatibility. Hopefully this will happen in a few years after everyone migrate to a better ASDF and/or to XCVB, but it would be very bad to do it now.
4. Replace asdf:*central-registry* by a symbol-macro with appropriate magic when you dereference it or setf it. Only the new variable with new semantics is handled by the new search procedure. Complex and still introduces subtle semantic issues.

I’ve been suggested the below features, but have rejected them, for the sake of keeping ASDF no more complex than strictly necessary.

• More syntactic sugar: synonyms for the configuration directives, such as (:add-directory X) for (:directory X), or (:add-directory-hierarchy X) or (:add-directory X :recurse t) for (:tree X).
• The possibility to register individual files instead of directories.
• Integrate Xach Beane’s tilde expander into the parser, or something similar that is shell-friendly or shell-compatible. I’d rather keep ASDF minimal. But maybe this precisely keeps it minimal by removing the need for evaluated entries that ASDF has? i.e. uses of USER-HOMEDIR-PATHNAME and $SBCL_HOME Hopefully, these are already superseded by the :default-registry
• Using the shell-unfriendly syntax /** instead of TEXINPUTS-like // to specify recursion down a filesystem tree in the environment variable. It isn’t that Lisp friendly either.

8.14 TODO

• Add examples

8.15 Credits for the source-registry

Thanks a lot to Stelian Ionescu for the initial idea.

Thanks to Rommel Martinez for the initial implementation attempt.

All bad design ideas and implementation bugs are mine, not theirs. But so are good design ideas and elegant implementation tricks.

— Francois-Rene Rideau fare@tunes.org, Mon, 22 Feb 2010 00:07:33 -0500

9.1 Configurations

Configurations specify mappings from input locations to output locations. Once again we rely on the XDG base directory specification for configuration. See XDG base directory.

1. Some hardcoded wrapping output translations configuration may be used. This allows special output translations (or usually, invariant directories) to be specified corresponding to the similar special entries in the source registry.
2. An application may explicitly initialize the output-translations configuration using the Configuration API in which case this takes precedence. (see Configuration API.) It may itself compute this configuration from the command-line, from a script, from its own configuration file, etc.
3. The source registry will be configured from the environment variable ASDF_OUTPUT_TRANSLATIONS if it exists.
4. The source registry will be configured from user configuration file $XDG_CONFIG_DIRS/common-lisp/asdf-output-translations.conf (which defaults to ~/.config/common-lisp/asdf-output-translations.conf) if it exists.
5. The source registry will be configured from user configuration directory $XDG_CONFIG_DIRS/common-lisp/asdf-output-translations.conf.d/ (which defaults to ~/.config/common-lisp/asdf-output-translations.conf.d/) if it exists.
6. The source registry will be configured from system configuration file /etc/common-lisp/asdf-output-translations.conf if it exists.
7. The source registry will be configured from system configuration directory /etc/common-lisp/asdf-output-translations.conf.d/ if it exists.

Each of these configurations is specified as a SEXP in a trivial domain-specific language (see Configuration DSL). Additionally, a more shell-friendly syntax is available for the environment variable (see Shell-friendly syntax for configuration).

When processing an entry in the above list of configuration methods, ASDF will stop unless that entry explicitly or implicitly specifies that it includes its inherited configuration.

Note that by default, a per-user cache is used for output files. This allows the seamless use of shared installations of software between several users, and takes files out of the way of the developers when they browse source code, at the expense of taking a small toll when developers have to clean up output files and find they need to get familiar with output-translations first.¹⁵

9.2 Backward Compatibility

We purposely do not provide backward compatibility with earlier versions of ASDF-Binary-Locations (8 Sept 2009), common-lisp-controller (7.0) or cl-launch (2.35), each of which had similar general capabilities. The APIs of these programs were not designed for easy user configuration through configuration files. Recent versions of common-lisp-controller (7.2) and cl-launch (3.000) use the new asdf-output-translations API as defined below. ASDF-Binary-Locations is fully superseded and not to be used anymore.

This incompatibility shouldn’t inconvenience many people. Indeed, few people use and customize these packages; these few people are experts who can trivially adapt to the new configuration. Most people are not experts, could not properly configure these features (except inasmuch as the default configuration of common-lisp-controller and/or cl-launch might have been doing the right thing for some users), and yet will experience software that “just works”, as configured by the system distributor, or by default.

Nevertheless, if you are a fan of ASDF-Binary-Locations, we provide a limited emulation mode:

Function: enable-asdf-binary-locations-compatibility &key centralize-lisp-binaries default-toplevel-directory include-per-user-information map-all-source-files source-to-target-mappings ¶

This function will initialize the new asdf-output-translations facility in a way that emulates the behaviour of the old ASDF-Binary-Locations facility. Where you would previously set global variables *centralize-lisp-binaries*, *default-toplevel-directory*, *include-per-user-information*, *map-all-source-files* or *source-to-target-mappings* you will now have to pass the same values as keyword arguments to this function. Note however that as an extension the :source-to-target-mappings keyword argument will accept any valid pathname designator for asdf-output-translations instead of just strings and pathnames.

If you insist, you can also keep using the old ASDF-Binary-Locations (the one available as an extension to load of top of ASDF, not the one built into a few old versions of ASDF), but first you must disable asdf-output-translations with (asdf:disable-output-translations), or you might experience “interesting” issues.

Also, note that output translation is enabled by default. To disable it, use (asdf:disable-output-translations).

9.3 Configuration DSL

Here is the grammar of the SEXP DSL for asdf-output-translations configuration:

;; A configuration is single SEXP starting with keyword :source-registry
;; followed by a list of directives.
CONFIGURATION := (:output-translations DIRECTIVE ...)

;; A directive is one of the following:
DIRECTIVE :=
    ;; INHERITANCE DIRECTIVE:
    ;; Your configuration expression MUST contain
    ;; exactly one of either of these:
    :inherit-configuration |
      ;; splices inherited configuration (often specified last)
    :ignore-inherited-configuration |
      ;; drop inherited configuration (specified anywhere)

    ;; forward compatibility directive (since ASDF 2.011.4), useful when
    ;; you want to use new configuration features but have to bootstrap a
    ;; the newer required ASDF from an older release that doesn't have
    ;; said features:
    :ignore-invalid-entries |

    ;; include a configuration file or directory
    (:include PATHNAME-DESIGNATOR) |

    ;; enable global cache in ~/.common-lisp/cache/sbcl-1.0.45-linux-amd64/
    ;; or something.
    :enable-user-cache |
    ;; Disable global cache. Map / to /
    :disable-cache |

    ;; add a single directory to be scanned (no recursion)
    (DIRECTORY-DESIGNATOR DIRECTORY-DESIGNATOR)

    ;; use a function to return the translation of a directory designator
    (DIRECTORY-DESIGNATOR (:function TRANSLATION-FUNCTION))

DIRECTORY-DESIGNATOR :=
    NIL | ; As source: skip this entry. As destination: same as source
    T | ; as source matches anything, as destination
        ; maps pathname to itself.
    ABSOLUTE-COMPONENT-DESIGNATOR ; same as in the source-registry language

TRANSLATION-FUNCTION :=
    SYMBOL | ;; symbol naming a function that takes two arguments:
             ;; the pathname to be translated and the matching
             ;; DIRECTORY-DESIGNATOR
    LAMBDA   ;; A form which evaluates to a function taking two arguments:
             ;; the pathname to be translated and the matching
             ;; DIRECTORY-DESIGNATOR

Relative components better be either relative or subdirectories of the path before them, or bust.

The last component, if not a pathname, is notionally completed by /**/*.*. You can specify more fine-grained patterns by using a pathname object as the last component e.g. #p"some/path/**/foo*/bar-*.fasl"

You may use #+features to customize the configuration file.

The second designator of a mapping may be nil, indicating that files are not mapped to anything but themselves (same as if the second designator was the same as the first).

When the first designator is t, the mapping always matches. When the first designator starts with :root, the mapping matches any host and device. In either of these cases, if the second designator isn’t t and doesn’t start with :root, then strings indicating the host and pathname are somehow copied in the beginning of the directory component of the source pathname before it is translated.

When the second designator is t, the mapping is the identity. When the second designator starts with :root, the mapping preserves the host and device of the original pathname. Notably, this allows you to map files to a subdirectory of the whichever directory the file is in. Though the syntax is not quite as easy to use as we’d like, you can have an (source destination) mapping entry such as follows in your configuration file, or you may use enable-asdf-binary-locations-compatibility with :centralize-lisp-binaries nil which will do the same thing internally for you:

#.(let ((wild-subdir
          (make-pathname :directory '(:relative :wild-inferiors)))
        (wild-file
          (make-pathname :name :wild :version :wild :type :wild)))
   `((:root ,wild-subdir ,wild-file)
     (:root ,wild-subdir :implementation ,wild-file)))

Starting with ASDF 2.011.4, you can use the simpler: `(:root (:root :**/ :implementation :*.*.*))

:include statements cause the search to recurse with the path specifications from the file specified.

If the translate-pathname mechanism cannot achieve a desired translation, the user may provide a function which provides the required algorithm. Such a translation function is specified by supplying a list as the second directory-designator the first element of which is the keyword :function, and the second element of which is either a symbol which designates a function or a lambda expression. The function designated by the second argument must take two arguments, the first being the pathname of the source file, the second being the wildcard that was matched. When invoked, the function should return the translated pathname.

An :inherit-configuration statement causes the search to recurse with the path specifications from the next configuration in the bulleted list. See Configurations, above.

• :enable-user-cache is the same as (t :user-cache).
• :disable-cache is the same as (t t).
• :user-cache uses the contents of variable asdf::*user-cache* which by default is the same as using (:home ".cache" "common-lisp" :implementation).

9.4 Configuration Directories

Configuration directories consist of files, each of which contains a list of directives without any enclosing (:output-translations ...) form. The files will be sorted by namestring as if by string< and the lists of directives of these files with be concatenated in order. An implicit :inherit-configuration will be included at the end of the list.

System-wide or per-user Common Lisp software distributions such as Debian packages or some future version of clbuild may then include files such as /etc/common-lisp/asdf-output-translations.conf.d/10-foo.conf or ~/.config/common-lisp/asdf-output-translations.conf.d/10-foo.conf to easily and modularly register configuration information about software being distributed.

The convention is that, for sorting purposes, the names of files in such a directory begin with two digits that determine the order in which these entries will be read. Also, the type of these files must be .conf, which not only simplifies the implementation by allowing for more portable techniques in finding those files, but also makes it trivial to disable a file, by renaming it to a different file type.

Directories may be included by specifying a directory pathname or namestring in an :include directive, e.g.:

        (:include "/foo/bar/")

9.5 Shell-friendly syntax for configuration

When processing the environment variable ASDF_OUTPUT_TRANSLATIONS:

• ASDF will skip to the next configuration if it’s an empty string.
• ASDF will READ the string as an SEXP in the DSL, if it begins with a parenthesis (.
• Otherwise ASDF will interpret the value as a list of directories (see below).

In the directory list format, directories should come in pairs, each pair indicating a mapping directive. Entries are separated by a : (colon) on Unix platforms (including Mac and cygwin), and by a ; (semicolon) on other platforms (mainly, Windows).

The magic empty entry, if it comes in what would otherwise be the first entry in a pair, indicates the splicing of inherited configuration; the next entry (if any) then starts a new pair. If the second entry in a pair is empty, it indicates that the directory in the first entry is to be left untranslated (which has the same effect as if the directory had been repeated).

For example, "/foo:/bar::/baz:" means: specify that outputs for things under directory /foo/ are translated to be under /bar/; then include the inherited configuration; then specify that outputs for things under directory /baz/ are not translated.

9.6 Semantics of Output Translations

From the specified configuration, a list of mappings is extracted in a straightforward way: mappings are collected in order, recursing through included or inherited configuration as specified. To this list is prepended some implementation-specific mappings, and is appended a global default.

The list is then compiled to a mapping table as follows: for each entry, in order, resolve the first designated directory into an actual directory pathname for source locations. If no mapping was specified yet for that location, resolve the second designated directory to an output location directory add a mapping to the table mapping the source location to the output location, and add another mapping from the output location to itself (unless a mapping already exists for the output location).

Based on the table, a mapping function is defined, mapping source pathnames to output pathnames: given a source pathname, locate the longest matching prefix in the source column of the mapping table. Replace that prefix by the corresponding output column in the same row of the table, and return the result. If no match is found, return the source pathname. (A global default mapping the filesystem root to itself may ensure that there will always be a match, with same fall-through semantics).

9.7 Caching Results

The implementation is allowed to either eagerly compute the information from the configurations and file system, or to lazily re-compute it every time, or to cache any part of it as it goes. To explicitly flush any information cached by the system, use the API below.

9.8 Output location API

The specified functions are exported from package ASDF.

Function: initialize-output-translations &optional PARAMETER ¶

will read the configuration and initialize all internal variables. You may extend or override configuration from the environment and configuration files with the given PARAMETER, which can be nil (no configuration override), or a SEXP (in the SEXP DSL), a string (as in the string DSL), a pathname (of a file or directory with configuration), or a symbol (fbound to function that when called returns one of the above).

Function: disable-output-translations ¶

will initialize output translations in a way that maps every pathname to itself, effectively disabling the output translation facility.

Function: clear-output-translations ¶

undoes any output translation configuration and clears any cache for the mapping algorithm. You might want to call this function (or better, clear-configuration) before you dump an image that would be resumed with a different configuration, and return an empty configuration. Note that this does not include clearing information about systems defined in the current image, only about where to look for systems not yet defined.

Function: ensure-output-translations &optional PARAMETER ¶

checks whether output translations have been initialized. If not, initialize them with the given PARAMETER. This function will be called before any attempt to operate on a system.

Function: apply-output-translations PATHNAME ¶

Applies the configured output location translations to PATHNAME (calls ensure-output-translations for the translations).

Every time you use ASDF’s output-files, or anything that uses it (that may compile, such as operate, perform, etc.), ensure-output-translations is called with parameter nil, which the first time around causes your configuration to be read. If you change a configuration file, you need to explicitly initialize-output-translations again, or maybe clear-output-translations (or clear-configuration), which will cause the initialization to happen next time around.

9.9 Credits for output translations

Thanks a lot to Peter van Eynde for Common Lisp Controller and to Bjorn Lindberg and Gary King for ASDF-Binary-Locations.

All bad design ideas and implementation bugs are to mine, not theirs. But so are good design ideas and elegant implementation tricks.

— Francois-Rene Rideau fare@tunes.org

10.1 ASDF errors

If ASDF detects an incorrect system definition, it will signal a generalised instance of SYSTEM-DEFINITION-ERROR.

Operations may go wrong (for example when source files contain errors). These are signalled using generalised instances of OPERATION-ERROR.

10.2 Compilation error and warning handling

ASDF checks for warnings and errors when a file is compiled. The variables *compile-file-warnings-behaviour* and *compile-file-failure-behaviour* control the handling of any such events. The valid values for these variables are :error, :warn, and :ignore.

11.1 Controlling file compilation

When declaring a component (system, module, file), you can specify a keyword argument :around-compile function. If left unspecified (and therefore unbound), the value will be inherited from the parent component if any, or with a default of nil if no value is specified in any transitive parent.

The argument must be either nil, an fbound symbol, a lambda-expression (e.g. (lambda (thunk) ...(funcall thunk ...) ...)) a function object (e.g. using #.#' but that’s discouraged because it prevents the introspection done by e.g. asdf-dependency-grovel), or a string that when read yields a symbol or a lambda-expression. nil means the normal compile-file function will be called. A non-nil value designates a function of one argument that will be called with a function that will invoke compile-file* with various arguments; the around-compile hook may supply additional keyword arguments to pass to that call to compile-file*.

One notable argument that is heeded by compile-file* is :compile-check, a function called when the compilation was otherwise a success, with the same arguments as compile-file; the function shall return true if the compilation and its resulting compiled file respected all system-specific invariants, and false (nil) if it broke any of those invariants; it may issue warnings or errors before it returns nil. (NB: The ability to pass such extra flags is only available starting with ASDF 2.22.3.) This feature is notably exercised by asdf-finalizers.

By using a string, you may reference a function, symbol and/or package that will only be created later during the build, but isn’t yet present at the time the defsystem form is evaluated. However, if your entire system is using such a hook, you may have to explicitly override the hook with nil for all the modules and files that are compiled before the hook is defined.

Using this hook, you may achieve such effects as: locally renaming packages, binding *readtables* and other syntax-controlling variables, handling warnings and other conditions, proclaiming consistent optimization settings, saving code coverage information, maintaining meta-data about compilation timings, setting gensym counters and PRNG seeds and other sources of non-determinism, overriding the source-location and/or timestamping systems, checking that some compile-time side-effects were properly balanced, etc.

Note that there is no around-load hook. This is on purpose. Some implementations such as ECL, GCL or MKCL link object files, which allows for no such hook. Other implementations allow for concatenating FASL files, which doesn’t allow for such a hook either. We aim to discourage something that’s not portable, and has some dubious impact on performance and semantics even when it is possible. Things you might want to do with an around-load hook are better done around-compile, though it may at times require some creativity (see e.g. the package-renaming system).

11.2 Controlling source file character encoding

Starting with ASDF 2.21, components accept a :encoding option so authors may specify which character encoding should be used to read and evaluate their source code. When left unspecified, the encoding is inherited from the parent module or system; if no encoding is specified at any point, or if nil is explicitly specified, an extensible protocol described below is followed, that ultimately defaults to :utf-8 since ASDF 3.

The protocol to determine the encoding is to call the function detect-encoding, which itself, if provided a valid file, calls the function specified by *encoding-detection-hook*, or else defaults to the *default-encoding*. The *encoding-detection-hook* is by default bound to function always-default-encoding, that always returns the contents of *default-encoding*. *default-encoding* is bound to :utf-8 by default (before ASDF 3, the default was :default).

Whichever encoding is returned must be a portable keyword, that will be translated to an implementation-specific external-format designator by function encoding-external-format, which itself simply calls the function specified *encoding-external-format-hook*; that function by default is default-encoding-external-format, that only recognizes :utf-8 and :default, and translates the former to the implementation-dependent *utf-8-external-format*, and the latter to itself (that itself is portable but has an implementation-dependent meaning).

In other words, there now are plenty of extension hooks, but by default ASDF enforces the previous de facto standard behaviour of using :utf-8, independently from whatever configuration the user may be using. Thus, system authors can now rely on :utf-8 being used while compiling their files, even if the user is currently using :koi8-r or :euc-jp as their interactive encoding. (Before ASDF 3, there was no such guarantee, :default was used, and only plain ASCII was safe to include in source code.)

Some legacy implementations only support 8-bit characters, and some implementations provide 8-bit only variants. On these implementations, the *utf-8-external-format* gracefully falls back to :default, and Unicode characters will be read as multi-character mojibake. To detect such situations, UIOP will push the :asdf-unicode feature on implementations that support Unicode, and you can use reader-conditionalization to protect any :encoding encoding statement, as in #+asdf-unicode :encoding #+asdf-unicode :utf-8. We recommend that you avoid using unprotected :encoding specifications until after ASDF 2.21 or later becomes widespread. As of May 2016, all maintained implementations provide ASDF 3.1, so you may prudently start using this and other features without such protection.

While it offers plenty of hooks for extension, and one such extension is available (see asdf-encodings below), ASDF itself only recognizes one encoding beside :default, and that is :utf-8, which is the de facto standard, already used by the vast majority of libraries that use more than ASCII. On implementations that do not support unicode, the feature :asdf-unicode is absent, and the :default external-format is used to read even source files declared as :utf-8. On these implementations, non-ASCII characters intended to be read as one CL character may thus end up being read as multiple CL characters. In most cases, this shouldn’t affect the software’s semantics: comments will be skipped just the same, strings with be read and printed with slightly different lengths, symbol names will be accordingly longer, but none of it should matter. But a few systems that actually depend on unicode characters may fail to work properly, or may work in a subtly different way. See for instance lambda-reader.

We invite you to embrace UTF-8 as the encoding for non-ASCII characters starting today, even without any explicit specification in your .asd files. Indeed, on some implementations and configurations, UTF-8 is already the :default, and loading your code may cause errors if it is encoded in anything but UTF-8. Therefore, even with the legacy behaviour, non-UTF-8 is guaranteed to break for some users, whereas UTF-8 is pretty much guaranteed not to break anywhere (provided you do not use a BOM), although it might be read incorrectly on some implementations. :utf-8 has been the default value of *default-encoding* since ASDF 3.

If you need non-standard character encodings for your source code, use the extension system asdf-encodings, by specifying :defsystem-depends-on ("asdf-encodings") in your defsystem. This extension system will register support for more encodings using the *encoding-external-format-hook* facility, so you can explicitly specify :encoding :latin1 in your .asd file. Using the *encoding-detection-hook* it will also eventually implement some autodetection of a file’s encoding from an emacs-style -*- mode: lisp ; coding: latin1 -*- declaration, or otherwise based on an analysis of octet patterns in the file. At this point, asdf-encoding only supports the encodings that are supported as part of your implementation. Since the list varies depending on implementations, we still recommend you use :utf-8 everywhere, which is the most portable (next to it is :latin1).

Recent versions of Quicklisp include asdf-encodings; if you’re not using it, you may get this extension using git: git clone https://gitlab.common-lisp.net/asdf/asdf-encodings.git or git clone git@gitlab.common-lisp.net:asdf/asdf-encodings.git. You can also browse the repository on https://gitlab.common-lisp.net/asdf/asdf-encodings.

When you use asdf-encodings, any .asd file loaded will use the autodetection algorithm to determine its encoding. If you depend on this detection happening, you should explicitly load asdf-encodings early in your build. Note that :defsystem-depends-on cannot be used here: by the time the :defsystem-depends-on is loaded, the enclosing defsystem form has already been read.

In practice, this means that the *default-encoding* is usually used for .asd files. Currently, this defaults to :utf-8, and you should be safe using Unicode characters in those files. This might matter, for instance, in meta-data about author’s names. Otherwise, the main data in these files is component (path)names, and we don’t recommend using non-ASCII characters for these, for the result probably isn’t very portable.

11.3 Miscellaneous Functions

These functions are exported by ASDF for your convenience.

Function: system-relative-pathname system name &key type ¶

It’s often handy to locate a file relative to some system. The system-relative-pathname function meets this need.

It takes two mandatory arguments system and name and a keyword argument type: system is name of a system, whereas name and optionally type specify a relative pathname, interpreted like a component pathname specifier by coerce-pathname. See Pathname specifiers.

It returns a pathname built from the location of the system’s source directory and the relative pathname. For example:

> (asdf:system-relative-pathname 'cl-ppcre "regex.data")
#P"/repository/other/cl-ppcre/regex.data"

Function: system-source-directory system-designator ¶

ASDF does not provide a turnkey solution for locating data (or other miscellaneous) files that are distributed together with the source code of a system. Programmers can use system-source-directory to find such files. Returns a pathname object. The system-designator may be a string, symbol, or ASDF system object.

Function: clear-system system-designator ¶

It is sometimes useful to force recompilation of a previously loaded system. For these cases, (asdf:clear-system :foo) will remove the system from the table of currently loaded systems: the next time the system foo or one that depends on it is re-loaded, foo will be loaded again.¹⁶

Note that this does not and cannot undo the previous loading of the system. Common Lisp has no provision for such an operation, and its reliance on irreversible side-effects to global data structures makes such a thing impossible in the general case. If the software being re-loaded is not conceived with hot upgrade in mind, re-loading may cause many errors, warnings or subtle silent problems, as packages, generic function signatures, structures, types, macros, constants, etc. are being redefined incompatibly. It is up to the user to make sure that reloading is possible and has the desired effect. In some cases, extreme measures such as recursively deleting packages, unregistering symbols, defining methods on update-instance-for-redefined-class and much more are necessary for reloading to happen smoothly. ASDF itself goes to extensive effort to make a hot upgrade possible with respect to its own code. If you want, you can reuse some of its utilities such as uiop:define-package and uiop:with-upgradability, and get inspiration (or disinspiration) from what it does in header.lisp and upgrade.lisp.

Function: register-preloaded-system name &rest keys &key version &allow-other-keys ¶

A system with name name, created by make-instance with extra keys keys (e.g. :version), is registered as preloaded. If version is t (default), then the version is copied from the defined system of the same name (if registered) or else is nil (this automatic copy of version is only available starting since ASDF 3.1.8).

A preloaded system is considered as having already been loaded into the current image, and if at some point some other system :depends-on it yet no source code is found, it is considered as already provided, and ASDF will not raise a missing-component error.

This function is particularly useful if you distribute your code as fasls with either compile-bundle-op or monolithic-compile-bundle-op, and want to register systems so that dependencies will work uniformly whether you’re using your software from source or from fasl.

Note that if the system was already defined or loaded from source code, its build information will remain active until you call clear-system on it, at which point a system without build information will be registered in its place.

Function: register-immutable-system name &rest keys ¶

A system with name name is registered as preloaded, and additionally is marked as immutable: that is, attempts to compile or load it will be succeed without actually reading, creating or loading any file, as if the system was passed as a force-not argument to all calls to plan or operate. There will be no search for an updated .asd file to override the loaded version, whether from the source-register or any other method.

If a version keyword argument is specified as t or left unspecified, then the version is copied from the defined system of the same name (if registered) or else is nil. This automatic copy of version is available starting since immutable systems have been available in ASDF 3.1.5.

This function, available since ASDF 3.1.5, is particularly useful if you distribute a large body of code as a precompiled image, and want to allow users to extend the image with further extension systems, but without making thousands of filesystem requests looking for inexistent (or worse, out of date) source code for all the systems that came bundled with the image but aren’t distributed as source code to regular users.

Function: run-shell-command control-string &rest args ¶

This function is obsolete and present only for the sake of backwards-compatibility: “If it’s not backwards, it’s not compatible”. We strongly discourage its use. Its current behaviour is only well-defined on Unix platforms (which include MacOS X and cygwin). On Windows, anything goes. The following documentation is only for the purpose of your migrating away from it in a way that preserves semantics.

Instead we recommend the use run-program, described in the next section, and available as part of ASDF since ASDF 3.

run-shell-command takes as arguments a format control-string and arguments to be passed to format after this control-string to produce a string. This string is a command that will be evaluated with a POSIX shell if possible; yet, on Windows, some implementations will use CMD.EXE, while others (like SBCL) will make an attempt at invoking a POSIX shell (and fail if it is not present).

11.4 Some Utility Functions

The below functions are not exported by ASDF itself, but by UIOP, available since ASDF 3. Some of them have precursors in ASDF 2, but we recommend that for active developments, you should rely on the package UIOP as included in ASDF 3. UIOP provides many, many more utility functions, and we recommend you read its README.md and sources for more information.

Function: parse-unix-namestring name &key type defaults dot-dot ensure-directory &allow-other-keys ¶

Coerce name into a pathname using standard Unix syntax.

Unix syntax is used whether or not the underlying system is Unix; on non-Unix systems it is only usable for relative pathnames. In order to manipulate relative pathnames portably, it is crucial to possess a portable pathname syntax independent of the underlying OS. This is what parse-unix-namestring provides, and why we use it in ASDF.

When given a pathname object, just return it untouched. When given nil, just return nil. When given a non-null symbol, first downcase its name and treat it as a string. When given a string, portably decompose it into a pathname as below.

#\/ separates directory components.

The last #\/-separated substring is interpreted as follows: 1- If type is :directory or ensure-directory is true, the string is made the last directory component, and its name and type are nil. if the string is empty, it’s the empty pathname with all slots nil. 2- If type is nil, the substring is a file-namestring, and its name and type are separated by split-name-type. 3- If type is a string, it is the given type, and the whole string is the name.

Directory components with an empty name the name . are removed. Any directory named .. is read as dot-dot, which must be one of :back or :up and defaults to :back.

host, device and version components are taken from defaults, which itself defaults to *nil-pathname*. *nil-pathname* is also used if defaults is nil. No host or device can be specified in the string itself, which makes it unsuitable for absolute pathnames outside Unix.

For relative pathnames, these components (and hence the defaults) won’t matter if you use merge-pathnames* but will matter if you use merge-pathnames, which is an important reason to always use merge-pathnames*.

Arbitrary keys are accepted, and the parse result is passed to ensure-pathname with those keys, removing type, defaults and dot-dot. When you’re manipulating pathnames that are supposed to make sense portably even though the OS may not be Unixish, we recommend you use :want-relative t so that parse-unix-namestring will throw an error if the pathname is absolute.

Function: merge-pathnames* specified &optional defaults ¶

This function is a replacement for merge-pathnames that uses the host and device from the defaults rather than the specified pathname when the latter is a relative pathname. This allows ASDF and its users to create and use relative pathnames without having to know beforehand what are the host and device of the absolute pathnames they are relative to.

Function: subpathname pathname subpath &key type ¶

This function takes a pathname and a subpath and a type. If subpath is already a pathname object (not namestring), and is an absolute pathname at that, it is returned unchanged; otherwise, subpath is turned into a relative pathname with given type as per parse-unix-namestring with :want-relative t :type type, then it is merged with the pathname-directory-pathname of pathname, as per merge-pathnames*.

We strongly encourage the use of this function for portably resolving relative pathnames in your code base.

Function: subpathname* pathname subpath &key type ¶

This function returns nil if the base pathname is nil, otherwise acts like subpathname.

Function: run-program command &key ignore-error-status force-shell input output error-output if-input-does-not-exist if-output-exists if-error-output-exists element-type external-format &allow-other-keys ¶

run-program takes a command argument that is either a list of a program name or path and its arguments, or a string to be executed by a shell. It spawns the command, waits for it to return, verifies that it exited cleanly (unless told not too below), and optionally captures and processes its output. It accepts many keyword arguments to configure its behaviour.

run-program returns three values: the first for the output, the second for the error-output, and the third for the return value. (Beware that before ASDF 3.0.2.11, it didn’t handle input or error-output, and returned only one value, the one for the output if any handler was specified, or else the exit code; please upgrade ASDF, or at least UIOP, to rely on the new enhanced behaviour.)

output is its most important argument; it specifies how the output is captured and processed. If it is nil, then the output is redirected to the null device, that will discard it. If it is :interactive, then it is inherited from the current process (beware: this may be different from your *standard-output*, and under SLIME will be on your *inferior-lisp* buffer). If it is t, output goes to your current *standard-output* stream. Otherwise, output should be a value that is a suitable first argument to slurp-input-stream (see below), or a list of such a value and keyword arguments. In this case, run-program will create a temporary stream for the program output; the program output, in that stream, will be processed by a call to slurp-input-stream, using output as the first argument (or if it’s a list the first element of output and the rest as keywords). The primary value resulting from that call (or nil if no call was needed) will be the first value returned by run-program. E.g., using :output :string will have it return the entire output stream as a string. And using :output '(:string :stripped t) will have it return the same string stripped of any ending newline.

error-output is similar to output, except that the resulting value is returned as the second value of run-program. t designates the *error-output*. Also :output means redirecting the error output to the output stream, in which case nil is returned.

input is similar to output, except that vomit-output-stream is used, no value is returned, and t designates the *standard-input*.

element-type and external-format are passed on to your Lisp implementation, when applicable, for creation of the output stream.

One and only one of the stream slurping or vomiting may or may not happen in parallel in parallel with the subprocess, depending on options and implementation, and with priority being given to output processing. Other streams are completely produced or consumed before or after the subprocess is spawned, using temporary files.

force-shell forces evaluation of the command through a shell, even if it was passed as a list rather than a string. If a shell is used, it is /bin/sh on Unix or CMD.EXE on Windows, except on implementations that (erroneously, IMNSHO) insist on consulting $SHELL like clisp.

ignore-error-status causes run-program to not raise an error if the spawned program exits in error. Following POSIX convention, an error is anything but a normal exit with status code zero. By default, an error of type subprocess-error is raised in this case.

run-program works on all platforms supported by ASDF, except Genera. See the source code for more documentation.

Function: slurp-input-stream processor input-stream &key ¶

slurp-input-stream is a generic function of two arguments, a target object and an input stream, and accepting keyword arguments. Predefined methods based on the target object are as follows:

• If the object is a function, the function is called with the stream as argument.
• If the object is a cons, its first element is applied to its rest appended by a list of the input stream.
• If the object is an output stream, the contents of the input stream are copied to it. If the linewise keyword argument is provided, copying happens line by line, and an optional prefix is printed before each line. Otherwise, copying happen based on a buffer of size buffer-size, using the specified element-type.
• If the object is 'string or :string, the content is captured into a string. Accepted keywords include the element-type and a flag stripped, which when true causes any single line ending to be removed as per uiop:stripln.
• If the object is :lines, the content is captured as a list of strings, one per line, without line ending. If the count keyword argument is provided, it is a maximum count of lines to be read.
• If the object is :line, the content is captured as with :lines above, and then its sub-object is extracted with the at argument, which defaults to 0, extracting the first line. A number will extract the corresponding line. See the documentation for uiop:access-at.
• If the object is :forms, the content is captured as a list of s-expressions, as read by the Lisp reader. If the count argument is provided, it is a maximum count of lines to be read. We recommend you control the syntax with such macro as uiop:with-safe-io-syntax.
• If the object is :form, the content is captured as with :forms above, and then its sub-object is extracted with the at argument, which defaults to 0, extracting the first form. A number will extract the corresponding form. See the documentation for uiop:access-at. We recommend you control the syntax with such macro as uiop:with-safe-io-syntax.

13.1 “Where do I report a bug?”

ASDF bugs are tracked on launchpad: https://launchpad.net/asdf.

If you’re unsure about whether something is a bug, or for general discussion, use the asdf-devel mailing list (see Mailing list).

13.2 Mailing list

Discussion of ASDF development is conducted on the mailing list asdf-devel@common-lisp.net. http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel

13.3 “What has changed between ASDF 1, ASDF 2, and ASDF 3?”

We released ASDF 2.000 on May 31st 2010, ASDF 3.0.0 on May 15th 2013, ASDF 3.1.2 on May 6th 2014. Releases of ASDF 2 and now ASDF 3 have since then been included in all actively maintained CL implementations that used to bundle ASDF 1, plus many implementations that previously did not. ASDF has been made to work with all actively maintained CL implementations and even a few implementations that are not actively maintained.

Furthermore, it is possible to upgrade from ASDF 1 to ASDF 2 or ASDF 3 on the fly (though we recommend instead upgrading your implementation or replacing its ASDF module). For this reason, we have stopped supporting ASDF 1 and ASDF 2. If you are using ASDF 1 or ASDF 2 and are experiencing any kind of issues or limitations, we recommend you upgrade to ASDF 3 — and we explain how to do that. See Loading ASDF.

Note that in the context of compatibility requirements, ASDF 2.27, released on Feb 1st 2013, and further releases up to 2.33, count as pre-releases of ASDF 3, and define the :asdf3 feature, though the first stable release of ASDF 3 was release 3.0.1. Significant new or improved functionality were added in ASDF 3.1; the :asdf3.1 feature is present in recent enough versions to detect this functionality; the first stable release since then was ASDF 3.1.2. New *features* are only added at major milestones, and the next one will probably be :asdf3.2.

• What are ASDF 1, ASDF 2, and ASDF 3?
• How do I detect the ASDF version?
• ASDF can portably name files in subdirectories
• Output translations
• Source Registry Configuration
• Usual operations are made easier to the user
• Many bugs have been fixed
• ASDF itself is versioned
• ASDF can be upgraded
• Decoupled release cycle
• Pitfalls of the transition to ASDF 2
• Pitfalls of the upgrade to ASDF 3
• What happened to the bundle operations?

(when (find-package :asdf)
  (let ((ver (symbol-value
                   (or (find-symbol (string :*asdf-version*) :asdf)
                       (find-symbol (string :*asdf-revision*) :asdf)))))
    (etypecase ver
      (string ver)
      (cons (with-output-to-string (s)
              (loop for (n . m) on ver
                    do (princ n s)
                       (when m (princ "." s)))))
      (null "1.0"))))

13.4 Issues with installing the proper version of ASDF

• “My Common Lisp implementation comes with an outdated version of ASDF. What to do?”
• “I’m a Common Lisp implementation vendor. When and how should I upgrade ASDF?”
• After upgrading ASDF, ASDF (and Quicklisp) can’t find my systems

13.5 Issues with configuring ASDF

• “How can I customize where fasl files are stored?”
• “How can I wholly disable the compiler output cache?”

mkdir -p ~/.config/common-lisp/asdf-output-translations.conf.d/
echo ':disable-cache' > \
~/.config/common-lisp/asdf-output-translations.conf.d/99-disable-cache.conf

export ASDF_OUTPUT_TRANSLATIONS=/:

(asdf:disable-output-translations)

13.6 Issues with using and extending ASDF to define systems

• “How can I cater for unit-testing in my system?”
• “How can I cater for documentation generation in my system?”
• “How can I maintain non-Lisp (e.g. C) source files?”
• “I want to put my module’s files at the top level. How do I do this?”
• How do I create a system definition where all the source files have a .cl extension?
• How do I mark a source file to be loaded only and not compiled?
• How do I work with readtables?
• How can I capture ASDF’s output?
• *LOAD-PATHNAME* and *LOAD-TRUENAME* have weird values, help!

(asdf:defsystem "spatial-trees"
  :components
  ((:module "base"
            :pathname ""
            :components
            ((:file "package")
             (:file "basedefs" :depends-on ("package"))
             (:file "rectangles" :depends-on ("package"))))
   (:module tree-impls
            :depends-on ("base")
            :pathname ""
            :components
            ((:file "r-trees")
             (:file "greene-trees" :depends-on ("r-trees"))
             (:file "rstar-trees" :depends-on ("r-trees"))
             (:file "rplus-trees" :depends-on ("r-trees"))
             (:file "x-trees" :depends-on ("r-trees" "rstar-trees"))))
   (:module viz
            :depends-on ("base")
            :pathname ""
            :components
            ((:static-file "spatial-tree-viz.lisp")))
   (:module tests
            :depends-on ("base")
            :pathname ""
            :components
            ((:static-file "spatial-tree-test.lisp")))
   (:static-file "LICENCE")
   (:static-file "TODO")))

(defsystem my-cl-system
  :default-component-class cl-source-file.cl
  ...)

;; Prologue: make sure we're using a sane package.
(defpackage :my-asdf-extension
   (:use :asdf :common-lisp)
   (:export #:cl-source-file.lis))
(in-package :my-asdf-extension)

(defclass cl-source-file.lis (cl-source-file)
  ((type :initform "lis")))

(defsystem my-cl-system
  :default-component-class my-asdf-extension:cl-source-file.lis
  ...)

(in-package :asdf)
(defclass cl-source-file.lis (cl-source-file)
   ((type :initform "lis")))
(defsystem my-cl-system
  :default-component-class cl-source-file.lis
  ...)

13.7 ASDF development FAQs

• How do I run the tests interactively in a REPL?

;; BEWARE! Some tests expect you to be in the .../asdf/test directory
;; If your REPL is not there yet, change your current directory:
;; under SLIME, you may: ,change-directory ~/common-lisp/asdf/test/
;; otherwise you may evaluate something like:
(require "asdf") (asdf:upgrade-asdf) ;load UIOP & update asdf.lisp
(uiop:chdir (asdf:system-relative-pathname :asdf "test/"))
(setf *default-pathname-defaults* (uiop:getcwd))

;; Load the test script support.
(load "script-support.lisp")

;; Initialize the script support for interaction.
;; This will also change your *package* to asdf-test
;; after frobbing the asdf-test package to make it usable.
;; NB: this function is also available from package cl-user,
;; and also available with the shorter name da in both packages.
(asdf-test:debug-asdf)

;; Now, you may experiment with test code from a .script file.
;; See the instructions given at the end of your failing test
;; to identify which form is needed, e.g.
(run-test-script "test-utilities.script")