     [file docstyle.texi not included]

ERT: Emacs Lisp Regression Testing
**********************************

Copyright © 2008, 2010--2025 Free Software Foundation, Inc.

     Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, with the Front-Cover Texts being ``A GNU Manual,'' and with the Back-Cover Texts as in (a) below.  A copy of the license is included in the section entitled ``GNU Free Documentation License''.

     (a) The FSF's Back-Cover Text is: ``You have the freedom to copy and modify this GNU manual.''

ERT is a tool for automated testing in Emacs Lisp.  Its main features are facilities for defining and running tests, reporting results, and debugging test failures interactively.  Tests are written in Emacs Lisp, just like the code under test, making it easy to write expressive and concise test cases.

ERT has unique features that take advantage of the dynamic and interactive nature of Emacs.  It is unopinionated about when or how tests are written: you can use it to verify newly written code, reproduce known bugs, prevent regressions, or explore the behavior of complex systems.  It can be used for both unit tests and larger integration tests.

Introduction
************

ERT allows you to define tests in addition to functions, macros, variables, and the other usual Lisp constructs.  Tests are simply Lisp code: code that invokes other code and checks whether it behaves as expected.

ERT keeps track of the tests that are defined and provides convenient commands to run them to verify whether the definitions that are currently loaded in Emacs pass the tests.

Some Lisp files have comments like the following (adapted from the package pp.el):

     ;; (pp-to-string '(quote quote))          ; expected: "'quote"
     ;; (pp-to-string '((quote a) (quote b)))  ; expected: "('a 'b)\n"
     ;; (pp-to-string '('a 'b))                ; same as above

The Lisp forms contained in these comments can be evaluated from time to time, e.g. with C-x C-e, to compare the output with the expected output.  ERT formalizes this and introduces a common convention, which simplifies Emacs development, since programmers no longer have to manually find and evaluate such comments.

An ERT test definition equivalent to the above comments is this:

     (ert-deftest pp-test-quote ()
       "Tests the rendering of `quote' symbols in `pp-to-string'."
       (should (equal (pp-to-string '(quote quote)) "'quote"))
       (should (equal (pp-to-string '((quote a) (quote b))) "('a 'b)\n"))
       (should (equal (pp-to-string '('a 'b)) "('a 'b)\n")))

If you know defun, the syntax of ert-deftest should look familiar: This example defines a test named pp-test-quote that will pass if the three calls to equal all return non-nil.

should is a macro with the same meaning as cl-assert but better error reporting.  See The should Macro.

Each test should have a name that describes what functionality it tests. Test names can be chosen arbitrarily---they are in a namespace separate from functions and variables---but should follow the usual Emacs Lisp convention of having a prefix that indicates which package they belong to.  Test names are displayed by ERT when reporting failures and can be used when selecting which tests to run.

The empty parentheses () in the first line don't currently have any meaning and are reserved for future extension.  They also make the syntax of ert-deftest more similar to that of defun.

The docstring describes what feature this test tests.  When running tests interactively, the first line of the docstring is displayed for tests that fail, so it is good if the first line makes sense on its own.

The body of a test can be arbitrary Lisp code.  It should have as few side effects as possible; each test should be written to clean up after itself, leaving Emacs in the same state as it was before the test.  Tests should clean up even if they fail.  See Tests and Their Environment.

How to Run Tests
****************

You can run tests either in the Emacs you are working in, or on the command line in a separate Emacs process in batch mode (i.e., with no user interface).  The former mode is convenient during interactive development, the latter is useful to make sure that tests pass independently of your customizations; and it allows you to invoke tests from makefiles, and to write scripts that run tests in several different Emacs versions.

Running Tests Interactively
===========================

You can run the tests that are currently defined in your Emacs with the command M-x ert RET t RET (which is an alias of ert-run-tests-interactively).  The t argument means to run all the defined tests, see Test Selectors, which also explains how to run only some specific part of the tests.  ERT will pop up a new buffer, the ERT results buffer, showing the results of the tests run. It looks like this:

     Selector: t
     Passed:  31
     Skipped: 0
     Failed:  2 (2 unexpected)
     Total:   33/33

     Started at:   2008-09-11 08:39:25-0700
     Finished.
     Finished at:  2008-09-11 08:39:27-0700

     FF...............................

     F addition-test
         (ert-test-failed
          ((should
            (=
             (+ 1 2)
             4))
           :form
           (= 3 4)
           :value nil))

     F list-test
         (ert-test-failed
          ((should
            (equal
             (list 'a 'b 'c)
             '(a b d)))
           :form
           (equal
            (a b c)
            (a b d))
           :value nil :explanation
           (list-elt 2
                     (different-atoms c d))))

At the top, there is a summary of the results: we ran all tests defined in the current Emacs (Selector: t), 31 of them passed, and 2 failed unexpectedly.  See Expected Failures, for an explanation of the term unexpected in this context.

The line of dots and Fs is a progress bar where each character represents one test; it fills while the tests are running.  A dot means that the test passed, an F means that it failed.  Below the progress bar, ERT shows details about each test that had an unexpected result.  In the example above, there are two failures, both due to failed should forms.  See Understanding Explanations, for more details.

The following key bindings are available in the ERT results buffer:

RET
     Each name of a function or macro in this buffer is a button; moving point to it and typing RET jumps to its definition.

TAB
S-TAB
     Cycle between buttons forward (forward-button) and backward (backward-button).

r
     Re-run the test near point on its own (ert-results-rerun-test-at-point).

d
     Re-run the test near point on its own with the debugger enabled (ert-results-rerun-test-at-point-debugging-errors).

R
     Re-run all tests (ert-results-rerun-all-tests).

.
     Jump to the definition of the test near point (ert-results-find-test-at-point-other-window).  This has the same effect as RET, but does not require point to be on the name of the test.

b
     Show the backtrace of a failed test (ert-results-pop-to-backtrace-for-test-at-point). See Backtraces (GNU Emacs Lisp Reference Manual), for more information about backtraces.

l
     Show the list of should forms executed in the test (ert-results-pop-to-should-forms-for-test-at-point).

m
     Show any messages that were generated (with the Lisp function message) in a test or any of the code that it invoked (ert-results-pop-to-messages-for-test-at-point).

L
     By default, long expressions in the failure details are abbreviated using print-length and print-level.  Increase the limits to show more of the expression by moving point to a test failure with this command (ert-results-toggle-printer-limits-for-test-at-point).

D
     Delete a test from the running Emacs session (ert-delete-test).

h
     Show the documentation of a test (ert-describe-test).

T
     Display test timings for the last run (ert-results-pop-to-timings).

M-x ert-delete-all-tests
     Delete all tests from the running session.

M-x ert-describe-test
     Prompt for a test and then show its documentation.

Running Tests in Batch Mode
===========================

ERT supports automated invocations from the command line or from scripts or makefiles.  There are two functions for this purpose, ert-run-tests-batch and ert-run-tests-batch-and-exit. They can be used like this:

     emacs -batch -l ert -l my-tests.el -f ert-run-tests-batch-and-exit

This command will start up Emacs in batch mode, load ERT, load my-tests.el, and run all tests defined in it.  It will exit with a zero exit status if all tests passed, or nonzero if any tests failed or if anything else went wrong.  It will also print progress messages and error diagnostics to standard output.

You can also redirect the above output to a log file, say output.log, and use the ert-summarize-tests-batch-and-exit function to produce a neat summary as shown below:

     emacs -batch -l ert -f ert-summarize-tests-batch-and-exit output.log

ERT attempts to limit the output size for failed tests by choosing conservative values for print-level and print-length when printing Lisp values.  This can in some cases make it difficult to see which portions of those values are incorrect.  Use ert-batch-print-level and ert-batch-print-length to customize that:

     emacs -batch -l ert -l my-tests.el \
           --eval "(let ((ert-batch-print-level 10) \
                         (ert-batch-print-length 120)) \
                     (ert-run-tests-batch-and-exit))"

Even modest settings for print-level and print-length can produce extremely long lines in backtraces, however, with attendant pauses in execution progress.  Set ert-batch-backtrace-line-length to t to use the value of backtrace-line-length, nil to stop any limitations on backtrace line lengths (that is, to get full backtraces), or a positive integer to limit backtrace line length to that number.

By default, ERT in batch mode is quite verbose, printing a line with result after each test.  This gives you progress information: how many tests have been executed and how many there are.  However, in some cases this much output may be undesirable.  In this case, set ert-quiet variable to a non-nil value:

     emacs -batch -l ert -l my-tests.el \
           --eval "(let ((ert-quiet t)) (ert-run-tests-batch-and-exit))"

In quiet mode ERT prints only unexpected results and summary.

You can specify selectors to only run a subset of your tests (see Test Selectors).  For example, the following would run all tests where the name of the test matches the regular expression ``to-match''.

     emacs -batch -l ert -l my-tests.el \
           -eval '(ert-run-tests-batch-and-exit "to-match")'

By default, ERT test failure summaries are quite brief in batch mode---only the names of the failed tests are listed.  If the EMACS_TEST_VERBOSE environment variable is set and is non-empty, the failure summaries will also include the data from the failing test.

ERT can produce JUnit test reports in batch mode.  If the environment variable EMACS_TEST_JUNIT_REPORT is set, ERT will produce for every test package my-tests.el a corresponding JUnit test report my-tests.xml.  The function ert-summarize-tests-batch-and-exit collects all these package test reports into a new JUnit test report, with the respective name of that environment variable.

Test Selectors
==============

Functions like ert accept a test selector, a Lisp expression specifying a set of tests.  Test selector syntax is similar to Common Lisp's type specifier syntax:

  * nil selects no tests.

  * t selects all tests.

  * :new selects all tests that have not been run yet.

  * :failed and :passed select tests according to their most recent result.

  * :expected, :unexpected select tests according to their most recent result.

  * A string is a regular expression that selects all tests with matching names.

  * A test (i.e., an object of ert-test data type, see its doc string for details) selects that test.

  * A symbol selects the test that the symbol names.

  * (member TESTS...) selects the elements of TESTS, a list of tests or symbols naming tests.

  * (eql TEST) selects TEST, a test or a symbol naming a test.

  * (and SELECTORS...) selects the tests that match all SELECTORS.

  * (or SELECTORS...) selects the tests that match any of the SELECTORS.

  * (not SELECTOR) selects all tests that do not match SELECTOR.

  * (tag TAG) selects all tests that have TAG on their tags list. (Tags are optional labels you can apply to tests when you define them.)

  * (satisfies PREDICATE) selects all tests that satisfy PREDICATE, a function that takes a test as argument and returns non-nil if it is selected.

Selectors that are frequently useful when selecting tests to run include t to run all tests that are currently defined in Emacs, "^foo-" to run all tests in package foo (this assumes that package foo uses the prefix foo- for its test names), result-based selectors such as (or :new :unexpected) to run all tests that have either not run yet or that had an unexpected result in the last run, and tag-based selectors such as (not (tag :causes-redisplay)) to run all tests that are not tagged :causes-redisplay.

How to Write Tests
******************

ERT lets you define tests in the same way you define functions.  You can type ert-deftest forms in a buffer and evaluate them there with eval-defun or compile-defun, or you can save the file and load it, optionally byte-compiling it first.

Just like find-function is only able to find where a function was defined if the function was loaded from a file, ERT is only able to find where a test was defined if the test was loaded from a file.

The should Macro
================

Test bodies can include arbitrary code; but to be useful, they need to check whether the code being tested (or code under test) does what it is supposed to do.  The macro should is similar to cl-assert from the cl package (see Assertions (Common Lisp Extensions)), but analyzes its argument form and records information that ERT can display to help debugging.

This test definition

     (ert-deftest addition-test ()
       (should (= (+ 1 2) 4)))

will produce this output when run via M-x ert:

     F addition-test
         (ert-test-failed
          ((should
            (=
             (+ 1 2)
             4))
           :form
           (= 3 4)
           :value nil))

In this example, should recorded the fact that (= (+ 1 2) 4) reduced to (= 3 4) before it reduced to nil.  When debugging why the test failed, it helps to know that the function + returned 3 here.  ERT records the return value for any predicate called directly within should.

In addition to should, ERT provides should-not, which checks that the predicate returns nil, and should-error, which checks that the form called within it signals an error.  An example use of should-error:

     (ert-deftest test-divide-by-zero ()
       (should-error (/ 1 0)
                     :type 'arith-error))

This checks that dividing one by zero signals an error of type arith-error.  The :type argument to should-error is optional; if absent, any type of error is accepted. should-error returns an error description of the error that was signaled, to allow additional checks to be made.  The error description has the format (ERROR-SYMBOL . DATA).

There is no should-not-error macro since tests that signal an error fail anyway, so should-not-error is effectively the default.

See Understanding Explanations, for more details on what should reports.

Expected Failures
=================

Some bugs are complicated to fix, or not very important, and are left as known bugs.  If there is a test case that triggers the bug and fails, ERT will alert you of this failure every time you run all tests.  For known bugs, this alert is a distraction.  The way to suppress it is to add :expected-result :failed to the test definition:

     (ert-deftest future-bug ()
       "Test `time-forward' with negative arguments.
     Since this functionality isn't implemented, the test is known to fail."
       :expected-result :failed
       (time-forward -1))

ERT will still display a small f in the progress bar as a reminder that there is a known bug, and will count the test as failed, but it will be quiet about it otherwise.

An alternative to marking the test as a known failure this way is to delete the test.  This is a good idea if there is no intent to fix it, i.e., if the behavior that was formerly considered a bug has become an accepted feature.

In general, however, it can be useful to keep tests that are known to fail.  If someone wants to fix the bug, they will have a very good starting point: an automated test case that reproduces the bug.  This makes it much easier to fix the bug, demonstrate that it is fixed, and prevent future regressions.

ERT displays the same kind of alerts for tests that pass unexpectedly as it displays for unexpected failures.  This way, if you make code changes that happen to fix a bug that you weren't aware of, you will know to remove the :expected-result clause of that test and close the corresponding bug report, if any.

Since :expected-result evaluates its argument when the test is loaded, tests can be marked as known failures only on certain Emacs versions, specific architectures, etc.:

     (ert-deftest foo ()
       "A test that is expected to fail on Emacs 23 but succeed elsewhere."
       :expected-result (if (string-match "GNU Emacs 23[.]" (emacs-version))
                            :failed
                          :passed)
       ...)

Tests and Their Environment
===========================

Sometimes, it doesn't make sense to run a test due to missing preconditions.  A required Emacs feature might not be compiled in, the function to be tested could call an external binary which might not be available on the test machine, you name it.  In this case, the macros skip-when or skip-unless could be used to skip the test. (The skip-when macro was added in Emacs 30.1.  If you need your tests to be compatible with older versions of Emacs, use skip-unless instead.)

For example, this test is skipped on MS-Windows and macOS:

     (ert-deftest test-gnu-linux ()
       "A test that is not relevant on MS-Windows and macOS."
       (skip-when (memq system-type '(windows-nt ns))
       ...))

This test is skipped if the feature dbusbind is not present in the running Emacs:

     (ert-deftest test-dbus ()
       "A test that checks D-BUS functionality."
       (skip-unless (featurep 'dbusbind))
       ...)

The outcome of running a test should not depend on the current state of the environment, and each test should leave its environment in the same state it found it in.  In particular, a test should not depend on any Emacs customization variables or hooks, and if it has to make any changes to Emacs's state or state external to Emacs (such as the file system), it should undo these changes before it returns, regardless of whether it passed or failed.

Tests should not depend on the environment because any such dependencies can make the test brittle or lead to failures that occur only under certain circumstances and are hard to reproduce.  Of course, the code under test may have settings that affect its behavior.  In that case, it is best to make the test let-bind all such setting variables to set up a specific configuration for the duration of the test.  The test can also set up a number of different configurations and run the code under test with each.

Tests that have side effects on their environment should restore it to its original state because any side effects that persist after the test can disrupt the workflow of the programmer running the tests.  If the code under test has side effects on Emacs's current state, such as on the current buffer or window configuration, the test should create a temporary buffer for the code to manipulate (using with-temp-buffer), or save and restore the window configuration (using save-window-excursion), respectively.  For aspects of the state that can not be preserved with such macros, cleanup should be performed with unwind-protect, to ensure that the cleanup occurs even if the test fails.

An exception to this are messages that the code under test prints with message and similar logging; tests should not bother restoring the *Message* buffer to its original state.

The above guidelines imply that tests should avoid calling highly customizable commands such as find-file, except, of course, if such commands are what they want to test.  The exact behavior of find-file depends on many settings such as find-file-wildcards, enable-local-variables, and auto-mode-alist.  It is difficult to write a meaningful test if its behavior can be affected by so many external factors.  Also, find-file has side effects that are hard to predict and thus hard to undo: It may create a new buffer or reuse an existing buffer if one is already visiting the requested file; and it runs find-file-hook, which can have arbitrary side effects.

Instead, it is better to use lower-level mechanisms with simple and predictable semantics like with-temp-buffer, insert or insert-file-contents-literally, and to activate any desired mode by calling the corresponding function directly, after binding the hook variables to nil.  This avoids the above problems.

Useful Techniques when Writing Tests
====================================

Testing simple functions that have no side effects and no dependencies on their environment is easy.  Such tests often look like this:

     (ert-deftest ert-test-mismatch ()
       (should (eql (cl-mismatch "" "") nil))
       (should (eql (cl-mismatch "" "a") 0))
       (should (eql (cl-mismatch "a" "a") nil))
       (should (eql (cl-mismatch "ab" "a") 1))
       (should (eql (cl-mismatch "Aa" "aA") 0))
       (should (eql (cl-mismatch '(a b c) '(a b d)) 2)))

This test calls the function cl-mismatch several times with various combinations of arguments and compares the return value to the expected return value.  (Some programmers prefer (should (eql EXPECTED ACTUAL)) over the (should (eql ACTUAL EXPECTED)) shown here.  ERT works either way.)

Here's a more complicated test:

     (ert-deftest ert-test-record-backtrace ()
       (let ((test (make-ert-test :body (lambda () (ert-fail "foo")))))
         (let ((result (ert-run-test test)))
           (should (ert-test-failed-p result))
           (with-temp-buffer
             (ert--print-backtrace (ert-test-failed-backtrace result))
             (goto-char (point-min))
             (end-of-line)
             (let ((first-line (buffer-substring-no-properties
                                (point-min) (point))))
               (should (equal first-line
                              "  signal(ert-test-failed (\"foo\"))")))))))

This test creates a test object using make-ert-test whose body will immediately signal failure.  It then runs that test and asserts that it fails.  Then, it creates a temporary buffer and invokes ert--print-backtrace to print the backtrace of the failed test to the current buffer.  Finally, it extracts the first line from the buffer and asserts that it matches what we expect.  It uses buffer-substring-no-properties and equal to ignore text properties; for a test that takes properties into account, buffer-substring and equal-including-properties could be used instead.

The reason why this test only checks the first line of the backtrace is that the remainder of the backtrace is dependent on ERT's internals as well as whether the code is running interpreted or compiled.  By looking only at the first line, the test checks a useful property---that the backtrace correctly captures the call to signal that results from the call to ert-fail---without being brittle.

This example also shows that writing tests is much easier if the code under test was structured with testing in mind.

For example, if ert-run-test accepted only symbols that name tests rather than test objects, the test would need a name for the failing test, which would have to be a temporary symbol generated with make-symbol, to avoid side effects on Emacs's state.  Choosing the right interface for ert-run-tests allows the test to be simpler.

Similarly, if ert--print-backtrace printed the backtrace to a buffer with a fixed name rather than the current buffer, it would be much harder for the test to undo the side effect.  Of course, some code somewhere needs to pick the buffer name.  But that logic is independent of the logic that prints backtraces, and keeping them in separate functions allows us to test them independently.

A lot of code that you will encounter in Emacs was not written with testing in mind.  Sometimes, the easiest way to write tests for such code is to restructure the code slightly to provide better interfaces for testing.  Usually, this makes the interfaces easier to use as well.

erts files
==========

Many relevant Emacs tests depend on comparing the contents of a buffer before and after executing a particular function.  These tests can be written the normal way---making a temporary buffer, inserting the ``before'' text, running the function, and then comparing with the expected ``after'' text.  However, this often leads to test code that's pretty difficult to read and write, especially when the text in question is multi-line.

So ert provides a function called ert-test-erts-file that takes two parameters: the name of a specially-formatted erts file, and (optionally) a function that performs the transform.

These erts files can be edited with the erts-mode major mode.

An erts file is divided into sections by the (=-=) separator.

Here's an example file containing two tests:

     Name: flet

     =-=
     (cl-flet ((bla (x)
     (* x x)))
     (bla 42))
     =-=
     (cl-flet ((bla (x)
                 (* x x)))
       (bla 42))
     =-=-=

     Name: defun

     =-=
     (defun x ()
       (print (quote ( thingy great
                       stuff))))
     =-=-=

A test starts with a line containing just =-= and ends with a line containing just =-=-=.  The test may be preceded by freeform text (for instance, comments), and also name/value pairs (see below for a list of them).

If there is a line with =-= inside the test, that designates the start of the ``after'' text.  Otherwise, the ``before'' and ``after'' texts are assumed to be identical, which you typically see when writing indentation tests.

ert-test-erts-file puts the ``before'' section into a temporary buffer, calls the transform function, and then compares with the ``after'' section.

Here's an example usage:

     (ert-test-erts-file "elisp.erts"
                         (lambda ()
                           (emacs-lisp-mode)
                           (indent-region (point-min) (point-max))))

A list of the name/value specifications that can appear before a test follows.  The general syntax is Name: Value, but continuation lines can be used (along the same lines as in mail---subsequent lines that start with a space are part of the value).

     Name: foo
     Code: (indent-region
             (point-min) (point-max))

Name
     All tests should have a name.  This name will appear in ERT output if the test fails, and helps to identify the failing test.

Code
     This is the code that will be run to do the transform.  This can also be passed in via the ert-test-erts-file call, but Code overrides that.  It's used not only in the following test, but in all subsequent tests in the file (until overridden by another Code specification).

No-Before-Newline
No-After-Newline
     These specifications say whether the ``before'' or ``after'' portions have a newline at the end.  (This would otherwise be impossible to specify.)

Point-Char
     Sometimes it's useful to be able to put point at a specific place before executing the transform function.  Point-Char: | will make ert-test-erts-file place point where | is in the ``before'' form (and remove that character), and will check that it's where the | character is in the ``after'' form (and issue a test failure if that isn't the case).  (This is used in all subsequent tests, unless overridden by a new Point-Char spec.)

Skip
     If this is present and value is a form that evaluates to a non-nil value, the test will be skipped.

If you need to use the literal line single line =-= in a test section, you can quote it with a \ character.

Syntax Highlighting Tests
=========================

Syntax highlighting is normally provided by the Font Lock minor mode that assigns face properties to parts of the buffer.  The ert-font-lock package makes it possible to introduce unit tests checking face assignment.  Test assertions are included in code-level comments directly and can be read either from inline strings or files. The parser expects the input string to contain at least one assertion.

The test assertion parser extracts tests from comment-only lines.  Every comment assertion line starts either with a caret (^) or an arrow (<-).  A single caret/arrow or carets should be followed immediately by the name of a face or a list of faces to be checked against the :face property at point.

The test then checks if the first non-assertion column above the caret contains a face expected by the assertion:

     var variable = 11;
     //   ^ font-lock-variable-name-face
     //             ^ font-lock-literal-face
     //               ^ font-lock-punctuation-face
     // this is not an assertion, it's just a comment
     //   ^ font-lock-comment-face

     // multiple carets per line
     // ^^^^     ^    ^ font-lock-comment-face

Both symbol-only :face property values and assertion face values are normalized to single element lists so the assertions below are equivalent:

     // single
     // ^ font-lock-comment-face
     // single
     // ^ (font-lock-comment-face)

Assertions can be negated:

     var variable = 11;
     //  ^ !font-lock-comment-face

It is possible to specify face lists in assertions:

     // TODO
     // ^^^^ (font-lock-comment-face hl-todo)
          var test = 1;
     // ^    ()
     // ^    nil
     //   negation works as expected
     //   ^  !nil

The arrow (<-) means that the first non-empty column of the assertion line will be used for the check:

     var variable = 1;
     // <- font-lock-keyword-face
       11;
        // <- font-lock-literal-face

The ert-font-lock-test-string function extracts ERT assertions from an inline string.  The javascript-mode symbol below specifies the major mode used for comments and font locking:

     (ert-deftest test-font-lock-test-string--correct ()
       (ert-font-lock-test-string
        "
     var abc = function(d) {
     // <- font-lock-keyword-face
     //   ^ font-lock-variable-name-face
         //        ^ font-lock-keyword-face
         //             ^ font-lock-variable-name-face
     };
     "
        'javascript-mode))

It is also possible to extract test assertions from a file:

     (ert-deftest test-font-lock-test-file--correct ()
       (ert-font-lock-test-file
        (ert-resource-file "correct.js")
        'javascript-mode))

The ert-font-lock-deftest macro simplifies inline test definition:

     (ert-font-lock-deftest test-macro-test--inline
       emacs-lisp-mode
       "
     (defun fun ())
     ;; ^ font-lock-keyword-face
     ;;      ^ font-lock-function-name-face")

The ert-font-lock-deftest-file macro reads assertions from a file:

     (ert-font-lock-deftest-file test-macro-test--file
       "Test reading correct assertions from a file."
       javascript-mode
       "correct.js")

The ert-font-lock-deftest and ert-font-lock-deftest-file macros accept the same keyword arguments as ert-deftest, i.e., :tag and :expected-result.

How to Debug Tests
******************

This section describes how to use ERT's features to understand why a test failed.

Understanding Explanations
==========================

Failed should forms are reported like this:

     F addition-test
         (ert-test-failed
          ((should
            (=
             (+ 1 2)
             4))
           :form
           (= 3 4)
           :value nil))

ERT shows what the should expression looked like and what values its subexpressions had: The source code of the assertion was (should (= (+ 1 2) 4)), which applied the function = to the arguments 3 and 4, resulting in the value nil.  In this case, the test is wrong; it should expect 3 rather than 4.

If a predicate like equal is used with should, ERT provides a so-called explanation:

     F list-test
         (ert-test-failed
          ((should
            (equal
             (list 'a 'b 'c)
             '(a b d)))
           :form
           (equal
            (a b c)
            (a b d))
           :value nil :explanation
           (list-elt 2
                     (different-atoms c d))))

In this case, the function equal was applied to the arguments (a b c) and (a b d).  ERT's explanation shows that the item at index 2 differs between the two lists; in one list, it is the atom c, in the other, it is the atom d.

In simple examples like the above, the explanation is unnecessary. But in cases where the difference is not immediately apparent, it can save time:

     F test1
         (ert-test-failed
          ((should
            (equal x y))
           :form
           (equal a a)
           :value nil :explanation
           (different-symbols-with-the-same-name a a)))

ERT only provides explanations for predicates that have an explanation function registered.  See Defining Explanation Functions.

Interactive Debugging
=====================

Debugging failed tests essentially works the same way as debugging any other problems with Lisp code.  Here are a few tricks specific to tests:

  * Re-run the failed test a few times to see if it fails in the same way each time.  It's good to find out whether the behavior is deterministic before spending any time looking for a cause.  In the ERT results buffer, r re-runs the selected test.

  * Use . to jump to the source code of the test to find out exactly what it does.  Perhaps the test is broken rather than the code under test.

  * If the test contains a series of should forms and you can't tell which one failed, use l, which shows you the list of all should forms executed during the test before it failed.

  * Use b to view the backtrace.  You can also use d to re-run the test with debugging enabled, this will enter the debugger and show the backtrace as well; but the top few frames shown there will not be relevant to you since they are ERT's own debugger hook.  b strips them out, so it is more convenient.

  * If the test or the code under testing prints messages using message, use m to see what messages it printed before it failed.  This can be useful to figure out how far it got.

  * You can instrument tests for debugging the same way you instrument defuns for debugging: go to the source code of the test and type C-u C-M-x.  Then, go back to the ERT buffer and re-run the test with r or d.

  * If you have been editing and rearranging tests, it is possible that ERT remembers an old test that you have since renamed or removed: renamings or removals of definitions in the source code leave around a stray definition under the old name in the running process (this is a common problem in Lisp).  In such a situation, hit D to let ERT forget about the obsolete test.

Extending ERT
*************

There are several ways to add functionality to ERT.

Defining Explanation Functions
==============================

The explanation function for a predicate is a function that takes the same arguments as the predicate and returns an explanation. The explanation should explain why the predicate, when invoked with the arguments given to the explanation function, returns the value that it returns.  The explanation can be any object but should have a comprehensible printed representation.  If the return value of the predicate needs no explanation for a given list of arguments, the explanation function should return nil.

To associate an explanation function with a predicate, add the property ert-explainer to the symbol that names the predicate. The value of the property should be the symbol that names the explanation function.

Low-Level Functions for Working with Tests
==========================================

Both ert-run-tests-interactively and ert-run-tests-batch are implemented on top of the lower-level test handling code in the sections of ert.el labeled ``Facilities for running a single test'', ``Test selectors'', and ``Facilities for running a whole set of tests''.

If you want to write code that works with ERT tests, you should take a look at this lower-level code.  Symbols that start with ert-- are internal to ERT, whereas those that start with ert- are meant to be usable by other code.  But there is no mature API yet.

Contributions to ERT are welcome.

Other Testing Concepts
**********************

For information on mocks, stubs, fixtures, or test suites, see below.

Other Tools for Emacs Lisp
==========================

Stubbing out functions or using so-called mocks can make it easier to write tests.  See https://en.wikipedia.org/wiki/Mock_object for an explanation of the corresponding concepts in object-oriented languages.

ERT does not have built-in support for mocks or stubs.  The package el-mock (see https://www.emacswiki.org/emacs/el-mock.el) offers mocks for Emacs Lisp and can be used in conjunction with ERT.

Fixtures and Test Suites
========================

In many ways, ERT is similar to frameworks for other languages like SUnit or JUnit.  However, two features commonly found in such frameworks are notably absent from ERT: fixtures and test suites.

Fixtures are mainly used (e.g., in SUnit or JUnit) to provide an environment for a set of tests, and consist of set-up and tear-down functions.

While fixtures are a useful syntactic simplification in other languages, this does not apply to Lisp, where higher-order functions and unwind-protect are available.  One way to implement and use a fixture in ERT is

     (defun my-fixture (body)
       (unwind-protect
           (progn [set up]
                  (funcall body))
         [tear down]))

     (ert-deftest my-test ()
       (my-fixture
        (lambda ()
          [test code])))

(Another way would be a with-my-fixture macro.)  This solves the set-up and tear-down part, and additionally allows any test to use any combination of fixtures, so it is more flexible than what other tools typically allow.

If the test needs access to the environment the fixture sets up, the fixture can be modified to pass arguments to the body.

These are well-known Lisp techniques.  Special syntax for them could be added but would provide only a minor simplification.

(If you are interested in such syntax, note that splitting set-up and tear-down into separate functions, like *Unit tools usually do, makes it impossible to establish dynamic let bindings as part of the fixture.  So, blindly imitating the way fixtures are implemented in other languages would be counter-productive in Lisp.)

The purpose of test suites is to group related tests together.

The most common use of this is to run just the tests for one particular module.  Since symbol prefixes are the usual way of separating module namespaces in Emacs Lisp, test selectors already solve this by allowing regexp matching on test names; e.g., the selector "^ert-" selects ERT's self-tests.

Other uses include grouping tests by their expected execution time, e.g., to run quick tests during interactive development and slow tests less often.  This can be achieved with the :tag argument to ert-deftest and tag test selectors.

GNU Free Documentation License
******************************

     [file doclicense.texi not included]