repository·main·Indexed 21 days ago
https://github.com/lambdaisland/kaochaA full-featured, next-generation test runner for Clojure that supports multiple test types (e.g., clojure.test, Midje) and build tools including Clojure CLI (tools.deps), Leiningen, Boot, and Babashka. Key features include test filtering, watch mode, pluggable reporting, randomized test order, fail-fast mode, and profiling. It is primarily configured via a tests.edn file and requires Clojure 1.9+ (or Clojure/ClojureScript 1.10+ for ClojureScript support).
A testable is a map representing a unit of testing. It contains the testable's type, a unique ID, type-specific information, and potentially nested testables.
Testables exist in three distinct states depending on the lifecycle stage:
:kaocha/tests key in the configuration.load step, testables are enriched with metadata and nested child testables (e.g., a namespace testable containing individual var testables).run step, testables are updated with execution results such as :kaocha.result/count, :kaocha.result/pass, :kaocha.result/fail, and :kaocha.result/error.;; Example of a top-level testable in the Configuration stage
{:kaocha/tests [{:kaocha.testable/type :kaocha.type/clojure.test
:kaocha.testable/id :unit
:kaocha/source-paths ["src"]
:kaocha/test-paths ["test"]}]}
;; Example of a testable in the Test Plan stage (nested)
{:kaocha.test-plan/tests [{:kaocha.testable/type :kaocha.type/clojure.test
:kaocha.testable/id :unit
:kaocha/source-paths ["src"]
:kaocha/test-paths ["test"]
:kaocha.testable/meta {}
:kaocha.test-plan/tests [{:kaocha.testable/type :kaocha.type/ns
:kaocha.testable/id :kaocha.runner-test
,,,
:kaocha.test-plan/tests [{:kaocha.testable/type :kaocha.testable/var
:kaocha.testable/id :kaocha.runner-test/main-test
,,,}]}]}]}
;; Example of a testable in the Test Result stage (with results)
{:kaocha.result/tests [{:kaocha.testable/type :kaocha.type/clojure.test
:kaocha.testable/id :unit
:kaocha/source-paths ["src"]
:kaocha/test-paths ["test"]
:kaocha.testable/meta {}
:kaocha.result/tests [{:kaocha.testable/type :kaocha.type/ns
:kaocha.testable/id :kaocha.runner-test
,,,
:kaocha.result/tests [{:kaocha.testable/type :kaocha.testable/var
:kaocha.testable/id :kaocha.runner-test/main-test
:kaocha.result/count 1
:kaocha.result/pass 1
:kaocha.result/fail 0
:kaocha.result/error 0
,,,}]}]}]}When an assertion fails in clojure.test, Kaocha provides enhanced, pretty-printed diffs to help you visualize the differences between the expected and actual values, especially for complex data structures like maps and vectors.
(defn my-fn []
{:xxx [1 2 3]
:blue :red
"hello" {:world :!}})
(deftest my-test
(is (= {:xxx [1 3 4]
"hello" {:world :?}}
{:xxx [1 2 3]
:blue :red
"hello" {:world :!}})))org.clojure/tools.cli before starting the main command line runner. This check ensures that the version of tools.cli in your project's dependency tree supports command line flags of the form --[no-]xxx. If an outdated version is detected, Kaocha will emit an error to stderr to prevent broken flag behavior.Kaocha operates in three conceptual stages: loading configuration, loading tests, and running tests. Each stage produces a data structure (in EDN format) that you can inspect using specific flags. This is useful for debugging configuration or test discovery issues.
# Print the merged/normalized configuration and exit
bin/kaocha --print-config
# Load tests, build the plan, print it, and exit
bin/kaocha --print-test-plan
# Print the final test result map and exit
bin/kaocha --print-resultKaocha randomizes the order of test suites, namespaces, and test vars by default to help detect unintended dependencies between tests.
--seed flag with that specific value.--no-randomize flag or configure it in tests.edn.# Reproduce a run with a specific seed
bin/kaocha --seed 10761431
# Disable randomization
bin/kaocha --no-randomizeYou can limit a test run to only include tests that have specific metadata associated with them. For clojure.test type tests, metadata can be associated with a test variable (using ^:key) or a test namespace (using ^:key on the ns declaration).
To focus tests, you must provide a metadata key that has a truthy value. Kaocha will then only execute tests where that key is present and truthy.
Kaocha supports the #meta-merge reader tag (similar to Aero's #merge but performing a deep merge via meta-merge semantics). This is useful for creating a hierarchy of configuration files, such as a base tests.edn and a user-specific tests.user.edn that overrides or augments specific parts of the tree.
You can control how specific keys are merged using metadata tags:
^:replace: Replaces the existing value/subtree with the new one.^:append: Appends the new value to the existing one.^:prepend: Prepends the new value to the existing one.Note on Default Replacement: By default, the following keys are replaced rather than merged:
:kaocha/reporter:kaocha/tests:kaocha/test-paths:kaocha/source-paths:kaocha/ns-patternsTo add to the default namespace patterns instead of replacing them, use ^:append:
#kaocha/v1 {:ns-patterns ^:append ["^test-"]}#kaocha/v1
#meta-merge [{:tests [{:id :unit
:test-paths ["test/unit"]}
{:id :features
:test-paths ["test/features"]}]
:kaocha/plugins [:kaocha.plugin.some.required/plugin
,,,]
}
#include "tests.user.edn"]Reporters dictate the output format generated by Kaocha. You can configure a reporter via the --reporter CLI flag or by setting the :reporter key in your tests.edn configuration file. Reporters can be a single namespaced symbol or a vector of symbols.
Available reporters include:
kaocha.report/dots: A concise, information-rich reporter that prints a sequence of symbols (e.g., . for pass, F for fail, E for error) representing the test progress. Best for general use.kaocha.report/documentation: Provides detailed, hierarchical output of namespaces, vars, and testing blocks. Highly recommended for CI environments.kaocha.report.progress/report: Displays a progress bar for each test suite, showing percentage and completion count. Bars turn red on failure.kaocha.report/tap: Outputs results using the Test Anything Protocol (TAP), useful for integration with external tools.kaocha.report/debug: Prints the raw clojure.test style events map (with some noise filtered out) for debugging purposes.;; Example configuration using documentation reporter
{:kaocha/reporter [kaocha.report/documentation]}Pending tests are used for tests that are not yet implemented or require fixing. Unlike skipped tests, pending tests are explicitly reported in the test results at the end of the run, providing a list of their test IDs and file/line information to ensure they are not forgotten.
To mark a test as pending, add the ^:kaocha/pending metadata to the test definition. This metadata is supported by any test type that allows setting metadata tags (e.g., ClojureScript, Cucumber).
(deftest ^:kaocha/pending my-test)Kaocha uses a data-driven approach where behavior is modified by passing data structures through hook functions. A hook is a function supplied at a specific point in the Kaocha process. It receives a data structure (such as config, test-plan, or testable), can perform side effects, and can return an updated version of that data structure for Kaocha to continue using.
Crucial Rule: Hooks should always return their first argument (or a modified version of it) so that Kaocha has the necessary data to proceed. If a hook only performs side effects, it must still return the original data structure unchanged.
(defn my-hook [data]
(println "Doing something...")
data) ;; Always return the dataA Kaocha test run follows a three-step lifecycle, transforming data structures at each stage. Understanding this flow is essential for building plugins or custom test types.
:kaocha/config).kaocha.testable/-load multimethod. The configuration transforms into a test plan (spec: :kaocha/test-plan), which contains a nested collection of "testables".:kaocha/result).