.. _library_mutation_testing:

``mutation_testing``
====================

This tool provides mutation testing support for loaded Logtalk entities.
It's still under development and breaking changes should be expected.

API documentation
-----------------

This tool API documentation is available at:

`../../apis/library_index.html#mutation-testing <../../apis/library_index.html#mutation-testing>`__

Loading
-------

This tool can be loaded using the query:

::

   | ?- logtalk_load(mutation_testing(loader)).

Testing
-------

To test this tool, load the ``tester.lgt`` file:

::

   | ?- logtalk_load(mutation_testing(tester)).

Some tests depend on the availability of the
``test_files/mutation-testing-report-schema.json`` file. This file can
be downloaded from:

https://github.com/stryker-mutator/mutation-testing-elements

Features
--------

- Deterministic mutant discovery per entity predicate directive,
  predicate/non-terminal clause/rule, and mutator.
- Configurable mutator sets and campaign size limits.
- Deterministic sampled execution (``sampling(all|count(N)|rate(R))``
  plus ``seed/1``).
- Mutation score computation (killed versus survived mutants).
- Mutation generation guided by code coverage stats.
- Threshold gating suitable for CI/CD checks.
- Exporting of mutation campaign reports in plain text and JSON formats.

Default mutators
----------------

The default mutators are:

- | ``fail_insertion``
  | Inserts failure in the selected predicate/non-terminal rule bodies.

- | ``body_goal_negation``
  | Negates the selected predicate/non-terminal clause behavior.

- | ``relational_operator_replacement``
  | Replaces a relational operator (e.g. ``>``, ``<``, ``@=<``, ``==``)
    with a complementary alternative.

- | ``arithmetic_operator_replacement``
  | Replaces an arithmetic operator in selected expressions.

- | ``truth_literal_flip``
  | Flips truth literals (``true`` <-> ``fail``) in selected goals.

- | ``head_arguments_mutation``
  | Mutates one compile-time bound head argument using
    ``type::mutation/3``.

- | ``head_arguments_reordering``
  | Reorders head arguments (swapping the first two arguments).

- | ``clauses_reordering``
  | Reorders predicate/non-terminal clauses/rules by swapping a clause
    with its successor.

- | ``scope_directive_replacement``
  | Replaces a matching predicate/non-terminal scope directive
    visibility with an alternative visibility.

- | ``predicate_directive_suppression``
  | Suppresses a matching predicate/non-terminal directive.

- | ``uses_directive_resource_deletion``
  | Deletes one matching resource from a ``uses/2`` directive.

Usage
-----

Mutation testing **requires** an existing set of tests for the code
being mutated. These tests are used to verify if they are able to kill
the generated mutants or if they are too weak allowing the mutants to
survive.

By default, the tool looks for the tests driver file (usually,
``tester.lgt``) in the directory of the entity, library, or directory
being tested. If the tests driver file is not found there, the tool
looks into the startup directory of the Logtalk process.

Both the name of the tests driver file and its directory can be set
using the ``tester_file_name/1`` and ``tester_directory/1`` options if
the defaults don't work in your particular case.

In this tool documentation, **campaign predicates** refers to the
following public predicates:

- ``entity/1-2``
- ``predicate/2-3``
- ``library/1-2``
- ``directory/1-2``

These predicates run mutants and report results in one step.

Run mutation testing for one entity using defaults:

::

   | ?- mutation_testing::entity(my_object).

List generated mutants for one entity:

::

   | ?- mutation_testing::entity_mutants(my_object, Mutants).

Run with custom options:

::

   | ?- mutation_testing::entity(my_object, [
       mutators([fail_insertion, body_goal_negation]),
       max_mutations_per_mutator(5),
       sampling(count(100)),
       seed(20260303),
       max_mutators(100),
       threshold(60.0),
       verbose(true),
       print_mutation(true)
   ]).

Run campaigns for all loaded entities from a specific library:

::

   | ?- mutation_testing::library(core).

Run campaigns for all loaded entities from a specific directory:

::

   | ?- mutation_testing::directory('/path/to/sources').

Pretty-print a report term using the default text format:

::

   | ?- mutation_testing::report_entity(my_object, Report, []),
        mutation_testing::format_report(Report).

Suppress report formatting output for campaign predicates:

::

   | ?- mutation_testing::entity(my_object, [format(none)]).

Execution uses ``lgtunit`` test sets only. When using
``timeout(Timeout)``, timeout handling is best-effort.

When running mutation campaigns inside another ``lgtunit`` test run,
per-mutant ``killed`` versus ``survived`` classification may be affected
by nested test-run message and event handling. Campaign summary
accounting remains deterministic.

Limitations
-----------

- Mutation points are currently mostly clause-level (one occurrence per
  clause per mutator), except ``head_arguments_mutation`` which creates
  one occurrence per compile-time bound head argument with supported
  type; internal expression-level candidate enumeration is not
  implemented.
- Equivalent mutant detection and duplicate mutant pruning are not
  implemented.
- Built-in mutation apply/revert currently targets loaded source
  entities (objects/categories), not protocols.
- Nested ``lgtunit`` runs may affect per-mutant ``killed`` vs
  ``survived`` status classification.

Options
-------

- | ``exclude_entities(Entities)``
  | List of loaded entities to exclude (default ``[]``).

- | ``include_entities(Entities)``
  | List of loaded entities to include (default ``[]`` meaning all).

- | ``max_mutators(Max)``
  | Maximum number of discovered mutators to use when ``mutators/1`` is
    not explicitly provided (``all`` or positive integer; default
    ``all``).

- | ``max_mutations_per_mutator(Max)``
  | Maximum number of mutations generated per mutator and
    predicate/non-terminal (``all`` or positive integer; default ``5``).

- | ``mutators(Mutators)``
  | Names of the mutators to use. When set to ``[]`` (default), mutators
    are auto-discovered and optionally limited by ``max_mutators/1``.

- | ``sampling(Sampling)``
  | Mutant selection strategy (``all``, ``count(N)``, or ``rate(R)``
    where ``0.0 =< R =< 1.0``; default ``all``). See below for full
    details.

- | ``seed(Seed)``
  | Pseudo-random generator seed for reproducible sampling (default
    ``123456789``; ignored when ``sampling(all)``).

- | ``timeout(Timeout)``
  | Maximum time in seconds for each mutant execution (positive number;
    default ``300``). Timeout handling is best-effort.

- | ``threshold(Threshold)``
  | Minimum mutation score in range ``0.0..100.0`` (default ``0.0``).

- | ``print_mutation(Boolean)``
  | When ``true``, prints original and mutated terms with source
    location for mutators (default ``false``).

- | ``verbose(Boolean)``
  | Print per-mutant test results (default ``false``).

- ``format(Format)`` Controls report formatting output (``none``,
  ``text``, or ``json``; default ``text``). When set to ``text`` or
  ``json``, a report file is generated automatically for campaign
  predicates. The ``json`` format implements the Stryker Mutation
  Testing Framework report format: https://stryker-mutator.io/

- ``report_file_name(FileName)`` Report output file base name or path
  without extension (atom; default ``mutation_test_report``). The
  extension is inferred from ``format/1`` (``text`` -> ``.txt``,
  ``json`` -> ``.json``). When not absolute, the file is saved in the
  tests driver directory.

- | ``tester_file_name(Tester)``
  | Name of the tests driver file for the code being tested (default
    ``tester.lgt``).

- | ``tester_directory(Directory)``
  | Full path to the directory containing the tests driver file for the
    code being tested (no default).

- | ``tester_options(Options)``
  | Additional options for the calls to the ``logtalk_tester`` script
    that are used to check mutations. Note that the most common options
    are already covered by explicit options (e.g., ``timeout/1``).
    Options without arguments are specified using the corresponding
    character. Options that take a value are specified using a compound
    term whose name is the option character and whose single argument is
    the option value (default ``[o(quiet)]``). Invalid options are
    ignored.

Sampling semantics
------------------

The ``sampling(Sampling)`` option controls **which generated mutants are
actually executed** in a mutation campaign. It is a post-generation
selection step:

1. Generate candidate mutants for the selected
   entities/predicates/mutators.
2. Apply ``sampling(...)`` to select a subset (or all).
3. Execute only the selected mutants.

Because of this design, ``sampling(...)`` affects campaign predicates
(``entity/2``, ``predicate/3``, ``library/2``, ``directory/2``).

The ``format(Format)`` option affects only campaign predicates that
execute and report in one step (``entity/2``, ``predicate/3``,
``library/2``, ``directory/2``). It does not affect the ``*_mutants``
predicates or the ``report_*`` predicates, which only compute and return
terms.

For campaign predicates, a report file is written automatically unless
``format(none)`` is used.

For ``report_library/3`` and ``report_directory/3``, call
``format_report/2-3`` explicitly if you want to persist the computed
report term. A single ``format_report(..., Report)`` call always
generates a single output document (including JSON).

The ``mutants/2-3`` predicates are still useful for inspecting the
generated deterministic mutant list itself; they are not
execution/reporting predicates.

Sampling modes
~~~~~~~~~~~~~~

- ``sampling(all)`` Execute all generated mutants.

- ``sampling(count(N))`` Shuffle the generated mutant list and execute
  up to ``N`` mutants. If ``N`` is greater than the total number of
  generated mutants, all mutants are executed.

- ``sampling(rate(R))`` Compute ``round(R * TotalGeneratedMutants)``,
  shuffle the mutant list, and execute that many mutants. If the
  computed value is greater than the total number of generated mutants,
  all mutants are executed.

The randomization used by ``count/1`` and ``rate/1`` is controlled by
the ``seed/1`` option, allowing reproducible selection.

Sampling usage in practice
~~~~~~~~~~~~~~~~~~~~~~~~~~

- **Control campaign cost** Run a representative subset when the full
  mutant set is large.

- **Trade off speed vs. confidence** Use smaller samples for quick
  feedback and larger/full runs for stronger confidence before release.

- **Enable reproducible sampling** With a fixed ``seed/1``, repeated
  runs with the same generated mutant set and same sampling options
  select the same subset. When all mutants are executed, (using
  ``sampling(all)``) changing the seed has no effect on selection.

- **Complement other limits** ``max_mutators/1`` and
  ``max_mutations_per_mutator/1`` constrain mutant generation;
  ``sampling/1`` controls execution selection after generation.

Defining new mutators
---------------------

Define a new mutator as parametric object implementing the ``expanding``
protocol, either the ``clause_mutator_protocol`` or the
``directive_mutator_protocol`` marker protocol, and importing the
``mutator_common`` category:

::

   :- object(my_mutator(_Entity_, _Predicate_, _TargetIndex_, _Occurrence_, _PrintMutation_),
       implements((expanding, clause_mutator_protocol)),
       imports(mutator_common)).

       % mutate code when loading it using this object as a hook object
       term_expansion(Term, Mutation) :-
           ...

       % compute a term mutation; generate multiple mutations by backtracking
       mutation(Term, Mutation) :-
           ...

       % reset any required internal state
       reset :-
           ...

   :- end_object.

New mutators are found by dynamically looking for objects that conform
to either the ``clause_mutator_protocol`` or the
``directive_mutator_protocol`` marker protocols.

For implementation examples, see the default mutators:

- `mutators/fail_insertion.lgt <mutators/fail_insertion.lgt>`__
- `mutators/body_goal_negation.lgt <mutators/body_goal_negation.lgt>`__
- `mutators/relational_operator_replacement.lgt <mutators/relational_operator_replacement.lgt>`__
- `mutators/arithmetic_operator_replacement.lgt <mutators/arithmetic_operator_replacement.lgt>`__
- `mutators/truth_literal_flip.lgt <mutators/truth_literal_flip.lgt>`__
- `mutators/head_arguments_mutation.lgt <mutators/head_arguments_mutation.lgt>`__
- `mutators/head_arguments_reordering.lgt <mutators/head_arguments_reordering.lgt>`__
- `mutators/clauses_reordering.lgt <mutators/clauses_reordering.lgt>`__
- `mutators/scope_directive_replacement.lgt <mutators/scope_directive_replacement.lgt>`__
