diff --git a/src/extensions/docs/index.rst b/src/extensions/docs/index.rst index 0ae5047f4..28bc0394a 100644 --- a/src/extensions/docs/index.rst +++ b/src/extensions/docs/index.rst @@ -63,6 +63,17 @@ Extensions `ubCode `__ VS Code extension. Getting IDE support for Sphinx-Needs in a Bazel context made easy. + .. grid-item-card:: + + Module Verification Report + ^^^ + One Need per module gives that module a report page whose sections behave + like ordinary RST. + :ref:`Module Verification Report`, + or jump straight to the + :ref:`live example`. + + .. grid-item-card:: Mounts @@ -80,4 +91,6 @@ Extensions Source Code Linker Extension Guide Sync TOML + Module Verification Report + Module Verification Report Example mounts_internals diff --git a/src/extensions/docs/module_verification_report.md b/src/extensions/docs/module_verification_report.md new file mode 100644 index 000000000..b36b2088b --- /dev/null +++ b/src/extensions/docs/module_verification_report.md @@ -0,0 +1,111 @@ + + +(module-verification-report)= +# Module Verification Report + +Give a module a verification report page by writing **one Need**. The report's +sections are ordinary RST sections: they appear in the sidebar and the local +ToC, they are `:ref:`-able from other pages, they land in the search index, and +they survive into non-HTML builders. + +--- + +## Authoring + +```rst +.. mod_ver_report:: Baselibs Verification Report + :id: mod_vrep__baselibs + :belongs_to: mod__baselibs + :covers: comp__baselibs_json, comp__baselibs_bit_manipulation + :safety: ASIL_B + :security: NO + :status: valid + :verification_method: test + :titles: + comp__baselibs_json = JSON Utilities + + Free-form introduction. It becomes the Need's description. +``` + +Scaling to N modules means adding N Needs — nothing else. + +### Options + +| Option | Meaning | +| ------ | ------- | +| *argument* | Report title. | +| `:id:` | Report Need id. Mandatory; also namespaces every generated anchor. | +| `:belongs_to:` | The module this report is about. | +| `:covers:` | The components in scope. A **real link field**, comma and/or whitespace separated. | +| `:titles:` | Optional presentation-only heading overrides, one `id = Heading` per line. | +| *anything else* | Forwarded verbatim to the Need. The metamodel decides what is valid. | + +Everything except `:titles:` ends up on the Need, so `covers_back` and the usual +link validation come for free. + +### Generated sections + +1. **Report Metadata** — the report's own fields. +2. **Verification Scope** — the covered components. +3. **One section per covered component** — a `:need:` reference plus a table of + everything related to it. +4. **Verification Evidence** — whatever links to the report via `contains` or + `evidence`. + +Anchors are namespaced with the report id, e.g. + +```rst +See :ref:`mod_vrep__baselibs__comp__baselibs_json`. +``` + +so they are stable across rebuilds and two reports on one page never collide. + +--- + +## Why the component list has to be written out + +Sphinx turns headings into sections exactly once, during the read phase, before +the Need graph exists. A report cannot therefore discover its own components +from the graph and still get real sections — the two happen at different times. + +So the list is authored, and drift is **detected rather than silently +corrected**: the metamodel check +`check_mod_ver_report_scope` compares the report's `:covers:` against the +module's `includes` in **both** directions and fails the build if they differ. +The fix is always a one-line edit. + +--- + +## Design rule for contributors + +> The extension emits RST. It never reads the Need model to compute an answer. + +The directive emits `needtable` filters and `:need:` references; sphinx-needs +resolves them after collection. That is why the extension has no `NeedsView`, no +registry, no `build-finished` pass and no build lifecycle hooks at all. + +If new report content needs Python that walks needs and computes something, the +line has been crossed. If it needs a new `needtable` filter, it has not. + +Two invariants are enforced by tests and must not be "cleaned up": + +- **The report body is a sibling of the Need, not its child.** sphinx-needs + parses Need content with `match_titles=False`; moving the body inside the Need + node silently removes every section, and the HTML still looks fine. +- **`setup()` registers directives and nothing else.** The single + `config-inited` handler exists only because directive registration is + last-one-wins. + +See `src/extensions/score_module_verification_report/README.md` for the +configuration values and the full rationale. diff --git a/src/extensions/docs/module_verification_report_example.rst b/src/extensions/docs/module_verification_report_example.rst new file mode 100644 index 000000000..4e1b43b6c --- /dev/null +++ b/src/extensions/docs/module_verification_report_example.rst @@ -0,0 +1,166 @@ +.. + # ******************************************************************************* + # Copyright (c) 2026 Contributors to the Eclipse Foundation + # + # See the NOTICE file(s) distributed with this work for additional + # information regarding copyright ownership. + # + # This program and the accompanying materials are made available under the + # terms of the Apache License Version 2.0 which is available at + # https://www.apache.org/licenses/LICENSE-2.0 + # + # SPDX-License-Identifier: Apache-2.0 + # ******************************************************************************* + +.. _module-verification-report-example: + +========================================== +Module Verification Report: Live Example +========================================== + +This page renders a real report. Everything below the "Rendered report" heading +is produced by a single ``mod_ver_report`` directive — look at the sidebar and +the local contents to see that its sections are ordinary RST sections. + +See :ref:`module-verification-report` for the reference documentation. + +.. note:: + + The architecture needs on this page exist only to give the example something + to point at. They are not part of the Docs-as-Code architecture. + +The module being reported on +============================ + +A module with two components: + +.. feat:: Example Baselibs Feature + :id: feat__example_baselibs + :version: 1 + :security: NO + :safety: QM + :status: valid + + Container feature for the example components. + +.. comp:: Example JSON Component + :id: comp__example_baselibs_json + :version: 1 + :security: NO + :safety: QM + :status: valid + :belongs_to: feat__example_baselibs + + Parses and serialises JSON. + +.. comp:: Example Bit Manipulation Component + :id: comp__example_baselibs_bits + :version: 1 + :security: NO + :safety: QM + :status: valid + :belongs_to: feat__example_baselibs + + Bit-level helpers. + +.. comp_arc_sta:: Example JSON Package Diagram + :id: comp_arc_sta__example_feature__json + :version: 1 + :security: NO + :safety: QM + :status: valid + :belongs_to: comp__example_baselibs_json + + An architecture view of the JSON component. It shows up in the JSON + component's table below because it links to that component. + +.. comp_arc_sta:: Example Bit Manipulation Package Diagram + :id: comp_arc_sta__example_feature__bits + :version: 1 + :security: NO + :safety: QM + :status: valid + :belongs_to: comp__example_baselibs_bits + + An architecture view of the bit manipulation component. + +.. workproduct:: Example Baselibs Test Report + :id: wp__example_baselibs_test_report + :version: 1 + :status: valid + + Stands in for the artefact backing the verification report. The report links + to it with ``evidence``, so it appears in the report's Verification Evidence + section. + +.. mod:: Example Baselibs Module + :id: mod__example_baselibs + :version: 1 + :security: NO + :safety: QM + :status: valid + :includes: comp__example_baselibs_json, comp__example_baselibs_bits + + The module the report below is about. + +What the author writes +====================== + +One Need. That is the whole input for the page you see below it: + +.. code-block:: rst + + .. mod_ver_report:: Example Baselibs Verification Report + :id: mod_vrep__example_feature__baselibs + :version: 1 + :belongs_to: mod__example_baselibs + :covers: comp__example_baselibs_json, comp__example_baselibs_bits + :safety: QM + :security: NO + :status: valid + :verification_method: test_and_inspection + :evidence: wp__example_baselibs_test_report + :titles: + comp__example_baselibs_json = JSON Utilities + comp__example_baselibs_bits = Bit Manipulation + + Verification report for the example Baselibs module. + +``:covers:`` is a real link field, so ``mod__example_baselibs`` gets a +``covered by`` backlink and the metamodel checks that the list matches the +module's ``includes`` in both directions. ``:titles:`` is optional; without it +the headings are derived from the component ids. + +Rendered report +=============== + +.. mod_ver_report:: Example Baselibs Verification Report + :id: mod_vrep__example_feature__baselibs + :version: 1 + :belongs_to: mod__example_baselibs + :covers: comp__example_baselibs_json, comp__example_baselibs_bits + :safety: QM + :security: NO + :status: valid + :verification_method: test_and_inspection + :evidence: wp__example_baselibs_test_report + :titles: + comp__example_baselibs_json = JSON Utilities + comp__example_baselibs_bits = Bit Manipulation + + Verification report for the example Baselibs module. + +The sections are real +===================== + +Each generated section carries an anchor namespaced with the report id, so it +can be referenced from anywhere like any other section: + +.. code-block:: rst + + See :ref:`mod_vrep__example_feature__baselibs__comp__example_baselibs_json`. + +Which renders as: :ref:`mod_vrep__example_feature__baselibs__comp__example_baselibs_json` +— the link text comes from the section title, because the target *is* a +section. The same anchors appear in the sidebar, in the local contents, in the +search index and in the LaTeX/PDF bookmarks. diff --git a/src/extensions/score_metamodel/checks/mod_ver_report.py b/src/extensions/score_metamodel/checks/mod_ver_report.py new file mode 100644 index 000000000..8028c59a0 --- /dev/null +++ b/src/extensions/score_metamodel/checks/mod_ver_report.py @@ -0,0 +1,97 @@ +# ******************************************************************************* +# Copyright (c) 2026 Contributors to the Eclipse Foundation +# +# See the NOTICE file(s) distributed with this work for additional +# information regarding copyright ownership. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# ******************************************************************************* +"""Scope rule for module verification reports. + +The report page rendered by ``score_module_verification_report`` needs its +component list authored on the Need, because sections must exist at read time. +That makes drift possible, so the rule below makes drift *detected*, never +silently corrected: the build fails and someone edits one line. + +The rule is bidirectional and lives here -- in the metamodel -- rather than in +the rendering extension, so it applies to all reports regardless of how (or +whether) they are rendered. + +The YAML ``graph_checks`` DSL compares an attribute of a linked need against a +constant; it cannot express set equality between two link fields, which is why +this one check is written in Python. +""" + +from collections.abc import Iterable +from typing import cast + +from score_metamodel import CheckLogger, graph_check +from sphinx.application import Sphinx +from sphinx_needs.data import NeedsView +from sphinx_needs.need_item import NeedItem + +REPORT_TYPE = "mod_ver_report" +MODULE_TYPE = "mod" +COMPONENT_TYPE = "comp" + + +def _link_ids(need: NeedItem, option: str) -> list[str]: + """Read a link field as a list of ids, tolerating the single-string form.""" + value: object = need.get(option, None) + if not value: + return [] + if isinstance(value, str): + return [value] + return [str(item) for item in cast("Iterable[object]", value)] + + +@graph_check +def check_mod_ver_report_scope(app: Sphinx, needs: NeedsView, log: CheckLogger) -> None: + """``mod_ver_report.covers`` must match ``mod.includes`` exactly. + + * a component included by the module but missing from ``:covers:`` has no + section in the report -- the report silently under-reports its scope. + * a component in ``:covers:`` that the module does not include claims + verification of something outside the module. + + Only ``comp`` targets participate; ``:covers:`` may also point at + requirements or other artifacts and those are left alone. + """ + for need in needs.values(): + if need["type"] != REPORT_TYPE: + continue + + covered_components = { + need_id + for need_id in _link_ids(need, "covers") + if need_id in needs and needs[need_id]["type"] == COMPONENT_TYPE + } + + for module_id in _link_ids(need, "belongs_to"): + module = needs.get(module_id, None) + if module is None or module["type"] != MODULE_TYPE: + # Wrong or dangling belongs_to target is reported by the + # regular link checks; nothing to add here. + continue + + included = set(_link_ids(module, "includes")) + + for missing in sorted(included - covered_components): + log.warning_for_option( + need, + "covers", + f"does not cover '{missing}', which is included by " + f"'{module_id}'. Add it to ':covers:' so the report gets a " + "section for it.", + ) + + for extra in sorted(covered_components - included): + log.warning_for_option( + need, + "covers", + f"covers '{extra}', which is not included by '{module_id}'.", + ) diff --git a/src/extensions/score_metamodel/tests/rst/graph/test_mod_ver_report_scope.rst b/src/extensions/score_metamodel/tests/rst/graph/test_mod_ver_report_scope.rst new file mode 100644 index 000000000..890bc05e5 --- /dev/null +++ b/src/extensions/score_metamodel/tests/rst/graph/test_mod_ver_report_scope.rst @@ -0,0 +1,99 @@ +.. + # ******************************************************************************* + # Copyright (c) 2026 Contributors to the Eclipse Foundation + # + # See the NOTICE file(s) distributed with this work for additional + # information regarding copyright ownership. + # + # This program and the accompanying materials are made available under the + # terms of the Apache License Version 2.0 which is available at + # https://www.apache.org/licenses/LICENSE-2.0 + # + # SPDX-License-Identifier: Apache-2.0 + # ******************************************************************************* + + +.. test_metadata:: + :id: test_metadata__metamodel_graph_mod_ver_report + :partially_verifies_list: tool_req__docs_verification_report_need + :test_type: requirements_based + :derivation_technique: requirements_based + + Tests that a module verification report covers exactly the components of its + module — drift in either direction is detected, never silently corrected. + +--- Setup + +.. feat:: Report Scope Feature + :id: feat__report_scope + :security: NO + :safety: QM + :status: valid + +.. comp:: Report Scope Component A + :id: comp__report_scope_a + :security: NO + :safety: QM + :status: valid + :belongs_to: feat__report_scope + +.. comp:: Report Scope Component B + :id: comp__report_scope_b + :security: NO + :safety: QM + :status: valid + :belongs_to: feat__report_scope + +.. comp:: Report Scope Component Outside The Module + :id: comp__report_scope_outside + :security: NO + :safety: QM + :status: valid + :belongs_to: feat__report_scope + +.. mod:: Report Scope Module + :id: mod__report_scope + :security: NO + :safety: QM + :status: valid + :includes: comp__report_scope_a, comp__report_scope_b + +--- + +.. Positive test: covers exactly the included components — no warning expected. + +.. mod_ver_report:: Complete Report + :id: mod_vrep__report_scope__complete + :safety: QM + :security: NO + :status: valid + :verification_method: test + :belongs_to: mod__report_scope + :covers: comp__report_scope_a, comp__report_scope_b + :expect_not: does not cover, which is not included by + +.. Negative test: an included component is missing from ':covers:'. + +.. mod_ver_report:: Report Missing A Component + :id: mod_vrep__report_scope__missing + :safety: QM + :security: NO + :status: valid + :verification_method: test + :belongs_to: mod__report_scope + :covers: comp__report_scope_a + :expect: does not cover 'comp__report_scope_b', which is included by 'mod__report_scope' + :expect_not: which is not included by + +.. Negative test: a covered component is not part of the module. + +.. mod_ver_report:: Report Covering Too Much + :id: mod_vrep__report_scope__extra + :safety: QM + :security: NO + :status: valid + :verification_method: test + :belongs_to: mod__report_scope + :covers: comp__report_scope_a, comp__report_scope_b, comp__report_scope_outside + :expect: covers 'comp__report_scope_outside', which is not included by 'mod__report_scope' + :expect_not: does not cover diff --git a/src/extensions/score_metamodel/tests/rst/options/test_options_verification_report.rst b/src/extensions/score_metamodel/tests/rst/options/test_options_verification_report.rst index c6f84faed..9e5c1e5bc 100644 --- a/src/extensions/score_metamodel/tests/rst/options/test_options_verification_report.rst +++ b/src/extensions/score_metamodel/tests/rst/options/test_options_verification_report.rst @@ -60,7 +60,7 @@ :branch_coverage_percent: 85 :applies_to_module_version: 1.0.0 :belongs_to: mod__verification_module - :covers: comp_req__verification__sample + :covers: comp__verification_component, comp_req__verification__sample .. Invalid coverage percentage value in module verification report @@ -74,3 +74,4 @@ :verification_method: inspection :line_coverage_percent: 150 :belongs_to: mod__verification_module + :covers: comp__verification_component diff --git a/src/extensions/score_module_verification_report/BUILD b/src/extensions/score_module_verification_report/BUILD new file mode 100644 index 000000000..8e4e30249 --- /dev/null +++ b/src/extensions/score_module_verification_report/BUILD @@ -0,0 +1,43 @@ +# ******************************************************************************* +# Copyright (c) 2026 Contributors to the Eclipse Foundation +# +# See the NOTICE file(s) distributed with this work for additional +# information regarding copyright ownership. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# ******************************************************************************* +load("@aspect_rules_py//py:defs.bzl", "py_library") +load("@docs_as_code_hub_env//:requirements.bzl", "all_requirements") +load("//:score_pytest.bzl", "score_pytest") + +filegroup( + name = "all_sources", + srcs = glob(["*.py"]), + visibility = ["//visibility:public"], +) + +py_library( + name = "score_module_verification_report", + srcs = [":all_sources"], + imports = ["."], + visibility = ["//visibility:public"], + data = ["//src/needs_templates:files"], + deps = all_requirements + [ + "@score_docs_as_code//src/helper_lib", + ], +) + +score_pytest( + name = "score_module_verification_report_test", + size = "medium", + srcs = glob(["tests/*.py"]), + pytest_config = "//:pyproject.toml", + deps = [ + ":score_module_verification_report", + "@score_docs_as_code//src/extensions/score_metamodel", + ], +) diff --git a/src/extensions/score_module_verification_report/README.md b/src/extensions/score_module_verification_report/README.md new file mode 100644 index 000000000..a5362cf9b --- /dev/null +++ b/src/extensions/score_module_verification_report/README.md @@ -0,0 +1,140 @@ + + +# `score_module_verification_report` + +Per-module verification report pages whose sections behave like ordinary RST. + +Resolves [#764](https://github.com/eclipse-score/docs-as-code/issues/764) — option **K**. + +## The governing design rule + +> **The extension emits RST. It never reads the Need model to compute an answer.** + +Rendering, not resolving. The directive emits `needtable` filters and `:need:` +references; sphinx-needs resolves all of them after collection, with its own +semantics, its own external-need handling and its own backlinks. The extension +therefore needs no `NeedsView`, no build lifecycle hook and no model +completeness at read time. The only thing it knows at read time is *which +sections exist*. + +**Test for future changes:** if adding report content requires new Python that +walks needs and computes something, the line has been crossed. If it requires a +new `needtable` filter in the template, it has not. + +## Authoring surface + +One Need per module. That is the whole consumer-facing API: + +```rst +.. mod_ver_report:: Baselibs Verification Report + :id: mod_vrep__baselibs + :belongs_to: mod__baselibs + :covers: comp__baselibs_json, comp__baselibs_bit_manipulation + :safety: ASIL_B + :security: NO + :status: valid + :verification_method: test + :titles: + comp__baselibs_json = JSON Utilities + + Free-form introduction; becomes the Need's description. +``` + +Scaling to N modules = adding N Needs. Nothing else. + +`:covers:` is a **real link field on the Need**, not an opaque directive option. +That is what lets the metamodel validate it, generate `covers_back` for free and +report through the normal warning pipeline — and it is why this extension owns +no consistency-checking code. + +`:titles:` is the only presentation-only option: an optional `id = Heading` per +line. Without it the heading is derived from the id +(`comp__baselibs_json` → "Baselibs Json"), which is a deliberate last-resort +fallback — the *real* title is rendered by the `:need:` reference inside the +section, resolved by sphinx-needs. + +## What gets emitted + +A `mod_ver_report` Need followed by a flat list of sections: + +| Section | Content | +| ---------------------- | ----------------------------------------------------------- | +| Report Metadata | `needtable` on the report itself | +| Verification Scope | `needtable` over the covered components | +| *one per component* | `:need:` reference plus a `needtable` of related needs | +| Verification Evidence | `needtable` over the report's `contains` / `evidence` backlinks | + +Every section is preceded by an explicit target namespaced with the report id +(`mod_vrep__baselibs__comp__baselibs_json`), so anchors are stable across +rebuilds and two reports on one page never collide. + +## Two things that look like implementation details but are not + +**1. The report body is a *sibling* of the Need, never its child.** + +Sphinx turns headings into `section` nodes exactly once, during the read phase. +Sections are what produce anchors, sidebar entries, `:ref:` targets, +search-index entries and PDF bookmarks. sphinx-needs parses Need content with +`match_titles=False`, so a heading inside a `.need` template can never become a +section. The directive therefore parses its generated RST with +`parse_text_to_nodes(..., allow_section_headings=True)` and returns the nodes +into the surrounding document. + +Moving the body inside the Need node — to make it render in the Need's box — +silently removes every section from the document and cannot be spotted by +looking at the HTML. Don't. + +**2. The extension owns no build lifecycle.** + +No `env-updated` re-read, no `build-finished` consistency pass, no +`env.*_registry` merged across parallel workers. `setup()` declares config +values and registers directives. + +There is exactly one `config-inited` handler and it calls `add_directive` +twice — directive registration is last-one-wins and sphinx-needs registers +`mod_ver_report` from its own `config-inited` handler. It touches no +environment state and reads no needs. +`tests/test_report_integration.py` asserts this stays true. + +## Validation lives in the metamodel + +`src/extensions/score_metamodel/checks/mod_ver_report.py` enforces the rule with +real content: **every component in `mod__x.includes` must appear in the report's +`:covers:`, and vice versa** — bidirectional, so an omitted component is caught +as well as an over-claimed one. + +Failure mode by design: drift is **detected, never silently corrected**. +Sections must exist at read time, so the list must be authored. The build fails +and someone edits one line. + +Everything else — mandatory fields, allowed link targets — is already declared +in `metamodel.yaml` under `mod_ver_report`. + +## Configuration + +The report layout and table columns are defined by the bundled Jinja template. +There are no report-specific Sphinx config values. The optional evidence table +uses the standard `contains` and `evidence` link fields when they are configured +by the consuming project. + +## Status + +Proof of concept. Known gaps: + +- `:covers:` is still `ANY` in `metamodel.yaml`; narrowing the allowed target + types is a separate, consumer-affecting change. +- Two reports placed at *different* heading depths on one page rely on docutils' + title-style bookkeeping and are untested. +- No `needflow` / `needpie` in the default template — both are pure additions to + the templates when wanted. diff --git a/src/extensions/score_module_verification_report/__init__.py b/src/extensions/score_module_verification_report/__init__.py new file mode 100644 index 000000000..479062a6b --- /dev/null +++ b/src/extensions/score_module_verification_report/__init__.py @@ -0,0 +1,81 @@ +# ******************************************************************************* +# Copyright (c) 2026 Contributors to the Eclipse Foundation +# +# See the NOTICE file(s) distributed with this work for additional +# information regarding copyright ownership. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# ******************************************************************************* +"""Module Verification Report. + +One Need per module gives that module a report page whose sections behave like +ordinary RST: real anchors, real ToC entries, real ``:ref:`` targets, present in +every builder. + +See ``directive.py`` for the design rules. In short: the extension emits RST +and never reads the Need model; scope is a real link field validated by the +metamodel. + +Lifecycle +--------- +The extension deliberately has **no** build lifecycle hooks: no ``env-updated`` +re-read, no ``build-finished`` consistency pass, no registry carried across +parallel workers. ``setup()`` declares config values and registers two +directives. + +The single ``config-inited`` handler exists only because directive registration +is last-one-wins and sphinx-needs registers ``mod_ver_report`` from its own +``config-inited`` handler (priority 500). The handler below runs at priority +900 and does nothing but call ``add_directive`` twice. It touches no +environment state and reads no needs. +""" + +from __future__ import annotations + +from score_module_verification_report.directive import ( + REPORT_NEED_DIRECTIVE, + REPORT_TYPE, + ModuleVerificationReportDirective, +) +from sphinx.application import Sphinx +from sphinx.config import Config +from sphinx_needs.directives.need import NeedDirective + +__all__ = ["setup"] + + +class ReportNeedDirective(NeedDirective): + """sphinx-needs' Need directive, reachable under an un-shadowed name. + + ``NeedDirective`` derives the need type from the directive name it was + invoked as. The public ``mod_ver_report`` name belongs to the report + directive, so the generated RST calls this alias instead and it restores + the intended type. + """ + + def run(self): # type: ignore[no-untyped-def] + self.name = REPORT_TYPE + return super().run() + + +def _register_directives(app: Sphinx, config: Config) -> None: + # Runs late on config-inited, after sphinx-needs registered a plain + # NeedDirective under REPORT_TYPE. Registration only -- no model access. + app.add_directive(REPORT_NEED_DIRECTIVE, ReportNeedDirective, override=True) + app.add_directive(REPORT_TYPE, ModuleVerificationReportDirective, override=True) + + +def setup(app: Sphinx) -> dict[str, object]: + app.setup_extension("sphinx_needs") + + app.connect("config-inited", _register_directives, priority=900) + + return { + "version": "0.1.0", + "parallel_read_safe": True, + "parallel_write_safe": True, + } diff --git a/src/extensions/score_module_verification_report/directive.py b/src/extensions/score_module_verification_report/directive.py new file mode 100644 index 000000000..0cb66cefe --- /dev/null +++ b/src/extensions/score_module_verification_report/directive.py @@ -0,0 +1,249 @@ +# ******************************************************************************* +# Copyright (c) 2026 Contributors to the Eclipse Foundation +# +# See the NOTICE file(s) distributed with this work for additional +# information regarding copyright ownership. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# ******************************************************************************* +"""The ``mod_ver_report`` directive. + +Design decisions that are load-bearing and must not be "cleaned up" +------------------------------------------------------------------- + +1. **The report body is a sibling of the Need, never its child.** + + Sphinx turns headings into ``section`` nodes exactly once, during the read + phase. Sections are what produce anchors, sidebar entries, ``:ref:`` + targets, search-index entries and PDF bookmarks. After reading, Sphinx + never looks for headings again. + + sphinx-needs parses Need content with ``match_titles=False``, so a heading + written inside a ``.need`` template can never become a section. Therefore + this directive parses the generated report text with + ``parse_text_to_nodes(..., allow_section_headings=True)`` and returns the + resulting nodes *into the surrounding document*. The generated + ``mod_ver_report`` Need is the first of those nodes; the sections follow it + as siblings. + + A reviewer may be tempted to ask for the body to be moved inside the Need + node so that it renders in the Need's box. Doing so silently removes every + section from the document and cannot be spotted by looking at the HTML. + Do not do it. + +2. **The extension emits RST. It never reads the Need model.** + + The only thing known at read time is *which sections exist*. Everything + with semantics -- titles, coverage, backlinks, external needs -- is deferred + to ``needtable`` filters and ``:need:`` references that sphinx-needs + resolves after collection. Consequently this directive needs no + ``NeedsView``, no lifecycle hook and no model completeness at read time. + + Test for future changes: if new report content needs Python that walks needs + and computes something, the line has been crossed. If it needs a new + ``needtable`` filter in the template, it has not. + +3. **Scope is validated by the metamodel, not here.** + + ``:covers:`` is passed through verbatim as a real link field on the Need. + sphinx-needs validates target existence and type, generates ``covers_back`` + for free, and reports through the normal warning pipeline. The + "every component of the module is covered, and vice versa" rule lives in + ``score_metamodel`` as a graph check. +""" + +from __future__ import annotations + +from collections.abc import Callable, Sequence +from typing import Any, ClassVar, Final, cast + +from docutils import nodes +from score_module_verification_report import rendering +from sphinx.util import logging +from sphinx.util.docutils import SphinxDirective +from sphinx_needs.utils import DummyOptionSpec + +logger = logging.getLogger(__name__) + +#: Need type / public directive name. +REPORT_TYPE: Final = "mod_ver_report" + +#: Internal directive name bound to sphinx-needs' ``NeedDirective`` for +#: ``REPORT_TYPE``. The public name is taken by this directive, so the +#: generated RST needs an un-shadowed way to actually create the Need. +REPORT_NEED_DIRECTIVE: Final = "mod_ver_report_need" + +#: Directive options that steer rendering and must not reach the Need. +PRESENTATION_OPTIONS: Final = ("titles",) + + +class ModuleVerificationReportDirective(SphinxDirective): + """Emit one ``mod_ver_report`` Need plus a flat list of real sections.""" + + required_arguments = 1 + optional_arguments = 0 + final_argument_whitespace = True + has_content = True + # Any option is accepted and forwarded to the Need. The metamodel decides + # which options are mandatory, which are links and what they may target. + # Annotated with docutils' own type: ``Directive.option_spec`` is a mutable + # class variable, so a narrower type here is an invalid override. + option_spec: ClassVar[dict[str, Callable[[str], Any]] | None] = DummyOptionSpec() + + options: dict[str, str | None] + + def _warn(self, message: str, subtype: str = "report") -> None: + logger.warning( + f"{REPORT_TYPE}: {message}", + location=self.get_location(), + type=REPORT_TYPE, + subtype=subtype, + ) + + def run(self) -> Sequence[nodes.Node]: + options = dict(self.options) + title = self.arguments[0].strip() + + presentation = {key: options.pop(key, None) for key in PRESENTATION_OPTIONS} + + report_id = (options.get("id") or "").strip() + if not report_id: + self._warn("missing mandatory ':id:' option; no report is rendered", "id") + return [] + try: + rendering.quote_for_filter(report_id) + except ValueError: + self._warn(f"{report_id!r} is not a usable need id", "id") + return [] + + components = rendering.parse_component_list(options.get("covers")) + for message in components.warnings: + self._warn(message, "covers") + + title_overrides, title_warnings = rendering.parse_title_overrides( + presentation["titles"] + ) + for message in title_warnings: + self._warn(message, "titles") + for unknown in sorted(set(title_overrides) - set(components.ids)): + self._warn( + f"title override for {unknown!r} which is not listed in ':covers:'", + "titles", + ) + + text = self._render(title, options, report_id, components.ids, title_overrides) + parsed = self.parse_text_to_nodes(text, allow_section_headings=True) + _promote_report_anchors( + parsed, + rendering.section_anchors(report_id, components.ids), + self.state.document, + ) + return parsed + + def _render( + self, + title: str, + options: dict[str, str | None], + report_id: str, + component_ids: list[str], + title_overrides: dict[str, str], + ) -> str: + module_ids = rendering.parse_component_list(options.get("belongs_to")).ids + module_id = module_ids[0] if module_ids else "" + return rendering.render_report_template( + directive_name=REPORT_NEED_DIRECTIVE, + title=title, + options=options, + content=list(self.content), + report_id=report_id, + module_id=module_id, + component_ids=component_ids, + title_overrides=title_overrides, + evidence_links=self._configured_evidence_links(), + config=self.config, + ) + + def _configured_evidence_links(self) -> list[str]: + """Keep only evidence links that are actually configured link fields. + + This reads the link *configuration* -- not the Need model -- so that a + project which does not define ``contains``/``evidence`` gets no section + instead of a filter over a field that does not exist. + + Both spellings are honoured: ``needs_links`` (a dict keyed by option + name, what ``score_metamodel`` writes) and the deprecated + ``needs_extra_links`` list. + """ + known = set(cast("dict[str, Any]", self.config.needs_links)) + known.update( + str(link["option"]) + for link in cast("list[dict[str, Any]]", self.config.needs_extra_links) + if isinstance(link, dict) and "option" in link + ) + return [link for link in ("contains", "evidence") if link in known] + + +def _promote_report_anchors( + parsed: Sequence[nodes.Node], anchors: list[str], document: nodes.document +) -> None: + """Make the namespaced anchor each generated section's *primary* id. + + ``.. _name:`` in front of a heading gives the section a second id, but + docutils only merges it during the ``PropagateTargets`` transform and then + appends it, so ``section["ids"][0]`` stays the id docutils derived from the + heading text. That id is what Sphinx uses for the ToC entry and the HTML + element, and it is neither namespaced nor stable: two reports with the same + heading text on one page make docutils disambiguate with a ``-1`` suffix. + + So the merge is done here instead, at read time, in document order. Ids + and names are moved -- not copied -- from the target onto the section, and + the document's id/name maps are repointed accordingly. Nothing is + invented: these are exactly the ids docutils created from the targets the + report emitted. + """ + wanted = {nodes.make_id(name) for name in anchors} + pending: nodes.target | None = None + + # ``parsed`` is the parser's own children list: removing a top-level + # target below mutates it, so iterate over a snapshot. + for root in list(parsed): + for node in root.findall(include_self=True): + if isinstance(node, nodes.target): + if not _is_plain_target(node): + continue + if wanted.intersection(node["ids"]): + pending = node + elif isinstance(node, nodes.section) and pending is not None: + _merge_target_into_section(pending, node, document) + pending = None + + +def _is_plain_target(target: nodes.target) -> bool: + """True for ``.. _name:`` targets that have not been resolved yet.""" + return not any(target.get(key) for key in ("refid", "refuri", "refname")) + + +def _merge_target_into_section( + target: nodes.target, section: nodes.section, document: nodes.document +) -> None: + ids = list(target["ids"]) + names = list(target["names"]) + + section["ids"] = ids + [i for i in section["ids"] if i not in ids] + section["names"] = names + [n for n in section["names"] if n not in names] + + for id_ in ids: + document.ids[id_] = section + for name in names: + document.nameids[name] = ids[0] + + # Emptied so that docutils' PropagateTargets transform does not merge the + # same ids a second time; removed so no stray anchor is left behind. + target["ids"] = [] + target["names"] = [] + if target.parent is not None: + target.parent.remove(target) diff --git a/src/extensions/score_module_verification_report/rendering.py b/src/extensions/score_module_verification_report/rendering.py new file mode 100644 index 000000000..b9b6d2c67 --- /dev/null +++ b/src/extensions/score_module_verification_report/rendering.py @@ -0,0 +1,251 @@ +# ******************************************************************************* +# Copyright (c) 2026 Contributors to the Eclipse Foundation +# +# See the NOTICE file(s) distributed with this work for additional +# information regarding copyright ownership. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# ******************************************************************************* +"""Pure rendering helpers for the module verification report. + +Everything in here is a *string transformation*. No function in this module +receives, reads or resolves a Need. That is deliberate and it is the governing +design rule of this extension: + + The extension emits RST. It never reads the Need model to compute an + answer. + +Rendering, not resolving. The emitted RST contains ``needtable`` filters and +``:need:`` references; sphinx-needs resolves those later with its own +semantics, its own external-need handling and its own backlinks. +""" + +from __future__ import annotations + +import json +import re +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any + +from jinja2 import Environment, FileSystemLoader, StrictUndefined + +# Conservative allow-list for anything that gets interpolated into a +# sphinx-needs filter string. Filter strings are evaluated as Python by +# sphinx-needs, so a need id is never pasted in unchecked. +NEED_ID_RE = re.compile(r"^[A-Za-z0-9_.\-]+$") + +# ``comp__foo[version==1]`` -- a link-version qualifier. sphinx-needs strips +# these itself on real link fields; we must not silently drop them while +# building sections, because the section would then be built for a component +# the author did not literally write. +VERSION_QUALIFIER_RE = re.compile(r"\[[^\]]*\]$") + +_SPLIT_RE = re.compile(r"[,\s]+") + +_TEMPLATE_NAME = "mod_ver_report_tiny.need" + + +def _template_environment() -> Environment: + """Return the environment for the shared report template. + + The template is a Bazel runfile of this extension. It is deliberately + rendered from the explicit directive inputs only; it receives no Sphinx + environment and no Need model. + """ + template_folder = Path(__file__).parents[2] / "needs_templates" + return Environment( + loader=FileSystemLoader(template_folder), + undefined=StrictUndefined, + keep_trailing_newline=True, + trim_blocks=True, + lstrip_blocks=True, + ) + + +def render_report_template( + *, + directive_name: str, + title: str, + options: dict[str, str | None], + content: list[str], + report_id: str, + module_id: str, + component_ids: list[str], + title_overrides: dict[str, str], + evidence_links: list[str], + config: Any, +) -> str: + """Render the report from the shared Jinja template. + + ``components`` is intentionally derived from the directive's declared + ``:covers:`` list. Passing Need objects here would turn K into a + read-phase model traversal and reintroduce the lifecycle problem this + implementation avoids. + """ + module_short = module_id.removeprefix("mod__") + components = [ + { + "id": component_id, + "title": title_overrides.get(component_id) or derive_title(component_id), + "slug": component_id.removeprefix(f"comp__{module_short}_"), + } + for component_id in component_ids + ] + return ( + _template_environment() + .get_template(_TEMPLATE_NAME) + .render( + report_need_directive=directive_name, + report_title=title, + report_options=options, + report_content=content, + report_id=report_id, + module_id=module_id, + module_short=module_short, + module_slug=module_short.replace("_", "").lower(), + feature_id=module_id.replace("mod__", "feat__", 1), + components=components, + evidence_filter=" or ".join( + f"{quote_for_filter(report_id)} in {link}_back" + for link in evidence_links + ), + scope_filter=( + "id in [" + + ", ".join( + quote_for_filter(component_id) for component_id in component_ids + ) + + "]" + if component_ids + else "False" + ), + ) + ) + + +@dataclass +class ComponentList: + """Result of parsing the ``:covers:`` option.""" + + ids: list[str] = field(default_factory=list) + warnings: list[str] = field(default_factory=list) + + +def quote_for_filter(need_id: str) -> str: + """Return ``need_id`` as a safely quoted literal for a needs filter string. + + Raises ``ValueError`` for anything that is not a plain need id. Callers + turn that into a Sphinx warning; nothing unvalidated ever reaches the + filter string. + """ + if not NEED_ID_RE.match(need_id): + raise ValueError(f"{need_id!r} is not a valid need id") + # json.dumps gives us a double-quoted, escaped literal that is also valid + # Python -- belt and braces on top of the allow-list above. + return json.dumps(need_id) + + +def parse_component_list(raw: str | None) -> ComponentList: + """Parse the ``:covers:`` option value into an ordered, de-duplicated list. + + Accepts comma and/or whitespace separated ids across multiple lines. + Version qualifiers are reported instead of being silently ignored, and + duplicates are dropped deterministically (first occurrence wins). + """ + result = ComponentList() + if not raw: + return result + + seen: set[str] = set() + for token in _SPLIT_RE.split(raw.strip()): + if not token: + continue + stripped = VERSION_QUALIFIER_RE.sub("", token) + if stripped != token: + result.warnings.append( + f"ignoring version qualifier on {token!r}; the report section " + f"is built for {stripped!r}" + ) + if not NEED_ID_RE.match(stripped): + result.warnings.append( + f"{stripped!r} is not a valid need id and is skipped" + ) + continue + if stripped in seen: + result.warnings.append(f"duplicate entry {stripped!r} is listed once") + continue + seen.add(stripped) + result.ids.append(stripped) + return result + + +def parse_title_overrides(raw: str | None) -> tuple[dict[str, str], list[str]]: + """Parse the ``:titles:`` option: one `` = `` per line.""" + overrides: dict[str, str] = {} + warnings: list[str] = [] + if not raw: + return overrides, warnings + for line in raw.splitlines(): + line = line.strip() + if not line: + continue + if "=" not in line: + warnings.append( + f"cannot parse title override {line!r}; expected 'id = Title'" + ) + continue + need_id, _, title = line.partition("=") + need_id, title = need_id.strip(), title.strip() + if not need_id or not title: + warnings.append( + f"cannot parse title override {line!r}; expected 'id = Title'" + ) + continue + overrides[need_id] = title + return overrides, warnings + + +def derive_title(need_id: str) -> str: + """Fallback used because K cannot access the merged Need model at read time.""" + _, _, tail = need_id.partition("__") + slug = tail or need_id + return slug.replace("_", " ").strip().title() or need_id + + +def anchor(report_id: str, slug: str) -> str: + """Deterministic, collision-free RST target name. + + Anchors are namespaced by the report id, so two reports on one page never + collide and docutils never has to disambiguate with an unstable ``-1`` + suffix. + """ + return f"{report_id}__{slug}".lower() + + +def _heading(title: str) -> str: + return f"{title}\n{UNDERLINE * max(len(title), 3)}\n" + + +def _section(target: str, title: str, body: str) -> str: + return f".. _{target}:\n\n{_heading(title)}\n{body.rstrip()}\n" + + +def section_anchors(report_id: str, component_ids: list[str]) -> list[str]: + """Every RST target name the report emits, in document order. + + The directive uses this to promote the namespaced anchor to be each + section's *primary* id, so the ToC entry, the HTML element id and the + ``:ref:`` target all agree and stay stable when two reports on one page + happen to use the same heading text. + """ + return [ + anchor(report_id, "feature"), + anchor(report_id, "verification-scope"), + anchor(report_id, "components"), + *(anchor(report_id, component_id) for component_id in component_ids), + anchor(report_id, "verification-evidence"), + ] diff --git a/src/extensions/score_module_verification_report/tests/test_rendering.py b/src/extensions/score_module_verification_report/tests/test_rendering.py new file mode 100644 index 000000000..3ac94d377 --- /dev/null +++ b/src/extensions/score_module_verification_report/tests/test_rendering.py @@ -0,0 +1,112 @@ +# ******************************************************************************* +# Copyright (c) 2026 Contributors to the Eclipse Foundation +# +# See the NOTICE file(s) distributed with this work for additional +# information regarding copyright ownership. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# ******************************************************************************* +"""Unit tests for the pure rendering helpers.""" + +import pytest +from score_module_verification_report import rendering + + +class TestQuoteForFilter: + """Acceptance test 10: filter expressions with user-supplied IDs.""" + + def test_plain_id_is_quoted(self) -> None: + assert rendering.quote_for_filter("comp__a") == '"comp__a"' + + @pytest.mark.parametrize( + "hostile", + [ + 'a" or True or "', + "a' or True or 'b", + "__import__('os')", + "a\nb", + "a; b", + "a b", + "", + ], + ) + def test_hostile_input_is_rejected(self, hostile: str) -> None: + with pytest.raises(ValueError): + rendering.quote_for_filter(hostile) + + def test_rejected_ids_never_reach_a_rendered_filter(self) -> None: + parsed = rendering.parse_component_list('comp__a, x"or(True)') + assert parsed.ids == ["comp__a"] + assert any("not a valid need id" in w for w in parsed.warnings) + + def test_bare_words_survive_but_stay_quoted(self) -> None: + # Whitespace is a separator, so a hostile value degenerates into + # separate tokens. Each is still quoted, so the worst case is a filter + # that matches nothing -- never one that evaluates injected code. + parsed = rendering.parse_component_list("comp__a or True") + assert parsed.ids == ["comp__a", "or", "True"] + assert all( + rendering.quote_for_filter(need_id).startswith('"') + for need_id in parsed.ids + ) + + +class TestParseComponentList: + def test_comma_and_whitespace_separated(self) -> None: + parsed = rendering.parse_component_list("comp__a, comp__b\n comp__c") + assert parsed.ids == ["comp__a", "comp__b", "comp__c"] + assert parsed.warnings == [] + + def test_empty(self) -> None: + assert rendering.parse_component_list(None).ids == [] + assert rendering.parse_component_list(" ").ids == [] + + def test_version_qualifier_warns_instead_of_being_ignored(self) -> None: + parsed = rendering.parse_component_list("comp__a[version==2]") + assert parsed.ids == ["comp__a"] + assert any("version qualifier" in w for w in parsed.warnings) + + def test_duplicates_are_deterministic(self) -> None: + """Acceptance test 9: duplicate component ids.""" + parsed = rendering.parse_component_list("comp__b, comp__a, comp__b") + assert parsed.ids == ["comp__b", "comp__a"] + assert any("duplicate" in w for w in parsed.warnings) + + +class TestTitles: + def test_overrides_are_parsed(self) -> None: + overrides, warnings = rendering.parse_title_overrides( + "comp__a = JSON Utilities\ncomp__b = Bit Manipulation\n" + ) + assert overrides == { + "comp__a": "JSON Utilities", + "comp__b": "Bit Manipulation", + } + assert warnings == [] + + def test_malformed_override_warns(self) -> None: + overrides, warnings = rendering.parse_title_overrides("comp__a JSON") + assert overrides == {} + assert len(warnings) == 1 + + def test_derive_title_is_the_documented_fallback(self) -> None: + assert rendering.derive_title("comp__baselibs_json") == "Baselibs Json" + assert rendering.derive_title("nounderscores") == "Nounderscores" + + +class TestAnchors: + def test_anchor_is_namespaced_by_report(self) -> None: + """Acceptance test 3: no collisions between two reports on one page.""" + a = rendering.anchor("mod_vrep__one", "comp__shared") + b = rendering.anchor("mod_vrep__two", "comp__shared") + assert a != b + assert a == "mod_vrep__one__comp__shared" + + def test_anchor_is_stable(self) -> None: + assert rendering.anchor("mod_vrep__x", "comp__y") == rendering.anchor( + "mod_vrep__x", "comp__y" + ) diff --git a/src/extensions/score_module_verification_report/tests/test_report_integration.py b/src/extensions/score_module_verification_report/tests/test_report_integration.py new file mode 100644 index 000000000..4911e2d8c --- /dev/null +++ b/src/extensions/score_module_verification_report/tests/test_report_integration.py @@ -0,0 +1,614 @@ +# ******************************************************************************* +# Copyright (c) 2026 Contributors to the Eclipse Foundation +# +# See the NOTICE file(s) distributed with this work for additional +# information regarding copyright ownership. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# ******************************************************************************* +"""Acceptance tests for the module verification report. + +"It looks right in the sidebar" is not evidence. These tests assert on the +doctree, on ``env.tocs`` and on rendered output. + +Acceptance test 7 (a component in ``mod.includes`` but missing from +``:covers:`` fails the build via the *metamodel* rule, not via extension code) +deliberately lives with the metamodel: +``src/extensions/score_metamodel/tests/rst/graph/test_mod_ver_report_scope.rst``. +""" + +from __future__ import annotations + +import json +import re +from collections.abc import Callable +from pathlib import Path +from typing import Any + +import pytest +from docutils import nodes +from sphinx.testing.util import SphinxTestApp + +EXTERNAL_NEEDS: dict[str, Any] = { + "current_version": "1.0", + "project": "external", + "versions": { + "1.0": { + "needs": { + "comp__external_thing": { + "docname": "index", + "id": "comp__external_thing", + "lineno": 1, + "status": "valid", + "title": "External Component", + "type": "comp", + "type_name": "comp", + } + } + } + }, +} + +CONF_PY = """ +extensions = ["sphinx_needs", "score_module_verification_report"] +needs_id_regex = "^[a-zA-Z0-9_]+$" +needs_types = [ + dict(directive="feat", title="Feature", prefix="feat__", color="#FFF", style="node"), + dict(directive="comp", title="Component", prefix="comp__", color="#FFF", style="node"), + dict(directive="mod", title="Module", prefix="mod__", color="#FFF", style="node"), + dict( + directive="mod_ver_report", + title="Module Verification Report", + prefix="mod_vrep__", + color="#FFF", + style="node", + ), + dict(directive="tcase", title="Test Case", prefix="tcase__", color="#FFF", style="node"), +] +needs_extra_options = ["safety", "security", "verification_method"] +needs_extra_links = [ + dict(option="belongs_to", incoming="belongs to", outgoing="belongs to"), + dict(option="includes", incoming="included by", outgoing="includes"), + dict(option="covers", incoming="covered by", outgoing="covers"), + dict(option="contains", incoming="contained by", outgoing="contains"), + dict(option="evidence", incoming="evidence for", outgoing="evidence"), +] +needs_external_needs = [ + dict(base_url="https://example.invalid/docs", json_path="external_needs.json") +] +suppress_warnings = ["app.add_directive", "epub.unknown_project_files"] +""" + +ARCHITECTURE = """ +.. comp:: JSON + :id: comp__baselibs_json + :safety: ASIL_B + :security: NO + :status: valid + +.. comp:: Bit Manipulation + :id: comp__baselibs_bit_manipulation + :safety: ASIL_B + :security: NO + :status: valid + +.. mod:: Baselibs + :id: mod__baselibs + :includes: comp__baselibs_json, comp__baselibs_bit_manipulation + +.. tcase:: A test case + :id: tcase__baselibs_1 + :status: valid +""" + +REPORT = """ +Baselibs +======== + +.. mod_ver_report:: Baselibs Verification Report + :id: mod_vrep__baselibs + :belongs_to: mod__baselibs + :covers: comp__baselibs_json, comp__baselibs_bit_manipulation + :safety: ASIL_B + :security: NO + :status: valid + :verification_method: test + :contains: tcase__baselibs_1 + :titles: + comp__baselibs_json = JSON Utilities + comp__baselibs_bit_manipulation = Bit Manipulation + + Verification report for the Baselibs module. +""" + + +def _write_sources(root: Path, docs: dict[str, str], conf: str = CONF_PY) -> None: + root.mkdir(parents=True, exist_ok=True) + (root / "conf.py").write_text(conf) + (root / "external_needs.json").write_text(json.dumps(EXTERNAL_NEEDS)) + for name, text in docs.items(): + path = root / f"{name}.rst" + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(text) + + +def _index(*docnames: str) -> str: + entries = "\n ".join(docnames) + return f"Docs\n====\n\n.. toctree::\n :maxdepth: 3\n\n {entries}\n" + + +AppFactory = Callable[..., SphinxTestApp] + + +@pytest.fixture +def build(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> AppFactory: + """Build a source tree and return the finished app.""" + + def _build( + docs: dict[str, str], + *, + buildername: str = "html", + parallel: int = 1, + srcdir: Path | None = None, + outdir: Path | None = None, + freshenv: bool = True, + conf: str = CONF_PY, + ) -> SphinxTestApp: + src = srcdir or (tmp_path / "src") + _write_sources(src, docs, conf) + monkeypatch.chdir(src) + app = SphinxTestApp( + freshenv=freshenv, + srcdir=src, + confdir=src, + outdir=outdir or (tmp_path / f"out-{buildername}"), + buildername=buildername, + parallel=parallel, + ) + app.build() + return app + + return _build + + +def _sections(doctree: nodes.document) -> list[nodes.section]: + return list(doctree.findall(nodes.section)) + + +def _section_ids(doctree: nodes.document) -> list[str]: + return [section["ids"][0] for section in _sections(doctree) if section["ids"]] + + +def _titles(doctree: nodes.document) -> list[str]: + return [section[0].astext() for section in _sections(doctree)] + + +# -------------------------------------------------------------------------- +# The core promise: real sections, produced during the read phase. +# -------------------------------------------------------------------------- + + +def test_report_emits_real_sections_as_siblings_of_the_need( + build: AppFactory, +) -> None: + app = build({"index": _index("report"), "report": REPORT + ARCHITECTURE}) + doctree = app.env.get_doctree("report") + + titles = _titles(doctree) + assert "Feature" in titles + assert "Components" in titles + assert "JSON Utilities" in titles # explicit override + assert "Bit Manipulation" in titles # explicit template title + assert "Verification Evidence" in titles + + # The Need is a sibling of the sections, not their container: no section + # may be a descendant of the Need node. + from sphinx_needs.nodes import Need + + for need in doctree.findall(Need): + assert not list(need.findall(nodes.section, include_self=False)), ( + "a section ended up inside the Need node; " + "sphinx-needs parses need content with match_titles=False, so those " + "headings would silently stop being sections" + ) + + +def test_generated_sections_follow_the_template_hierarchy(build: AppFactory) -> None: + """The Jinja template emits top-level report areas and nested details.""" + app = build({"index": _index("report"), "report": REPORT + ARCHITECTURE}) + doctree = app.env.get_doctree("report") + + page = next(iter(doctree.findall(nodes.section))) # "Baselibs" + generated = [child for child in page.children if isinstance(child, nodes.section)] + assert [child[0].astext() for child in generated] == [ + "Feature", + "Verification Scope", + "Components", + "JSON Utilities", + "Bit Manipulation", + "Verification Evidence", + ] + component = next( + section for section in generated if section[0].astext() == "JSON Utilities" + ) + assert "Component Requirements Statistics" in _titles(component) + + +# -------------------------------------------------------------------------- +# Acceptance test 1: :ref: from another page resolves and links. +# -------------------------------------------------------------------------- + + +def test_ref_from_another_page_resolves(build: AppFactory) -> None: + other = ( + "Other\n=====\n\n" + "See :ref:`mod_vrep__baselibs__comp__baselibs_json` for details.\n" + ) + app = build( + { + "index": _index("report", "other"), + "report": REPORT + ARCHITECTURE, + "other": other, + } + ) + assert "undefined label" not in app.warning.getvalue() + + html = (Path(app.outdir) / "other.html").read_text() + assert 'href="report.html#mod-vrep-baselibs-comp-baselibs-json"' in html + # An implicit :ref: takes its text from the section title -- proof that the + # target really is a section and not a synthesized anchor. + assert "JSON Utilities" in html + + +# -------------------------------------------------------------------------- +# Acceptance test 2: local ToC entries for every component section. +# -------------------------------------------------------------------------- + + +def test_toc_contains_every_generated_section(build: AppFactory) -> None: + app = build({"index": _index("report"), "report": REPORT + ARCHITECTURE}) + + toc = app.env.tocs["report"] + toc_titles = [ref.astext() for ref in toc.findall(nodes.reference)] + for expected in ( + "Feature", + "Verification Scope", + "Components", + "JSON Utilities", + "Bit Manipulation", + "Verification Evidence", + ): + assert expected in toc_titles + + toc_anchors = [ref["anchorname"] for ref in toc.findall(nodes.reference)] + assert "#mod-vrep-baselibs-comp-baselibs-json" in toc_anchors + + +# -------------------------------------------------------------------------- +# Acceptance test 3: stable, collision-free anchors. +# -------------------------------------------------------------------------- + + +SECOND_REPORT = """ +.. mod_ver_report:: Second Verification Report + :id: mod_vrep__second + :belongs_to: mod__baselibs + :covers: comp__baselibs_json + :safety: ASIL_B + :security: NO + :status: valid + :verification_method: test + :titles: + comp__baselibs_json = JSON Utilities +""" + + +def test_two_reports_on_one_page_do_not_collide(build: AppFactory) -> None: + app = build( + {"index": _index("report"), "report": REPORT + SECOND_REPORT + ARCHITECTURE} + ) + ids = _section_ids(app.env.get_doctree("report")) + + assert "mod-vrep-baselibs-comp-baselibs-json" in ids + assert "mod-vrep-second-comp-baselibs-json" in ids + # Same heading text twice, but no docutils "-1" disambiguation suffix. + assert not [i for i in ids if re.search(r"-\d+$", i)] + assert len(ids) == len(set(ids)) + + +def test_anchors_are_stable_across_rebuilds(tmp_path: Path, build: AppFactory) -> None: + src, out = tmp_path / "src", tmp_path / "out" + docs = {"index": _index("report"), "report": REPORT + ARCHITECTURE} + + first = build(docs, srcdir=src, outdir=out) + before = _section_ids(first.env.get_doctree("report")) + first.cleanup() + + second = build(docs, srcdir=src, outdir=out, freshenv=False) + assert _section_ids(second.env.get_doctree("report")) == before + + +# -------------------------------------------------------------------------- +# Acceptance test 4: heading depth follows placement. +# -------------------------------------------------------------------------- + + +def test_heading_depth_at_root_and_nested(build: AppFactory) -> None: + # The report placed below an existing sub-heading. REPORT's own page title + # is dropped so only the directive is re-used. + nested = ( + "Page\n====\n\nChapter\n-------\n\n" + "\n".join(REPORT.splitlines()[3:]) + "\n" + ) + app = build( + { + "index": _index("rootlevel", "nested"), + "rootlevel": REPORT + ARCHITECTURE, + "nested": nested + ARCHITECTURE, + } + ) + + def depth_of(docname: str, title: str) -> int: + doctree = app.env.get_doctree(docname) + for section in doctree.findall(nodes.section): + if section[0].astext() == title: + depth = 0 + node = section.parent + while node is not None: + if isinstance(node, nodes.section): + depth += 1 + node = node.parent + return depth + raise AssertionError(f"{title!r} not found in {docname}") + + # Root level: directly below the page title. + assert depth_of("rootlevel", "JSON Utilities") == 1 + # Nested below an existing heading: one level deeper. + assert depth_of("nested", "JSON Utilities") == 2 + assert depth_of("nested", "Feature") == 2 + + +# -------------------------------------------------------------------------- +# Acceptance test 5: a non-HTML builder keeps the sections. +# -------------------------------------------------------------------------- + + +def test_singlehtml_builder_keeps_sections(build: AppFactory) -> None: + app = build( + {"index": _index("report"), "report": REPORT + ARCHITECTURE}, + buildername="singlehtml", + ) + html = (Path(app.outdir) / "index.html").read_text() + assert 'id="mod-vrep-baselibs-comp-baselibs-json"' in html + assert "JSON Utilities" in html + + +def test_latex_builder_keeps_sections(build: AppFactory) -> None: + app = build( + {"index": _index("report"), "report": REPORT + ARCHITECTURE}, + buildername="latex", + ) + tex = next(Path(app.outdir).glob("*.tex")).read_text() + assert "JSON Utilities" in tex + assert "Components" in tex + + +# -------------------------------------------------------------------------- +# Acceptance test 6: identical under -j 1 and parallel reading. +# -------------------------------------------------------------------------- + + +def test_structure_identical_serial_and_parallel( + tmp_path: Path, build: AppFactory +) -> None: + # Sphinx only forks when there are more than five documents to read. + filler = {f"filler{i}": f"Filler {i}\n=========\n\ntext\n" for i in range(8)} + docs = { + "index": _index("report", *filler), + "report": REPORT + ARCHITECTURE, + **filler, + } + + serial = build(docs, srcdir=tmp_path / "s1", outdir=tmp_path / "o1", parallel=1) + serial_ids = _section_ids(serial.env.get_doctree("report")) + serial_titles = _titles(serial.env.get_doctree("report")) + serial.cleanup() + + parallel = build(docs, srcdir=tmp_path / "s2", outdir=tmp_path / "o2", parallel=4) + assert _section_ids(parallel.env.get_doctree("report")) == serial_ids + assert _titles(parallel.env.get_doctree("report")) == serial_titles + + +# -------------------------------------------------------------------------- +# Acceptance test 8: a needtable resolves an external Need. +# -------------------------------------------------------------------------- + + +EXTERNAL_REPORT = """ +External +======== + +.. mod_ver_report:: External Verification Report + :id: mod_vrep__external + :belongs_to: mod__baselibs + :covers: comp__external_thing + :safety: QM + :security: NO + :status: valid + :verification_method: inspection + :titles: + comp__external_thing = External Component +""" + + +def test_needtable_resolves_an_external_need(build: AppFactory) -> None: + app = build( + {"index": _index("report"), "report": EXTERNAL_REPORT + ARCHITECTURE}, + ) + html = (Path(app.outdir) / "report.html").read_text() + # Resolved by sphinx-needs from needs.json, including its external URL -- + # the extension itself never looked at the need. + assert "External Component" in html + assert "https://example.invalid/docs" in html + + +# -------------------------------------------------------------------------- +# Acceptance tests 9 + 10 at build level. +# -------------------------------------------------------------------------- + + +DUPLICATE_REPORT = """ +Dupes +===== + +.. mod_ver_report:: Duplicate Report + :id: mod_vrep__dupes + :belongs_to: mod__baselibs + :covers: comp__baselibs_json, comp__baselibs_json, comp__baselibs_bit_manipulation + :safety: QM + :security: NO + :status: valid + :verification_method: test + :titles: + comp__baselibs_json = Same Title + comp__baselibs_bit_manipulation = Same Title +""" + + +def test_duplicates_and_title_collisions_are_deterministic( + build: AppFactory, +) -> None: + app = build({"index": _index("report"), "report": DUPLICATE_REPORT + ARCHITECTURE}) + + assert "duplicate entry 'comp__baselibs_json'" in app.warning.getvalue() + + doctree = app.env.get_doctree("report") + titles = _titles(doctree) + assert titles.count("Same Title") == 2 # collision allowed ... + ids = _section_ids(doctree) + assert "mod-vrep-dupes-comp-baselibs-json" in ids # ... anchors still differ + assert "mod-vrep-dupes-comp-baselibs-bit-manipulation" in ids + assert len(ids) == len(set(ids)) + + +HOSTILE_REPORT = """ +Hostile +======= + +.. mod_ver_report:: Hostile Report + :id: mod_vrep__hostile + :belongs_to: mod__baselibs + :covers: comp__baselibs_json, comp__baselibs_bit_manipulation, evil"or(1) + :safety: QM + :security: NO + :status: valid + :verification_method: test +""" + + +def test_hostile_component_id_is_rejected_not_interpolated( + build: AppFactory, +) -> None: + app = build({"index": _index("report"), "report": HOSTILE_REPORT + ARCHITECTURE}) + warnings = app.warning.getvalue() + assert "is not a valid need id and is skipped" in warnings + + titles = _titles(app.env.get_doctree("report")) + assert "Baselibs Json" in titles or "Json" in " ".join(titles) + # The rejected token never became a section. + assert not [t for t in titles if "evil" in t.lower()] + + +# -------------------------------------------------------------------------- +# Version qualifiers are reported, not silently stripped. +# -------------------------------------------------------------------------- + + +VERSIONED_REPORT = """ +Versioned +========= + +.. mod_ver_report:: Versioned Report + :id: mod_vrep__versioned + :belongs_to: mod__baselibs + :covers: comp__baselibs_json[version==1], comp__baselibs_bit_manipulation + :safety: QM + :security: NO + :status: valid + :verification_method: test +""" + + +def test_version_qualifier_warns(build: AppFactory) -> None: + app = build({"index": _index("report"), "report": VERSIONED_REPORT + ARCHITECTURE}) + assert "ignoring version qualifier" in app.warning.getvalue() + assert "Baselibs Json" in _titles(app.env.get_doctree("report")) + + +# -------------------------------------------------------------------------- +# The extension owns no build lifecycle beyond registering its directives. +# -------------------------------------------------------------------------- + + +NEEDS_EXTRA_LINKS_BLOCK = """needs_extra_links = [ + dict(option="belongs_to", incoming="belongs to", outgoing="belongs to"), + dict(option="includes", incoming="included by", outgoing="includes"), + dict(option="covers", incoming="covered by", outgoing="covers"), + dict(option="contains", incoming="contained by", outgoing="contains"), + dict(option="evidence", incoming="evidence for", outgoing="evidence"), +]""" + +NEEDS_LINKS_BLOCK = """needs_links = { + "belongs_to": dict(incoming="belongs to", outgoing="belongs to"), + "includes": dict(incoming="included by", outgoing="includes"), + "covers": dict(incoming="covered by", outgoing="covers"), + "contains": dict(incoming="contained by", outgoing="contains"), + "evidence": dict(incoming="evidence for", outgoing="evidence"), +}""" + +NO_EVIDENCE_LINKS_BLOCK = """needs_extra_links = [ + dict(option="belongs_to", incoming="belongs to", outgoing="belongs to"), + dict(option="includes", incoming="included by", outgoing="includes"), + dict(option="covers", incoming="covered by", outgoing="covers"), +]""" + + +def test_evidence_section_honours_the_needs_links_dict(build: AppFactory) -> None: + """``score_metamodel`` writes ``needs_links``, not ``needs_extra_links``. + + Looking only at the deprecated list silently drops the Verification + Evidence section in every real project. + """ + conf = CONF_PY.replace(NEEDS_EXTRA_LINKS_BLOCK, NEEDS_LINKS_BLOCK) + assert conf != CONF_PY + app = build({"index": _index("report"), "report": REPORT + ARCHITECTURE}, conf=conf) + assert "Verification Evidence" in _titles(app.env.get_doctree("report")) + + +def test_evidence_section_is_skipped_when_the_links_are_not_configured( + build: AppFactory, +) -> None: + conf = CONF_PY.replace(NEEDS_EXTRA_LINKS_BLOCK, NO_EVIDENCE_LINKS_BLOCK) + assert conf != CONF_PY + report = REPORT.replace(" :contains: tcase__baselibs_1\n", "") + app = build({"index": _index("report"), "report": report + ARCHITECTURE}, conf=conf) + titles = _titles(app.env.get_doctree("report")) + assert "Verification Evidence" not in titles + assert "Components" in titles + + +def test_extension_has_no_build_lifecycle_hooks() -> None: + import inspect + + import score_module_verification_report as ext + + source = inspect.getsource(ext) + connects = re.findall(r'app\.connect\(\s*"([^"]+)"', source) + assert connects == ["config-inited"], ( + "The extension must own no build lifecycle: no env-updated re-read, no " + "build-finished consistency pass, no registry. The single config-inited " + f"handler registers directives only. Found: {connects}" + ) diff --git a/src/extensions/score_sphinx_bundle/BUILD b/src/extensions/score_sphinx_bundle/BUILD index 113803b97..0cb93f404 100644 --- a/src/extensions/score_sphinx_bundle/BUILD +++ b/src/extensions/score_sphinx_bundle/BUILD @@ -37,6 +37,7 @@ py_library( "@score_docs_as_code//src/extensions/score_mounts", "@score_docs_as_code//src/extensions/score_source_code_linker", "@score_docs_as_code//src/extensions/score_metrics", + "@score_docs_as_code//src/extensions/score_module_verification_report", "@score_docs_as_code//src/extensions/score_sync_toml", "@score_docs_as_code//src/helper_lib", ], diff --git a/src/extensions/score_sphinx_bundle/__init__.py b/src/extensions/score_sphinx_bundle/__init__.py index f3399e507..6c0546aae 100644 --- a/src/extensions/score_sphinx_bundle/__init__.py +++ b/src/extensions/score_sphinx_bundle/__init__.py @@ -41,6 +41,7 @@ "needs_config_writer", "score_sync_toml", "score_metrics", + "score_module_verification_report", "broken_link_fix", ] diff --git a/src/needs_templates/mod_ver_report_tiny.need b/src/needs_templates/mod_ver_report_tiny.need new file mode 100644 index 000000000..706250e31 --- /dev/null +++ b/src/needs_templates/mod_ver_report_tiny.need @@ -0,0 +1,237 @@ +{# + Module verification report template. + + This is the read-phase K adaptation of the report template originally used + by the complex-template PoC. In contrast to that PoC, ``components`` is + supplied from the report's explicit ``:covers:`` declaration and the report + headings are real RST sections, not rubrics inside a Need node. +#} +.. {{ report_need_directive }}:: {{ report_title }} +{% for name, value in report_options.items() %} + :{{ name }}: {{ value or "" }} +{% endfor %} +{% if report_content %} + +{% for line in report_content %} + {{ line }} +{% endfor %} +{% endif %} + +.. raw:: html + + + +.. _{{ report_id }}__feature: + +Feature ++++++++++++++++++ + +This document provides a verification report of the {{ module_short }} feature +and its associated components. + +.. needtable:: + :filter: id == "{{ feature_id }}" + :columns: title as "Name";id as "Id";safety;security;status + :style: table + +Feature Requirements Statistics +------------------------------- + +.. grid:: 1 2 2 2 + :gutter: 3 + + .. grid-item:: + + .. needpie:: Feature Requirements Status + :labels: valid, invalid + :colors: #37a12d, #ca2828 + :legend: + + type == "feat_req" and "{{ feature_id }}" in satisfied_by and status == "valid" + type == "feat_req" and "{{ feature_id }}" in satisfied_by and status == "invalid" + + .. grid-item:: + + .. needpie:: Feature Requirements Test Coverage + :labels: fully covered, partially covered, not covered + :colors: #37a12d, #f0a500, #ca2828 + :legend: + + type == "feat_req" and "{{ feature_id }}" in satisfied_by and len(fully_verifies_back) > 0 + type == "feat_req" and "{{ feature_id }}" in satisfied_by and len(partially_verifies_back) > 0 and len(fully_verifies_back) == 0 + type == "feat_req" and "{{ feature_id }}" in satisfied_by and len(fully_verifies_back) == 0 and len(partially_verifies_back) == 0 + +.. dropdown:: Show requirements table + :animate: fade-in + + .. needtable:: + :filter: type == "feat_req" and "{{ feature_id }}" in satisfied_by + :style: table + :columns: id;title;safety;status;fully_verifies_back;partially_verifies_back + :colwidths: 13,22,8,10,23,24 + :sort: id + +Feature Architecture Statistics +------------------------------- + +.. needtable:: + :filter: type in ["feat_arc_sta", "feat_arc_dyn"] and "{{ feature_id }}" in belongs_to + :style: table + :columns: id;title;safety;status;tags + :sort: id + +Feature Inspection Statistics +----------------------------- + +Presence of the feature-level inspection work products. + +.. _{{ report_id }}__verification-scope: + +Verification Scope +++++++++++++++++++ + +.. needtable:: + :filter: {{ scope_filter }} + :columns: id;title;type;status;safety;security + :style: table + +.. _{{ report_id }}__components: + +Components +++++++++++ + +.. needtable:: + :filter: type == "comp" and "{{ module_id }}" in includes_back + :columns: id as "Component";safety;security;status + :style: table + :sort: id + +{% macro component_work_product(component, work_product, kind) %} + * - :need:`{{ work_product }}` + - {{ kind }} + - .. needtable:: + :filter: type == "document" and "{{ component.slug|replace("_", "")|lower }}" in id.replace("_", "").lower() and "{{ work_product }}" in realizes + :columns: id + :style: table + - .. needtable:: + :filter: type == "document" and "{{ component.slug|replace("_", "")|lower }}" in id.replace("_", "").lower() and "{{ work_product }}" in realizes + :columns: status + :style: table +{% endmacro %} + +{% for component in components %} +.. _{{ report_id }}__{{ component.id }}: + +{{ component.title }} +{{ "+" * [component.title|length, 3]|max }} + +:need:`{{ component.id }}` + +Component Requirements Statistics +--------------------------------- + +.. grid:: 1 2 2 2 + :gutter: 3 + + .. grid-item:: + + .. needpie:: {{ component.title }} Requirements Status + :labels: valid, invalid + :colors: #37a12d, #ca2828 + :legend: + + type == "comp_req" and "{{ component.id }}" in satisfied_by and status == "valid" + type == "comp_req" and "{{ component.id }}" in satisfied_by and status == "invalid" + + .. grid-item:: + + .. needpie:: {{ component.title }} Requirements Test Coverage + :labels: fully covered, partially covered, not covered + :colors: #37a12d, #f0a500, #ca2828 + :legend: + + type == "comp_req" and "{{ component.id }}" in satisfied_by and len(fully_verifies_back) > 0 + type == "comp_req" and "{{ component.id }}" in satisfied_by and len(partially_verifies_back) > 0 and len(fully_verifies_back) == 0 + type == "comp_req" and "{{ component.id }}" in satisfied_by and len(fully_verifies_back) == 0 and len(partially_verifies_back) == 0 + +Requirements Traceability +------------------------- + +.. dropdown:: Show requirements table + :animate: fade-in + + .. needtable:: + :filter: type == "comp_req" and "{{ component.id }}" in satisfied_by + :style: table + :columns: id;title;safety;status;fully_verifies_back;partially_verifies_back + :colwidths: 13,22,8,10,23,24 + :sort: id + +Architectural Elements +---------------------- + +.. dropdown:: Show architectural elements table + :animate: fade-in + + .. needtable:: + :filter: type in ["comp_arc_sta", "comp_arc_dyn"] and "{{ component.id }}" in belongs_to + :style: table + :columns: id;title;safety;status;tags + :colwidths: 25,30,10,15,20 + :sort: id + +Verification & Safety Analysis Documents +---------------------------------------- + +.. dropdown:: Show work products table + :animate: fade-in + + .. list-table:: + :header-rows: 1 + :widths: 30 25 25 20 + :class: wp-doc-table + + * - Work Product + - Kind + - Realized by + - Status +{{ component_work_product(component, "wp__requirements_inspect", "Requirements Inspection") }} +{{ component_work_product(component, "wp__sw_arch_verification", "Architecture Inspection") }} +{{ component_work_product(component, "wp__sw_implementation_inspection", "Implementation Inspection") }} +{{ component_work_product(component, "wp__sw_component_dfa", "DFA") }} +{{ component_work_product(component, "wp__sw_component_fmea", "FMEA") }} +{% endfor %} + +{% if evidence_filter %} +.. _{{ report_id }}__verification-evidence: + +Verification Evidence ++++++++++++++++++++++ + +.. needtable:: + :filter: {{ evidence_filter }} + :columns: id;title;type;status + :style: table +{% endif %}