diff --git a/src/extensions/docs/index.rst b/src/extensions/docs/index.rst index 0ae5047f4..28bc0394a 100644 --- a/src/extensions/docs/index.rst +++ b/src/extensions/docs/index.rst @@ -63,6 +63,17 @@ Extensions `ubCode `__ VS Code extension. Getting IDE support for Sphinx-Needs in a Bazel context made easy. + .. grid-item-card:: + + Module Verification Report + ^^^ + One Need per module gives that module a report page whose sections behave + like ordinary RST. + :ref:`Module Verification Report`, + or jump straight to the + :ref:`live example`. + + .. grid-item-card:: Mounts @@ -80,4 +91,6 @@ Extensions Source Code Linker Extension Guide Sync TOML + Module Verification Report + Module Verification Report Example mounts_internals diff --git a/src/extensions/docs/module_verification_report.md b/src/extensions/docs/module_verification_report.md new file mode 100644 index 000000000..f8154a175 --- /dev/null +++ b/src/extensions/docs/module_verification_report.md @@ -0,0 +1,151 @@ + + +(module-verification-report)= +# Module Verification Report + +Give a module a verification report page by writing **one Need**. The report's +sections are ordinary RST sections: they appear in the sidebar and the local +ToC, they are `:ref:`-able from other pages, they land in the search index, and +they survive into non-HTML builders. + +--- + +## Authoring + +```rst +.. mod_ver_report:: Baselibs Verification Report + :id: mod_vrep__baselibs + :belongs_to: mod__baselibs + :covers: comp__baselibs_json, comp__baselibs_bit_manipulation + :safety: ASIL_B + :security: NO + :status: valid + :verification_method: test + :titles: + comp__baselibs_json = JSON Utilities + + Free-form introduction. It becomes the Need's description. +``` + +Scaling to N modules means adding N Needs — nothing else. + +### Options + +| Option | Meaning | +| ------ | ------- | +| *argument* | Report title. | +| `:id:` | Report Need id. Mandatory; also namespaces every generated anchor. | +| `:belongs_to:` | The module this report is about. | +| `:covers:` | The components in scope. A **real link field**, comma and/or whitespace separated. | +| `:titles:` | Heading for each component section, one `component id = Heading` per line. The only option that does not reach the Need. | +| *anything else* | Forwarded verbatim to the Need. The metamodel decides what is valid. | + +Everything except `:titles:` ends up on the Need, so `covers_back` and the usual +link validation come for free. The feature the statistics are about is derived +from the module (`mod__x` → `feat__x`). + +### Generated sections + +A flat list, in this order: + +1. **Feature** — the verified feature. +2. **Feature Requirements Statistics** — status and test-coverage pie charts + plus a requirements table. +3. **Feature Architecture Statistics** — status and inspection pie charts plus + an architecture elements table. +4. **Feature Inspection Statistics** — feature-level work products and the + documents realising them. +5. **Component Overview** — the covered components. +6. **One section per covered component** — component table, requirements and + architecture statistics, requirements traceability, test coverage, + architectural elements, and verification/safety-analysis documents. + +Inside a component section, the sub-parts are `rubric` directives rather than +sub-sections: no navigation is needed below the component level, and a flat list +keeps the sidebar readable. + +Every number on the page comes out of a `needtable` or `needpie` filter that +sphinx-needs evaluates after need collection. The extension computes none of +them. + +### Changing what a report says + +The body is a Jinja template, not Python. It is a Sphinx-Needs template file +(`mod_ver_report.need`) and lives in `needs_template_folder` alongside every +other need template; the extension sets that config to the folder shipped with +Docs-as-Code unless your `conf.py` already set it. + +To change the report, put your own `mod_ver_report.need` into your template +folder: + +```python +needs_template_folder = "docs/_needs_templates" +``` + +Your folder is searched first and the shipped one is the fallback, so a folder +without the file still builds. The file is rendered by the directive rather +than by Sphinx-Needs' `:template:` option — `:template:` renders into the +Need's content, where headings can never become sections. + +Anchors are namespaced with the report id, e.g. + +```rst +See :ref:`mod_vrep__baselibs__comp__baselibs_json`. +``` + +so they are stable across rebuilds and two reports on one page never collide. + +--- + +## Why the component list has to be written out + +Sphinx turns headings into sections exactly once, during the read phase, before +the Need graph exists. A report cannot therefore discover its own components +from the graph and still get real sections — the two happen at different times. + +So the list is authored, and drift is **detected rather than silently +corrected**: the metamodel check +`check_mod_ver_report_scope` compares the report's `:covers:` against the +module's `includes` in **both** directions and fails the build if they differ. +The fix is always a one-line edit. + +--- + +## Design rule for contributors + +> The extension emits RST. It never reads the Need model to compute an answer. + +The directive emits `needtable` filters and `:need:` references; sphinx-needs +resolves them after collection. That is why the extension has no `NeedsView`, no +registry, no `build-finished` pass and no build lifecycle hooks at all. + +If new report content needs Python that walks needs and computes something, the +line has been crossed. If it needs a new `needtable` filter in the template, it +has not. + +This is also why the template does not use the upstream `linked_needs()` render +helper to discover components: that reads the Need graph during rendering, which +is what forces a second read pass. + +Two invariants are enforced by tests and must not be "cleaned up": + +- **The report body is a sibling of the Need, not its child.** sphinx-needs + parses Need content with `match_titles=False`; moving the body inside the Need + node silently removes every section, and the HTML still looks fine. +- **`setup()` registers directives and nothing else.** The single + `config-inited` handler exists only because directive registration is + last-one-wins. + +See `src/extensions/score_module_verification_report/README.md` for the +configuration values and the full rationale. diff --git a/src/extensions/docs/module_verification_report_example.rst b/src/extensions/docs/module_verification_report_example.rst new file mode 100644 index 000000000..73d6a3422 --- /dev/null +++ b/src/extensions/docs/module_verification_report_example.rst @@ -0,0 +1,225 @@ +.. + # ******************************************************************************* + # Copyright (c) 2026 Contributors to the Eclipse Foundation + # + # See the NOTICE file(s) distributed with this work for additional + # information regarding copyright ownership. + # + # This program and the accompanying materials are made available under the + # terms of the Apache License Version 2.0 which is available at + # https://www.apache.org/licenses/LICENSE-2.0 + # + # SPDX-License-Identifier: Apache-2.0 + # ******************************************************************************* + +.. _module-verification-report-example: + +========================================== +Module Verification Report: Live Example +========================================== + +This page renders a real report. Everything below the "Rendered report" heading +is produced by a single ``mod_ver_report`` directive — look at the sidebar and +the local contents to see that its sections are ordinary RST sections. + +See :ref:`module-verification-report` for the reference documentation. + +.. note:: + + The needs on this page exist only to give the example something to point at. + They are not part of the Docs-as-Code architecture. + +The world being reported on +=========================== + +A feature with a requirement and an architecture element, a module with two +components, a component requirement covered by a test, and an inspection +document realising a work product: + +.. feat:: Example Baselibs Feature + :id: feat__example_baselibs + :version: 1 + :security: NO + :safety: QM + :status: valid + + Container feature for the example components. + +.. feat_req:: Example Baselibs Feature Requirement + :id: feat_req__example_feature__baselibs + :version: 1 + :reqtype: Functional + :security: NO + :safety: QM + :status: valid + :valid_from: v1.0 + :satisfied_by: feat__example_baselibs + + The feature shall provide basic library utilities. + +.. feat_arc_sta:: Example Baselibs Feature Package Diagram + :id: feat_arc_sta__example_feature__baselibs + :version: 1 + :security: NO + :safety: QM + :status: valid + :tags: inspected + :includes: logic_arc_int__example_feature__baselibs + :belongs_to: feat__example_baselibs + + Feature-level architecture view. Carries the ``inspected`` tag, so it shows + up as inspected in the feature architecture statistics. + +.. logic_arc_int:: Example Baselibs Logical Interface + :id: logic_arc_int__example_feature__baselibs + :version: 1 + :security: NO + :safety: QM + :status: valid + + Referenced by the feature package diagram above. + +.. comp:: Example JSON Component + :id: comp__example_baselibs_json + :version: 1 + :security: NO + :safety: QM + :status: valid + :belongs_to: feat__example_baselibs + + Parses and serialises JSON. + +.. comp:: Example Bit Manipulation Component + :id: comp__example_baselibs_bits + :version: 1 + :security: NO + :safety: QM + :status: valid + :belongs_to: feat__example_baselibs + + Bit-level helpers. + +.. comp_req:: Example JSON Round-Trip Requirement + :id: comp_req__example_feature__json_roundtrip + :version: 1 + :reqtype: Functional + :security: NO + :safety: QM + :status: valid + :satisfied_by: comp__example_baselibs_json + + The component shall serialise and parse a document without loss. + +.. comp_arc_sta:: Example JSON Package Diagram + :id: comp_arc_sta__example_feature__json + :version: 1 + :security: NO + :safety: QM + :status: valid + :tags: inspected + :belongs_to: comp__example_baselibs_json + + Architecture view of the JSON component. + +.. comp_arc_sta:: Example Bit Manipulation Package Diagram + :id: comp_arc_sta__example_feature__bits + :version: 1 + :security: NO + :safety: QM + :status: valid + :belongs_to: comp__example_baselibs_bits + + Architecture view of the bit manipulation component. Deliberately *not* + tagged ``inspected``, so the inspection pie chart is not all green. + +.. mod:: Example Baselibs Module + :id: mod__example_baselibs + :version: 1 + :security: NO + :safety: QM + :status: valid + :includes: comp__example_baselibs_json, comp__example_baselibs_bits + + The module the report below is about. + +.. document:: Example JSON Requirements Inspection + :id: doc__example_json_req_inspect + :version: 1 + :security: NO + :safety: QM + :status: valid + :realizes: wp__requirements_inspect + + The document realising the ``wp__requirements_inspect`` work product from + the process description. It is matched into the JSON component's work + product table because its id contains the component slug. + +What the author writes +====================== + +One Need. That is the whole input for the page you see below it: + +.. code-block:: rst + + .. mod_ver_report:: Example Baselibs Verification Report + :id: mod_vrep__example_feature__baselibs + :version: 1 + :belongs_to: mod__example_baselibs + :covers: comp__example_baselibs_json, comp__example_baselibs_bits + :safety: QM + :security: NO + :status: valid + :verification_method: test_and_inspection + :titles: + comp__example_baselibs_json = JSON Utilities + comp__example_baselibs_bits = Bit Manipulation + + Verification report for the example Baselibs module. + +``:covers:`` is a real link field, so ``mod__example_baselibs`` gets a +``covered by`` backlink and the metamodel checks that the list matches the +module's ``includes`` in both directions. + +``:titles:`` is the one presentation-only option: it names each component +section. Without it the headings are derived from the component ids +(``comp__example_baselibs_json`` → "Example Baselibs Json"). The feature the +statistics are about is derived the same way the upstream template derives it, +by rewriting ``mod__example_baselibs`` to ``feat__example_baselibs``. + +Rendered report +=============== + +.. mod_ver_report:: Example Baselibs Verification Report + :id: mod_vrep__example_feature__baselibs + :version: 1 + :belongs_to: mod__example_baselibs + :covers: comp__example_baselibs_json, comp__example_baselibs_bits + :safety: QM + :security: NO + :status: valid + :verification_method: test_and_inspection + :titles: + comp__example_baselibs_json = JSON Utilities + comp__example_baselibs_bits = Bit Manipulation + + Verification report for the example Baselibs module. + +The sections are real +===================== + +Each generated section carries an anchor namespaced with the report id, so it +can be referenced from anywhere like any other section: + +.. code-block:: rst + + See :ref:`mod_vrep__example_feature__baselibs__comp__example_baselibs_json`. + +Which renders as: +:ref:`mod_vrep__example_feature__baselibs__comp__example_baselibs_json` +— the link text comes from the section title, because the target *is* a +section. The same anchors appear in the sidebar, in the local contents, in the +search index and in the LaTeX/PDF bookmarks. + +The rubrics *inside* a component section (Requirements Statistics, Test +Coverage, …) are deliberately not sections: no navigation is needed below the +component level, and a flat list keeps the sidebar readable. diff --git a/src/extensions/score_metamodel/checks/mod_ver_report.py b/src/extensions/score_metamodel/checks/mod_ver_report.py new file mode 100644 index 000000000..8028c59a0 --- /dev/null +++ b/src/extensions/score_metamodel/checks/mod_ver_report.py @@ -0,0 +1,97 @@ +# ******************************************************************************* +# Copyright (c) 2026 Contributors to the Eclipse Foundation +# +# See the NOTICE file(s) distributed with this work for additional +# information regarding copyright ownership. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# ******************************************************************************* +"""Scope rule for module verification reports. + +The report page rendered by ``score_module_verification_report`` needs its +component list authored on the Need, because sections must exist at read time. +That makes drift possible, so the rule below makes drift *detected*, never +silently corrected: the build fails and someone edits one line. + +The rule is bidirectional and lives here -- in the metamodel -- rather than in +the rendering extension, so it applies to all reports regardless of how (or +whether) they are rendered. + +The YAML ``graph_checks`` DSL compares an attribute of a linked need against a +constant; it cannot express set equality between two link fields, which is why +this one check is written in Python. +""" + +from collections.abc import Iterable +from typing import cast + +from score_metamodel import CheckLogger, graph_check +from sphinx.application import Sphinx +from sphinx_needs.data import NeedsView +from sphinx_needs.need_item import NeedItem + +REPORT_TYPE = "mod_ver_report" +MODULE_TYPE = "mod" +COMPONENT_TYPE = "comp" + + +def _link_ids(need: NeedItem, option: str) -> list[str]: + """Read a link field as a list of ids, tolerating the single-string form.""" + value: object = need.get(option, None) + if not value: + return [] + if isinstance(value, str): + return [value] + return [str(item) for item in cast("Iterable[object]", value)] + + +@graph_check +def check_mod_ver_report_scope(app: Sphinx, needs: NeedsView, log: CheckLogger) -> None: + """``mod_ver_report.covers`` must match ``mod.includes`` exactly. + + * a component included by the module but missing from ``:covers:`` has no + section in the report -- the report silently under-reports its scope. + * a component in ``:covers:`` that the module does not include claims + verification of something outside the module. + + Only ``comp`` targets participate; ``:covers:`` may also point at + requirements or other artifacts and those are left alone. + """ + for need in needs.values(): + if need["type"] != REPORT_TYPE: + continue + + covered_components = { + need_id + for need_id in _link_ids(need, "covers") + if need_id in needs and needs[need_id]["type"] == COMPONENT_TYPE + } + + for module_id in _link_ids(need, "belongs_to"): + module = needs.get(module_id, None) + if module is None or module["type"] != MODULE_TYPE: + # Wrong or dangling belongs_to target is reported by the + # regular link checks; nothing to add here. + continue + + included = set(_link_ids(module, "includes")) + + for missing in sorted(included - covered_components): + log.warning_for_option( + need, + "covers", + f"does not cover '{missing}', which is included by " + f"'{module_id}'. Add it to ':covers:' so the report gets a " + "section for it.", + ) + + for extra in sorted(covered_components - included): + log.warning_for_option( + need, + "covers", + f"covers '{extra}', which is not included by '{module_id}'.", + ) diff --git a/src/extensions/score_metamodel/tests/rst/graph/test_mod_ver_report_scope.rst b/src/extensions/score_metamodel/tests/rst/graph/test_mod_ver_report_scope.rst new file mode 100644 index 000000000..890bc05e5 --- /dev/null +++ b/src/extensions/score_metamodel/tests/rst/graph/test_mod_ver_report_scope.rst @@ -0,0 +1,99 @@ +.. + # ******************************************************************************* + # Copyright (c) 2026 Contributors to the Eclipse Foundation + # + # See the NOTICE file(s) distributed with this work for additional + # information regarding copyright ownership. + # + # This program and the accompanying materials are made available under the + # terms of the Apache License Version 2.0 which is available at + # https://www.apache.org/licenses/LICENSE-2.0 + # + # SPDX-License-Identifier: Apache-2.0 + # ******************************************************************************* + + +.. test_metadata:: + :id: test_metadata__metamodel_graph_mod_ver_report + :partially_verifies_list: tool_req__docs_verification_report_need + :test_type: requirements_based + :derivation_technique: requirements_based + + Tests that a module verification report covers exactly the components of its + module — drift in either direction is detected, never silently corrected. + +--- Setup + +.. feat:: Report Scope Feature + :id: feat__report_scope + :security: NO + :safety: QM + :status: valid + +.. comp:: Report Scope Component A + :id: comp__report_scope_a + :security: NO + :safety: QM + :status: valid + :belongs_to: feat__report_scope + +.. comp:: Report Scope Component B + :id: comp__report_scope_b + :security: NO + :safety: QM + :status: valid + :belongs_to: feat__report_scope + +.. comp:: Report Scope Component Outside The Module + :id: comp__report_scope_outside + :security: NO + :safety: QM + :status: valid + :belongs_to: feat__report_scope + +.. mod:: Report Scope Module + :id: mod__report_scope + :security: NO + :safety: QM + :status: valid + :includes: comp__report_scope_a, comp__report_scope_b + +--- + +.. Positive test: covers exactly the included components — no warning expected. + +.. mod_ver_report:: Complete Report + :id: mod_vrep__report_scope__complete + :safety: QM + :security: NO + :status: valid + :verification_method: test + :belongs_to: mod__report_scope + :covers: comp__report_scope_a, comp__report_scope_b + :expect_not: does not cover, which is not included by + +.. Negative test: an included component is missing from ':covers:'. + +.. mod_ver_report:: Report Missing A Component + :id: mod_vrep__report_scope__missing + :safety: QM + :security: NO + :status: valid + :verification_method: test + :belongs_to: mod__report_scope + :covers: comp__report_scope_a + :expect: does not cover 'comp__report_scope_b', which is included by 'mod__report_scope' + :expect_not: which is not included by + +.. Negative test: a covered component is not part of the module. + +.. mod_ver_report:: Report Covering Too Much + :id: mod_vrep__report_scope__extra + :safety: QM + :security: NO + :status: valid + :verification_method: test + :belongs_to: mod__report_scope + :covers: comp__report_scope_a, comp__report_scope_b, comp__report_scope_outside + :expect: covers 'comp__report_scope_outside', which is not included by 'mod__report_scope' + :expect_not: does not cover diff --git a/src/extensions/score_metamodel/tests/rst/options/test_options_verification_report.rst b/src/extensions/score_metamodel/tests/rst/options/test_options_verification_report.rst index c6f84faed..9e5c1e5bc 100644 --- a/src/extensions/score_metamodel/tests/rst/options/test_options_verification_report.rst +++ b/src/extensions/score_metamodel/tests/rst/options/test_options_verification_report.rst @@ -60,7 +60,7 @@ :branch_coverage_percent: 85 :applies_to_module_version: 1.0.0 :belongs_to: mod__verification_module - :covers: comp_req__verification__sample + :covers: comp__verification_component, comp_req__verification__sample .. Invalid coverage percentage value in module verification report @@ -74,3 +74,4 @@ :verification_method: inspection :line_coverage_percent: 150 :belongs_to: mod__verification_module + :covers: comp__verification_component diff --git a/src/extensions/score_module_verification_report/BUILD b/src/extensions/score_module_verification_report/BUILD new file mode 100644 index 000000000..07e791f2d --- /dev/null +++ b/src/extensions/score_module_verification_report/BUILD @@ -0,0 +1,46 @@ +# ******************************************************************************* +# Copyright (c) 2026 Contributors to the Eclipse Foundation +# +# See the NOTICE file(s) distributed with this work for additional +# information regarding copyright ownership. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# ******************************************************************************* +load("@aspect_rules_py//py:defs.bzl", "py_library") +load("@docs_as_code_hub_env//:requirements.bzl", "all_requirements") +load("//:score_pytest.bzl", "score_pytest") + +filegroup( + name = "all_sources", + srcs = glob(["*.py"]), + visibility = ["//visibility:public"], +) + +py_library( + name = "score_module_verification_report", + srcs = [":all_sources"], + # The report body is a Sphinx-Needs template. Keep it beside the extension + # in the Bazel runfiles tree; the extension derives the folder from its own + # __file__ and hands it to Sphinx as ``needs_template_folder``. + data = ["@score_docs_as_code//src/needs_templates:files"], + imports = ["."], + visibility = ["//visibility:public"], + deps = all_requirements + [ + "@score_docs_as_code//src/helper_lib", + ], +) + +score_pytest( + name = "score_module_verification_report_test", + size = "medium", + srcs = glob(["tests/*.py"]), + pytest_config = "//:pyproject.toml", + deps = [ + ":score_module_verification_report", + "@score_docs_as_code//src/extensions/score_metamodel", + ], +) diff --git a/src/extensions/score_module_verification_report/README.md b/src/extensions/score_module_verification_report/README.md new file mode 100644 index 000000000..e7ca19260 --- /dev/null +++ b/src/extensions/score_module_verification_report/README.md @@ -0,0 +1,192 @@ + + +# `score_module_verification_report` + +Per-module verification report pages whose sections behave like ordinary RST. + +Resolves [#764](https://github.com/eclipse-score/docs-as-code/issues/764) — option **K**. + +## The governing design rule + +> **The extension emits RST. It never reads the Need model to compute an answer.** + +Rendering, not resolving. The directive emits `needtable` filters and `:need:` +references; sphinx-needs resolves all of them after collection, with its own +semantics, its own external-need handling and its own backlinks. The extension +therefore needs no `NeedsView`, no build lifecycle hook and no model +completeness at read time. The only thing it knows at read time is *which +sections exist*. + +**Test for future changes:** if adding report content requires new Python that +walks needs and computes something, the line has been crossed. If it requires a +new `needtable` filter in the template, it has not. + +## Authoring surface + +One Need per module. That is the whole consumer-facing API: + +```rst +.. mod_ver_report:: Baselibs Verification Report + :id: mod_vrep__baselibs + :belongs_to: mod__baselibs + :covers: comp__baselibs_json, comp__baselibs_bit_manipulation + :safety: ASIL_B + :security: NO + :status: valid + :verification_method: test + :titles: + comp__baselibs_json = JSON Utilities + + Free-form introduction; becomes the Need's description. +``` + +Scaling to N modules = adding N Needs. Nothing else. + +`:covers:` and `:belongs_to:` are **real link fields on the Need**, not opaque +directive options. That is what lets the metamodel validate them, generate +backlinks for free and report through the normal warning pipeline — and it is +why this extension owns no consistency-checking code. The feature the +statistics are about is derived from the module (`mod__x` → `feat__x`), exactly +as the upstream template does it; it feeds an `id == ...` filter, so the worst +case is an empty feature table. + +`:titles:` is the only option that does not reach the Need: `component id = +Heading`, one per line. Without it the heading is derived from the id +(`comp__baselibs_json` → "Baselibs Json") — a deliberate last-resort fallback; +the real title is resolved by sphinx-needs in the component table. + +## What gets emitted + +A `mod_ver_report` Need followed by a **flat** list of sections. The content +follows the standard module verification report (the upstream +`mod_ver_report_tiny.need` template): + +| Section | Content | +| ------- | ------- | +| Feature | `needtable` on the verified feature | +| Feature Requirements Statistics | status + test-coverage `needpie`, plus a requirements `needtable` in a dropdown | +| Feature Architecture Statistics | status + inspection `needpie`, plus an elements `needtable` | +| Feature Inspection Statistics | work products and the documents realising them | +| Component Overview | `needtable` over the covered components | +| *one per component* | component table, requirements + architecture statistics, requirements traceability, test coverage, architectural elements, and verification/safety-analysis documents | + +Inside a component section the sub-parts stay `rubric` directives. Constraint 3 +says no subsection nesting is needed, and nothing below a component level needs +its own anchor. + +Every section is preceded by an explicit target namespaced with the report id +(`mod_vrep__baselibs__comp__baselibs_json`), so anchors are stable across +rebuilds and two reports on one page never collide. + +## The report body is a template + +[`src/needs_templates/mod_ver_report.need`](../../needs_templates/mod_ver_report.need) +holds the whole report. It is a Sphinx-Needs template file — `.need` extension, +living in `needs_template_folder` next to every other need template in this +repository. `setup()` sets `needs_template_folder` to the shipped folder unless +the project set it itself. + +It is rendered by the **directive**, not by Sphinx-Needs' `:template:` option. +That difference is load-bearing: `:template:` renders into the Need's content, +which Sphinx-Needs parses with `match_titles=False`, so headings written there +can never become sections. The directive renders the file with a context built +only from the directive's options and from configuration, and returns the parsed +result into the surrounding document. + +Changing what a report *says* means editing the template, not Python. A project +overrides it by dropping its own `mod_ver_report.need` into its +`needs_template_folder`; that folder is searched first and the shipped one is +the fallback, so a project template folder that does not contain the file still +builds. + +The whole context is `report_id`, `module_id`, `module_slug`, `feature_id` and +`components` (each with `id`, `title` and `slug`), plus a `q` filter that safely +quotes a need id for a filter string. Column lists, table filters, the +work-product rows and the section list all live in the template — the extension +has **no configuration values of its own**. + +### Differences from the upstream `.need` template + +The upstream template is a sphinx-needs `.need` template, rendered *inside* the +Need node. Two changes were required: + +- **Top-level `rubric` directives became sections.** A rubric looks like a + heading and is nothing like one: no anchor, no ToC entry, no `:ref:` target, + no search-index entry, and nothing at all in non-HTML builders. This is the + entire point of the approach. +- **`linked_needs(module_id, "includes")` is gone.** Reading the Need graph + while rendering is what forces the second read pass with its silent-failure + mode. The component list comes from the report's own `covers` link field; the + metamodel check enforces that it matches the module's `includes`. + +## Two things that look like implementation details but are not + +**1. The report body is a *sibling* of the Need, never its child.** + +Sphinx turns headings into `section` nodes exactly once, during the read phase. +Sections are what produce anchors, sidebar entries, `:ref:` targets, +search-index entries and PDF bookmarks. sphinx-needs parses Need content with +`match_titles=False`, so a heading inside a `.need` template can never become a +section. The directive therefore parses its generated RST with +`parse_text_to_nodes(..., allow_section_headings=True)` and returns the nodes +into the surrounding document. + +Moving the body inside the Need node — to make it render in the Need's box — +silently removes every section from the document and cannot be spotted by +looking at the HTML. Don't. + +**2. The extension owns no build lifecycle.** + +No `env-updated` re-read, no `build-finished` consistency pass, no +`env.*_registry` merged across parallel workers. `setup()` declares config +values and registers directives. + +There is exactly one `config-inited` handler and it calls `add_directive` +twice — directive registration is last-one-wins and sphinx-needs registers +`mod_ver_report` from its own `config-inited` handler. It touches no +environment state and reads no needs. +`tests/test_report_integration.py` asserts this stays true. + +## Validation lives in the metamodel + +`src/extensions/score_metamodel/checks/mod_ver_report.py` enforces the rule with +real content: **every component in `mod__x.includes` must appear in the report's +`:covers:`, and vice versa** — bidirectional, so an omitted component is caught +as well as an over-claimed one. + +Failure mode by design: drift is **detected, never silently corrected**. +Sections must exist at read time, so the list must be authored. The build fails +and someone edits one line. + +Everything else — mandatory fields, allowed link targets — is already declared +in `metamodel.yaml` under `mod_ver_report`. + +## Status + +Proof of concept. Known gaps: + +- **Work-product documents are matched by id substring.** `document` needs carry + no link to the component they belong to, so the template's `document_filter` + macro falls back to `{slug} in id.replace("_", "").lower()`. This *will* + produce false positives (`json` also matches `jsonschema`). It is a filter in + a template, not hand-written Python — replace it as soon as the metamodel + models the link. +- **LCOV coverage is not integrated.** Reading a coverage report during + directive execution introduces an untracked Sphinx dependency and breaks + incremental correctness and Bazel reproducibility. The section renders a note + saying so. +- `:covers:` is still `ANY` in `metamodel.yaml`; narrowing the allowed target + types is a separate, consumer-affecting change. +- Two reports placed at *different* heading depths on one page rely on docutils' + title-style bookkeeping and are untested. diff --git a/src/extensions/score_module_verification_report/__init__.py b/src/extensions/score_module_verification_report/__init__.py new file mode 100644 index 000000000..90c1c64d5 --- /dev/null +++ b/src/extensions/score_module_verification_report/__init__.py @@ -0,0 +1,96 @@ +# ******************************************************************************* +# Copyright (c) 2026 Contributors to the Eclipse Foundation +# +# See the NOTICE file(s) distributed with this work for additional +# information regarding copyright ownership. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# ******************************************************************************* +"""Module Verification Report. + +One Need per module gives that module a report page whose sections behave like +ordinary RST: real anchors, real ToC entries, real ``:ref:`` targets, present in +every builder. + +See ``directive.py`` for the design rules. In short: the extension emits RST +and never reads the Need model; scope is a real link field validated by the +metamodel. + +Lifecycle +--------- +The extension deliberately has **no** build lifecycle hooks: no ``env-updated`` +re-read, no ``build-finished`` consistency pass, no registry carried across +parallel workers, and no configuration of its own -- the template is the place +to change what a report says. ``setup()`` points ``needs_template_folder`` at +the shipped templates and registers two directives. + +The single ``config-inited`` handler exists only because directive registration +is last-one-wins and sphinx-needs registers ``mod_ver_report`` from its own +``config-inited`` handler (priority 500). The handler below runs at priority +900 and does nothing but call ``add_directive`` twice. It touches no +environment state and reads no needs. +""" + +from __future__ import annotations + +from score_module_verification_report import rendering +from score_module_verification_report.directive import ( + REPORT_NEED_DIRECTIVE, + REPORT_TYPE, + ModuleVerificationReportDirective, +) +from sphinx.application import Sphinx +from sphinx.config import Config +from sphinx_needs.directives.need import NeedDirective + +from src.helper_lib import config_setdefault + +__all__ = ["setup"] + + +class ReportNeedDirective(NeedDirective): + """sphinx-needs' Need directive, reachable under an un-shadowed name. + + ``NeedDirective`` derives the need type from the directive name it was + invoked as. The public ``mod_ver_report`` name belongs to the report + directive, so the generated RST calls this alias instead and it restores + the intended type. + """ + + def run(self): # type: ignore[no-untyped-def] + self.name = REPORT_TYPE + return super().run() + + +def _register_directives(app: Sphinx, config: Config) -> None: + # Runs late on config-inited, after sphinx-needs registered a plain + # NeedDirective under REPORT_TYPE. Registration only -- no model access. + app.add_directive(REPORT_NEED_DIRECTIVE, ReportNeedDirective, override=True) + app.add_directive(REPORT_TYPE, ModuleVerificationReportDirective, override=True) + + +def setup(app: Sphinx) -> dict[str, object]: + app.setup_extension("sphinx_needs") + # The default template uses grid/dropdown (sphinx_design) and needpie. + app.setup_extension("sphinx_design") + + # The report body is a Sphinx-Needs template file: ``.need`` extension, + # living in ``needs_template_folder`` next to every other need template. + # Point that config at the folder shipped with docs-as-code unless the + # project set it itself -- in which case the project's folder is searched + # first and the shipped one is the fallback. + config_setdefault( + app.config, "needs_template_folder", str(rendering.shipped_template_folder()) + ) + + app.connect("config-inited", _register_directives, priority=900) + + return { + "version": "0.1.0", + "parallel_read_safe": True, + "parallel_write_safe": True, + } diff --git a/src/extensions/score_module_verification_report/directive.py b/src/extensions/score_module_verification_report/directive.py new file mode 100644 index 000000000..f354bba55 --- /dev/null +++ b/src/extensions/score_module_verification_report/directive.py @@ -0,0 +1,245 @@ +# ******************************************************************************* +# Copyright (c) 2026 Contributors to the Eclipse Foundation +# +# See the NOTICE file(s) distributed with this work for additional +# information regarding copyright ownership. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# ******************************************************************************* +"""The ``mod_ver_report`` directive. + +Design decisions that are load-bearing and must not be "cleaned up" +------------------------------------------------------------------- + +1. **The report body is a sibling of the Need, never its child.** + + Sphinx turns headings into ``section`` nodes exactly once, during the read + phase. Sections are what produce anchors, sidebar entries, ``:ref:`` + targets, search-index entries and PDF bookmarks. After reading, Sphinx + never looks for headings again. + + sphinx-needs parses Need content with ``match_titles=False``, so a heading + written inside a ``.need`` template can never become a section. Therefore + this directive parses the generated report text with + ``parse_text_to_nodes(..., allow_section_headings=True)`` and returns the + resulting nodes *into the surrounding document*. The generated + ``mod_ver_report`` Need is the first of those nodes; the sections follow it + as siblings. + + A reviewer may be tempted to ask for the body to be moved inside the Need + node so that it renders in the Need's box. Doing so silently removes every + section from the document and cannot be spotted by looking at the HTML. + Do not do it. + +2. **The extension emits RST. It never reads the Need model.** + + The only thing known at read time is *which sections exist*. Everything + with semantics -- titles, coverage, backlinks, external needs -- is deferred + to ``needtable`` filters and ``:need:`` references that sphinx-needs + resolves after collection. Consequently this directive needs no + ``NeedsView``, no lifecycle hook and no model completeness at read time. + + Test for future changes: if new report content needs Python that walks needs + and computes something, the line has been crossed. If it needs a new + ``needtable`` filter in the template, it has not. + +3. **Scope is validated by the metamodel, not here.** + + ``:covers:`` is passed through verbatim as a real link field on the Need. + sphinx-needs validates target existence and type, generates ``covers_back`` + for free, and reports through the normal warning pipeline. The + "every component of the module is covered, and vice versa" rule lives in + ``score_metamodel`` as a graph check. +""" + +from __future__ import annotations + +import re +from collections.abc import Callable, Sequence +from typing import Any, ClassVar, Final + +from docutils import nodes +from score_module_verification_report import rendering +from sphinx.util import logging +from sphinx.util.docutils import SphinxDirective +from sphinx_needs.utils import DummyOptionSpec + +logger = logging.getLogger(__name__) + +#: Need type / public directive name. +REPORT_TYPE: Final = "mod_ver_report" + +#: Internal directive name bound to sphinx-needs' ``NeedDirective`` for +#: ``REPORT_TYPE``. The public name is taken by this directive, so the +#: generated RST needs an un-shadowed way to actually create the Need. +REPORT_NEED_DIRECTIVE: Final = "mod_ver_report_need" + +#: Explicit section headings; the only option that does not reach the Need. +TITLES_OPTION: Final = "titles" + +#: ``.. ___:`` -- the targets the template emits. +TARGET_RE = re.compile(r"^\.\. _(\S+):$", re.MULTILINE) + + +class ModuleVerificationReportDirective(SphinxDirective): + """Emit one ``mod_ver_report`` Need plus a flat list of real sections.""" + + required_arguments = 1 + optional_arguments = 0 + final_argument_whitespace = True + has_content = True + # Any option is accepted and forwarded to the Need. The metamodel decides + # which options are mandatory, which are links and what they may target. + # Annotated with docutils' own type: ``Directive.option_spec`` is a mutable + # class variable, so a narrower type here is an invalid override. + option_spec: ClassVar[dict[str, Callable[[str], Any]] | None] = DummyOptionSpec() + + options: dict[str, str | None] + + def _warn(self, message: str, subtype: str = "report") -> None: + logger.warning( + f"{REPORT_TYPE}: {message}", + location=self.get_location(), + type=REPORT_TYPE, + subtype=subtype, + ) + + def _ids(self, option: str) -> list[str]: + ids, warnings = rendering.parse_ids(self.options.get(option)) + for message in warnings: + self._warn(message, option) + return ids + + def run(self) -> Sequence[nodes.Node]: + options = dict(self.options) + titles_raw = options.pop(TITLES_OPTION, None) + + report_id = (options.get("id") or "").strip() + try: + rendering.quote_for_filter(report_id) + except ValueError: + self._warn( + f"missing or unusable ':id:' ({report_id!r}); nothing rendered", "id" + ) + return [] + + component_ids = self._ids("covers") + module_id = next(iter(self._ids("belongs_to")), "") + + titles, warnings = rendering.parse_titles(titles_raw) + for message in warnings: + self._warn(message, TITLES_OPTION) + for unknown in sorted(set(titles) - set(component_ids)): + self._warn( + f"title for {unknown!r}, which is not in ':covers:'", TITLES_OPTION + ) + + text = rendering.render_need( + REPORT_NEED_DIRECTIVE, + self.arguments[0].strip(), + options, + list(self.content), + ) + text += "\n" + rendering.render_report( + { + "report_id": report_id, + "module_id": module_id, + "module_slug": rendering.derive_slug(module_id, ""), + "feature_id": rendering.derive_feature_id(module_id), + "components": [ + { + "id": component_id, + "title": titles.get(component_id) + or rendering.derive_title(component_id), + "slug": rendering.derive_slug(component_id, module_id), + } + for component_id in component_ids + ], + }, + self.config.needs_template_folder or None, + ) + + parsed = self.parse_text_to_nodes(text, allow_section_headings=True) + _promote_report_anchors( + parsed, _emitted_anchors(text, report_id), self.state.document + ) + return parsed + + +def _emitted_anchors(text: str, report_id: str) -> set[str]: + """The section targets the template emitted, as docutils will id them. + + Scanning the rendered text rather than duplicating the section list in + Python keeps the template the single owner of the report's structure. + """ + prefix = f"{report_id}__" + return { + nodes.make_id(name) + for name in TARGET_RE.findall(text) + if name.startswith(prefix) + } + + +def _promote_report_anchors( + parsed: Sequence[nodes.Node], wanted: set[str], document: nodes.document +) -> None: + """Make the namespaced anchor each generated section's *primary* id. + + ``.. _name:`` in front of a heading gives the section a second id, but + docutils only merges it during the ``PropagateTargets`` transform and then + appends it, so ``section["ids"][0]`` stays the id docutils derived from the + heading text. That id is what Sphinx uses for the ToC entry and the HTML + element, and it is neither namespaced nor stable: two reports with the same + heading text on one page make docutils disambiguate with a ``-1`` suffix. + + So the merge is done here instead, at read time, in document order. Ids + and names are moved -- not copied -- from the target onto the section, and + the document's id/name maps are repointed accordingly. Nothing is + invented: these are exactly the ids docutils created from the targets the + report emitted. + """ + pending: nodes.target | None = None + + # ``parsed`` is the parser's own children list: removing a top-level + # target below mutates it, so iterate over a snapshot. + for root in list(parsed): + for node in root.findall(include_self=True): + if isinstance(node, nodes.target): + if not _is_plain_target(node): + continue + if wanted.intersection(node["ids"]): + pending = node + elif isinstance(node, nodes.section) and pending is not None: + _merge_target_into_section(pending, node, document) + pending = None + + +def _is_plain_target(target: nodes.target) -> bool: + """True for ``.. _name:`` targets that have not been resolved yet.""" + return not any(target.get(key) for key in ("refid", "refuri", "refname")) + + +def _merge_target_into_section( + target: nodes.target, section: nodes.section, document: nodes.document +) -> None: + ids = list(target["ids"]) + names = list(target["names"]) + + section["ids"] = ids + [i for i in section["ids"] if i not in ids] + section["names"] = names + [n for n in section["names"] if n not in names] + + for id_ in ids: + document.ids[id_] = section + for name in names: + document.nameids[name] = ids[0] + + # Emptied so that docutils' PropagateTargets transform does not merge the + # same ids a second time; removed so no stray anchor is left behind. + target["ids"] = [] + target["names"] = [] + if target.parent is not None: + target.parent.remove(target) diff --git a/src/extensions/score_module_verification_report/rendering.py b/src/extensions/score_module_verification_report/rendering.py new file mode 100644 index 000000000..14e32c896 --- /dev/null +++ b/src/extensions/score_module_verification_report/rendering.py @@ -0,0 +1,203 @@ +# ******************************************************************************* +# Copyright (c) 2026 Contributors to the Eclipse Foundation +# +# See the NOTICE file(s) distributed with this work for additional +# information regarding copyright ownership. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# ******************************************************************************* +"""Rendering helpers for the module verification report. + +Everything here is a string transformation. No function in this module +receives, reads or resolves a Need. That is the governing design rule: + + The extension emits RST. It never reads the Need model to compute an + answer. + +The report body lives in ``src/needs_templates/mod_ver_report.need`` -- a +Sphinx-Needs template file, resolved through ``needs_template_folder``. The +template owns the report's content *and* its section structure; this module +only builds the context and safely quotes the need ids that end up inside +filter strings. +""" + +from __future__ import annotations + +import json +import re +from functools import lru_cache +from pathlib import Path +from typing import Any + +from jinja2 import Environment, FileSystemLoader, StrictUndefined + +#: Sphinx-Needs templates end in ``.need`` and live in ``needs_template_folder``. +TEMPLATE_NAME = "mod_ver_report.need" + +# Conservative allow-list for anything interpolated into a sphinx-needs filter +# string. Filters are evaluated as Python, so a need id is never pasted in +# unchecked. +NEED_ID_RE = re.compile(r"^[A-Za-z0-9_.\-]+$") + +# ``comp__foo[version==1]``. sphinx-needs strips these itself on real link +# fields; we must not silently drop them while building sections, or the +# section is built for something the author did not write. +VERSION_QUALIFIER_RE = re.compile(r"\[[^\]]*\]$") + +_SPLIT_RE = re.compile(r"[,\s]+") + + +def quote_for_filter(need_id: str) -> str: + """Return *need_id* as a safely quoted literal for a needs filter string. + + Raises ``ValueError`` for anything that is not a plain need id; callers + turn that into a Sphinx warning. + """ + if not NEED_ID_RE.match(need_id): + raise ValueError(f"{need_id!r} is not a valid need id") + return json.dumps(need_id) + + +def parse_ids(raw: str | None) -> tuple[list[str], list[str]]: + """Parse a link option into an ordered, de-duplicated list of ids. + + Returns ``(ids, warnings)``. Accepts comma and/or whitespace separated ids + across lines. Version qualifiers are reported rather than silently + ignored, and duplicates are dropped deterministically. + """ + ids: list[str] = [] + warnings: list[str] = [] + for token in _SPLIT_RE.split((raw or "").strip()): + if not token: + continue + need_id = VERSION_QUALIFIER_RE.sub("", token) + if need_id != token: + warnings.append( + f"ignoring version qualifier on {token!r}; the report section " + f"is built for {need_id!r}" + ) + if not NEED_ID_RE.match(need_id): + warnings.append(f"{need_id!r} is not a valid need id and is skipped") + elif need_id in ids: + warnings.append(f"duplicate entry {need_id!r} is listed once") + else: + ids.append(need_id) + return ids, warnings + + +def parse_titles(raw: str | None) -> tuple[dict[str, str], list[str]]: + """Parse the ``:titles:`` option: one `` = `` per line.""" + titles: dict[str, str] = {} + warnings: list[str] = [] + for line in (raw or "").splitlines(): + need_id, sep, title = line.partition("=") + need_id, title = need_id.strip(), title.strip() + if not line.strip(): + continue + if not sep or not need_id or not title: + warnings.append( + f"cannot parse title {line.strip()!r}; expected 'id = Title'" + ) + else: + titles[need_id] = title + return titles, warnings + + +def derive_title(need_id: str) -> str: + """Last-resort heading for a component without an explicit ``:titles:``. + + Intentionally dumb. The real title lives on the Need and is rendered by + the component table inside the section, resolved by sphinx-needs. + ``comp__baselibs_json`` becoming "Baselibs Json" is an accepted fallback. + """ + slug = need_id.partition("__")[2] or need_id + return slug.replace("_", " ").strip().title() or need_id + + +def derive_feature_id(module_id: str) -> str: + """``mod__baselibs`` -> ``feat__baselibs``, as the upstream template does. + + An exact, total rewrite of one id into another, feeding an ``id == ...`` + filter -- so the worst case is an empty feature table, never a wrong match. + """ + return ( + "feat__" + module_id.removeprefix("mod__") + if module_id.startswith("mod__") + else "" + ) + + +def derive_slug(need_id: str, module_id: str) -> str: + """Matching key for the work-product documents of *need_id*. + + Mirrors the upstream template: drop the type prefix and the module name, + then remove underscores and lowercase. This is substring matching and can + produce false positives; see the README's "Known gaps". + """ + tail = need_id.partition("__")[2] or need_id + module_short = module_id.removeprefix("mod__") + if module_short: + tail = tail.removeprefix(f"{module_short}_") + return tail.replace("_", "").lower() + + +def render_need( + directive_name: str, + title: str, + options: dict[str, str | None], + content: list[str], +) -> str: + """Render the ``mod_ver_report`` Need itself. + + Options are passed through verbatim: the metamodel -- not this extension -- + decides which are mandatory, which are links and what they may point at. + """ + lines = [f".. {directive_name}:: {title}"] + lines += [f" :{k}: {'' if v is None else v}" for k, v in options.items()] + if content: + lines.append("") + lines += [f" {line}" if line else "" for line in content] + return "\n".join(lines) + "\n" + + +def shipped_template_folder() -> Path: + """The Sphinx-Needs template folder shipped with docs-as-code. + + Derived from this file's location so it resolves in the workspace, in Bazel + runfiles and in the sandbox alike -- the same approach + ``score_sphinx_bundle`` uses for the very same directory. + """ + folder = Path(__file__).parents[2] / "needs_templates" + if not folder.is_dir(): + raise FileNotFoundError(f"Needs template folder does not exist: {folder}") + return folder + + +@lru_cache(maxsize=4) +def _environment(folders: tuple[str, ...]) -> Environment: + env = Environment( + loader=FileSystemLoader(list(folders)), + undefined=StrictUndefined, + keep_trailing_newline=True, + autoescape=False, # noqa: S701 - RST output, not HTML + ) + env.filters["q"] = quote_for_filter + return env + + +def render_report(context: dict[str, Any], template_folder: str | None) -> str: + """Render the report body. + + *template_folder* is the project's ``needs_template_folder``. It is + searched first, so a project overrides the report by dropping its own + ``mod_ver_report.need`` in there; the shipped folder is the fallback. + """ + shipped = str(shipped_template_folder()) + folders = dict.fromkeys( + [template_folder, shipped] if template_folder else [shipped] + ) + return _environment(tuple(folders)).get_template(TEMPLATE_NAME).render(**context) diff --git a/src/extensions/score_module_verification_report/tests/test_rendering.py b/src/extensions/score_module_verification_report/tests/test_rendering.py new file mode 100644 index 000000000..b5cd4809c --- /dev/null +++ b/src/extensions/score_module_verification_report/tests/test_rendering.py @@ -0,0 +1,200 @@ +# ******************************************************************************* +# Copyright (c) 2026 Contributors to the Eclipse Foundation +# +# See the NOTICE file(s) distributed with this work for additional +# information regarding copyright ownership. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# ******************************************************************************* +"""Unit tests for the rendering helpers and the report template.""" + +import pytest +from score_module_verification_report import rendering + +CONTEXT = { + "report_id": "mod_vrep__x", + "module_id": "mod__x", + "module_slug": "x", + "feature_id": "feat__x", + "components": [{"id": "comp__x_json", "title": "JSON", "slug": "json"}], +} + + +@pytest.fixture(scope="module") +def rendered() -> str: + return rendering.render_report(CONTEXT, None) + + +class TestQuoteForFilter: + """Acceptance test 10: filter expressions with user-supplied IDs.""" + + def test_plain_id_is_quoted(self) -> None: + assert rendering.quote_for_filter("comp__a") == '"comp__a"' + + @pytest.mark.parametrize( + "hostile", + ['a" or True or "', "a' or 'b", "__import__('os')", "a\nb", "a; b", "a b", ""], + ) + def test_hostile_input_is_rejected(self, hostile: str) -> None: + with pytest.raises(ValueError): + rendering.quote_for_filter(hostile) + + def test_rejected_ids_never_reach_a_rendered_filter(self) -> None: + ids, warnings = rendering.parse_ids('comp__a, x"or(True)') + assert ids == ["comp__a"] + assert any("not a valid need id" in w for w in warnings) + + +class TestParseIds: + def test_comma_and_whitespace_separated(self) -> None: + ids, warnings = rendering.parse_ids("comp__a, comp__b\n comp__c") + assert ids == ["comp__a", "comp__b", "comp__c"] + assert warnings == [] + + def test_empty(self) -> None: + assert rendering.parse_ids(None) == ([], []) + assert rendering.parse_ids(" ") == ([], []) + + def test_version_qualifier_warns_instead_of_being_ignored(self) -> None: + ids, warnings = rendering.parse_ids("comp__a[version==2]") + assert ids == ["comp__a"] + assert any("version qualifier" in w for w in warnings) + + def test_duplicates_are_deterministic(self) -> None: + """Acceptance test 9: duplicate component ids.""" + ids, warnings = rendering.parse_ids("comp__b, comp__a, comp__b") + assert ids == ["comp__b", "comp__a"] + assert any("duplicate" in w for w in warnings) + + +class TestParseTitles: + def test_titles_are_parsed(self) -> None: + titles, warnings = rendering.parse_titles("comp__a = JSON\ncomp__b = Bits\n") + assert titles == {"comp__a": "JSON", "comp__b": "Bits"} + assert warnings == [] + + def test_malformed_line_warns(self) -> None: + titles, warnings = rendering.parse_titles("comp__a JSON") + assert titles == {} + assert len(warnings) == 1 + + +class TestDerivations: + def test_title_fallback(self) -> None: + assert rendering.derive_title("comp__baselibs_json") == "Baselibs Json" + assert rendering.derive_title("nounderscores") == "Nounderscores" + + def test_feature_is_derived_from_the_module(self) -> None: + assert rendering.derive_feature_id("mod__baselibs") == "feat__baselibs" + + def test_feature_derivation_gives_up_rather_than_guessing(self) -> None: + assert rendering.derive_feature_id("") == "" + assert rendering.derive_feature_id("something_else") == "" + + def test_component_slug_drops_the_module_name(self) -> None: + assert ( + rendering.derive_slug("comp__baselibs_bit_manipulation", "mod__baselibs") + == "bitmanipulation" + ) + assert rendering.derive_slug("mod__baselibs", "") == "baselibs" + + +class TestTemplateLocation: + def test_template_is_a_need_file_in_the_needs_template_folder(self) -> None: + """Sphinx-Needs templates end in ``.need`` and live in that one folder.""" + folder = rendering.shipped_template_folder() + assert folder.name == "needs_templates" + assert rendering.TEMPLATE_NAME.endswith(".need") + assert (folder / rendering.TEMPLATE_NAME).is_file() + + +class TestTemplate: + """The template owns the report's content and its section structure.""" + + @pytest.mark.parametrize( + "heading", + [ + "Feature", + "Feature Requirements Statistics", + "Feature Architecture Statistics", + "Feature Inspection Statistics", + "Component Overview", + "JSON", + ], + ) + def test_section_is_present(self, rendered: str, heading: str) -> None: + assert f"\n{heading}\n{'+' * len(heading)}\n" in rendered + + def test_every_section_carries_a_namespaced_target(self, rendered: str) -> None: + """Acceptance test 3: no collisions between two reports on one page.""" + targets = [line for line in rendered.splitlines() if line.startswith(".. _")] + assert targets, "no section targets emitted" + assert all(t.startswith(".. _mod_vrep__x__") for t in targets) + assert ".. _mod_vrep__x__comp__x_json:" in targets + + def test_all_headings_use_one_underline_char(self, rendered: str) -> None: + """Flat by construction: same style -> siblings, whatever the placement.""" + assert {line[0] for line in rendered.splitlines() if set(line) == {"+"}} == { + "+" + } + + @pytest.mark.parametrize( + "rubric", + [ + "Component Requirements Statistics", + "Component Architecture Statistics", + "Requirements Traceability", + "Test Coverage", + "Architectural Elements", + "Verification & Safety Analysis Documents", + ], + ) + def test_component_internals_stay_rubrics(self, rendered: str, rubric: str) -> None: + """No navigation is needed below a component, so no nesting is emitted.""" + assert f".. rubric:: {rubric}" in rendered + assert f"\n{rubric}\n{'+' * len(rubric)}\n" not in rendered + + def test_statistics_are_needpie_filters(self, rendered: str) -> None: + assert ".. needpie:: Feature Requirements Status" in rendered + assert ( + 'type == "feat_req" and "feat__x" in satisfied_by and status == "valid"' + in rendered + ) + assert ( + 'type == "comp_req" and "comp__x_json" in satisfied_by ' + "and len(fully_verifies_back) > 0" in rendered + ) + + def test_work_product_rows_are_generated(self, rendered: str) -> None: + assert ":need:`wp__sw_component_fmea`" in rendered + assert ( + 'type == "document" and "wp__sw_component_dfa" in realizes ' + 'and "json" in id.replace("_", "").lower()' in rendered + ) + + def test_no_feature_renders_a_sentence_not_a_broken_filter(self) -> None: + text = rendering.render_report({**CONTEXT, "feature_id": ""}, None) + assert "feature sections\nstay empty" in text + assert "needpie:: Feature" not in text + + def test_no_components_renders_a_sentence(self) -> None: + text = rendering.render_report({**CONTEXT, "components": []}, None) + assert "does not declare any covered components" in text + + +class TestRenderNeed: + def test_need_options_are_passed_through_verbatim(self) -> None: + text = rendering.render_need( + "mod_ver_report_need", + "Baselibs Verification Report", + {"id": "mod_vrep__baselibs", "covers": "comp__a, comp__b", "flag": None}, + ["Intro.", "", "More."], + ) + assert ".. mod_ver_report_need:: Baselibs Verification Report" in text + assert " :covers: comp__a, comp__b" in text + assert " :flag: " in text + assert " Intro." in text diff --git a/src/extensions/score_module_verification_report/tests/test_report_integration.py b/src/extensions/score_module_verification_report/tests/test_report_integration.py new file mode 100644 index 000000000..525c2f182 --- /dev/null +++ b/src/extensions/score_module_verification_report/tests/test_report_integration.py @@ -0,0 +1,717 @@ +# ******************************************************************************* +# Copyright (c) 2026 Contributors to the Eclipse Foundation +# +# See the NOTICE file(s) distributed with this work for additional +# information regarding copyright ownership. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# ******************************************************************************* +"""Acceptance tests for the module verification report. + +"It looks right in the sidebar" is not evidence. These tests assert on the +doctree, on ``env.tocs`` and on rendered output. + +Acceptance test 7 (a component in ``mod.includes`` but missing from +``:covers:`` fails the build via the *metamodel* rule, not via extension code) +deliberately lives with the metamodel: +``src/extensions/score_metamodel/tests/rst/graph/test_mod_ver_report_scope.rst``. +""" + +from __future__ import annotations + +import json +import re +from collections.abc import Callable +from pathlib import Path +from typing import Any + +import pytest +from docutils import nodes +from score_module_verification_report import rendering +from sphinx.testing.util import SphinxTestApp + +EXTERNAL_NEEDS: dict[str, Any] = { + "current_version": "1.0", + "project": "external", + "versions": { + "1.0": { + "needs": { + "comp__external_thing": { + "docname": "index", + "id": "comp__external_thing", + "lineno": 1, + "status": "valid", + "title": "External Component", + "type": "comp", + "type_name": "comp", + } + } + } + }, +} + +CONF_PY = """ +extensions = ["sphinx_needs", "score_module_verification_report"] +needs_id_regex = "^[a-zA-Z0-9_]+$" +needs_types = [ + dict(directive=d, title=d, prefix=d + "__", color="#FFF", style="node") + for d in ( + "feat", + "feat_req", + "feat_arc_sta", + "comp", + "comp_req", + "comp_arc_sta", + "mod", + "mod_ver_report", + "document", + "wp", + "tcase", + ) +] +needs_extra_options = ["safety", "security", "verification_method"] +needs_links = { + name: dict(incoming=name, outgoing=name) + for name in ( + "belongs_to", + "includes", + "covers", + "contains", + "satisfied_by", + "realizes", + "fully_verifies", + "partially_verifies", + ) +} +needs_external_needs = [ + dict(base_url="https://example.invalid/docs", json_path="external_needs.json") +] +suppress_warnings = ["app.add_directive", "epub.unknown_project_files"] +""" + +#: The world the report reports on: a feature, two components, requirements, +#: architecture elements, a test, and the work products the template lists. +ARCHITECTURE = """ +.. feat:: Baselibs Feature + :id: feat__baselibs + :safety: ASIL_B + :security: NO + :status: valid + +.. feat_req:: A feature requirement + :id: feat_req__baselibs_one + :safety: ASIL_B + :status: valid + :satisfied_by: feat__baselibs + +.. feat_arc_sta:: A feature architecture element + :id: feat_arc_sta__baselibs + :safety: ASIL_B + :status: valid + :tags: inspected + :belongs_to: feat__baselibs + +.. comp:: JSON + :id: comp__baselibs_json + :safety: ASIL_B + :security: NO + :status: valid + :belongs_to: feat__baselibs + +.. comp:: Bit Manipulation + :id: comp__baselibs_bit_manipulation + :safety: ASIL_B + :security: NO + :status: valid + :belongs_to: feat__baselibs + +.. comp_req:: A JSON requirement + :id: comp_req__baselibs_json_one + :safety: ASIL_B + :status: valid + :satisfied_by: comp__baselibs_json + +.. comp_arc_sta:: A JSON architecture element + :id: comp_arc_sta__baselibs_json + :safety: ASIL_B + :status: valid + :tags: inspected + :belongs_to: comp__baselibs_json + +.. tcase:: A test case + :id: tcase__baselibs_1 + :status: valid + :fully_verifies: comp_req__baselibs_json_one + +.. mod:: Baselibs + :id: mod__baselibs + :includes: comp__baselibs_json, comp__baselibs_bit_manipulation + +.. wp:: Requirements Inspection + :id: wp__requirements_inspect + :status: valid + +.. wp:: Architecture Inspection + :id: wp__sw_arch_verification + :status: valid + +.. wp:: Implementation Inspection + :id: wp__sw_implementation_inspection + :status: valid + +.. wp:: Component DFA + :id: wp__sw_component_dfa + :status: valid + +.. wp:: Component FMEA + :id: wp__sw_component_fmea + :status: valid + +.. document:: Baselibs JSON requirements inspection + :id: doc__baselibs_json_requirements_inspection + :status: valid + :realizes: wp__requirements_inspect +""" + +REPORT = """ +Baselibs +======== + +.. mod_ver_report:: Baselibs Verification Report + :id: mod_vrep__baselibs + :belongs_to: mod__baselibs + :covers: comp__baselibs_json, comp__baselibs_bit_manipulation + :safety: ASIL_B + :security: NO + :status: valid + :verification_method: test + :contains: tcase__baselibs_1 + :titles: + comp__baselibs_json = JSON Utilities + + Verification report for the Baselibs module. +""" + +#: The flat list of sections a report emits, in order. +EXPECTED_SECTIONS = [ + "Feature", + "Feature Requirements Statistics", + "Feature Architecture Statistics", + "Feature Inspection Statistics", + "Component Overview", + "JSON Utilities", + "Baselibs Bit Manipulation", +] + + +def _write_sources(root: Path, docs: dict[str, str], conf: str = CONF_PY) -> None: + root.mkdir(parents=True, exist_ok=True) + (root / "conf.py").write_text(conf) + (root / "external_needs.json").write_text(json.dumps(EXTERNAL_NEEDS)) + for name, text in docs.items(): + path = root / f"{name}.rst" + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(text) + + +def _index(*docnames: str) -> str: + entries = "\n ".join(docnames) + return f"Docs\n====\n\n.. toctree::\n :maxdepth: 3\n\n {entries}\n" + + +AppFactory = Callable[..., SphinxTestApp] + + +@pytest.fixture +def build(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> AppFactory: + """Build a source tree and return the finished app.""" + + def _build( + docs: dict[str, str], + *, + buildername: str = "html", + parallel: int = 1, + srcdir: Path | None = None, + outdir: Path | None = None, + freshenv: bool = True, + conf: str = CONF_PY, + ) -> SphinxTestApp: + src = srcdir or (tmp_path / "src") + _write_sources(src, docs, conf) + monkeypatch.chdir(src) + app = SphinxTestApp( + freshenv=freshenv, + srcdir=src, + confdir=src, + outdir=outdir or (tmp_path / f"out-{buildername}"), + buildername=buildername, + parallel=parallel, + ) + app.build() + return app + + return _build + + +def _sections(doctree: nodes.document) -> list[nodes.section]: + return list(doctree.findall(nodes.section)) + + +def _section_ids(doctree: nodes.document) -> list[str]: + return [section["ids"][0] for section in _sections(doctree) if section["ids"]] + + +def _titles(doctree: nodes.document) -> list[str]: + return [section[0].astext() for section in _sections(doctree)] + + +# -------------------------------------------------------------------------- +# The core promise: real sections, produced during the read phase. +# -------------------------------------------------------------------------- + + +def test_report_emits_real_sections_as_siblings_of_the_need( + build: AppFactory, +) -> None: + app = build({"index": _index("report"), "report": REPORT + ARCHITECTURE}) + doctree = app.env.get_doctree("report") + + titles = _titles(doctree) + for expected in EXPECTED_SECTIONS: + assert expected in titles + + # The Need is a sibling of the sections, not their container: no section + # may be a descendant of the Need node. + from sphinx_needs.nodes import Need + + for need in doctree.findall(Need): + assert not list(need.findall(nodes.section, include_self=False)), ( + "a section ended up inside the Need node; " + "sphinx-needs parses need content with match_titles=False, so those " + "headings would silently stop being sections" + ) + + +def test_generated_sections_are_flat(build: AppFactory) -> None: + """Constraint 3: a flat list of top-level entries is sufficient.""" + app = build({"index": _index("report"), "report": REPORT + ARCHITECTURE}) + doctree = app.env.get_doctree("report") + + page = next(iter(doctree.findall(nodes.section))) # "Baselibs" + generated = [child for child in page.children if isinstance(child, nodes.section)] + assert [child[0].astext() for child in generated] == EXPECTED_SECTIONS + for section in generated: + assert not list(section.findall(nodes.section, include_self=False)) + + +# -------------------------------------------------------------------------- +# Acceptance test 1: :ref: from another page resolves and links. +# -------------------------------------------------------------------------- + + +def test_ref_from_another_page_resolves(build: AppFactory) -> None: + other = ( + "Other\n=====\n\n" + "See :ref:`mod_vrep__baselibs__comp__baselibs_json` for details.\n" + ) + app = build( + { + "index": _index("report", "other"), + "report": REPORT + ARCHITECTURE, + "other": other, + } + ) + assert "undefined label" not in app.warning.getvalue() + + html = (Path(app.outdir) / "other.html").read_text() + assert 'href="report.html#mod-vrep-baselibs-comp-baselibs-json"' in html + # An implicit :ref: takes its text from the section title -- proof that the + # target really is a section and not a synthesized anchor. + assert "JSON Utilities" in html + + +# -------------------------------------------------------------------------- +# Acceptance test 2: local ToC entries for every component section. +# -------------------------------------------------------------------------- + + +def test_toc_contains_every_generated_section(build: AppFactory) -> None: + app = build({"index": _index("report"), "report": REPORT + ARCHITECTURE}) + + toc = app.env.tocs["report"] + toc_titles = [ref.astext() for ref in toc.findall(nodes.reference)] + for expected in EXPECTED_SECTIONS: + assert expected in toc_titles + + toc_anchors = [ref["anchorname"] for ref in toc.findall(nodes.reference)] + assert "#mod-vrep-baselibs-comp-baselibs-json" in toc_anchors + + +# -------------------------------------------------------------------------- +# Acceptance test 3: stable, collision-free anchors. +# -------------------------------------------------------------------------- + + +SECOND_REPORT = """ +.. mod_ver_report:: Second Verification Report + :id: mod_vrep__second + :belongs_to: mod__baselibs + :covers: comp__baselibs_json + :safety: ASIL_B + :security: NO + :status: valid + :verification_method: test + :titles: + comp__baselibs_json = JSON Utilities +""" + + +def test_two_reports_on_one_page_do_not_collide(build: AppFactory) -> None: + app = build( + {"index": _index("report"), "report": REPORT + SECOND_REPORT + ARCHITECTURE} + ) + ids = _section_ids(app.env.get_doctree("report")) + + assert "mod-vrep-baselibs-comp-baselibs-json" in ids + assert "mod-vrep-second-comp-baselibs-json" in ids + # Same heading text twice, but no docutils "-1" disambiguation suffix. + assert not [i for i in ids if re.search(r"-\d+$", i)] + assert len(ids) == len(set(ids)) + + +def test_anchors_are_stable_across_rebuilds(tmp_path: Path, build: AppFactory) -> None: + src, out = tmp_path / "src", tmp_path / "out" + docs = {"index": _index("report"), "report": REPORT + ARCHITECTURE} + + first = build(docs, srcdir=src, outdir=out) + before = _section_ids(first.env.get_doctree("report")) + first.cleanup() + + second = build(docs, srcdir=src, outdir=out, freshenv=False) + assert _section_ids(second.env.get_doctree("report")) == before + + +# -------------------------------------------------------------------------- +# Acceptance test 4: heading depth follows placement. +# -------------------------------------------------------------------------- + + +def test_heading_depth_at_root_and_nested(build: AppFactory) -> None: + # The report placed below an existing sub-heading. REPORT's own page title + # is dropped so only the directive is re-used. + nested = ( + "Page\n====\n\nChapter\n-------\n\n" + "\n".join(REPORT.splitlines()[3:]) + "\n" + ) + app = build( + { + "index": _index("rootlevel", "nested"), + "rootlevel": REPORT + ARCHITECTURE, + "nested": nested + ARCHITECTURE, + } + ) + + def depth_of(docname: str, title: str) -> int: + doctree = app.env.get_doctree(docname) + for section in doctree.findall(nodes.section): + if section[0].astext() == title: + depth = 0 + node = section.parent + while node is not None: + if isinstance(node, nodes.section): + depth += 1 + node = node.parent + return depth + raise AssertionError(f"{title!r} not found in {docname}") + + # Root level: directly below the page title. + assert depth_of("rootlevel", "JSON Utilities") == 1 + # Nested below an existing heading: one level deeper. + assert depth_of("nested", "JSON Utilities") == 2 + # ... and still flat among themselves at that level. + assert depth_of("nested", "Component Overview") == 2 + + +# -------------------------------------------------------------------------- +# Acceptance test 5: a non-HTML builder keeps the sections. +# -------------------------------------------------------------------------- + + +def test_singlehtml_builder_keeps_sections(build: AppFactory) -> None: + app = build( + {"index": _index("report"), "report": REPORT + ARCHITECTURE}, + buildername="singlehtml", + ) + html = (Path(app.outdir) / "index.html").read_text() + assert 'id="mod-vrep-baselibs-comp-baselibs-json"' in html + assert "JSON Utilities" in html + + +def test_latex_builder_keeps_sections(build: AppFactory) -> None: + app = build( + {"index": _index("report"), "report": REPORT + ARCHITECTURE}, + buildername="latex", + ) + tex = next(Path(app.outdir).glob("*.tex")).read_text() + assert "JSON Utilities" in tex + assert "Component Overview" in tex + assert "Feature Requirements Statistics" in tex + + +# -------------------------------------------------------------------------- +# Acceptance test 6: identical under -j 1 and parallel reading. +# -------------------------------------------------------------------------- + + +def test_structure_identical_serial_and_parallel( + tmp_path: Path, build: AppFactory +) -> None: + # Sphinx only forks when there are more than five documents to read. + filler = {f"filler{i}": f"Filler {i}\n=========\n\ntext\n" for i in range(8)} + docs = { + "index": _index("report", *filler), + "report": REPORT + ARCHITECTURE, + **filler, + } + + serial = build(docs, srcdir=tmp_path / "s1", outdir=tmp_path / "o1", parallel=1) + serial_ids = _section_ids(serial.env.get_doctree("report")) + serial_titles = _titles(serial.env.get_doctree("report")) + serial.cleanup() + + parallel = build(docs, srcdir=tmp_path / "s2", outdir=tmp_path / "o2", parallel=4) + assert _section_ids(parallel.env.get_doctree("report")) == serial_ids + assert _titles(parallel.env.get_doctree("report")) == serial_titles + + +# -------------------------------------------------------------------------- +# Acceptance test 8: a needtable resolves an external Need. +# -------------------------------------------------------------------------- + + +EXTERNAL_REPORT = """ +External +======== + +.. mod_ver_report:: External Verification Report + :id: mod_vrep__external + :belongs_to: mod__baselibs + :covers: comp__external_thing + :safety: QM + :security: NO + :status: valid + :verification_method: inspection +""" + + +def test_needtable_resolves_an_external_need(build: AppFactory) -> None: + app = build( + {"index": _index("report"), "report": EXTERNAL_REPORT + ARCHITECTURE}, + ) + html = (Path(app.outdir) / "report.html").read_text() + # Resolved by sphinx-needs from needs.json, including its external URL -- + # the extension itself never looked at the need. + assert "External Component" in html + assert "https://example.invalid/docs" in html + + +# -------------------------------------------------------------------------- +# Acceptance tests 9 + 10 at build level. +# -------------------------------------------------------------------------- + + +DUPLICATE_REPORT = """ +Dupes +===== + +.. mod_ver_report:: Duplicate Report + :id: mod_vrep__dupes + :belongs_to: mod__baselibs + :covers: comp__baselibs_json, comp__baselibs_json, comp__baselibs_bit_manipulation + :safety: QM + :security: NO + :status: valid + :verification_method: test + :titles: + comp__baselibs_json = Same Title + comp__baselibs_bit_manipulation = Same Title +""" + + +def test_duplicates_and_title_collisions_are_deterministic( + build: AppFactory, +) -> None: + app = build({"index": _index("report"), "report": DUPLICATE_REPORT + ARCHITECTURE}) + + assert "duplicate entry 'comp__baselibs_json'" in app.warning.getvalue() + + doctree = app.env.get_doctree("report") + titles = _titles(doctree) + assert titles.count("Same Title") == 2 # collision allowed ... + ids = _section_ids(doctree) + assert "mod-vrep-dupes-comp-baselibs-json" in ids # ... anchors still differ + assert "mod-vrep-dupes-comp-baselibs-bit-manipulation" in ids + assert len(ids) == len(set(ids)) + + +HOSTILE_REPORT = """ +Hostile +======= + +.. mod_ver_report:: Hostile Report + :id: mod_vrep__hostile + :belongs_to: mod__baselibs + :covers: comp__baselibs_json, comp__baselibs_bit_manipulation, evil"or(1) + :safety: QM + :security: NO + :status: valid + :verification_method: test +""" + + +def test_hostile_component_id_is_rejected_not_interpolated( + build: AppFactory, +) -> None: + app = build({"index": _index("report"), "report": HOSTILE_REPORT + ARCHITECTURE}) + warnings = app.warning.getvalue() + assert "is not a valid need id and is skipped" in warnings + + titles = _titles(app.env.get_doctree("report")) + assert "Baselibs Json" in titles or "Json" in " ".join(titles) + # The rejected token never became a section. + assert not [t for t in titles if "evil" in t.lower()] + + +# -------------------------------------------------------------------------- +# Version qualifiers are reported, not silently stripped. +# -------------------------------------------------------------------------- + + +VERSIONED_REPORT = """ +Versioned +========= + +.. mod_ver_report:: Versioned Report + :id: mod_vrep__versioned + :belongs_to: mod__baselibs + :covers: comp__baselibs_json[version==1], comp__baselibs_bit_manipulation + :safety: QM + :security: NO + :status: valid + :verification_method: test +""" + + +def test_version_qualifier_warns(build: AppFactory) -> None: + app = build({"index": _index("report"), "report": VERSIONED_REPORT + ARCHITECTURE}) + assert "ignoring version qualifier" in app.warning.getvalue() + assert "Baselibs Json" in _titles(app.env.get_doctree("report")) + + +# -------------------------------------------------------------------------- +# The report content itself: every number comes out of a needs filter. +# -------------------------------------------------------------------------- + + +def test_report_content_is_resolved_by_sphinx_needs(build: AppFactory) -> None: + app = build({"index": _index("report"), "report": REPORT + ARCHITECTURE}) + html = (Path(app.outdir) / "report.html").read_text() + + # Feature sections resolved the feature the report declares. + assert "feat__baselibs" in html + assert "feat_req__baselibs_one" in html + assert "feat_arc_sta__baselibs" in html + + # Component sections resolved the component's requirements, architecture + # and the test that verifies the requirement (a backlink sphinx-needs + # computed, not something the extension went looking for). + assert "comp_req__baselibs_json_one" in html + assert "comp_arc_sta__baselibs_json" in html + assert "tcase__baselibs_1" in html + + # Work products and the document realising one of them. + assert "wp__sw_component_dfa" in html + assert "doc__baselibs_json_requirements_inspection" in html + + # Statistics are needpie images, one per configured chart. + assert html.count("= 8 + assert "need_pie" in html + + +def test_feature_is_derived_from_the_module(build: AppFactory) -> None: + """``mod__baselibs`` -> ``feat__baselibs``, as the upstream template does.""" + assert "feat__baselibs" not in REPORT + app = build({"index": _index("report"), "report": REPORT + ARCHITECTURE}) + assert "feat_req__baselibs_one" in (Path(app.outdir) / "report.html").read_text() + + +def test_component_internals_are_rubrics_not_sections(build: AppFactory) -> None: + """Flat sections: nothing below a component competes for the sidebar.""" + app = build({"index": _index("report"), "report": REPORT + ARCHITECTURE}) + doctree = app.env.get_doctree("report") + + component = next( + s for s in doctree.findall(nodes.section) if s[0].astext() == "JSON Utilities" + ) + assert not list(component.findall(nodes.section, include_self=False)) + rubrics = [r.astext() for r in component.findall(nodes.rubric)] + assert "Requirements Traceability" in rubrics + assert "Verification & Safety Analysis Documents" in rubrics + + +def test_needs_template_folder_is_set_and_points_at_the_template( + build: AppFactory, +) -> None: + assert "needs_template_folder" not in CONF_PY + app = build({"index": _index("report"), "report": REPORT + ARCHITECTURE}) + + folder = Path(app.config.needs_template_folder) + assert folder.is_dir() + assert (folder / rendering.TEMPLATE_NAME).is_file() + + +def test_project_template_folder_wins_over_the_shipped_one( + build: AppFactory, tmp_path: Path +) -> None: + """A project overrides the report by dropping its own ``.need`` in place.""" + folder = tmp_path / "our_needs_templates" + folder.mkdir() + (folder / rendering.TEMPLATE_NAME).write_text( + "Only one section, for {{ report_id }}\n" + "+++++++++++++++++++++++++++++++++++++++++++++\n" + ) + conf = CONF_PY + f"needs_template_folder = {str(folder)!r}\n" + app = build({"index": _index("report"), "report": REPORT + ARCHITECTURE}, conf=conf) + assert _titles(app.env.get_doctree("report")) == [ + "Baselibs", + "Only one section, for mod_vrep__baselibs", + ] + + +def test_shipped_template_is_the_fallback_for_a_project_folder( + build: AppFactory, tmp_path: Path +) -> None: + """A project folder without our template must not break the build.""" + folder = tmp_path / "unrelated_needs_templates" + folder.mkdir() + (folder / "something_else.need").write_text("unrelated\n") + conf = CONF_PY + f"needs_template_folder = {str(folder)!r}\n" + app = build({"index": _index("report"), "report": REPORT + ARCHITECTURE}, conf=conf) + assert _titles(app.env.get_doctree("report")) == ["Baselibs", *EXPECTED_SECTIONS] + + +def test_extension_has_no_build_lifecycle_hooks() -> None: + import inspect + + import score_module_verification_report as ext + + source = inspect.getsource(ext) + connects = re.findall(r'app\.connect\(\s*"([^"]+)"', source) + assert connects == ["config-inited"], ( + "The extension must own no build lifecycle: no env-updated re-read, no " + "build-finished consistency pass, no registry. The single config-inited " + f"handler registers directives only. Found: {connects}" + ) diff --git a/src/extensions/score_sphinx_bundle/BUILD b/src/extensions/score_sphinx_bundle/BUILD index 113803b97..0cb93f404 100644 --- a/src/extensions/score_sphinx_bundle/BUILD +++ b/src/extensions/score_sphinx_bundle/BUILD @@ -37,6 +37,7 @@ py_library( "@score_docs_as_code//src/extensions/score_mounts", "@score_docs_as_code//src/extensions/score_source_code_linker", "@score_docs_as_code//src/extensions/score_metrics", + "@score_docs_as_code//src/extensions/score_module_verification_report", "@score_docs_as_code//src/extensions/score_sync_toml", "@score_docs_as_code//src/helper_lib", ], diff --git a/src/extensions/score_sphinx_bundle/__init__.py b/src/extensions/score_sphinx_bundle/__init__.py index f3399e507..6c0546aae 100644 --- a/src/extensions/score_sphinx_bundle/__init__.py +++ b/src/extensions/score_sphinx_bundle/__init__.py @@ -41,6 +41,7 @@ "needs_config_writer", "score_sync_toml", "score_metrics", + "score_module_verification_report", "broken_link_fix", ] diff --git a/src/needs_templates/mod_ver_report.need b/src/needs_templates/mod_ver_report.need new file mode 100644 index 000000000..96235ff5c --- /dev/null +++ b/src/needs_templates/mod_ver_report.need @@ -0,0 +1,299 @@ +{#- + Module verification report body. + + This is a Jinja template and it lives in the Sphinx-Needs template folder + (``needs_template_folder``) with the ``.need`` extension, alongside every + other need template in this repository. + + It is rendered by the ``mod_ver_report`` *directive*, not by Sphinx-Needs' + ``:template:`` option, and that difference is load-bearing: ``:template:`` + renders into the Need's content, which Sphinx-Needs parses with + ``match_titles=False``. Headings written there can never become sections, so + they would produce no anchors, no ToC entries, no ``:ref:`` targets and + nothing at all in non-HTML builders. The directive renders this file itself + and returns the result into the surrounding document instead. + + Adapted from ``mod_ver_report_tiny.need``. Two deliberate differences: + + * Top-level ``rubric`` directives became real sections, for the reason above. + Rubrics are kept only *inside* a component section, where no navigation is + required. + * The component list comes from the report Need's ``covers`` link field via + the directive, not from a ``linked_needs()`` render helper. Reading the + Need graph while rendering is what forces a second read pass; the + metamodel checks instead that ``covers`` matches the module's ``includes`` + in both directions. + + Everything with semantics is still deferred to Sphinx-Needs: every number on + this page comes out of a ``needtable``/``needpie`` filter evaluated after + need collection. + + Context: ``report_id``, ``module_id``, ``module_slug``, ``feature_id`` and + ``components`` (each with ``id``, ``title`` and ``slug``). Plus a ``q`` + filter that safely quotes a need id for a filter string. +-#} +{#- Everything the report shows is decided here, not in Python. -#} +{% set feature_workproducts = [ + ("wp__requirements_inspect", "Requirements Inspection"), + ("wp__sw_arch_verification", "Architecture Inspection"), + ] %} +{% set component_workproducts = [ + ("wp__requirements_inspect", "Requirements Inspection"), + ("wp__sw_arch_verification", "Architecture Inspection"), + ("wp__sw_implementation_inspection", "Implementation Inspection"), + ("wp__sw_component_dfa", "DFA"), + ("wp__sw_component_fmea", "FMEA"), + ] %} +{#- Documents carry no link to their component, so they are matched on the id + substring. This can produce false positives; see the README. -#} +{% macro document_filter(workproduct_id, slug) -%} +type == "document" and {{ workproduct_id|q }} in realizes and {{ slug|q }} in id.replace("_", "").lower() +{%- endmacro %} +{#- A real section: explicit target namespaced with the report id, so anchors + are stable and two reports on one page never collide. One underline + character throughout keeps every generated heading a sibling. -#} +{% macro heading(slug, title) -%} +.. _{{ report_id }}__{{ slug }}: + +{{ title }} +{{ "+" * title|length }} +{%- endmacro %} +{% macro workproduct_table(workproducts, slug) -%} +.. list-table:: + :header-rows: 1 + :widths: 30 25 25 20 + :class: wp-doc-table + + * - Work Product + - Kind + - Realized by + - Status +{%- for wp_id, kind in workproducts %} + * - :need:`{{ wp_id }}` + - {{ kind }} + - .. needtable:: + :filter: {{ document_filter(wp_id, slug) }} + :columns: id + :style: table + - .. needtable:: + :filter: {{ document_filter(wp_id, slug) }} + :columns: status + :style: table +{%- endfor %} +{%- endmacro %} +{% macro status_pie(title, type_filter, scope_filter) -%} +.. needpie:: {{ title }} + :labels: valid, invalid + :colors: #37a12d, #ca2828 + :legend: + + {{ type_filter }} and {{ scope_filter }} and status == "valid" + {{ type_filter }} and {{ scope_filter }} and status == "invalid" +{%- endmacro %} +{% macro coverage_pie(title, type_filter, scope_filter) -%} +.. needpie:: {{ title }} + :labels: fully covered, partially covered, not covered + :colors: #37a12d, #f0a500, #ca2828 + :legend: + + {{ type_filter }} and {{ scope_filter }} and len(fully_verifies_back) > 0 + {{ type_filter }} and {{ scope_filter }} and len(partially_verifies_back) > 0 and len(fully_verifies_back) == 0 + {{ type_filter }} and {{ scope_filter }} and len(fully_verifies_back) == 0 and len(partially_verifies_back) == 0 +{%- endmacro %} +{% macro inspection_pie(title, type_filter, scope_filter) -%} +.. needpie:: {{ title }} + :labels: inspected, not inspected + :colors: #37a12d, #ca2828 + :legend: + + {{ type_filter }} and {{ scope_filter }} and "inspected" in tags + {{ type_filter }} and {{ scope_filter }} and "inspected" not in tags +{%- endmacro %} +{% macro requirements_table(type_filter, scope_filter) -%} +.. dropdown:: Show requirements table + :animate: fade-in + + .. needtable:: + :filter: {{ type_filter }} and {{ scope_filter }} + :style: table + :columns: id;title;safety;status;fully_verifies_back;partially_verifies_back + :colwidths: 13,22,8,10,23,24 + :sort: id +{%- endmacro %} +{% macro architecture_table(type_filter, scope_filter) -%} +.. dropdown:: Show architectural elements table + :animate: fade-in + + .. needtable:: + :filter: {{ type_filter }} and {{ scope_filter }} + :style: table + :columns: id;title;safety;status;tags + :colwidths: 25,30,10,15,20 + :sort: id +{%- endmacro %} +.. raw:: html + + + +{{ heading("feature", "Feature") }} + +{% if feature_id %} +.. needtable:: + :filter: id == {{ feature_id|q }} + :columns: title as "Name";id as "Id";safety;security;status + :style: table +{% else %} +No feature could be derived from ``{{ module_id }}``; the feature sections +stay empty. +{% endif %} + +{{ heading("feature-requirements-statistics", "Feature Requirements Statistics") }} + +{% if feature_id %} +.. grid:: 1 2 2 2 + :gutter: 3 + + .. grid-item:: + + {{ status_pie("Feature Requirements Status", 'type == "feat_req"', feature_id|q ~ " in satisfied_by")|indent(6) }} + + .. grid-item:: + + {{ coverage_pie("Feature Requirements Test Coverage", 'type == "feat_req"', feature_id|q ~ " in satisfied_by")|indent(6) }} + +{{ requirements_table('type == "feat_req"', feature_id|q ~ " in satisfied_by") }} +{% else %} +No feature declared. +{% endif %} + +{{ heading("feature-architecture-statistics", "Feature Architecture Statistics") }} + +{% if feature_id %} +.. grid:: 1 2 2 2 + :gutter: 3 + + .. grid-item:: + + {{ status_pie("Feature Architecture Elements Status", 'type in ["feat_arc_sta", "feat_arc_dyn"]', feature_id|q ~ " in belongs_to")|indent(6) }} + + .. grid-item:: + + {{ inspection_pie("Feature Architecture Elements Inspection Status", 'type in ["feat_arc_sta", "feat_arc_dyn"]', feature_id|q ~ " in belongs_to")|indent(6) }} + +{{ architecture_table('type in ["feat_arc_sta", "feat_arc_dyn"]', feature_id|q ~ " in belongs_to") }} +{% else %} +No feature declared. +{% endif %} + +{{ heading("feature-inspection-statistics", "Feature Inspection Statistics") }} + +Presence of the feature-level inspection work products. + +.. dropdown:: Show work products table + :animate: fade-in + + {{ workproduct_table(feature_workproducts, module_slug)|indent(3) }} + +{{ heading("component-overview", "Component Overview") }} + +{% if components %} +.. needtable:: + :filter: id in [{{ components|map(attribute="id")|map("q")|join(", ") }}] + :columns: id as "Component";safety;security;status + :style: table + :sort: id +{% else %} +This report does not declare any covered components. +{% endif %} +{% for component in components %} + +{{ heading(component.id, component.title) }} + +.. rubric:: Component + +.. needtable:: + :filter: id == {{ component.id|q }} or {{ component.id|q }} in belongs_to + :columns: id;title;type;status;safety;security + :style: table + +.. rubric:: Component Requirements Statistics + +.. grid:: 1 2 2 2 + :gutter: 3 + + .. grid-item:: + + {{ status_pie(component.title ~ " Requirements Status", 'type == "comp_req"', component.id|q ~ " in satisfied_by")|indent(6) }} + + .. grid-item:: + + {{ coverage_pie(component.title ~ " Requirements Test Coverage", 'type == "comp_req"', component.id|q ~ " in satisfied_by")|indent(6) }} + +.. rubric:: Component Architecture Statistics + +.. grid:: 1 2 2 2 + :gutter: 3 + + .. grid-item:: + + {{ status_pie(component.title ~ " Architecture Elements Status", 'type in ["comp_arc_sta", "comp_arc_dyn"]', component.id|q ~ " in belongs_to")|indent(6) }} + + .. grid-item:: + + {{ inspection_pie(component.title ~ " Architecture Elements Inspection Status", 'type in ["comp_arc_sta", "comp_arc_dyn"]', component.id|q ~ " in belongs_to")|indent(6) }} + +.. rubric:: Requirements Traceability + +All requirements of this component with their verification status and the +tests that fully or partially verify them. + +{{ requirements_table('type == "comp_req"', component.id|q ~ " in satisfied_by") }} + +.. rubric:: Test Coverage + +.. note:: + + Per-source-file line and branch coverage is not part of this extension. + Reading an LCOV report while the directive runs would introduce an + untracked Sphinx dependency and break incremental correctness and Bazel + reproducibility. + +.. rubric:: Architectural Elements + +Architectural elements of this component with their inspection status. +Elements that have been formally inspected carry the ``inspected`` tag. + +{{ architecture_table('type in ["comp_arc_sta", "comp_arc_dyn"]', component.id|q ~ " in belongs_to") }} + +.. rubric:: Verification & Safety Analysis Documents + +Presence of the standard verification and safety analysis work products for +this component. An empty cell means the corresponding document is missing. + +.. dropdown:: Show work products table + :animate: fade-in + + {{ workproduct_table(component_workproducts, component.slug)|indent(3) }} +{% endfor %}