diff --git a/docs/how-to/test_to_doc_links.rst b/docs/how-to/test_to_doc_links.rst index 81ec104af..87164b3ff 100644 --- a/docs/how-to/test_to_doc_links.rst +++ b/docs/how-to/test_to_doc_links.rst @@ -20,21 +20,44 @@ Reference Docs in Tests This guide explains how to annotate test cases so that docs-as-code automatically creates traceability links between tests and requirements. -Details of implementation -------------------------- +How to annotate tests +--------------------- -The mechanism is language-agnostic: -The ``score_source_code_linker`` extension parses ``test.xml`` files, extracts test metadata -(name, file, line, result, and verification properties), -and creates backlinks on the referenced requirements. +To link a test to the requirements it verifies, add the test metadata using +the mechanism provided by its test framework: -The extension will look for ``test.xml`` files in both ``bazel-testlogs/`` -as well as a folder named ``tests-report/``. This folder can be created manually if the tests -require some pre-run step or are matrix tests or similar. +Python (pytest) +^^^^^^^^^^^^^^^ + Use the ``@add_test_properties`` decorator. The test docstring supplies + the ``Description`` metadata. + +C++ (gTest) +^^^^^^^^^^^ + Use ``RecordProperty``. Put shared properties in ``SetUp()`` and + per-test properties inside each ``TEST_F``. + +Rust +^^^^ + There is currently no provided official way to add this metadata in Rust. + Use the advanced JUnit XML path below until Rust support is available. +See the `Verification Templates `_ +for complete examples and the required metadata. -Required Properties -------------------- +Advanced usage: JUnit XML for other languages +---------------------------------------------- + +This section is only relevant when your language or test framework does not +have one of the integrations above. In that case, produce JUnit XML with the +metadata described below. The generated test results are processed +automatically and create GitHub links from the requirements to the testcases. + +The extension looks for files named ``test.xml`` under ``bazel-testlogs/`` or +``tests-report/`` at the workspace root. Create ``tests-report/`` manually when +the test framework needs a separate pre-run step or produces matrix results. + +Required properties +^^^^^^^^^^^^^^^^^^^ Every linked test must declare the following properties (see :need:`gd_guidl__verification_specification` for detailed values): @@ -50,13 +73,12 @@ Every linked test must declare the following properties ``Description`` A human-readable explanation of the test objective and expected outcome. - *In Python tests, a docstring takes the place of the description attribute.* What should a test.xml look like? ---------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Each testcase has file as well as it's line number as additional attributes. -Then as described above, each testcase also has properties inside the XML. +Each testcase must include its source file and line number as attributes, +along with the verification properties. .. code-block:: xml @@ -80,35 +102,22 @@ Then as described above, each testcase also has properties inside the XML. -If you are not working in Python and using the provided pytest plugin, please ensure that the xml that is written in the end -looks like this, otherwise the extension will not be able to parse the xml correctly. - -What happens when properties are missing -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +When properties or attributes are missing, testcases are still generated, +but a testcase can only be linked to requirements if either ``PartiallyVerifies`` +or ``FullyVerifies`` is filled. -When properties or attributes are missing for some or all tests inside the test.xml, testcases are still generated. -However, testcases can only be linked to the requirements if either Partially- or FullyVerifies is filled. - -Language-Specific Annotations ------------------------------ - -Each language uses a different mechanism to attach properties to test cases, -but all produce the same JUnit XML output that the linker consumes. - -C++ (gTest) - Use ``RecordProperty`` — shared properties go in ``SetUp()``, per-test properties - inside each ``TEST_F``. - See :need:`gd_req__verification_link_tests_cpp`. - -Rust - Currently there is no provided official way to do this in Rust. - We are working with the rust community to figure this out. - -Python (pytest) - Use the ``@add_test_properties`` decorator; the docstring serves as ``Description``. - See :need:`gd_req__verification_link_tests_python`. +Testcase result annotations +--------------------------- -See :need:`gd_temp__verification_specification` for code templates. +GitHub test links are decorated with their result, for example ``(passed)``, +``(failed)``, ``(skipped)``, or ``(disabled)``. The annotation is applied to +rendered links that target a testcase need, including the ``testlink`` entries +shown on requirements. The status text inherits the surrounding theme colour +and uses CSS classes with colours selected for the current S-CORE light and +dark themes. These colours are not configurable for arbitrary Sphinx themes. +Testcases without a result are left unchanged; links whose GitHub URL +identifies multiple testcases are also left unchanged because their result +would be ambiguous. Running Tests and Building Docs @@ -120,7 +129,7 @@ Running Tests and Building Docs bazel test //... -2. Build the documentation — the linker picks up ``bazel-testlogs/`` automatically: +2. Build the documentation — the generated test results are picked up automatically: .. code-block:: bash @@ -183,4 +192,4 @@ Limitations ----------- - Tests must be executed by Bazel before building docs so ``test.xml`` files exist. -- Not compatible with Esbonio / live preview (no ``bazel-testlogs/`` available). +- Not compatible with Esbonio / live preview because generated test results are unavailable there. diff --git a/src/extensions/docs/source_code_linker.md b/src/extensions/docs/source_code_linker.md index 5ea233246..6cf0952c5 100644 --- a/src/extensions/docs/source_code_linker.md +++ b/src/extensions/docs/source_code_linker.md @@ -381,6 +381,7 @@ score_source_code_linker/ ├── needlinks.py # CodeLink dataclass & JSON encoder/decoder ├── testlink.py # DataForTestLink definition & logic ├── xml_parser.py # Parses XML files into test case data +├── testcase_annotations.py # Adds result annotations to GitHub testcase links ├── tests/ # Testsuite, containing unit & integration tests │ └── ... ``` diff --git a/src/extensions/score_source_code_linker/__init__.py b/src/extensions/score_source_code_linker/__init__.py index ef1acfd90..c887dffd6 100644 --- a/src/extensions/score_source_code_linker/__init__.py +++ b/src/extensions/score_source_code_linker/__init__.py @@ -51,6 +51,9 @@ load_repo_source_links_json, store_repo_source_links_json, ) +from src.extensions.score_source_code_linker.testcase_annotations import ( + annotate_testcase_results, +) from src.extensions.score_source_code_linker.testlink import ( DataForTestLink, load_data_of_test_case_json, @@ -140,15 +143,24 @@ def setup_source_code_linker(app: Sphinx, ws_root: Path | None): description="Skip rescanning source code files via the source code linker.", ) - # Define need_string_links here to not have it in conf.py - # source_code_link and testlinks have the same schema + # Define need_string_links here to not have them in conf.py. Test links + # carry the result as an additional field in their serialized value. app.config.needs_string_links.setdefault( - "source_code_linker", + "source_code_linker_pure", { "regex": r"(?P.+)<>(?P.+)", "link_url": "{{url}}", "link_name": "{{name}}", - "options": ["source_code_link", "testlink"], + "options": ["source_code_link"], + }, + ) + app.config.needs_string_links.setdefault( + "test_code_linker", + { + "regex": r"(?P.+)<>(?P.+)<>(?P.+)", + "link_url": "{{url}}", + "link_name": "{{name}} ({{result}})", + "options": ["testlink"], }, ) @@ -323,6 +335,10 @@ def setup_once(app: Sphinx): # Priority=515 to ensure it's called after the test linker & combined connection app.connect("env-updated", inject_links_into_needs, priority=525) + # sphinx-needs resolves NeedIncoming and need metadata links on the same event. + # Run after those resolvers so the GitHub references are available to annotate. + app.connect("doctree-resolved", annotate_testcase_results, priority=800) + def setup(app: Sphinx) -> dict[str, str | bool]: # Esbonio will execute setup() on every iteration. @@ -460,7 +476,7 @@ def _render_test_link( type="score_source_code_linker", ) return str(link.name) - return f"{base}<>{link.name}" + return f"{base}<>{link.name}<>{link.result}" def _warn_missing_need(source_code_links: object) -> None: diff --git a/src/extensions/score_source_code_linker/testcase_annotations.py b/src/extensions/score_source_code_linker/testcase_annotations.py new file mode 100644 index 000000000..9e4d050b2 --- /dev/null +++ b/src/extensions/score_source_code_linker/testcase_annotations.py @@ -0,0 +1,135 @@ +# ******************************************************************************* +# Copyright (c) 2026 Contributors to the Eclipse Foundation +# +# See the NOTICE file(s) distributed with this work for additional +# information regarding copyright ownership. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# ******************************************************************************* +"""Color execution results in rendered GitHub testcase links. + +The ``testlink`` string-link configuration renders each link as +`` ()``. This hook replaces the plain result suffix with +the corresponding colored HTML span after sphinx-needs has created the link. +""" + +from __future__ import annotations + +from html import escape + +from docutils import nodes + +# Known result values get semantic classes so the stylesheet can provide +# readable colours for the current S-CORE light and dark themes. +RESULT_CLASSES = { + "passed": "score-testcase-result--passed", + "failed": "score-testcase-result--failed", + "skipped": "score-testcase-result--skipped", + "disabled": "score-testcase-result--disabled", +} +# The event handler is expected to be idempotent for a doctree. This marker +# prevents a second invocation from appending the same status again. +_ANNOTATED_ATTR = "score_source_code_linker_testcase_result_annotated" + +# The styles are inserted into a document only when that document contains an +# annotation. Keeping them in one block avoids repeating inline style rules on +# every testcase link and lets the same classes handle theme changes. +_TESTCASE_STATUS_CSS = """ + +""" + + +def _result_node(result_text: str, result_class: str) -> nodes.raw: + escaped_result = escape(result_text, quote=True) + status_html = ( + f' ({escaped_result})' + ) + return nodes.raw("", status_html, format="html") + + +def _color_existing_result_suffix(ref: nodes.reference) -> bool: + """Replace a recognized ``(result)`` suffix with a colored node.""" + if not ref.children or not isinstance(ref.children[-1], nodes.Text): + return False + + last_text = ref.children[-1] + text = last_text.astext() + for result_text, result_class in RESULT_CLASSES.items(): + suffix = f" ({result_text})" + if text.endswith(suffix): + prefix = text[: -len(suffix)] + ref.replace(last_text, nodes.Text(prefix)) + ref.append(_result_node(result_text, result_class)) + return True + return False + + +def annotate_testcase_results(app, doctree, docname): + """Color rendered testcase result suffixes using the S-CORE theme palette. + + The handler runs after sphinx-needs' own ``doctree-resolved`` handlers. + It therefore sees the external references generated for GitHub ``testlink`` + metadata, whose labels already contain the result. + """ + # CSS applies to the whole document, so one style block is enough even if + # the document contains many annotated references. + css_added = False + + for ref in list(doctree.findall(nodes.reference)): + if ref.get(_ANNOTATED_ATTR): + # A repeated event invocation must not append another badge. + continue + + if not _color_existing_result_suffix(ref): + continue + + if not css_added: + doctree.insert(0, nodes.raw("", _TESTCASE_STATUS_CSS, format="html")) + css_added = True + ref[_ANNOTATED_ATTR] = True diff --git a/src/extensions/score_source_code_linker/tests/test_source_code_link_integration.py b/src/extensions/score_source_code_linker/tests/test_source_code_link_integration.py index dae4a7d3a..2100bc459 100644 --- a/src/extensions/score_source_code_linker/tests/test_source_code_link_integration.py +++ b/src/extensions/score_source_code_linker/tests/test_source_code_link_integration.py @@ -250,8 +250,25 @@ def basic_conf(): color="#BFD8D2", style="node", ), + dict( + directive="testcase", + title="Testcase", + prefix="testcase__", + color="#D9EAD3", + style="node", + ), +] +needs_extra_options = [ + "source_code_link", + "testlink", + "name", + "file", + "line", + "test_type", + "derivation_technique", + "result", + "result_text", ] -needs_extra_options = ["source_code_link", "testlink"] needs_extra_links = [{ "option": "partially_verifies", "incoming": "paritally_verified_by", @@ -432,7 +449,9 @@ def make_test_link(testlinks: list[DataForTestLink]): url="", hash="", ) - return ", ".join(f"{get_github_link(metadata, n)}<>{n.name}" for n in testlinks) + return ", ".join( + f"{get_github_link(metadata, n)}<>{n.name}<>{n.result}" for n in testlinks + ) def compare_json_files( @@ -552,6 +571,11 @@ def test_source_link_integration_ok( expected_test_link = make_test_link(example_test_link_text_all_ok[treq_id]) actual_test_code_link = treq_info.get("testlink", "no test link") assert expected_test_link == actual_test_code_link, treq_id + + # The rendered testlink references point at the testcase external needs. + # Verify the annotation survives the complete Sphinx HTML rendering path. + rendered_html = (app.outdir / "index.html").read_text(encoding="utf-8") + assert "(passed)" in rendered_html finally: app.cleanup() diff --git a/src/extensions/score_source_code_linker/tests/test_testcase_annotations.py b/src/extensions/score_source_code_linker/tests/test_testcase_annotations.py new file mode 100644 index 000000000..43007fde5 --- /dev/null +++ b/src/extensions/score_source_code_linker/tests/test_testcase_annotations.py @@ -0,0 +1,97 @@ +# ******************************************************************************* +# Copyright (c) 2026 Contributors to the Eclipse Foundation +# +# See the NOTICE file(s) distributed with this work for additional +# information regarding copyright ownership. +# +# This program and the accompanying materials are made available under the +# terms of the Apache License Version 2.0 which is available at +# https://www.apache.org/licenses/LICENSE-2.0 +# +# SPDX-License-Identifier: Apache-2.0 +# ******************************************************************************* +"""Unit tests for testcase result annotations.""" + +from __future__ import annotations + +from docutils import nodes + +from score_pytest.attribute_plugin import add_test_properties +from src.extensions.score_source_code_linker.testcase_annotations import ( + annotate_testcase_results, +) + + +def _doctree_with_reference(*, refid=None, refuri=None, text="testcase"): + doc = nodes.document(None, None) # type: ignore[arg-type] + attrs = {} + if refid is not None: + attrs["refid"] = refid + if refuri is not None: + attrs["refuri"] = refuri + ref = nodes.reference("", "", nodes.Text(text), **attrs) + doc.append(ref) + return doc, ref + + +@add_test_properties( + partially_verifies=["tool_req__docs_test_link_testcase"], + test_type="requirements-based", + derivation_technique="requirements-analysis", +) +def test_annotates_external_github_testlink(): + """Color the result already present in an external testcase link.""" + url = "https://github.com/example/repo/blob/abc/tests/test.py#L42" + doc, ref = _doctree_with_reference(refuri=url, text="test_requirement (passed)") + annotate_testcase_results(None, doc, "requirements") + + html = ref.children[-1].astext() + assert "score-testcase-result--passed" in html + assert "(passed)" in html + assert len(ref.children) == 2 + assert ref.children[0].astext() == "test_requirement" + assert ".score-testcase-result--passed" in doc.astext() + assert 'html[data-theme="dark"]' in doc.astext() + + +@add_test_properties( + partially_verifies=["tool_req__docs_test_link_testcase"], + test_type="requirements-based", + derivation_technique="requirements-analysis", +) +def test_does_not_annotate_reference_without_result_suffix(): + """Leave references without a rendered result suffix unchanged.""" + doc, ref = _doctree_with_reference( + refid="testcase__foo", text="Some testcase title" + ) + annotate_testcase_results(None, doc, "requirements") + assert len(ref.children) == 1 + + +@add_test_properties( + partially_verifies=["tool_req__docs_test_link_testcase"], + test_type="requirements-based", + derivation_technique="requirements-analysis", +) +def test_does_not_annotate_link_without_result_suffix(): + """Leave links unchanged when they have no result suffix.""" + doc, ref = _doctree_with_reference( + refuri="https://github.com/example/repo/blob/abc/src/other.py#L1" + ) + annotate_testcase_results(None, doc, "requirements") + assert len(ref.children) == 1 + + +@add_test_properties( + partially_verifies=["tool_req__docs_test_link_testcase"], + test_type="requirements-based", + derivation_technique="requirements-analysis", +) +def test_does_not_annotate_same_reference_twice(): + """Keep repeated handler invocations from duplicating an annotation.""" + url = "https://github.com/example/repo/blob/abc/tests/test.py#L42" + doc, ref = _doctree_with_reference(refuri=url, text="testcase (passed)") + annotate_testcase_results(None, doc, "requirements") + annotate_testcase_results(None, doc, "requirements") + + assert len(ref.children) == 2