diff --git a/README.md b/README.md index 627f2c31..27d580e8 100644 --- a/README.md +++ b/README.md @@ -45,10 +45,16 @@ This guide walks you through both installation and usage. 1. [Branch Option](#branch-option) 3. [Path Scan](#path-scan) 1. [Terraform Plan Scan](#terraform-plan-scan) - 4. [Commit History Scan](#commit-history-scan) + 4. [Binary Scan](#binary-scan) + 1. [Supported Artifacts](#supported-artifacts) + 2. [Binary Scan Options](#binary-scan-options) + 3. [How Components Are Identified](#how-components-are-identified) + 4. [Unidentified Components](#unidentified-components) + 5. [Limitations](#binary-scan-limitations) + 5. [Commit History Scan](#commit-history-scan) 1. [Commit Range Option (Diff Scanning)](#commit-range-option-diff-scanning) - 5. [Pre-Commit Scan](#pre-commit-scan) - 6. [Pre-Push Scan](#pre-push-scan) + 6. [Pre-Commit Scan](#pre-commit-scan) + 7. [Pre-Push Scan](#pre-push-scan) 2. [Scan Results](#scan-results) 1. [Show/Hide Secrets](#showhide-secrets) 2. [Soft Fail](#soft-fail) @@ -67,6 +73,9 @@ This guide walks you through both installation and usage. 6. [Ignoring via a config file](#ignoring-via-a-config-file) 9. [Report command](#report-command) 1. [Generating SBOM Report](#generating-sbom-report) + 1. [Repository](#repository) + 2. [Local Project](#local-project) + 3. [Built Artifact](#built-artifact) 10. [Import command](#import-command) 11. [Scan logs](#scan-logs) 12. [Syntax Help](#syntax-help) @@ -983,6 +992,127 @@ If you just have a configuration file, you can generate a plan by doing the foll `cycode scan -t iac path ~/PATH/TO/YOUR/{tfplan}.json` +### Binary Scan + +A binary scan examines a built Java artifact instead of a source tree. +Point the CLI at a JAR, WAR, EAR or Spring Boot fat JAR and it identifies the open-source components packaged inside it, then scans them exactly as an SCA scan of the source would. + +This closes the gap between what was scanned and what was shipped. +It is the only option when you have no source at all: a vendor-supplied JAR, a legacy EAR whose build job no longer exists, or a release gate that should check the actual deployable rather than the commit that supposedly produced it. + +To scan an artifact, execute the following: + +`cycode scan -t sca binary {{path}}` + +For example: + +```shell +# a single deployable +cycode scan -t sca binary ./dist/payments.war + +# every Java archive under a directory +cycode scan -t sca binary ./dist + +# recurse further into deeply nested archives +cycode scan -t sca --max-depth 5 binary ./dist/payments.ear +``` + +Everything an SCA scan normally supports still applies: `--severity-threshold`, `--soft-fail`, `cycode ignore` rules, `--export`, `--cycode-report` and the usual exit codes. + +> [!IMPORTANT] +> The artifact never leaves your machine. +> The CLI opens it locally and uploads only the resulting component inventory, which is typically a few tens of kilobytes regardless of how large the artifact is. + +You can also extract archives encountered during an ordinary path scan, using `--include-binaries`: + +`cycode scan -t sca --include-binaries path ./target` + +This is off by default. +Extracting a large tree of artifacts takes real time, and a path scan that silently got slower would be a worse surprise than an opt-in flag. + +#### Supported Artifacts + +| Artifact | Recognized layouts | +|---|---| +| JAR | `lib/*.jar`, embedded Maven metadata | +| WAR | `WEB-INF/lib/*.jar` | +| EAR | nested `*.war` and `*.jar` modules, `APP-INF/lib/*.jar` | +| Spring Boot fat JAR | `BOOT-INF/lib/*.jar` | + +Nested archives are opened recursively, so a JAR inside a WAR inside an EAR is scanned. + +#### Binary Scan Options + +> [!NOTE] +> These options belong to the `scan` command, so they must appear **before** the `binary` subcommand: +> `cycode scan -t sca --max-depth 5 binary app.ear` + +| Option | Default | Description | +|---|---|---| +| `--max-depth` | `3` | Nested-archive recursion limit. An EAR containing WARs containing JARs is depth 3. | +| `--offline` | off | Identify components from embedded metadata only. Acknowledges and silences the partial-results warning. | +| `--maven-central` | off | Look up archives that embedded metadata cannot identify on Maven Central by SHA-1. Sends only the digest, never the archive. Cannot be combined with `--offline`. | +| `--project-name` | inferred | Override the platform identity when the artifact is detached from its source repository. | +| `--keep-bom` | off | Write the generated component inventory beside each artifact, for inspection or audit. | +| `--include-binaries` | off | On `scan path` only. Extract any Java archives encountered during the walk. | + +Platform identity is inferred from the Git remote when you run inside a repository, and falls back to the artifact filename otherwise. +`--monitor` is refused on a bare filename identity, because monitoring keyed on `app.jar` would merge unrelated projects into one and quietly corrupt the trend data. +Use `--project-name` or run from inside the repository the artifact was built from. + +#### How Components Are Identified + +Components are identified from metadata the build itself wrote into the archive: + +| Source | Confidence | Notes | +|---|---|---| +| `META-INF/maven/.../pom.properties` | exact | Written by Maven. Authoritative group, artifact and version. | +| Maven Central digest lookup (`--maven-central`) | exact | The SHA-1 of the archive, matched against what Maven Central published. Opt-in, because it is the one step that sends anything about the artifact off the machine. | +| `META-INF/MANIFEST.MF` attributes | low | `Implementation-Title`, OSGi `Bundle-SymbolicName` and similar, with the group taken from `Implementation-Vendor-Id`. Used only when all three parts are shaped like Maven coordinates; a product name or a build banner is not one. | + +Every finding reports which source identified its component and where inside the artifact that component sits, so a hit on a large EAR points at a specific nested JAR rather than the whole file. + +> [!NOTE] +> Findings from a low-confidence match are printed and exported, but do **not** affect the exit code. +> A wrong coordinate produces a wrong vulnerability list, and a fabricated CVE breaking a release costs more trust than the extra coverage is worth. + +#### Unidentified Components + +Archives that carry no usable metadata are listed in their own section by path, digest and size, and appear under an `unidentified` key in `--output json` so CI can assert on coverage: + +``` +╭─ 🔎 Unidentified (1) ────────────────────────────────────────────╮ +│ Path SHA-1 Size │ +│ payments.war > WEB-INF/lib/internal-shim.jar 5fe08079… 142 B │ +╰──────────────────────────────────────────────────────────────────╯ + +3 identified (1 low confidence) | 1 unidentified | 17 vulnerabilities +``` + +The coverage line counts manifest-only matches as identified but calls them out, and `--output json` reports the same number as `binary.low_confidence_components`. + +We do not guess a component's identity from its filename. +`internal-shim.jar` is not evidence of anything, and an admitted gap is more useful than an invented coordinate. +Unidentified components do not set the exit code on their own. + +#### Binary Scan Limitations + +Read this before relying on a binary scan as your only check. + +- **Relocated and shaded classes are not detected.** + When a build rewrites `com.google.common` into `com.acme.shaded.common` and merges it into the parent JAR, there is no separate JAR to identify and no metadata left to read. + Those components will not appear in the results at all. + Detecting them requires class-level fingerprinting, which this feature does not do. +- **Source scanning gives a truer picture.** + A source scan resolves the real dependency graph from your lockfiles. + A binary scan sees what is physically packaged, and infers relationships from archive nesting plus any embedded `pom.xml` files it finds. + Where you have source, scan the source; use binary scanning for the artifacts you cannot scan any other way. +- **Coverage is reported, not assumed.** + The coverage line is always printed and always true. + If it says components could not be identified, the scan is genuinely incomplete for those components rather than clean. +- **Java only, for now.** + .NET, npm and container artifacts are not supported yet. + ### Commit History Scan > [!NOTE] @@ -1533,6 +1663,7 @@ The following commands are available for use with this command: |------------------|-----------------------------------------------------------------| | `path` | Generate SBOM report for provided path in the command | | `repository-url` | Generate SBOM report for provided repository URI in the command | +| `binary` | Generate SBOM report for a built Java artifact (JAR, WAR, EAR) | ### Repository @@ -1558,6 +1689,23 @@ The `path` subcommand supports the following additional options: | `--gradle-all-sub-projects` | Run the Gradle restore command for all sub-projects (use from the root of a multi-project Gradle build). | | `--maven-settings-file` | For Maven only, allows using a custom [settings.xml](https://maven.apache.org/settings.html) file when building the dependency tree. | +### Built Artifact + +To create an SBOM report for a built Java artifact, without scanning it for vulnerabilities:\ +`cycode report sbom --format --output-file binary ` + +For example:\ +`cycode report sbom --format spdx-2.3 --output-file payments-sbom.json binary ./dist/payments.war` + +This answers the compliance case directly: an SBOM of what you actually shipped, rather than of what was committed. +It uses the same extraction as [Binary Scan](#binary-scan), so the [limitations](#binary-scan-limitations) documented there apply here too — in particular, shaded and relocated components will be missing from the SBOM. + +The `binary` subcommand supports the following additional option: + +| Option | Description | +|---------------|----------------------------------------------------------------------------------------------| +| `--max-depth` | Nested-archive recursion limit. Defaults to 3. An EAR containing WARs containing JARs is depth 3. | + # Import Command ## Importing SBOM diff --git a/cycode/cli/apps/report/sbom/__init__.py b/cycode/cli/apps/report/sbom/__init__.py index 77d081e8..b0a188aa 100644 --- a/cycode/cli/apps/report/sbom/__init__.py +++ b/cycode/cli/apps/report/sbom/__init__.py @@ -1,5 +1,6 @@ import typer +from cycode.cli.apps.report.sbom.binary.binary_command import binary_command from cycode.cli.apps.report.sbom.path.path_command import path_command from cycode.cli.apps.report.sbom.repository_url.repository_url_command import repository_url_command from cycode.cli.apps.report.sbom.sbom_command import sbom_command @@ -10,6 +11,7 @@ app.command(name='repository-url', short_help='Generate SBOM report for provided repository URI in the command.')( repository_url_command ) +app.command(name='binary', short_help='Generate SBOM report for a built Java artifact (JAR, WAR, EAR).')(binary_command) # backward compatibility app.command(hidden=True, name='repository_url')(repository_url_command) diff --git a/cycode/cli/apps/report/sbom/binary/__init__.py b/cycode/cli/apps/report/sbom/binary/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/cycode/cli/apps/report/sbom/binary/binary_command.py b/cycode/cli/apps/report/sbom/binary/binary_command.py new file mode 100644 index 00000000..c1356b3e --- /dev/null +++ b/cycode/cli/apps/report/sbom/binary/binary_command.py @@ -0,0 +1,115 @@ +import time +from pathlib import Path +from typing import Annotated + +import typer + +from cycode.cli import consts +from cycode.cli.apps.report.sbom.common import create_sbom_report, send_report_feedback +from cycode.cli.exceptions.handle_report_sbom_errors import handle_report_exception +from cycode.cli.files_collector.binary.collector import collect_binary_documents +from cycode.cli.files_collector.zip_documents import zip_documents +from cycode.cli.utils.get_api_client import get_report_cycode_client +from cycode.cli.utils.progress_bar import SbomReportProgressBarSection + +_REPORT_COMMAND_TYPE = 'binary' + + +def binary_command( + ctx: typer.Context, + path: Annotated[ + Path, + typer.Argument( + exists=True, + resolve_path=True, + help='Path to the built artifact to generate an SBOM for.', + show_default=False, + ), + ], + max_depth: Annotated[ + int, + typer.Option('--max-depth', help='Nested-archive recursion limit.', min=1), + ] = consts.BINARY_MAX_DEPTH, + maven_central: Annotated[ + bool, + typer.Option( + '--maven-central', + help='Look archives that embedded metadata cannot identify up on Maven Central by SHA-1. ' + 'Sends the digest of each such archive, never the archive itself, to search.maven.org.', + ), + ] = False, +) -> None: + """:package: [bold cyan]Generate an SBOM for a built Java artifact.[/] + + Reads a JAR, WAR, EAR or Spring Boot fat JAR and produces an SBOM of the open-source components inside it, + without scanning them for vulnerabilities. Answers the compliance case directly: an SBOM of what shipped, + rather than of what was committed. + + Example usage: + * `cycode report sbom --format cyclonedx-1.4-json binary app.war` + * `cycode report sbom --format spdx-2.3-json binary app.ear` + + Format conversion happens server-side, so every format the path command supports is supported here too. + + """ + ctx.obj['binary_max_depth'] = max_depth + ctx.obj['maven_central'] = maven_central + + client = get_report_cycode_client(ctx) + report_parameters = ctx.obj['report_parameters'] + output_format = report_parameters.output_format + output_file = ctx.obj['output_file'] + + progress_bar = ctx.obj['progress_bar'] + progress_bar.start() + + start_scan_time = time.time() + report_execution_id = -1 + + try: + # the only difference from the path command: our collector in place of the manifest walk. Everything from + # zip_documents onward is reused verbatim, and the server generates the document. + collection = collect_binary_documents( + ctx, + (str(path),), + stop_on_error=ctx.obj.get('stop_on_error', False), + progress_bar_section=SbomReportProgressBarSection.PREPARE_LOCAL_FILES, + ) + ctx.obj['binary_result'] = collection + + if not collection.documents: + raise typer.BadParameter( + f'No supported binary artifacts were found at {str(path)!r}. ' + 'Supported artifacts are .jar, .war and .ear files.', + param_hint='PATH', + ) + + zipped_documents = zip_documents(consts.SCA_SCAN_TYPE, collection.documents) + report_execution = client.request_sbom_report_execution(report_parameters, zip_file=zipped_documents) + report_execution_id = report_execution.id + + create_sbom_report(progress_bar, client, report_execution_id, output_file, output_format) + + send_report_feedback( + client=client, + start_scan_time=start_scan_time, + report_type='SBOM', + report_command_type=_REPORT_COMMAND_TYPE, + request_report_parameters=report_parameters.to_dict(without_entity_type=False), + report_execution_id=report_execution_id, + request_zip_file_size=zipped_documents.size, + ) + except Exception as e: + progress_bar.stop() + + send_report_feedback( + client=client, + start_scan_time=start_scan_time, + report_type='SBOM', + report_command_type=_REPORT_COMMAND_TYPE, + request_report_parameters=report_parameters.to_dict(without_entity_type=False), + report_execution_id=report_execution_id, + error_message=str(e), + ) + + handle_report_exception(ctx, e) diff --git a/cycode/cli/apps/scan/__init__.py b/cycode/cli/apps/scan/__init__.py index 629c3b8f..abd0e83e 100644 --- a/cycode/cli/apps/scan/__init__.py +++ b/cycode/cli/apps/scan/__init__.py @@ -1,5 +1,6 @@ import typer +from cycode.cli.apps.scan.binary.binary_command import binary_command from cycode.cli.apps.scan.commit_history.commit_history_command import commit_history_command from cycode.cli.apps.scan.path.path_command import path_command from cycode.cli.apps.scan.pre_commit.pre_commit_command import pre_commit_command @@ -23,6 +24,7 @@ app.command(name='path', short_help='Scan the files in the paths provided in the command.')(path_command) app.command(name='repository', short_help='Scan the Git repository included files.')(repository_command) +app.command(name='binary', short_help='Scan built Java artifacts (JAR, WAR, EAR, Spring Boot).')(binary_command) app.command(name='commit-history', short_help='Scan commit history or perform diff scanning between specific commits.')( commit_history_command ) diff --git a/cycode/cli/apps/scan/binary/__init__.py b/cycode/cli/apps/scan/binary/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/cycode/cli/apps/scan/binary/binary_command.py b/cycode/cli/apps/scan/binary/binary_command.py new file mode 100644 index 00000000..89f1ccbc --- /dev/null +++ b/cycode/cli/apps/scan/binary/binary_command.py @@ -0,0 +1,58 @@ +from pathlib import Path +from typing import Annotated + +import typer + +from cycode.cli.apps.scan.binary.identity import ( + assert_monitor_has_an_explicit_identity, + resolve_platform_identity, +) +from cycode.cli.apps.scan.code_scanner import scan_binary_artifacts +from cycode.cli.logger import logger + + +def binary_command( + ctx: typer.Context, + paths: Annotated[ + list[Path], + typer.Argument( + exists=True, + resolve_path=True, + help='Paths to the built artifacts to scan', + show_default=False, + ), + ], +) -> None: + """:package: [bold cyan]Scan built Java artifacts for open-source vulnerabilities.[/] + + Opens a JAR, WAR, EAR or Spring Boot fat JAR, identifies the open-source components inside it, and scans them + exactly as a source scan would. The artifact never leaves your machine: only the component inventory is + uploaded. + + Example usage: + * `cycode scan -t sca binary app.war`: Scan a single deployable. + * `cycode scan -t sca binary dist/`: Scan every Java archive under a directory. + * `cycode scan -t sca --max-depth 5 binary app.ear`: Recurse further into nested archives. + + Components are identified from embedded Maven metadata. Anything that cannot be identified is reported in its + own section rather than guessed at. Relocated and shaded classes are not detected: where source is available, + a source scan gives a truer dependency graph. + + """ + tuple_paths = tuple(str(path) for path in paths) + + identity = resolve_platform_identity(ctx, tuple_paths) + if ctx.obj.get('monitor'): + assert_monitor_has_an_explicit_identity(identity) + + ctx.obj['binary_identity'] = identity + + progress_bar = ctx.obj['progress_bar'] + progress_bar.start() + + logger.debug( + 'Starting binary scan process, %s', + {'paths': paths, 'identity': identity.value, 'identity_source': identity.source}, + ) + + scan_binary_artifacts(ctx, tuple_paths) diff --git a/cycode/cli/apps/scan/binary/identity.py b/cycode/cli/apps/scan/binary/identity.py new file mode 100644 index 00000000..c3283b80 --- /dev/null +++ b/cycode/cli/apps/scan/binary/identity.py @@ -0,0 +1,66 @@ +"""What the platform files a binary scan under. + +Identity lives with the command rather than with the collector: it is a question about how results are recorded, +not about how an archive is read, and keeping it here leaves the collector free of any dependency on the scan app. +""" + +import os +from dataclasses import dataclass + +import typer + +from cycode.cli.apps.scan.remote_url_resolver import get_remote_url_scan_parameter +from cycode.cli.files_collector.binary.collector import find_supported_artifacts + +IDENTITY_FROM_PROJECT_NAME = 'project-name' +IDENTITY_FROM_GIT_REMOTE = 'git-remote' +IDENTITY_FROM_FILENAME = 'filename' + + +@dataclass(frozen=True) +class PlatformIdentity: + """What the platform will file these results under, and where that came from.""" + + value: str + source: str + + @property + def is_explicit(self) -> bool: + """True when a human or a repository named this project, rather than it being taken off a filename.""" + return self.source in (IDENTITY_FROM_PROJECT_NAME, IDENTITY_FROM_GIT_REMOTE) + + +def resolve_platform_identity(ctx: typer.Context, paths: tuple[str, ...]) -> PlatformIdentity: + """Inside a repository, results attach to that repository exactly as a path scan does. + + Detached from one, the artifact filename is the identity, which is fine for a one-off assessment and is + explicitly not fine for monitoring. + """ + project_name = ctx.obj.get('project_name') + if project_name: + return PlatformIdentity(value=project_name, source=IDENTITY_FROM_PROJECT_NAME) + + remote_url = get_remote_url_scan_parameter(paths) + if remote_url: + return PlatformIdentity(value=remote_url, source=IDENTITY_FROM_GIT_REMOTE) + + artifacts = find_supported_artifacts(paths) + filename = os.path.basename(artifacts[0]) if artifacts else os.path.basename(paths[0]) + return PlatformIdentity(value=filename, source=IDENTITY_FROM_FILENAME) + + +def assert_monitor_has_an_explicit_identity(identity: PlatformIdentity) -> None: + """Refuse --monitor on a bare filename identity. + + Monitoring keyed on `app.jar` would merge unrelated teams into one project and quietly corrupt the trend data, + which is worse than refusing: the damage is invisible until someone acts on the numbers. + """ + if identity.is_explicit: + return + + raise typer.BadParameter( + f'--monitor needs an explicit project identity, but the only identity available is the artifact filename ' + f'({identity.value!r}). Run from inside the Git repository this artifact was built from, or pass ' + f'--project-name to name the project yourself.', + param_hint='--monitor', + ) diff --git a/cycode/cli/apps/scan/code_scanner.py b/cycode/cli/apps/scan/code_scanner.py index 667138fa..91d9f586 100644 --- a/cycode/cli/apps/scan/code_scanner.py +++ b/cycode/cli/apps/scan/code_scanner.py @@ -19,16 +19,19 @@ from cycode.cli.config import configuration_manager from cycode.cli.exceptions import custom_exceptions from cycode.cli.exceptions.handle_scan_errors import handle_scan_exception +from cycode.cli.files_collector.binary.collector import collect_binary_documents from cycode.cli.files_collector.path_documents import get_relevant_documents from cycode.cli.files_collector.sca.sca_file_collector import add_sca_dependencies_tree_documents_if_needed from cycode.cli.files_collector.zip_documents import zip_documents from cycode.cli.models import CliError, Document, LocalScanResult +from cycode.cli.printers.utils import binary_report from cycode.cli.utils.path_utils import get_absolute_path, get_path_by_os from cycode.cli.utils.progress_bar import ScanProgressBarSection from cycode.cli.utils.scan_batch import run_parallel_batched_scan from cycode.cli.utils.scan_utils import ( generate_unique_scan_id, is_cycodeignore_allowed_by_scan_config, + set_issue_detected, set_issue_detected_by_scan_results, should_use_presigned_upload, ) @@ -106,11 +109,61 @@ def scan_disk_files(ctx: typer.Context, paths: tuple[str, ...]) -> None: documents.append(entrypoint_document) add_sca_dependencies_tree_documents_if_needed(ctx, scan_type, documents) + _add_binary_documents_if_needed(ctx, paths, documents) scan_documents(ctx, documents, get_scan_parameters(ctx, paths)) except Exception as e: handle_scan_exception(ctx, e) +def _add_binary_documents_if_needed(ctx: typer.Context, paths: tuple[str, ...], documents: list[Document]) -> None: + """``--include-binaries``: extract any Java archives met during the walk and append their BOMs. + + Off by default and never implicit. Extraction of a large artifact tree costs real time, and a path scan that + silently got slower would be a worse surprise than an opt-in flag. + """ + if not ctx.obj.get('include_binaries'): + return + + # the local-files section is already complete by this point, so the collector must not reopen it + collection = collect_binary_documents( + ctx, paths, stop_on_error=ctx.obj.get('stop_on_error', False), progress_bar_section=None + ) + if not collection.documents: + logger.debug('No binary artifacts found during the walk, %s', {'paths': paths}) + return + + logger.debug( + 'Adding binary artifact documents, %s', + {'artifacts': len(collection.results_by_artifact), 'documents': len(collection.documents)}, + ) + ctx.obj['binary_result'] = collection + documents.extend(collection.documents) + + +def scan_binary_artifacts(ctx: typer.Context, paths: tuple[str, ...]) -> None: + """Scan built artifacts by synthesising a CycloneDX document for each and feeding it to the normal SCA path. + + A sibling of scan_disk_files rather than a variation on it: the archive itself never becomes a Document, so the + binary filter in file_excluder stays correct and untouched. What reaches scan_documents is exactly one + synthesised bom.json per artifact, which the SCA engine already knows how to consume. + """ + try: + collection = collect_binary_documents(ctx, paths, stop_on_error=ctx.obj.get('stop_on_error', False)) + + # the printers read the unidentified section and the coverage line from here + ctx.obj['binary_result'] = collection + + for artifact_path, error in collection.failures.items(): + logger.warning('Could not read an artifact, %s', {'path': artifact_path, 'error': error}) + + if not collection.documents: + logger.warning('No supported binary artifacts were found, %s', {'paths': paths}) + + scan_documents(ctx, collection.documents, get_scan_parameters(ctx, paths)) + except Exception as e: + handle_scan_exception(ctx, e) + + def _should_use_sync_flow(command_scan_type: str, scan_type: str, sync_option: bool) -> bool: """Decide whether to use sync flow or async flow for the scan. @@ -297,10 +350,24 @@ def scan_documents( progress_bar.update(ScanProgressBarSection.GENERATE_REPORT) progress_bar.stop() - set_issue_detected_by_scan_results(ctx, local_scan_results) + _set_issue_detected(ctx, local_scan_results) print_local_scan_results(ctx, local_scan_results, errors) +def _set_issue_detected(ctx: typer.Context, local_scan_results: list['LocalScanResult']) -> None: + """Decide whether this scan fails the build. + + For a binary scan, findings whose component was identified only from manifest attributes are excluded: a + fabricated CVE from a guessed coordinate breaking someone's release is the fastest way to lose trust in the + feature. They are still printed, still exported, still counted. + """ + if binary_report.get_binary_collection(ctx) is None: + set_issue_detected_by_scan_results(ctx, local_scan_results) + return + + set_issue_detected(ctx, binary_report.has_gating_detections(ctx, local_scan_results)) + + def _perform_scan_v4_async( cycode_client: 'ScanClient', zipped_documents: 'InMemoryZip', diff --git a/cycode/cli/apps/scan/scan_command.py b/cycode/cli/apps/scan/scan_command.py index 427f2d78..cb1261cd 100644 --- a/cycode/cli/apps/scan/scan_command.py +++ b/cycode/cli/apps/scan/scan_command.py @@ -5,6 +5,7 @@ import click import typer +from cycode.cli import consts from cycode.cli.apps.activation_manager import report_cli_activation, should_report_cli_activation from cycode.cli.apps.sca_options import ( GradleAllSubProjectsOption, @@ -26,6 +27,7 @@ _EXPORT_RICH_HELP_PANEL = 'Export options' _SCA_RICH_HELP_PANEL = 'SCA options' _SECRET_RICH_HELP_PANEL = 'Secret options' +_BINARY_RICH_HELP_PANEL = 'Binary options' def _single_value_callback(ctx: typer.Context, param: typer.CallbackParam, value: list) -> list: @@ -105,6 +107,58 @@ def scan_command( no_restore: NoRestoreOption = False, gradle_all_sub_projects: GradleAllSubProjectsOption = False, maven_settings_file: MavenSettingsFileOption = None, + max_depth: Annotated[ + int, + typer.Option( + '--max-depth', + help='Nested-archive recursion limit. An EAR containing WARs containing JARs is depth 3.', + min=1, + rich_help_panel=_BINARY_RICH_HELP_PANEL, + ), + ] = consts.BINARY_MAX_DEPTH, + offline: Annotated[ + bool, + typer.Option( + '--offline', + help='Identify components from embedded metadata only, without resolving unknown digests. ' + 'Acknowledges and silences the partial-results warning.', + rich_help_panel=_BINARY_RICH_HELP_PANEL, + ), + ] = False, + maven_central: Annotated[ + bool, + typer.Option( + '--maven-central', + help='Look archives that embedded metadata cannot identify up on Maven Central by SHA-1. ' + 'Sends the digest of each such archive, never the archive itself, to search.maven.org.', + rich_help_panel=_BINARY_RICH_HELP_PANEL, + ), + ] = False, + project_name: Annotated[ + Optional[str], + typer.Option( + '--project-name', + help='Override the platform identity when the artifact is detached from its source repository.', + show_default='inferred from the Git remote, else the artifact filename', + rich_help_panel=_BINARY_RICH_HELP_PANEL, + ), + ] = None, + include_binaries: Annotated[ + bool, + typer.Option( + '--include-binaries', + help='On `scan path` only. Extract and scan any Java archives encountered during the walk.', + rich_help_panel=_BINARY_RICH_HELP_PANEL, + ), + ] = False, + keep_bom: Annotated[ + bool, + typer.Option( + '--keep-bom', + help='Write the synthesised CycloneDX document beside each artifact for inspection.', + rich_help_panel=_BINARY_RICH_HELP_PANEL, + ), + ] = False, export_type: Annotated[ ExportTypeOption, typer.Option( @@ -139,6 +193,7 @@ def scan_command( * `cycode scan path `: Scan a specific local directory or file. * `cycode scan repository `: Scan Git related files in a local Git repository. * `cycode scan commit-history `: Scan the commit history of a local Git repository. + * `cycode scan -t sca binary `: Scan a built Java artifact (JAR, WAR, EAR, Spring Boot). """ if export_file and export_type is None: @@ -151,6 +206,11 @@ def scan_command( 'Export file must be specified when --export-type is provided.', param_hint='--export-file', ) + if offline and maven_central: + raise typer.BadParameter( + '--offline identifies from embedded metadata only; it cannot be combined with --maven-central.', + param_hint='--maven-central', + ) # _single_value_callback validated exactly one value was provided; unwrap from list scan_type = scan_type[0] @@ -163,6 +223,12 @@ def scan_command( ctx.obj['severity_threshold'] = severity_threshold ctx.obj['monitor'] = monitor ctx.obj['report'] = report + ctx.obj['binary_max_depth'] = max_depth + ctx.obj['offline'] = offline + ctx.obj['maven_central'] = maven_central + ctx.obj['project_name'] = project_name + ctx.obj['keep_bom'] = keep_bom + ctx.obj['include_binaries'] = include_binaries apply_sca_restore_options_to_context(ctx, no_restore, gradle_all_sub_projects, maven_settings_file) scan_client = get_scan_cycode_client(ctx) diff --git a/cycode/cli/apps/scan/scan_parameters.py b/cycode/cli/apps/scan/scan_parameters.py index f362d419..0c21875c 100644 --- a/cycode/cli/apps/scan/scan_parameters.py +++ b/cycode/cli/apps/scan/scan_parameters.py @@ -35,6 +35,12 @@ def get_scan_parameters(ctx: typer.Context, paths: Optional[tuple[str, ...]] = N ctx.obj['remote_url'] = remote_url scan_parameters['remote_url'] = remote_url + # An artifact detached from its source repository has no remote URL, so the platform is told the project name + # explicitly instead. Additive: nothing else in the CLI sets this today. + project_name = ctx.obj.get('project_name') + if project_name: + scan_parameters['project_name'] = project_name + # Include branch information if available (for repository scans) branch = ctx.obj.get('branch') if branch: diff --git a/cycode/cli/consts.py b/cycode/cli/consts.py index 7272dae3..807c0a4e 100644 --- a/cycode/cli/consts.py +++ b/cycode/cli/consts.py @@ -227,6 +227,17 @@ PRESIGNED_LINK_UPLOADED_ZIP_MAX_SIZE_LIMIT_IN_BYTES = 5 * 1024 * 1024 * 1024 # 5 GB (S3 presigned POST limit) PRESIGNED_UPLOAD_SCAN_TYPES = {SAST_SCAN_TYPE, SECRET_SCAN_TYPE} +# binary composition analysis: limits applied while reading untrusted archives (JAR/WAR/EAR) +BINARY_MAX_DEPTH = 3 +BINARY_MAVEN_CENTRAL_SEARCH_URL = 'https://search.maven.org/solrsearch/select' +BINARY_DIGEST_LOOKUP_TIMEOUT_IN_SECONDS = 10 +BINARY_MAX_ENTRY_COUNT = 100_000 +BINARY_MAX_ENTRY_SIZE_IN_BYTES = 512 * 1024 * 1024 # 512 MB +BINARY_MAX_TOTAL_SIZE_IN_BYTES = 2 * 1024 * 1024 * 1024 # 2 GB +BINARY_MAX_COMPRESSION_RATIO = 200 +# small, highly compressible files legitimately beat the ratio ceiling, so it only applies past this floor +BINARY_COMPRESSION_RATIO_FLOOR_IN_BYTES = 1024 * 1024 # 1 MB + DEFAULT_ZIP_MAX_SIZE_LIMIT_IN_BYTES = 20 * 1024 * 1024 ZIP_MAX_SIZE_LIMIT_IN_BYTES = { SCA_SCAN_TYPE: 200 * 1024 * 1024, diff --git a/cycode/cli/exceptions/custom_exceptions.py b/cycode/cli/exceptions/custom_exceptions.py index a9a1505f..896d9558 100644 --- a/cycode/cli/exceptions/custom_exceptions.py +++ b/cycode/cli/exceptions/custom_exceptions.py @@ -80,6 +80,41 @@ def __str__(self) -> str: return self.error_message +class BinaryExtractionError(CycodeError): + """Raised when an artifact cannot be read safely enough to identify what is inside it.""" + + def __init__(self, error_message: str) -> None: + self.error_message = error_message + super().__init__(self.error_message) + + def __str__(self) -> str: + return self.error_message + + +class MalformedArchiveError(BinaryExtractionError): + """The bytes are not a readable archive: truncated directory, bad header, or not a zip at all.""" + + +class UnsafeArchiveEntryError(BinaryExtractionError): + """An entry name would escape the archive root, or names a location we refuse to reproduce.""" + + +class ArchiveLimitExceededError(BinaryExtractionError): + """An archive breached one of the resource ceilings in consts.""" + + +class ArchiveEntryCountLimitError(ArchiveLimitExceededError): ... + + +class ArchiveEntrySizeLimitError(ArchiveLimitExceededError): ... + + +class ArchiveTotalSizeLimitError(ArchiveLimitExceededError): ... + + +class ArchiveCompressionRatioLimitError(ArchiveLimitExceededError): ... + + class AuthProcessError(CycodeError): def __init__(self, error_message: str) -> None: self.error_message = error_message diff --git a/cycode/cli/exceptions/handle_scan_errors.py b/cycode/cli/exceptions/handle_scan_errors.py index 56af186c..4f588aa3 100644 --- a/cycode/cli/exceptions/handle_scan_errors.py +++ b/cycode/cli/exceptions/handle_scan_errors.py @@ -32,6 +32,13 @@ def handle_scan_exception(ctx: typer.Context, err: Exception, *, return_exceptio message='File collection failed. ' 'Use --no-restore to skip dependency restoration, or fix the underlying issue.', ), + custom_exceptions.BinaryExtractionError: CliError( + soft_fail=False, + code='binary_extraction_error', + message=f'\n{err!s}\n' + 'The artifact could not be read safely enough to identify what is inside it. ' + 'If this is a legitimate build artifact, please report it.', + ), custom_exceptions.TfplanKeyError: CliError( soft_fail=True, code='key_error', diff --git a/cycode/cli/files_collector/binary/__init__.py b/cycode/cli/files_collector/binary/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/cycode/cli/files_collector/binary/base_extractor.py b/cycode/cli/files_collector/binary/base_extractor.py new file mode 100644 index 00000000..031fab2a --- /dev/null +++ b/cycode/cli/files_collector/binary/base_extractor.py @@ -0,0 +1,97 @@ +"""The contract every binary extractor implements. + +Shaped to mirror ``BaseRestoreDependencies`` so the two read as siblings to anyone already familiar with the SCA +collector: a predicate that claims a file, a step that reads it, and a step that turns what was read into something +the BOM builder can use. +""" + +from abc import ABC, abstractmethod +from dataclasses import dataclass, field +from typing import Optional + +LOGICAL_PATH_SEPARATOR = ' > ' + +# which tier produced a component, recorded on it so an analyst can weigh the match +EVIDENCE_POM_PROPERTIES = 'pom.properties' +EVIDENCE_DIGEST = 'digest' +EVIDENCE_MANIFEST = 'manifest.mf' + +CONFIDENCE_EXACT = 'exact' +CONFIDENCE_AMBIGUOUS = 'ambiguous' + + +def join_logical_path(parent: Optional[str], name: str) -> str: + """Build the containment chain shown to users, e.g. ``app.ear > web.war > WEB-INF/lib/guava.jar``.""" + if not parent: + return name + + return f'{parent}{LOGICAL_PATH_SEPARATOR}{name}' + + +@dataclass(frozen=True) +class ArchiveEntry: + """One file recovered from an archive, with its containment chain.""" + + logical_path: str # 'app.ear > web.war > WEB-INF/lib/guava.jar' + name: str # 'guava-31.1-jre.jar' + size: int # uncompressed bytes + sha1: str + depth: int # how many archives had to be opened to reach it; the scanned artifact itself is 0 + parent: Optional[str] # logical_path of the containing archive + payload: Optional[bytes] = None # populated only for entries we must parse + sha256: Optional[str] = None + is_archive: bool = False + was_opened: bool = False # an archive we declined to open still gets reported, just not walked + + +@dataclass(frozen=True) +class IdentifiedComponent: + group: str + artifact: str + version: str + sha1: str + logical_path: str + parent: Optional[str] + evidence: str # one of the EVIDENCE_* constants + confidence: str # one of the CONFIDENCE_* constants + sha256: Optional[str] = None + + @property + def purl(self) -> str: + # tier 3 recovers a name and a version but rarely a groupId; a purl may legitimately have no namespace, + # and inventing one would be worse than omitting it + if not self.group: + return f'pkg:maven/{self.artifact}@{self.version}' + + return f'pkg:maven/{self.group}/{self.artifact}@{self.version}' + + +@dataclass(frozen=True) +class UnidentifiedArtifact: + logical_path: str + sha1: str + size: int + + +@dataclass +class ExtractionResult: + components: list[IdentifiedComponent] = field(default_factory=list) + unidentified: list[UnidentifiedArtifact] = field(default_factory=list) + archives_opened: int = 0 + max_depth_reached: int = 0 + resolver_available: bool = True + resolver_unavailability_reason: Optional[str] = None + # bom-ref -> the refs it depends on, containment overlaid with real edges recovered from embedded poms + dependency_edges: dict[str, list[str]] = field(default_factory=dict) + has_real_edges: bool = False + + +class BinaryExtractor(ABC): + @abstractmethod + def handles(self, path: str) -> bool: ... + + @abstractmethod + def extract(self, path: str, max_depth: int) -> list[ArchiveEntry]: ... + + @abstractmethod + def identify(self, entries: list[ArchiveEntry]) -> ExtractionResult: ... diff --git a/cycode/cli/files_collector/binary/collector.py b/cycode/cli/files_collector/binary/collector.py new file mode 100644 index 00000000..05f06ff9 --- /dev/null +++ b/cycode/cli/files_collector/binary/collector.py @@ -0,0 +1,267 @@ +"""Orchestration: artifact paths in, scannable documents out. + +This is the only module in the binary package that knows about ``typer.Context``. Everything below it is a pure +function over bytes, which is what makes the extraction and identification logic unit-testable without a network, +an authenticated client, or a CLI invocation. +""" + +import os +import re +from dataclasses import dataclass, field +from typing import Optional +from xml.sax.saxutils import escape + +import typer + +from cycode.cli import consts +from cycode.cli.files_collector.binary import cyclonedx_builder +from cycode.cli.files_collector.binary.base_extractor import CONFIDENCE_AMBIGUOUS, BinaryExtractor, ExtractionResult +from cycode.cli.files_collector.binary.java_extractor import JavaArchiveExtractor +from cycode.cli.files_collector.binary.maven_central import MavenCentralDigestResolver +from cycode.cli.files_collector.binary.resolver import DigestResolver, NullDigestResolver +from cycode.cli.models import Document +from cycode.cli.utils.path_utils import get_path_by_os +from cycode.cli.utils.progress_bar import ProgressBarSection, ScanProgressBarSection +from cycode.logger import get_logger + +logger = get_logger('Binary Collector') + +BOM_FILE_NAME = 'bom.json' +POM_FILE_NAME = 'pom.xml' + +# the cyclonedx-maven-plugin writes /target/bom.json, and we mirror that exactly +BOM_DIRECTORY_NAME = 'target' + +_SAFE_ARTIFACT_ID = re.compile(r'[^A-Za-z0-9._-]') + +SYNTHETIC_POM_TEMPLATE = """ + + + 4.0.0 + com.cycode.binary + {artifact_id} + 0 + {name} + +""" + + +def build_artifact_root(artifact_path: str) -> str: + """The directory the synthesised documents are presented under, derived from the artifact they describe.""" + relative = artifact_path + if os.path.isabs(artifact_path): + try: + relative = os.path.relpath(artifact_path, os.getcwd()) + except ValueError: + # different drives on Windows; provenance beyond the filename is not recoverable + relative = os.path.basename(artifact_path) + + if relative.startswith(os.pardir): + relative = os.path.basename(artifact_path) + + return relative + + +def build_document_path(artifact_path: str) -> str: + """Where the synthesised BOM is presented, e.g. 'dist/payments.war/target/bom.json'.""" + return get_path_by_os(os.path.join(build_artifact_root(artifact_path), BOM_DIRECTORY_NAME, BOM_FILE_NAME)) + + +def build_manifest_path(artifact_path: str) -> str: + """Where the synthesised manifest is presented, e.g. 'dist/payments.war/pom.xml'.""" + return get_path_by_os(os.path.join(build_artifact_root(artifact_path), POM_FILE_NAME)) + + +def build_synthetic_manifest(artifact_name: str) -> str: + """A minimal pom.xml that gives the BOM the project context the scan service requires. + + Phase 0 established this is not optional. A bom.json that arrives alone is accepted by api/v4/scans/cli -- a + scan id comes back and the scan completes with no warning -- but produces no detections at all. The identical + document beside a pom.xml yields the full set, Log4Shell included. The manifest declares no dependencies of its + own, so every finding still comes from the BOM; it exists purely to make the engine route the document. + """ + artifact_id = _SAFE_ARTIFACT_ID.sub('-', artifact_name) or 'artifact' + return SYNTHETIC_POM_TEMPLATE.format(artifact_id=escape(artifact_id), name=escape(artifact_name)) + + +@dataclass +class BinaryCollectionResult: + documents: list[Document] = field(default_factory=list) + results_by_artifact: dict[str, ExtractionResult] = field(default_factory=dict) + failures: dict[str, str] = field(default_factory=dict) + + @property + def identified_count(self) -> int: + return sum(len(result.components) for result in self.results_by_artifact.values()) + + @property + def low_confidence_count(self) -> int: + """Components named by a manifest attribute only. Counted as identified, but a reader must be able to tell.""" + return sum( + 1 + for result in self.results_by_artifact.values() + for component in result.components + if component.confidence == CONFIDENCE_AMBIGUOUS + ) + + @property + def unidentified_count(self) -> int: + return sum(len(result.unidentified) for result in self.results_by_artifact.values()) + + @property + def resolver_available(self) -> bool: + return all(result.resolver_available for result in self.results_by_artifact.values()) + + @property + def resolver_unavailability_reason(self) -> Optional[str]: + """The reason as of the last artifact. One resolver serves the whole run and its counts accumulate, so an + earlier snapshot would under-report how many digests were affected.""" + reasons = [ + result.resolver_unavailability_reason + for result in self.results_by_artifact.values() + if result.resolver_unavailability_reason + ] + return reasons[-1] if reasons else None + + +def get_extractors(resolver: Optional[DigestResolver] = None) -> list[BinaryExtractor]: + """Registered extractors, in the order they are offered a path. + + .NET, npm and container support arrive as additional entries here rather than as new architecture. + """ + return [JavaArchiveExtractor(resolver=resolver or NullDigestResolver())] + + +def get_resolver(ctx: typer.Context) -> DigestResolver: + """Tier 2 is opt-in: nothing about an artifact leaves the machine unless the user asked for it to.""" + if ctx.obj.get('maven_central'): + return MavenCentralDigestResolver() + + return NullDigestResolver() + + +def find_supported_artifacts(paths: tuple[str, ...]) -> list[str]: + """Every file under the given paths that some extractor claims. Directories are walked.""" + extractors = get_extractors() + artifacts = [] + + for path in paths: + if os.path.isfile(path): + candidates = [path] + else: + candidates = [ + os.path.join(directory, name) for directory, _, names in os.walk(path) for name in sorted(names) + ] + + artifacts.extend( + candidate for candidate in candidates if any(extractor.handles(candidate) for extractor in extractors) + ) + + return artifacts + + +def collect_binary_documents( + ctx: typer.Context, + paths: tuple[str, ...], + stop_on_error: bool = False, + progress_bar_section: Optional['ProgressBarSection'] = ScanProgressBarSection.PREPARE_LOCAL_FILES, +) -> BinaryCollectionResult: + """Extract, identify and synthesise one CycloneDX document per artifact. + + A failure on one artifact does not stop the others unless ``--stop-on-error`` was given: a sweep over an + artifact repository should report what it could read, not abort on the first unreadable file. + + The scan and report flows drive different progress bars, so the section to advance is passed in rather than + assumed. Pass None when the caller has already completed that section -- as the path scan has by the time + --include-binaries runs -- because reopening a finished section breaks the bar. + """ + max_depth = ctx.obj.get('binary_max_depth', consts.BINARY_MAX_DEPTH) + keep_bom = ctx.obj.get('keep_bom', False) + + progress_bar = ctx.obj.get('progress_bar') if progress_bar_section is not None else None + artifacts = find_supported_artifacts(paths) + + logger.debug('Collecting binary artifacts, %s', {'count': len(artifacts), 'max_depth': max_depth}) + if progress_bar and artifacts: + progress_bar.set_section_length(progress_bar_section, len(artifacts)) + + collection = BinaryCollectionResult() + # one resolver for the whole run, so a transport failure on the first artifact is not retried on every other + extractors = get_extractors(get_resolver(ctx)) + + for artifact_path in artifacts: + try: + documents, result = _collect_one(artifact_path, extractors, max_depth, keep_bom) + except Exception as e: + logger.debug('Failed to read an artifact, %s', {'path': artifact_path}, exc_info=e) + collection.failures[artifact_path] = str(e) + if stop_on_error: + raise + else: + collection.documents.extend(documents) + collection.results_by_artifact[artifact_path] = result + + if progress_bar: + progress_bar.update(progress_bar_section) + + return collection + + +def _collect_one( + artifact_path: str, + extractors: list[BinaryExtractor], + max_depth: int, + keep_bom: bool, +) -> tuple[list[Document], ExtractionResult]: + extractor = next(extractor for extractor in extractors if extractor.handles(artifact_path)) + + entries = extractor.extract(artifact_path, max_depth) + result = extractor.identify(entries) + + artifact_name = os.path.basename(artifact_path) + content = cyclonedx_builder.build_bom_json(artifact_name, result) + document_path = build_document_path(artifact_path) + + logger.debug( + 'Synthesised a BOM, %s', + { + 'path': document_path, + 'components': len(result.components), + 'unidentified': len(result.unidentified), + 'archives_opened': result.archives_opened, + }, + ) + + if keep_bom: + _write_bom_beside_artifact(artifact_path, content) + + documents = [ + Document(document_path, content, is_git_diff_format=False, absolute_path=artifact_path), + Document( + build_manifest_path(artifact_path), + build_synthetic_manifest(artifact_name), + is_git_diff_format=False, + absolute_path=artifact_path, + ), + ] + + return documents, result + + +def _write_bom_beside_artifact(artifact_path: str, content: str) -> Optional[str]: + """``--keep-bom``: the synthesised document, on disk, for inspection and audit.""" + output_path = f'{artifact_path}.{BOM_FILE_NAME}' + try: + # O_NOFOLLOW: the tree being scanned is untrusted by this feature's own threat model, and an unpacked + # vendor drop can carry a symlink at this exact path pointing anywhere the user can write + descriptor = os.open(output_path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC | os.O_NOFOLLOW, 0o600) + with os.fdopen(descriptor, 'w', encoding='utf-8') as handle: + handle.write(content) + except OSError as e: + # an unwritable directory is not a reason to fail a scan that has otherwise succeeded + logger.warning('Could not write the BOM, %s', {'path': output_path, 'error': str(e)}) + return None + + logger.debug('Wrote the synthesised BOM, %s', {'path': output_path}) + return output_path diff --git a/cycode/cli/files_collector/binary/cyclonedx_builder.py b/cycode/cli/files_collector/binary/cyclonedx_builder.py new file mode 100644 index 00000000..6eef2d3a --- /dev/null +++ b/cycode/cli/files_collector/binary/cyclonedx_builder.py @@ -0,0 +1,151 @@ +"""Assembly of a CycloneDX 1.4 document from what the extractor identified. + +Hand-written rather than generated: decision 11 rules out a new runtime dependency, and the subset of the +specification a binary-derived BOM needs is small and stable. The output is the one new artifact this feature +produces, and it is the only thing that leaves the machine. + +Every field is deterministic and the ordering is stable, so two scans of the same artifact produce byte-identical +documents. That is what makes the golden-file tests meaningful and what lets a user diff two builds. +""" + +import json +from datetime import datetime, timezone +from typing import Optional + +from cycode import __version__ +from cycode.cli.files_collector.binary.base_extractor import ( + ExtractionResult, + IdentifiedComponent, +) + +CYCLONEDX_BOM_FORMAT = 'CycloneDX' +CYCLONEDX_SPEC_VERSION = '1.4' + +GRAPH_CONTAINMENT = 'containment' +GRAPH_CONTAINMENT_WITH_REAL_EDGES = 'containment+partial' + +SOURCE_PROPERTY = 'cycode:source' +GRAPH_PROPERTY = 'cycode:graph' +COVERAGE_PROPERTY = 'cycode:coverage' +EVIDENCE_PROPERTY = 'cycode:evidence' +CONFIDENCE_PROPERTY = 'cycode:confidence' +PATH_PROPERTY = 'cycode:path' + +BINARY_EXTRACTION_SOURCE = 'binary-extraction' + +_TIMESTAMP_FORMAT = '%Y-%m-%dT%H:%M:%SZ' + + +def _now() -> str: + return datetime.now(timezone.utc).strftime(_TIMESTAMP_FORMAT) + + +def _property(name: str, value: str) -> dict[str, str]: + return {'name': name, 'value': value} + + +def _hashes(component: IdentifiedComponent) -> list[dict[str, str]]: + hashes = [{'alg': 'SHA-1', 'content': component.sha1}] + if component.sha256: + hashes.append({'alg': 'SHA-256', 'content': component.sha256}) + + return hashes + + +def build_component(component: IdentifiedComponent, paths: list[str]) -> dict: + """One CycloneDX component. ``paths`` lists every place in the artifact this coordinate was found.""" + body = { + 'bom-ref': component.purl, + 'type': 'library', + 'name': component.artifact, + 'version': component.version, + 'purl': component.purl, + 'hashes': _hashes(component), + 'properties': [ + _property(EVIDENCE_PROPERTY, component.evidence), + _property(CONFIDENCE_PROPERTY, component.confidence), + _property(PATH_PROPERTY, ', '.join(paths)), + ], + } + + if component.group: + body['group'] = component.group + + return body + + +def _deduplicate(components: list[IdentifiedComponent]) -> tuple[list[IdentifiedComponent], dict[str, list[str]]]: + """Collapse repeats of one coordinate, keeping every path it was found at. + + A deployable that ships the same jar in two places is describing one component, not two, and emitting it twice + would double-count it in the platform. + """ + ordered: list[IdentifiedComponent] = [] + paths: dict[str, list[str]] = {} + + for component in components: + if component.purl not in paths: + ordered.append(component) + paths[component.purl] = [] + + if component.logical_path not in paths[component.purl]: + paths[component.purl].append(component.logical_path) + + return ordered, paths + + +def build_bom( + artifact_name: str, + result: ExtractionResult, + timestamp: Optional[str] = None, +) -> dict: + """Assemble the document handed to the SCA pipeline.""" + components, paths = _deduplicate(result.components) + root_ref = artifact_name + + identified_count = len(components) + total_count = identified_count + len(result.unidentified) + graph_kind = GRAPH_CONTAINMENT_WITH_REAL_EDGES if result.has_real_edges else GRAPH_CONTAINMENT + + return { + 'bomFormat': CYCLONEDX_BOM_FORMAT, + 'specVersion': CYCLONEDX_SPEC_VERSION, + 'version': 1, + 'metadata': { + 'timestamp': timestamp or _now(), + 'tools': [{'vendor': 'Cycode', 'name': 'cycode-cli', 'version': __version__}], + 'component': {'bom-ref': root_ref, 'type': 'application', 'name': artifact_name}, + 'properties': [ + _property(SOURCE_PROPERTY, BINARY_EXTRACTION_SOURCE), + _property(GRAPH_PROPERTY, graph_kind), + _property(COVERAGE_PROPERTY, f'{identified_count}/{total_count}'), + ], + }, + 'components': [build_component(component, paths[component.purl]) for component in components], + 'dependencies': _build_dependencies(root_ref, components, result.dependency_edges), + } + + +def _build_dependencies( + root_ref: str, + components: list[IdentifiedComponent], + edges: dict[str, list[str]], +) -> list[dict]: + """Every ref gets an entry, so a leaf reads as "depends on nothing" rather than as missing data.""" + known_refs = {component.purl for component in components} + + entries = [] + for ref in [root_ref, *sorted(known_refs)]: + depends_on = sorted(target for target in edges.get(ref, []) if target in known_refs) + entries.append({'ref': ref, 'dependsOn': depends_on}) + + return entries + + +def to_json(bom: dict) -> str: + """Serialise with stable formatting. Key order is the insertion order above, which is the readable one.""" + return json.dumps(bom, indent=2, ensure_ascii=False) + + +def build_bom_json(artifact_name: str, result: ExtractionResult, timestamp: Optional[str] = None) -> str: + return to_json(build_bom(artifact_name, result, timestamp)) diff --git a/cycode/cli/files_collector/binary/identifiers/__init__.py b/cycode/cli/files_collector/binary/identifiers/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/cycode/cli/files_collector/binary/identifiers/manifest_mf.py b/cycode/cli/files_collector/binary/identifiers/manifest_mf.py new file mode 100644 index 00000000..bfd8d448 --- /dev/null +++ b/cycode/cli/files_collector/binary/identifiers/manifest_mf.py @@ -0,0 +1,127 @@ +"""Tier 3: attributes from ``META-INF/MANIFEST.MF``. + +A manifest reliably yields a name and a version but rarely a groupId, so every match from here is ambiguous and is +marked as such. We read what the jar declares about itself and stop there; deriving a groupId from a package prefix +would be a guess, and a fabricated coordinate produces a fabricated CVE list. + +What the jar declares is also checked for shape before it becomes a coordinate. Real manifests carry a product name +in ``Implementation-Title`` ("Sun Java System Application Server"), a build banner in ``Implementation-Version`` +("20100905 1938 [3.0.6 (2010-08-24)]") and no vendor id at all, and each of those would otherwise be reported as an +identified component. A purl assembled from them matches nothing in any index, so it is not a low-confidence +answer, it is a wrong one dressed as an answer. Those manifests yield nothing, and the jar is reported unidentified. + +The format wraps lines at 72 bytes and continues them with a leading single space. The wrap is applied to *bytes*, +so a multi-byte character can be split across the boundary: continuation is therefore joined before decoding, which +is the step naive parsers get wrong. +""" + +import re +from dataclasses import dataclass +from typing import Optional + +from cycode.cli.files_collector.binary.identifiers.pom_properties import MavenCoordinates + +_CONTINUATION_PREFIX = b' ' +_ATTRIBUTE_SEPARATOR = b': ' + +IMPLEMENTATION_TITLE = 'Implementation-Title' +IMPLEMENTATION_VERSION = 'Implementation-Version' +IMPLEMENTATION_VENDOR_ID = 'Implementation-Vendor-Id' +BUNDLE_SYMBOLIC_NAME = 'Bundle-SymbolicName' +BUNDLE_VERSION = 'Bundle-Version' +AUTOMATIC_MODULE_NAME = 'Automatic-Module-Name' + +# the characters Maven accepts in a groupId or artifactId; a product name with spaces or "::" is not a coordinate +_COORDINATE_PATTERN = re.compile(r'^[A-Za-z0-9_][A-Za-z0-9._-]*$') +# a version starts with a digit and carries no whitespace: 2.14.1, 1.0M10, 9.4.44.v20210927, 3.0.0-SNAPSHOT +_VERSION_PATTERN = re.compile(r'^[0-9][A-Za-z0-9._+-]*$') + + +@dataclass(frozen=True) +class ManifestIdentity: + coordinates: MavenCoordinates + source_attribute: str + + +def parse_manifest(payload: bytes) -> dict[str, str]: + """Return the attributes of the manifest's main section. + + Per-entry sections follow the first blank line and describe individual files rather than the artifact, so they + are not read. + """ + joined_lines: list[bytes] = [] + for raw_line in payload.split(b'\n'): + line = raw_line[:-1] if raw_line.endswith(b'\r') else raw_line + + if not line.strip(): + break # end of the main section + + if line.startswith(_CONTINUATION_PREFIX) and joined_lines: + joined_lines[-1] += line[1:] + else: + joined_lines.append(line) + + attributes = {} + for line in joined_lines: + name, separator, value = line.partition(_ATTRIBUTE_SEPARATOR) + if not separator: + continue + + try: + attributes[name.decode('utf-8').strip()] = value.decode('utf-8').strip() + except UnicodeDecodeError: + continue + + return attributes + + +def _without_directives(symbolic_name: str) -> str: + """``com.acme.thing;singleton:=true`` is the bundle ``com.acme.thing``.""" + return symbolic_name.split(';')[0].strip() + + +def is_coordinate_shaped(value: Optional[str]) -> bool: + return bool(value) and _COORDINATE_PATTERN.match(value) is not None + + +def is_version_shaped(value: Optional[str]) -> bool: + return bool(value) and _VERSION_PATTERN.match(value) is not None + + +def identify(payload: bytes) -> Optional[ManifestIdentity]: + """Best available coordinates, or None when the manifest says nothing usable. + + ``Implementation-Vendor-Id`` is the only source of a group because it is a declared group id, not an inference. + Without it there is no coordinate: a purl with no namespace cannot be matched to anything, so emitting one would + only inflate the identified count. The first attribute pair that is shaped like a coordinate wins; a pair that is + not falls through to the next rather than disqualifying the manifest. + """ + attributes = parse_manifest(payload) + group = attributes.get(IMPLEMENTATION_VENDOR_ID, '').strip() + if not is_coordinate_shaped(group): + return None + + candidates = ( + (IMPLEMENTATION_TITLE, attributes.get(IMPLEMENTATION_TITLE), attributes.get(IMPLEMENTATION_VERSION)), + ( + BUNDLE_SYMBOLIC_NAME, + _without_directives(attributes.get(BUNDLE_SYMBOLIC_NAME, '')), + attributes.get(BUNDLE_VERSION) or attributes.get(IMPLEMENTATION_VERSION), + ), + ( + AUTOMATIC_MODULE_NAME, + attributes.get(AUTOMATIC_MODULE_NAME), + attributes.get(IMPLEMENTATION_VERSION) or attributes.get(BUNDLE_VERSION), + ), + ) + + for source_attribute, artifact, version in candidates: + artifact = (artifact or '').strip() + version = (version or '').strip() + if is_coordinate_shaped(artifact) and is_version_shaped(version): + return ManifestIdentity( + coordinates=MavenCoordinates(group=group, artifact=artifact, version=version), + source_attribute=source_attribute, + ) + + return None diff --git a/cycode/cli/files_collector/binary/identifiers/pom_properties.py b/cycode/cli/files_collector/binary/identifiers/pom_properties.py new file mode 100644 index 00000000..40ab3b49 --- /dev/null +++ b/cycode/cli/files_collector/binary/identifiers/pom_properties.py @@ -0,0 +1,68 @@ +"""Tier 1: exact coordinates from ``META-INF/maven///pom.properties``. + +Written by the build that produced the jar, so this is authoritative rather than inferred. A shaded or uber jar +carries one of these per aggregated project, which is why callers get a list rather than a single answer. +""" + +from dataclasses import dataclass +from typing import Optional + +_COMMENT_PREFIXES = ('#', '!') +_KEY_VALUE_SEPARATORS = ('=', ':') + +_GROUP_KEY = 'groupId' +_ARTIFACT_KEY = 'artifactId' +_VERSION_KEY = 'version' + + +@dataclass(frozen=True) +class MavenCoordinates: + group: str + artifact: str + version: str + + +def parse_properties(payload: bytes) -> dict[str, str]: + """Parse the subset of the Java properties format that Maven actually emits here. + + Maven writes a fixed four-line file, so the exotic corners of the format (escaped separators, multi-line values) + do not arise. Anything unparseable is skipped rather than raising: a malformed properties file in one jar must + not fail the scan of an entire deployable. + """ + try: + text = payload.decode('utf-8-sig') + except UnicodeDecodeError: + return {} + + properties = {} + for raw_line in text.splitlines(): + line = raw_line.strip() + if not line or line.startswith(_COMMENT_PREFIXES): + continue + + separator_index = min( + (line.find(separator) for separator in _KEY_VALUE_SEPARATORS if line.find(separator) != -1), + default=-1, + ) + if separator_index <= 0: + continue + + key = line[:separator_index].strip() + value = line[separator_index + 1 :].strip() + if key and value: + properties[key] = value + + return properties + + +def identify(payload: bytes) -> Optional[MavenCoordinates]: + """Coordinates from one pom.properties, or None when it does not carry a complete set.""" + properties = parse_properties(payload) + + group = properties.get(_GROUP_KEY) + artifact = properties.get(_ARTIFACT_KEY) + version = properties.get(_VERSION_KEY) + if not group or not artifact or not version: + return None + + return MavenCoordinates(group=group, artifact=artifact, version=version) diff --git a/cycode/cli/files_collector/binary/identifiers/pom_xml.py b/cycode/cli/files_collector/binary/identifiers/pom_xml.py new file mode 100644 index 00000000..5161468f --- /dev/null +++ b/cycode/cli/files_collector/binary/identifiers/pom_xml.py @@ -0,0 +1,117 @@ +"""Real dependency edges from an embedded ``META-INF/maven///pom.xml``. + +Containment tells us a jar sits inside a war. A pom tells us which component actually depends on which, and that is +a materially better graph. These are untrusted XML documents lifted out of a customer artifact, so parsing them is +the one place in this feature where a parser feature becomes an attack. + +``xml.etree`` expands internal entities, which makes a billion-laughs document a live denial of service on the +Python versions this project supports, and ``defusedxml`` is not a dependency we may add. A document type +declaration is therefore refused outright before the parser ever sees the bytes. A Maven pom never legitimately +carries one, so this costs nothing and closes entity expansion, quadratic blowup and external entity resolution +together. +""" + +import logging +import re +import xml.etree.ElementTree as ET +from dataclasses import dataclass +from typing import Optional + +logger = logging.getLogger(__name__) + +# matches a document type or entity declaration however it is spaced or cased +_DECLARATION_PATTERN = re.compile(r' str: + """Version is frequently a property reference we cannot resolve, so edges are matched on group:artifact.""" + return f'{self.group}:{self.artifact}' + + +def _local_name(tag: str) -> str: + """``{http://maven.apache.org/POM/4.0.0}dependency`` is a ``dependency``.""" + return tag.rsplit('}', 1)[-1] + + +def _child_text(element: ET.Element, name: str) -> Optional[str]: + for child in element: + if _local_name(child.tag) == name: + return (child.text or '').strip() or None + + return None + + +def parse_xml(payload: bytes) -> ET.Element: + """Parse an untrusted XML document, refusing anything with a document type declaration. + + The guard runs on *decoded text*, not on the raw bytes. Scanning bytes for `` list[MavenDependency]: + """Direct dependencies declared by a pom. + + Only ``project/dependencies`` is read. Entries under ``dependencyManagement`` declare versions for modules that + may never be depended on, and treating them as edges would invent relationships the build never made. + """ + try: + root = parse_xml(payload) + except UnsafeXmlError as e: + logger.debug('Skipping an embedded pom: %s', e) + return [] + + dependencies = [] + for child in root: + if _local_name(child.tag) != _DEPENDENCIES_TAG: + continue + + for element in child: + if _local_name(element.tag) != _DEPENDENCY_TAG: + continue + + scope = (_child_text(element, 'scope') or '').lower() + if scope in _NON_SHIPPED_SCOPES: + continue + + group = _child_text(element, 'groupId') + artifact = _child_text(element, 'artifactId') + if not group or not artifact: + continue + + dependencies.append( + MavenDependency(group=group, artifact=artifact, version=_child_text(element, 'version')) + ) + + return dependencies diff --git a/cycode/cli/files_collector/binary/java_extractor.py b/cycode/cli/files_collector/binary/java_extractor.py new file mode 100644 index 00000000..177f204e --- /dev/null +++ b/cycode/cli/files_collector/binary/java_extractor.py @@ -0,0 +1,490 @@ +"""Extraction of Java deployables: JAR, WAR, EAR and Spring Boot fat JARs, including nested archives. + +Everything here is a pure function over bytes. No network, no auth, no Cycode API, no temp files. +""" + +import hashlib +import logging +import os +from collections import defaultdict +from typing import Optional + +from cycode.cli import consts +from cycode.cli.exceptions.custom_exceptions import BinaryExtractionError +from cycode.cli.files_collector.binary.base_extractor import ( + CONFIDENCE_AMBIGUOUS, + CONFIDENCE_EXACT, + EVIDENCE_DIGEST, + EVIDENCE_MANIFEST, + EVIDENCE_POM_PROPERTIES, + ArchiveEntry, + BinaryExtractor, + ExtractionResult, + IdentifiedComponent, + UnidentifiedArtifact, + join_logical_path, +) +from cycode.cli.files_collector.binary.identifiers import manifest_mf, pom_properties, pom_xml +from cycode.cli.files_collector.binary.identifiers.pom_properties import MavenCoordinates +from cycode.cli.files_collector.binary.resolver import DigestResolver, NullDigestResolver +from cycode.cli.files_collector.binary.safe_zip import ArchiveBudget, ArchiveLimits, SafeZip, SafeZipEntry + +logger = logging.getLogger(__name__) + +JAVA_ARCHIVE_EXTENSIONS = ('.jar', '.war', '.ear') + +# where a Java deployable keeps the libraries it ships with +LIBRARY_DIRECTORIES = ( + 'WEB-INF/lib/', # WAR + 'BOOT-INF/lib/', # Spring Boot fat JAR + 'APP-INF/lib/', # legacy WebLogic-style EAR module + 'lib/', # plain JAR and EAR conventions +) + +MANIFEST_ENTRY_NAME = 'META-INF/MANIFEST.MF' +MANIFEST_FILE_NAME = 'MANIFEST.MF' +POM_PROPERTIES_FILE_NAME = 'pom.properties' +POM_XML_FILE_NAME = 'pom.xml' +_MAVEN_METADATA_FILE_NAMES = (POM_PROPERTIES_FILE_NAME, POM_XML_FILE_NAME) +_MAVEN_METADATA_PATH_DEPTH = 5 # META-INF/maven/// + +_READ_CHUNK_SIZE_IN_BYTES = 64 * 1024 +_MAVEN_PURL_PREFIX = 'pkg:maven/' + + +def is_java_archive_name(name: str) -> bool: + return name.lower().endswith(JAVA_ARCHIVE_EXTENSIONS) + + +def is_library_entry(name: str, container_extension: str) -> bool: + """True when an entry is a shipped library rather than an incidental file that happens to be a zip.""" + if not is_java_archive_name(name): + return False + + if name.startswith(LIBRARY_DIRECTORIES): + return True + + # EAR modules are not confined to a lib directory; a WAR or JAR sits wherever application.xml points + return container_extension == '.ear' + + +def is_metadata_entry(name: str) -> bool: + """True for the embedded files identification reads in phase 2. Everything else is skipped unread.""" + if name.upper() == MANIFEST_ENTRY_NAME: + return True + + parts = name.split('/') + return ( + len(parts) == _MAVEN_METADATA_PATH_DEPTH + and parts[0] == 'META-INF' + and parts[1] == 'maven' + and parts[4] in _MAVEN_METADATA_FILE_NAMES + ) + + +def compute_file_digest(path: str) -> str: + """SHA-1 of a file on disk, streamed. Used to seed the cycle guard with the artifact we were pointed at.""" + # SHA-1 is identification, not security: it is the digest Maven Central and every artifact index key on, so + # it is the only algorithm a coordinate lookup can use. usedforsecurity=False declares that intent to the + # runtime and to ruff's S324. SHA-256 is emitted alongside it in the BOM for anyone who wants a strong digest. + digest = hashlib.sha1(usedforsecurity=False) + with open(path, 'rb') as handle: + while True: + chunk = handle.read(_READ_CHUNK_SIZE_IN_BYTES) + if not chunk: + break + + digest.update(chunk) + + return digest.hexdigest() + + +class JavaArchiveExtractor(BinaryExtractor): + """Reads a Java archive and everything shipped inside it, down to a bounded depth.""" + + def __init__(self, limits: Optional[ArchiveLimits] = None, resolver: Optional[DigestResolver] = None) -> None: + self._limits = limits or ArchiveLimits() + self._resolver = resolver or NullDigestResolver() + + def handles(self, path: str) -> bool: + return is_java_archive_name(path) + + def extract(self, path: str, max_depth: int = consts.BINARY_MAX_DEPTH) -> list[ArchiveEntry]: + """Walk the artifact, returning the archives it ships and the metadata needed to identify them. + + Class files and resources are never returned: they are not components, and carrying them would mean holding + an entire deployable in memory to no purpose. + """ + if not os.path.isfile(path): + raise BinaryExtractionError(f'{path!r} is not a file.') + + root_name = os.path.basename(path) + root_sha1 = compute_file_digest(path) + root_entry = ArchiveEntry( + logical_path=root_name, + name=root_name, + size=os.path.getsize(path), + sha1=root_sha1, + depth=0, + parent=None, + is_archive=True, + was_opened=True, + ) + + entries = [root_entry] + budget = ArchiveBudget(self._limits) + visited_digests = {root_sha1} + + with SafeZip.open(path, budget=budget, source_name=root_name) as archive: + self._walk( + archive=archive, + container_logical_path=root_name, + container_extension=_extension_of(root_name), + depth=1, + max_depth=max_depth, + budget=budget, + visited_digests=visited_digests, + entries=entries, + ) + + return entries + + def identify(self, entries: list[ArchiveEntry]) -> ExtractionResult: + """Run the identification ladder over what extraction found. + + Tier 1 reads embedded Maven metadata, tier 2 resolves the digests tier 1 could not place, tier 3 falls back + to manifest attributes, and whatever survives all three is reported as unidentified rather than guessed at. + The first tier to produce coordinates wins and records itself in ``evidence``. + """ + metadata_by_container = defaultdict(list) + for entry in entries: + if not entry.is_archive and entry.parent is not None: + metadata_by_container[entry.parent].append(entry) + + archives = [entry for entry in entries if entry.is_archive] + if not archives: + return ExtractionResult(resolver_available=self._resolver.available) + + root = archives[0] + shipped = [entry for entry in archives if entry.depth > 0] + + # a deployable is described by the BOM's metadata component and is never one of its own components. A jar + # that ships nothing is a library, and a library is precisely the thing being assessed: it is a candidate + # like any shipped jar, and failing to name it is a coverage gap that must be reported, not dropped. A + # scan that cannot name the one archive it was pointed at and still claims full coverage is lying. + candidates = shipped or ([root] if _is_library_root(root) else []) + + components, unidentified = self._run_identification_ladder(candidates, metadata_by_container) + + edges, has_real_edges = self._build_dependency_graph(root, candidates, components, metadata_by_container) + + return ExtractionResult( + components=components, + unidentified=unidentified, + archives_opened=sum(1 for entry in entries if entry.was_opened), + # expressed as a count of archives on the deepest chain, so it compares directly against --max-depth + max_depth_reached=max((entry.depth for entry in entries if entry.is_archive), default=-1) + 1, + resolver_available=self._resolver.available, + resolver_unavailability_reason=None if self._resolver.available else self._resolver.unavailability_reason, + dependency_edges=edges, + has_real_edges=has_real_edges, + ) + + def _run_identification_ladder( + self, + candidates: list[ArchiveEntry], + metadata_by_container: dict[str, list[ArchiveEntry]], + ) -> tuple[list[IdentifiedComponent], list[UnidentifiedArtifact]]: + components: list[IdentifiedComponent] = [] + + # tier 1 + needs_resolution: list[ArchiveEntry] = [] + for archive in candidates: + tier_one = self._identify_from_pom_properties(archive, metadata_by_container) + if tier_one: + components.extend(tier_one) + else: + needs_resolution.append(archive) + + # tier 2, as a single batch: every digest tier 1 could not place. The resolver is always asked; whether it + # can still attempt a lookup after an earlier failure is its own state to keep, and gating on + # ``available`` here would silently skip every artifact after the first one that hit trouble + resolved: dict[str, str] = {} + if needs_resolution: + resolved = self._resolver.resolve([archive.sha1 for archive in needs_resolution]) + + needs_manifest: list[ArchiveEntry] = [] + for archive in needs_resolution: + coordinates = parse_maven_purl(resolved.get(archive.sha1)) + if coordinates: + components.append(_component(archive, coordinates, EVIDENCE_DIGEST, CONFIDENCE_EXACT)) + else: + needs_manifest.append(archive) + + # tier 3, then tier 4 + unidentified: list[UnidentifiedArtifact] = [] + for archive in needs_manifest: + tier_three = self._identify_from_manifest(archive, metadata_by_container) + if tier_three: + components.append(tier_three) + else: + unidentified.append( + UnidentifiedArtifact(logical_path=archive.logical_path, sha1=archive.sha1, size=archive.size) + ) + + return components, unidentified + + def _identify_from_pom_properties( + self, + archive: ArchiveEntry, + metadata_by_container: dict[str, list[ArchiveEntry]], + ) -> list[IdentifiedComponent]: + """Tier 1. A shaded jar aggregates several projects and carries a pom.properties for each, so all are kept.""" + components = [] + for entry in metadata_by_container.get(archive.logical_path, []): + if entry.name != POM_PROPERTIES_FILE_NAME or entry.payload is None: + continue + + coordinates = pom_properties.identify(entry.payload) + if coordinates: + components.append(_component(archive, coordinates, EVIDENCE_POM_PROPERTIES, CONFIDENCE_EXACT)) + + return components + + def _identify_from_manifest( + self, + archive: ArchiveEntry, + metadata_by_container: dict[str, list[ArchiveEntry]], + ) -> Optional[IdentifiedComponent]: + """Tier 3. Always ambiguous: a manifest names the jar but rarely its groupId.""" + for entry in metadata_by_container.get(archive.logical_path, []): + if entry.name.upper() != MANIFEST_FILE_NAME or entry.payload is None: + continue + + identity = manifest_mf.identify(entry.payload) + if identity: + return _component(archive, identity.coordinates, EVIDENCE_MANIFEST, CONFIDENCE_AMBIGUOUS) + + return None + + def _build_dependency_graph( + self, + root: ArchiveEntry, + candidates: list[ArchiveEntry], + components: list[IdentifiedComponent], + metadata_by_container: dict[str, list[ArchiveEntry]], + ) -> tuple[dict[str, list[str]], bool]: + """Containment as the base tree, with real edges from embedded poms overlaid on top. + + Where the two disagree the real edge wins and the containment edge is dropped, because a jar that is + genuinely a transitive dependency of another is not a direct child of the application. + """ + refs_by_path: dict[str, list[str]] = defaultdict(list) + for component in components: + refs_by_path[component.logical_path].append(component.purl) + + archive_by_path = {archive.logical_path: archive for archive in candidates} + ref_by_coordinate = {f'{component.group}:{component.artifact}': component.purl for component in components} + root_ref = root.name + + containment: dict[str, set] = defaultdict(set) + for component in components: + container = self._container_ref(component.parent, root, refs_by_path, archive_by_path, root_ref) + if container != component.purl: + containment[container].add(component.purl) + + real: dict[str, set] = defaultdict(set) + for archive in candidates: + source_refs = refs_by_path.get(archive.logical_path) + if not source_refs: + continue + + for dependency in self._declared_dependencies(archive, metadata_by_container): + target_ref = ref_by_coordinate.get(dependency.coordinate_key) + if target_ref and target_ref != source_refs[0]: + real[source_refs[0]].add(target_ref) + + claimed = {target for targets in real.values() for target in targets} + for targets in containment.values(): + targets -= claimed + + merged: dict[str, list[str]] = {} + for ref in set(containment) | set(real): + merged[ref] = sorted(containment.get(ref, set()) | real.get(ref, set())) + + return merged, bool(claimed) + + def _declared_dependencies( + self, + archive: ArchiveEntry, + metadata_by_container: dict[str, list[ArchiveEntry]], + ) -> list[pom_xml.MavenDependency]: + dependencies = [] + for entry in metadata_by_container.get(archive.logical_path, []): + if entry.name == POM_XML_FILE_NAME and entry.payload is not None: + dependencies.extend(pom_xml.parse_dependencies(entry.payload)) + + return dependencies + + def _container_ref( + self, + parent_path: Optional[str], + root: ArchiveEntry, + refs_by_path: dict[str, list[str]], + archive_by_path: dict[str, ArchiveEntry], + root_ref: str, + ) -> str: + """The nearest identified ancestor, falling back to the artifact itself. + + An unidentified intermediate jar must not break the chain: its children still shipped inside the artifact, + so they attach to the nearest thing we can name. + """ + current = parent_path + while current and current != root.logical_path: + refs = refs_by_path.get(current) + if refs: + return refs[0] + + ancestor = archive_by_path.get(current) + current = ancestor.parent if ancestor else None + + return root_ref + + def _walk( + self, + archive: SafeZip, + container_logical_path: str, + container_extension: str, + depth: int, + max_depth: int, + budget: ArchiveBudget, + visited_digests: set[str], + entries: list[ArchiveEntry], + ) -> None: + for entry in archive.entries(): + if is_library_entry(entry.name, container_extension): + self._handle_nested_archive( + archive=archive, + entry=entry, + container_logical_path=container_logical_path, + depth=depth, + max_depth=max_depth, + budget=budget, + visited_digests=visited_digests, + entries=entries, + ) + elif is_metadata_entry(entry.name): + content = archive.read(entry) + entries.append( + ArchiveEntry( + logical_path=join_logical_path(container_logical_path, entry.name), + name=os.path.basename(entry.name), + size=content.size, + sha1=content.sha1, + sha256=content.sha256, + depth=depth, + parent=container_logical_path, + payload=content.data, + ) + ) + + def _handle_nested_archive( + self, + archive: SafeZip, + entry: SafeZipEntry, + container_logical_path: str, + depth: int, + max_depth: int, + budget: ArchiveBudget, + visited_digests: set[str], + entries: list[ArchiveEntry], + ) -> None: + content = archive.read(entry) + logical_path = join_logical_path(container_logical_path, entry.name) + + if depth >= max_depth: + logger.debug('Not opening %s: depth limit of %s reached', logical_path, max_depth) + should_open = False + elif content.sha1 in visited_digests: + logger.debug('Not opening %s: an archive with the same digest was already opened', logical_path) + should_open = False + else: + should_open = True + + # the bytes are dropped once we have recursed: a component is identified by its digest and its metadata, + # never by keeping the whole jar around + entries.append( + ArchiveEntry( + logical_path=logical_path, + name=os.path.basename(entry.name), + size=content.size, + sha1=content.sha1, + sha256=content.sha256, + depth=depth, + parent=container_logical_path, + is_archive=True, + was_opened=should_open, + ) + ) + + if not should_open: + return + + visited_digests.add(content.sha1) + + with SafeZip.open(content.data, budget=budget, source_name=entry.name) as nested: + self._walk( + archive=nested, + container_logical_path=logical_path, + container_extension=_extension_of(entry.name), + depth=depth + 1, + max_depth=max_depth, + budget=budget, + visited_digests=visited_digests, + entries=entries, + ) + + +def _is_library_root(root: ArchiveEntry) -> bool: + """A ``.jar`` that ships no libraries is a library. A WAR or EAR is an application, whatever it contains.""" + return _extension_of(root.name) == '.jar' + + +def parse_maven_purl(purl: Optional[str]) -> Optional[MavenCoordinates]: + """Read back a ``pkg:maven`` purl, as tier 2 returns them. The namespace is optional.""" + if not purl or not purl.startswith(_MAVEN_PURL_PREFIX): + return None + + remainder = purl[len(_MAVEN_PURL_PREFIX) :].split('?')[0].split('#')[0] + coordinates, separator, version = remainder.rpartition('@') + if not separator or not version or not coordinates: + return None + + group, _, artifact = coordinates.rpartition('/') + if not artifact: + return None + + return MavenCoordinates(group=group, artifact=artifact, version=version) + + +def _component( + archive: ArchiveEntry, + coordinates: MavenCoordinates, + evidence: str, + confidence: str, +) -> IdentifiedComponent: + return IdentifiedComponent( + group=coordinates.group, + artifact=coordinates.artifact, + version=coordinates.version, + sha1=archive.sha1, + sha256=archive.sha256, + logical_path=archive.logical_path, + parent=archive.parent, + evidence=evidence, + confidence=confidence, + ) + + +def _extension_of(name: str) -> str: + return os.path.splitext(name)[1].lower() diff --git a/cycode/cli/files_collector/binary/maven_central.py b/cycode/cli/files_collector/binary/maven_central.py new file mode 100644 index 00000000..41931493 --- /dev/null +++ b/cycode/cli/files_collector/binary/maven_central.py @@ -0,0 +1,147 @@ +"""Tier 2 against Maven Central: an unidentified archive looked up by the SHA-1 of its bytes. + +Opt-in, because it is the one step in this feature that sends anything about the artifact off the machine. What +leaves is a digest, never the archive, but a digest still reveals which jars a customer ships, and plenty of build +environments have no egress to a third party at all. The Cycode backend index replaces this behind the same seam +when it exists; nothing that calls a ``DigestResolver`` has to change. + +A hit is exact: the bytes on disk are the bytes Maven Central published under that coordinate. When one digest maps +to several coordinates -- a relocated artifact republished byte-for-byte under a new group -- the index's own +ranking is taken, and the alternatives are logged rather than guessed between. +""" + +import logging +from typing import Optional +from urllib.parse import quote + +import requests + +from cycode.cli import consts +from cycode.cli.files_collector.binary.resolver import DigestResolver +from cycode.cyclient.cycode_client_base import get_http_session +from cycode.cyclient.headers import get_cli_user_agent + +logger = logging.getLogger(__name__) + +_SHA1_FIELD = '1' +_ATTEMPTS_PER_DIGEST = 2 # the index is intermittently slow; one retry, never a storm + + +def _purl_of(group: str, artifact: str, version: str) -> str: + return f'pkg:maven/{group}/{artifact}@{version}' + + +def _coordinates_from(document: dict) -> Optional[tuple[str, str, str]]: + group, artifact, version = document.get('g'), document.get('a'), document.get('v') + if not group or not artifact or not version: + return None + + return str(group), str(artifact), str(version) + + +class MavenCentralDigestResolver(DigestResolver): + """One request per digest: the search API does not return the digest it matched, so a batched query would + come back as an unordered pile of coordinates with no way to hand each one back to its archive. + + search.maven.org answers a hit or a miss in well under a second but is intermittently slow, so a timeout is + retried once and then counted against that digest while the run carries on. A connection-level failure - DNS, + refused, TLS - will not fix itself within the run, so it stops the run and every remaining digest counts as + failed. Any failed digest at all makes the resolver report itself unavailable: the results are partial for + those digests, and the warning says how many and why. + """ + + def __init__( + self, + session: Optional[requests.Session] = None, + search_url: str = consts.BINARY_MAVEN_CENTRAL_SEARCH_URL, + timeout_in_seconds: float = consts.BINARY_DIGEST_LOOKUP_TIMEOUT_IN_SECONDS, + ) -> None: + self._session = session + self._search_url = search_url + self._timeout = timeout_in_seconds + self._asked = 0 + self._failed = 0 + self._last_error: Optional[str] = None + self._aborted = False + + @property + def available(self) -> bool: + return self._failed == 0 + + @property + def unavailability_reason(self) -> str: + return ( + f'Maven Central lookup failed for {self._failed} of {self._asked} digests ({self._last_error}); ' + f'those were not resolved.' + ) + + def resolve(self, digests: list[str]) -> dict[str, str]: + resolved: dict[str, str] = {} + for index, digest in enumerate(digests): + if self._aborted: + remaining = len(digests) - index + self._asked += remaining + self._failed += remaining + break + + self._asked += 1 + purl = self._lookup(digest.lower()) + if purl: + resolved[digest] = purl + + logger.debug( + 'Resolved digests on Maven Central, %s', + {'asked': len(digests), 'resolved': len(resolved), 'failed_so_far': self._failed}, + ) + return resolved + + def _lookup(self, digest: str) -> Optional[str]: + query = quote(f'{_SHA1_FIELD}:"{digest}"', safe='') + url = f'{self._search_url}?q={query}&rows=5&wt=json' + + body = None + for attempt in range(_ATTEMPTS_PER_DIGEST): + try: + response = self._get_session().get( + url, headers={'User-Agent': get_cli_user_agent()}, timeout=self._timeout + ) + response.raise_for_status() + body = response.json() + break + except requests.Timeout as e: + self._last_error = str(e) or e.__class__.__name__ + if attempt + 1 < _ATTEMPTS_PER_DIGEST: + logger.debug('Maven Central lookup timed out, retrying once, %s', {'digest': digest}) + except requests.ConnectionError as e: + self._last_error = str(e) or e.__class__.__name__ + self._aborted = True + break + except (requests.RequestException, ValueError) as e: + # ValueError covers a non-JSON body, which is what a captive portal or a proxy error page returns + self._last_error = str(e) or e.__class__.__name__ + break + + if body is None: + self._failed += 1 + logger.warning('Maven Central lookup failed, %s', {'digest': digest, 'error': self._last_error}) + return None + + section = body.get('response') if isinstance(body, dict) else None + documents = section.get('docs', []) if isinstance(section, dict) else [] + candidates = [purl for purl in (_coordinates_from(d) for d in documents if isinstance(d, dict)) if purl] + if not candidates: + return None + + if len(candidates) > 1: + logger.debug( + 'A digest is published under several coordinates; taking the first, %s', + {'digest': digest, 'coordinates': [_purl_of(*c) for c in candidates]}, + ) + + return _purl_of(*candidates[0]) + + def _get_session(self) -> requests.Session: + if self._session is None: + self._session = get_http_session() + + return self._session diff --git a/cycode/cli/files_collector/binary/resolver.py b/cycode/cli/files_collector/binary/resolver.py new file mode 100644 index 00000000..a259bb74 --- /dev/null +++ b/cycode/cli/files_collector/binary/resolver.py @@ -0,0 +1,44 @@ +"""Tier 2: resolving a digest to a coordinate. + +The Cycode index this is designed for does not exist yet, so the default is a no-op implementation. That is +deliberate rather than incomplete: ``NullDigestResolver`` reporting ``available = False`` exercises the exact +degradation path the product promises - the warning, the coverage counters, the partial marker - from day one. The +path is therefore tested and proven in production before that index ever arrives, instead of being written blind +alongside it. ``MavenCentralDigestResolver`` is the opt-in implementation that exists today. +""" + +from abc import ABC, abstractmethod + + +class DigestResolver(ABC): + @abstractmethod + def resolve(self, digests: list[str]) -> dict[str, str]: + """Map sha1 to purl. Missing keys mean unresolved. Never raises.""" + + @property + @abstractmethod + def available(self) -> bool: + """False when resolution could not be attempted or did not complete, which makes the results partial.""" + + @property + def unavailability_reason(self) -> str: + """One sentence for the degradation warning, read only when ``available`` is False.""" + return 'Digest lookup, which could identify the rest, is not available.' + + +class NullDigestResolver(DigestResolver): + """The default. The Cycode digest index lands behind this seam when the backend endpoint exists.""" + + def resolve(self, digests: list[str]) -> dict[str, str]: + return {} + + @property + def available(self) -> bool: + return False + + @property + def unavailability_reason(self) -> str: + return ( + 'Digest lookup, which could identify the rest, is not available in this release; ' + '--maven-central looks unidentified archives up by SHA-1 on search.maven.org.' + ) diff --git a/cycode/cli/files_collector/binary/safe_zip.py b/cycode/cli/files_collector/binary/safe_zip.py new file mode 100644 index 00000000..81717aa3 --- /dev/null +++ b/cycode/cli/files_collector/binary/safe_zip.py @@ -0,0 +1,298 @@ +"""Hardened reader for untrusted zip-based archives. + +Nothing here ever writes to disk. Entries are streamed into memory, capped as they are read, and nested archives +are reopened from the bytes already in hand. ``ZipFile.extract`` and ``ZipFile.extractall`` are never called: their +path handling is not ours to trust, and by never materialising a path we make zip slip unreachable rather than +merely defended against. +""" + +import hashlib +import io +import os +import re +import zipfile +import zlib +from collections.abc import Iterator +from dataclasses import dataclass, field +from types import TracebackType +from typing import IO, Optional, Union + +from cycode.cli import consts +from cycode.cli.exceptions.custom_exceptions import ( + ArchiveCompressionRatioLimitError, + ArchiveEntryCountLimitError, + ArchiveEntrySizeLimitError, + ArchiveTotalSizeLimitError, + MalformedArchiveError, + UnsafeArchiveEntryError, +) + +_READ_CHUNK_SIZE_IN_BYTES = 64 * 1024 + +_DRIVE_LETTER_PATTERN = re.compile(r'^[A-Za-z]:') + +_WINDOWS_RESERVED_NAMES = frozenset( + {'CON', 'PRN', 'AUX', 'NUL'} + | {f'COM{ordinal}' for ordinal in range(1, 10)} + | {f'LPT{ordinal}' for ordinal in range(1, 10)} +) + +# unix file-type bits, carried in the high 16 bits of ZipInfo.external_attr +_UNIX_FILE_TYPE_MASK = 0o170000 +_UNIX_REGULAR_FILE = 0o100000 + +# archives written by MS-DOS tooling record no unix mode at all, so a zero file type means "no opinion" +_UNIX_FILE_TYPE_UNSET = 0 + +# raised from inside zipfile when the bytes it was handed are not the archive they claim to be +_CORRUPT_ARCHIVE_ERRORS = (zipfile.BadZipFile, zlib.error, EOFError, OSError, ValueError) + + +@dataclass(frozen=True) +class ArchiveLimits: + """Resource ceilings applied while reading an archive. Injectable so tests can prove a control cheaply.""" + + max_entry_count: int = consts.BINARY_MAX_ENTRY_COUNT + max_entry_size_in_bytes: int = consts.BINARY_MAX_ENTRY_SIZE_IN_BYTES + max_total_size_in_bytes: int = consts.BINARY_MAX_TOTAL_SIZE_IN_BYTES + max_compression_ratio: int = consts.BINARY_MAX_COMPRESSION_RATIO + compression_ratio_floor_in_bytes: int = consts.BINARY_COMPRESSION_RATIO_FLOOR_IN_BYTES + + +DEFAULT_ARCHIVE_LIMITS = ArchiveLimits() + + +class ArchiveBudget: + """Uncompressed-byte budget shared by an archive and every archive nested inside it. + + Sharing matters: a nest of archives that are each individually modest can still sum past the total cap, and a + per-archive counter would never notice. + """ + + def __init__(self, limits: ArchiveLimits = DEFAULT_ARCHIVE_LIMITS) -> None: + self.limits = limits + self.consumed_bytes = 0 + + def consume(self, count: int) -> None: + self.consumed_bytes += count + if self.consumed_bytes > self.limits.max_total_size_in_bytes: + raise ArchiveTotalSizeLimitError( + f'Archive expands past the total uncompressed limit of {self.limits.max_total_size_in_bytes} bytes.' + ) + + +@dataclass(frozen=True) +class SafeZipEntry: + """A regular-file entry that has already passed name validation.""" + + name: str + size: int # uncompressed size as declared by the archive, which may be a lie + compress_size: int + # the central-directory record this entry came from. Reads go through it rather than through the name, + # because ZipFile resolves a name to whichever duplicate happens to be last, which lets a crafted archive + # hide a vulnerable jar behind a patched one carrying the same name. + info: Optional[zipfile.ZipInfo] = field(default=None, repr=False, compare=False) + + +@dataclass(frozen=True) +class EntryContent: + """The result of reading one entry. ``data`` is None when the caller asked for digests only.""" + + data: Optional[bytes] + sha1: str + sha256: str + size: int # bytes actually read, which is the number to trust + + +def validate_entry_name(name: str) -> None: + """Refuse any entry name we would not be willing to reproduce on disk, on any operating system. + + We never write these paths out, but they are surfaced to users and may be joined by callers later, so they are + validated at the point of entry rather than at the point of use. + """ + if not name: + raise UnsafeArchiveEntryError('Archive contains an entry with an empty name.') + + if '\x00' in name: + raise UnsafeArchiveEntryError('Archive contains an entry whose name embeds a null byte.') + + # zip names are meant to use forward slashes; treat a backslash as a separator rather than a literal character, + # because that is how Windows will read it + normalized = name.replace('\\', '/') + + if normalized.startswith('//'): + raise UnsafeArchiveEntryError(f'Archive entry uses a UNC path: {name!r}.') + + if normalized.startswith('/'): + raise UnsafeArchiveEntryError(f'Archive entry uses an absolute path: {name!r}.') + + if _DRIVE_LETTER_PATTERN.match(normalized): + raise UnsafeArchiveEntryError(f'Archive entry uses a drive letter: {name!r}.') + + for segment in normalized.split('/'): + if segment == '..': + raise UnsafeArchiveEntryError(f'Archive entry escapes the archive root: {name!r}.') + + if not segment or segment == '.': + continue + + # CON, CON.txt and CON.tar.gz are all the reserved device on Windows + if segment.split('.')[0].upper() in _WINDOWS_RESERVED_NAMES: + raise UnsafeArchiveEntryError(f'Archive entry uses a Windows reserved device name: {name!r}.') + + +def is_regular_file(info: zipfile.ZipInfo) -> bool: + """True for plain files. Symlinks, FIFOs, sockets and devices are all excluded by the same check.""" + file_type = (info.external_attr >> 16) & _UNIX_FILE_TYPE_MASK + return file_type in (_UNIX_FILE_TYPE_UNSET, _UNIX_REGULAR_FILE) + + +class SafeZip: + """A read-only view over an archive that refuses anything hostile before the caller ever sees it.""" + + def __init__(self, zip_file: zipfile.ZipFile, budget: ArchiveBudget, source_name: str) -> None: + self._zip_file = zip_file + self._budget = budget + self._source_name = source_name + + @property + def limits(self) -> ArchiveLimits: + return self._budget.limits + + @property + def source_name(self) -> str: + return self._source_name + + @classmethod + def open( + cls, + source: Union[str, bytes, IO[bytes]], + budget: Optional[ArchiveBudget] = None, + source_name: Optional[str] = None, + ) -> 'SafeZip': + """Open a path, a bytes blob, or any binary stream. Pass a shared ``budget`` when recursing.""" + if budget is None: + budget = ArchiveBudget() + + if isinstance(source, bytes): + stream: Union[str, IO[bytes]] = io.BytesIO(source) + name = source_name or '' + elif isinstance(source, str): + stream = source + name = source_name or os.path.basename(source) + else: + stream = source + name = source_name or '' + + try: + zip_file = zipfile.ZipFile(stream) + entry_count = len(zip_file.infolist()) + except _CORRUPT_ARCHIVE_ERRORS as e: + raise MalformedArchiveError(f'{name!r} is not a readable archive: {e}') from e + + if entry_count > budget.limits.max_entry_count: + zip_file.close() + raise ArchiveEntryCountLimitError( + f'{name!r} declares {entry_count} entries, past the limit of {budget.limits.max_entry_count}.' + ) + + return cls(zip_file, budget, name) + + # ruff would rather see typing.Self here, but that is 3.11+ and typing_extensions is not a + # declared dependency of this project + def __enter__(self) -> 'SafeZip': # noqa: PYI034 + return self + + def __exit__( + self, + exc_type: Optional[type[BaseException]], + exc_value: Optional[BaseException], + traceback: Optional[TracebackType], + ) -> None: + self.close() + + def close(self) -> None: + self._zip_file.close() + + def entries(self) -> Iterator[SafeZipEntry]: + """Yield every regular-file entry, in archive order. + + Directories and non-regular entries are skipped silently; an unsafe name aborts the whole archive, because + an artifact that contains one is not an artifact we are willing to report on. + """ + try: + infos = self._zip_file.infolist() + except _CORRUPT_ARCHIVE_ERRORS as e: + raise MalformedArchiveError(f'{self._source_name!r} has an unreadable directory: {e}') from e + + for info in infos: + if info.is_dir(): + continue + + validate_entry_name(info.filename) + + if not is_regular_file(info): + continue + + yield SafeZipEntry(name=info.filename, size=info.file_size, compress_size=info.compress_size, info=info) + + def read(self, entry: SafeZipEntry, buffer: bool = True) -> EntryContent: + """Stream one entry, hashing as we go and aborting the moment a ceiling is breached. + + ``buffer=False`` returns digests without holding the bytes, for entries we only need to identify. + """ + if entry.size > self.limits.max_entry_size_in_bytes: + raise ArchiveEntrySizeLimitError( + f'{entry.name!r} declares {entry.size} bytes, past the per-entry limit of ' + f'{self.limits.max_entry_size_in_bytes}.' + ) + + # SHA-1 is identification, not security: it is what artifact indexes key on, and usedforsecurity=False + # declares that. SHA-256 is computed alongside it so the BOM always carries a strong digest too. + sha1 = hashlib.sha1(usedforsecurity=False) + sha256 = hashlib.sha256() + chunks: Optional[list[bytes]] = [] if buffer else None + read_bytes = 0 + + try: + # entry.info, never entry.name: see the note on SafeZipEntry.info + with self._zip_file.open(entry.info if entry.info is not None else entry.name) as handle: + while True: + chunk = handle.read(_READ_CHUNK_SIZE_IN_BYTES) + if not chunk: + break + + read_bytes += len(chunk) + self._check_entry_size(entry, read_bytes) + self._check_compression_ratio(entry, read_bytes) + self._budget.consume(len(chunk)) + + sha1.update(chunk) + sha256.update(chunk) + if chunks is not None: + chunks.append(chunk) + except _CORRUPT_ARCHIVE_ERRORS as e: + raise MalformedArchiveError(f'{entry.name!r} in {self._source_name!r} could not be read: {e}') from e + + return EntryContent( + data=b''.join(chunks) if chunks is not None else None, + sha1=sha1.hexdigest(), + sha256=sha256.hexdigest(), + size=read_bytes, + ) + + def _check_entry_size(self, entry: SafeZipEntry, read_bytes: int) -> None: + if read_bytes > self.limits.max_entry_size_in_bytes: + raise ArchiveEntrySizeLimitError( + f'{entry.name!r} expands past the per-entry limit of {self.limits.max_entry_size_in_bytes} bytes.' + ) + + def _check_compression_ratio(self, entry: SafeZipEntry, read_bytes: int) -> None: + if read_bytes < self.limits.compression_ratio_floor_in_bytes or entry.compress_size <= 0: + return + + if read_bytes / entry.compress_size > self.limits.max_compression_ratio: + raise ArchiveCompressionRatioLimitError( + f'{entry.name!r} expands at more than {self.limits.max_compression_ratio}:1, which is a ' + f'decompression bomb rather than a build artifact.' + ) diff --git a/cycode/cli/printers/json_printer.py b/cycode/cli/printers/json_printer.py index acb7912f..84a25a11 100644 --- a/cycode/cli/printers/json_printer.py +++ b/cycode/cli/printers/json_printer.py @@ -3,6 +3,7 @@ from cycode.cli.models import CliError, CliResult from cycode.cli.printers.printer_base import PrinterBase +from cycode.cli.printers.utils import binary_report from cycode.cyclient.models import DetectionSchema if TYPE_CHECKING: @@ -45,10 +46,47 @@ def print_scan_results( # FIXME(MarshalX): we don't care about scan IDs in JSON output due to clumsy JSON root structure inlined_errors = [err._asdict() for err in errors.values()] - self.console.print_json(self._get_json_scan_result(scan_ids, detections_dict, report_urls, inlined_errors)) + self._print_degradation_warning_to_stderr() + self.console.print_json( + self._get_json_scan_result( + scan_ids, detections_dict, report_urls, inlined_errors, self._get_binary_section(local_scan_results) + ) + ) + + def _print_degradation_warning_to_stderr(self) -> None: + """The human running this still deserves the warning; stdout still has to stay parseable. + + The same fact is available machine-readably as ``binary.partial``, so a CI job never has to read stderr. + """ + collection = binary_report.get_binary_collection(self.ctx) + if collection is None or not binary_report.should_warn_about_degradation(self.ctx, collection): + return + + for line in binary_report.get_degradation_lines(collection): + self.console_err.print(f'[yellow]:warning: {line}[/]', highlight=False) + + def _get_binary_section(self, local_scan_results: list['LocalScanResult']) -> Optional[dict]: + """Coverage numbers a CI job can assert on, so a team can gate on identification rather than guess at it.""" + collection = binary_report.get_binary_collection(self.ctx) + if collection is None: + return None + + return { + 'identified': collection.identified_count, + 'low_confidence_components': collection.low_confidence_count, + 'unidentified': [entry._asdict() for entry in binary_report.get_unidentified(collection)], + 'low_confidence_detections': binary_report.count_low_confidence(self.ctx, local_scan_results), + 'resolver_available': collection.resolver_available, + 'partial': binary_report.should_warn_about_degradation(self.ctx, collection), + } def _get_json_scan_result( - self, scan_ids: list[str], detections: dict, report_urls: list[str], errors: list[dict] + self, + scan_ids: list[str], + detections: dict, + report_urls: list[str], + errors: list[dict], + binary: Optional[dict] = None, ) -> str: result = { 'scan_ids': scan_ids, @@ -57,6 +95,11 @@ def _get_json_scan_result( 'errors': errors, } + # additive: nothing existing changes shape, and the keys are absent entirely for non-binary scans + if binary is not None: + result['unidentified'] = binary['unidentified'] + result['binary'] = binary + return self.get_data_json(result) @staticmethod diff --git a/cycode/cli/printers/rich_printer.py b/cycode/cli/printers/rich_printer.py index 10cf561c..a8eafb0f 100644 --- a/cycode/cli/printers/rich_printer.py +++ b/cycode/cli/printers/rich_printer.py @@ -8,6 +8,7 @@ from cycode.cli import consts from cycode.cli.cli_types import SeverityOption from cycode.cli.printers.text_printer import TextPrinter +from cycode.cli.printers.utils import binary_report from cycode.cli.printers.utils.code_snippet_syntax import get_code_snippet_syntax from cycode.cli.printers.utils.detection_data import ( get_detection_clickable_cwe_cve, @@ -18,6 +19,7 @@ from cycode.cli.printers.utils.rich_helpers import get_columns_in_1_to_3_ratio, get_markdown_panel, get_panel if TYPE_CHECKING: + from cycode.cli.files_collector.binary.collector import BinaryCollectionResult from cycode.cli.models import CliError, Detection, Document, LocalScanResult @@ -29,6 +31,7 @@ def print_scan_results( ) -> None: if not errors and all(result.issue_detected == 0 for result in local_scan_results): self.console.print(self.NO_DETECTIONS_MESSAGE) + self.print_binary_report(local_scan_results) return detections, _ = sort_and_group_detections_from_scan_result(local_scan_results) @@ -41,9 +44,31 @@ def print_scan_results( detections_count, ) + self.print_binary_report(local_scan_results) self.print_scan_results_summary(local_scan_results) self.print_report_urls_and_errors(local_scan_results, errors) + def _print_unidentified_section(self, collection: 'BinaryCollectionResult') -> None: + unidentified = binary_report.get_unidentified(collection) + if not unidentified: + return + + table = Table(show_header=True, box=None, padding=(0, 2)) + table.add_column('Path', style='', overflow='fold') + table.add_column('SHA-1', style='dim') + table.add_column('Size', style='dim', justify='right') + + for entry in unidentified: + # Table renders markup, so an untrusted entry name is sanitised before it becomes a cell + table.add_row( + binary_report.for_display(entry.logical_path), + f'{entry.sha1[:12]}...', + binary_report.format_size(entry.size), + ) + + self.console.line() + self.console.print(get_panel(table, title=f'\U0001f50e Unidentified ({len(unidentified)})')) + def _get_details_table(self, detection: 'Detection') -> Table: details_table = Table(show_header=False, box=None, padding=(0, 1)) @@ -82,8 +107,7 @@ def _add_scan_related_rows(self, details_table: Table, detection: 'Detection') - def __add_secret_scan_related_rows(details_table: Table, detection: 'Detection') -> None: details_table.add_row('Secret SHA', detection.detection_details.get('sha512')) - @staticmethod - def __add_sca_scan_related_rows(details_table: Table, detection: 'Detection') -> None: + def __add_sca_scan_related_rows(self, details_table: Table, detection: 'Detection') -> None: detection_details = detection.detection_details details_table.add_row('CVEs', get_detection_clickable_cwe_cve(consts.SCA_SCAN_TYPE, detection)) @@ -100,6 +124,22 @@ def __add_sca_scan_related_rows(details_table: Table, detection: 'Detection') -> if not detection.has_alert: details_table.add_row('License', detection_details.get('license')) + self.__add_binary_evidence_rows(details_table, detection) + + def __add_binary_evidence_rows(self, details_table: Table, detection: 'Detection') -> None: + """Where inside the artifact the component sits, and how confidently we named it.""" + evidence = binary_report.get_detection_evidence(self.ctx, detection) + if evidence is None: + return + + source = binary_report.for_display(evidence.evidence) + + details_table.add_row('Found in', binary_report.for_display(evidence.logical_path)) + if evidence.is_ambiguous: + details_table.add_row('Identified by', f'[yellow]{source} - low confidence, does not affect exit code[/]') + else: + details_table.add_row('Identified by', f'{source} (exact)') + @staticmethod def __add_iac_scan_related_rows(details_table: Table, detection: 'Detection') -> None: details_table.add_row('IaC Provider', detection.detection_details.get('infra_provider')) diff --git a/cycode/cli/printers/text_printer.py b/cycode/cli/printers/text_printer.py index 51da53c5..e097bcfc 100644 --- a/cycode/cli/printers/text_printer.py +++ b/cycode/cli/printers/text_printer.py @@ -4,11 +4,13 @@ from cycode.cli.cli_types import SeverityOption from cycode.cli.models import CliError, CliResult, Document from cycode.cli.printers.printer_base import PrinterBase +from cycode.cli.printers.utils import binary_report from cycode.cli.printers.utils.code_snippet_syntax import get_code_snippet_syntax, get_detection_line from cycode.cli.printers.utils.detection_data import get_detection_title from cycode.cli.printers.utils.detection_ordering.common_ordering import sort_and_group_detections_from_scan_result if TYPE_CHECKING: + from cycode.cli.files_collector.binary.collector import BinaryCollectionResult from cycode.cli.models import Detection, LocalScanResult @@ -35,15 +37,73 @@ def print_scan_results( ) -> None: if not errors and all(result.issue_detected == 0 for result in local_scan_results): self.console.print(self.NO_DETECTIONS_MESSAGE) + # a clean scan still owes the user its coverage numbers: "no issues" and "we could not read half of it" + # are very different statements + self.print_binary_report(local_scan_results) return detections, _ = sort_and_group_detections_from_scan_result(local_scan_results) for detection, document in detections: self.__print_document_detection(document, detection) + self.print_binary_report(local_scan_results) self.print_scan_results_summary(local_scan_results) self.print_report_urls_and_errors(local_scan_results, errors) + def print_binary_report(self, local_scan_results: list['LocalScanResult']) -> None: + """The unidentified section, the degradation warning and the coverage line. + + Printed after findings and before the summary. Silent for every scan that is not a binary scan. + """ + collection = binary_report.get_binary_collection(self.ctx) + if collection is None: + return + + if binary_report.should_warn_about_degradation(self.ctx, collection): + self._print_degradation_warning(collection) + + self._print_unidentified_section(collection) + self._print_low_confidence_note(local_scan_results) + self._print_coverage_summary(collection, local_scan_results) + + def _print_degradation_warning(self, collection: 'BinaryCollectionResult') -> None: + self.console_err.line() + for line in binary_report.get_degradation_lines(collection): + self.console_err.print(f'[yellow]:warning: {line}[/]', highlight=False) + + def _print_unidentified_section(self, collection: 'BinaryCollectionResult') -> None: + unidentified = binary_report.get_unidentified(collection) + if not unidentified: + return + + self.console.line() + self.console.print(f'[bold]UNIDENTIFIED ({len(unidentified)})[/]') + for entry in unidentified: + # the name comes from inside an untrusted archive; it is markup- and control-character-sanitised + self.console.print(f' {binary_report.for_display(entry.logical_path)}', highlight=False) + self.console.print( + f' [dim]sha1 {entry.sha1[:8]}... {binary_report.format_size(entry.size)}[/]', highlight=False + ) + + def _print_low_confidence_note(self, local_scan_results: list['LocalScanResult']) -> None: + low_confidence = binary_report.count_low_confidence(self.ctx, local_scan_results) + if not low_confidence: + return + + self.console.line() + self.console.print( + f'[dim]{low_confidence} finding(s) come from a component identified by manifest attributes only. ' + f'They are marked low confidence and do not affect the exit code.[/]', + highlight=False, + ) + + def _print_coverage_summary( + self, collection: 'BinaryCollectionResult', local_scan_results: list['LocalScanResult'] + ) -> None: + vulnerabilities = binary_report.count_detections(local_scan_results) + self.console.line() + self.console.print(f'[bold]{binary_report.get_coverage_summary(collection, vulnerabilities)}[/]') + def __print_document_detection(self, document: 'Document', detection: 'Detection') -> None: self.__print_detection_summary(detection, document.path) self.__print_detection_code_segment(detection, document) @@ -70,8 +130,23 @@ def __print_detection_summary(self, detection: 'Detection', document_path: str) f'violation: [b bright_red]{title}[/]{detection_commit_id_message}\n', *self.__get_intermediate_summary_lines(detection), f'[dodger_blue1]File: {clickable_document_path}[/]', + *self._get_binary_evidence_lines(detection), ) + def _get_binary_evidence_lines(self, detection: 'Detection') -> list[str]: + """Where inside the artifact the component sits, and how confidently we named it.""" + evidence = binary_report.get_detection_evidence(self.ctx, detection) + if evidence is None: + return [] + + lines = [f'\n[dodger_blue1]Found in: {evidence.logical_path}[/]'] + if evidence.is_ambiguous: + lines.append(f'\n[yellow]Identified by: {evidence.evidence} (low confidence, does not affect exit code)[/]') + else: + lines.append(f'\n[dim]Identified by: {evidence.evidence} (exact)[/]') + + return lines + def __get_intermediate_summary_lines(self, detection: 'Detection') -> list[str]: intermediate_summary_lines = [] diff --git a/cycode/cli/printers/utils/binary_report.py b/cycode/cli/printers/utils/binary_report.py new file mode 100644 index 00000000..795a4a8c --- /dev/null +++ b/cycode/cli/printers/utils/binary_report.py @@ -0,0 +1,220 @@ +"""Shared reporting logic for binary scans. + +The printers differ in how they draw; what they say about a binary scan should not differ at all. Everything a +printer needs to know -- which findings came from an inferred coordinate, which archives could not be identified, +whether results are partial -- is computed here once and rendered three ways. +""" + +import re +from typing import TYPE_CHECKING, NamedTuple, Optional + +import typer +from rich.markup import escape + +from cycode.cli.files_collector.binary.base_extractor import CONFIDENCE_AMBIGUOUS +from cycode.cli.files_collector.binary.resolver import NullDigestResolver + +if TYPE_CHECKING: + from cycode.cli.files_collector.binary.collector import BinaryCollectionResult + from cycode.cli.models import Detection, LocalScanResult + +BINARY_RESULT_CONTEXT_KEY = 'binary_result' + +_PACKAGE_NAME_KEY = 'package_name' +_PACKAGE_VERSION_KEY = 'package_version' + +# C0 and C1 control characters, including ESC and the newlines rich does not strip +_CONTROL_CHARACTERS = re.compile(r'[\x00-\x1f\x7f-\x9f]') + +_MAX_DISPLAYED_LENGTH = 180 + + +def for_display(value: str) -> str: + """Make a string taken from inside an untrusted archive safe to put in front of a human. + + Entry names, manifest attributes and Maven coordinates are all authored by whoever built the artifact, and an + artifact we are asked to assess is by definition not trusted. Three things have to be neutralised before any + of it reaches a console: + + * rich markup, or an entry named ``[link=javascript:...]x[/link].jar`` becomes a live link in a terminal and a + raw ``href`` in an exported HTML report; + * ANSI escapes and newlines, which rich does not strip, and which would let an artifact clear the screen or + forge extra output lines -- including a fake coverage summary claiming full identification; + * unbounded length, which would let one entry name push the real findings off the screen. + + The BOM keeps the true, unmodified name. This is presentation only. + """ + cleaned = _CONTROL_CHARACTERS.sub('', value) + if len(cleaned) > _MAX_DISPLAYED_LENGTH: + cleaned = f'{cleaned[:_MAX_DISPLAYED_LENGTH]}...' + + return escape(cleaned) + + +class ComponentEvidence(NamedTuple): + """How a component we reported a finding against was identified.""" + + logical_path: str + evidence: str + confidence: str + + @property + def is_ambiguous(self) -> bool: + return self.confidence == CONFIDENCE_AMBIGUOUS + + +class UnidentifiedEntry(NamedTuple): + logical_path: str + sha1: str + size: int + + +def get_binary_collection(ctx: typer.Context) -> Optional['BinaryCollectionResult']: + """The collection result, or None when this was not a binary scan.""" + if not ctx.obj: + return None + + return ctx.obj.get(BINARY_RESULT_CONTEXT_KEY) + + +def _component_key(group: str, artifact: str, version: str) -> tuple[str, str]: + # the platform reports maven packages as 'group:artifact' + name = f'{group}:{artifact}' if group else artifact + return name, version + + +def build_component_index(collection: 'BinaryCollectionResult') -> dict[tuple[str, str], ComponentEvidence]: + index: dict[tuple[str, str], ComponentEvidence] = {} + + for result in collection.results_by_artifact.values(): + for component in result.components: + key = _component_key(component.group, component.artifact, component.version) + index.setdefault( + key, + ComponentEvidence( + logical_path=component.logical_path, + evidence=component.evidence, + confidence=component.confidence, + ), + ) + + return index + + +def get_detection_evidence(ctx: typer.Context, detection: 'Detection') -> Optional[ComponentEvidence]: + """Where inside the artifact this finding's component lives, and how confidently we named it. + + Without this a Log4Shell hit on a 40 MB EAR gives a developer nowhere to start. + """ + collection = get_binary_collection(ctx) + if not collection: + return None + + details = detection.detection_details or {} + key = (details.get(_PACKAGE_NAME_KEY), details.get(_PACKAGE_VERSION_KEY)) + + return build_component_index(collection).get(key) + + +def is_low_confidence(ctx: typer.Context, detection: 'Detection') -> bool: + """True for a finding whose component was only ever identified by a manifest attribute.""" + evidence = get_detection_evidence(ctx, detection) + return bool(evidence and evidence.is_ambiguous) + + +def has_gating_detections(ctx: typer.Context, local_scan_results: list['LocalScanResult']) -> bool: + """Whether any finding is confident enough to fail a build. + + A wrong CVE from a guessed coordinate breaking someone's release costs more trust than the extra recall is + worth, so tier 3 findings print but never gate. They are still reported, still exported, still counted. + """ + for local_scan_result in local_scan_results: + for document_detections in local_scan_result.document_detections: + for detection in document_detections.detections: + if not is_low_confidence(ctx, detection): + return True + + return False + + +def count_low_confidence(ctx: typer.Context, local_scan_results: list['LocalScanResult']) -> int: + return sum( + 1 + for local_scan_result in local_scan_results + for document_detections in local_scan_result.document_detections + for detection in document_detections.detections + if is_low_confidence(ctx, detection) + ) + + +def get_unidentified(collection: 'BinaryCollectionResult') -> list[UnidentifiedEntry]: + """Every archive we could not name, in a stable order.""" + entries = [ + UnidentifiedEntry(logical_path=item.logical_path, sha1=item.sha1, size=item.size) + for result in collection.results_by_artifact.values() + for item in result.unidentified + ] + + return sorted(entries, key=lambda entry: entry.logical_path) + + +def should_warn_about_degradation(ctx: typer.Context, collection: 'BinaryCollectionResult') -> bool: + """Warn only when resolution was unavailable AND it would have made a difference. + + ``--offline`` is the user acknowledging the trade-off, so it silences the warning rather than suppressing the + coverage numbers: the counts stay true either way. + """ + if ctx.obj and ctx.obj.get('offline'): + return False + + return not collection.resolver_available and collection.unidentified_count > 0 + + +def get_coverage_summary(collection: 'BinaryCollectionResult', vulnerabilities: int) -> str: + """One line, always present, always true. + + A manifest-only match counts as identified, but "9 identified" with three of them guessed from a manifest is a + different result from nine exact matches, so the guessed ones are called out inline rather than folded in. + """ + identified = f'{collection.identified_count} identified' + if collection.low_confidence_count: + identified += f' ({collection.low_confidence_count} low confidence)' + + return f'{identified} | {collection.unidentified_count} unidentified | {vulnerabilities} vulnerabilities' + + +def get_degradation_lines(collection: 'BinaryCollectionResult') -> list[str]: + """The partial-coverage warning. + + It leads with the coverage gap rather than with the missing capability, because the gap is what the reader has + to act on. It also says "not available in this release" rather than "unavailable": digest lookup is not a + service that went down, it is a tier that has not shipped yet, and wording it as an outage would train people + to wait for it to clear. + """ + total = collection.identified_count + collection.unidentified_count + reason = collection.resolver_unavailability_reason or NullDigestResolver().unavailability_reason + return [ + f'{collection.unidentified_count} of {total} components could not be identified ' + f'from embedded metadata - results are PARTIAL.', + reason, + 'Run with --offline to acknowledge and silence this warning.', + ] + + +def format_size(size_in_bytes: int) -> str: + if size_in_bytes < 1024: + return f'{size_in_bytes} B' + + if size_in_bytes < 1024 * 1024: + return f'{size_in_bytes / 1024:.0f} KB' + + return f'{size_in_bytes / (1024 * 1024):.1f} MB' + + +def count_detections(local_scan_results: list['LocalScanResult']) -> int: + return sum( + 1 + for local_scan_result in local_scan_results + for document_detections in local_scan_result.document_detections + for detection in document_detections.detections + ) diff --git a/cycode/cyclient/cycode_client_base.py b/cycode/cyclient/cycode_client_base.py index 4e7eebe4..1d63e3d8 100644 --- a/cycode/cyclient/cycode_client_base.py +++ b/cycode/cyclient/cycode_client_base.py @@ -58,6 +58,11 @@ def _get_session() -> requests.Session: return session +def get_http_session() -> requests.Session: + """The shared session, for callers outside the Cycode API that still need its trust-store and proxy handling.""" + return _get_session() + + def _get_request_function() -> Callable: return _get_session().request diff --git a/tests/cli/files_collector/binary/__init__.py b/tests/cli/files_collector/binary/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/tests/cli/files_collector/binary/cyclonedx_assertions.py b/tests/cli/files_collector/binary/cyclonedx_assertions.py new file mode 100644 index 00000000..31df7a28 --- /dev/null +++ b/tests/cli/files_collector/binary/cyclonedx_assertions.py @@ -0,0 +1,92 @@ +"""A structural validator for CycloneDX 1.4 documents. + +Deviation from the plan, recorded deliberately: BCA-P2-08 called for validating generated output against the +vendored CycloneDX 1.4 JSON schema. That needs ``jsonschema``, which is not installed and which decision 11 forbids +adding, and it needs the schema file itself, which CI has no egress to fetch. What is asserted here instead is the +part of the specification this feature can actually violate: required fields, enumerated values, digest shapes, +bom-ref uniqueness and referential integrity of the dependency graph. + +If the reviewers want true schema validation, the schema can be committed under ``tests/test_files/`` and +``jsonschema`` added as a dev dependency in a follow-up. That is a deliberate deferral, not an oversight. +""" + +import re + +# CycloneDX 1.4 component type enumeration +_COMPONENT_TYPES = frozenset( + {'application', 'framework', 'library', 'container', 'operating-system', 'device', 'firmware', 'file'} +) + +# the subset of the hash algorithm enumeration this feature emits +_HASH_ALGORITHMS = frozenset({'MD5', 'SHA-1', 'SHA-256', 'SHA-384', 'SHA-512'}) + +_HASH_LENGTHS = {'MD5': 32, 'SHA-1': 40, 'SHA-256': 64, 'SHA-384': 96, 'SHA-512': 128} + +_HEX = re.compile(r'^[a-fA-F0-9]+$') +_PURL = re.compile(r'^pkg:[a-zA-Z][a-zA-Z0-9.+-]*/.+$') +_TIMESTAMP = re.compile(r'^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$') + + +def _assert_properties(properties: list) -> None: + for entry in properties: + assert isinstance(entry.get('name'), str), f'property needs a name: {entry}' + assert entry['name'], f'property name must not be empty: {entry}' + assert isinstance(entry.get('value'), str), f'property needs a string value: {entry}' + + +def _assert_hashes(hashes: list) -> None: + for entry in hashes: + algorithm = entry.get('alg') + content = entry.get('content') + + assert algorithm in _HASH_ALGORITHMS, f'unknown hash algorithm: {algorithm}' + assert isinstance(content, str), f'hash content must be a string: {content}' + assert _HEX.match(content), f'hash content is not hex: {content}' + assert len(content) == _HASH_LENGTHS[algorithm], f'{algorithm} digest has the wrong length: {content}' + + +def _assert_component(component: dict) -> None: + assert component.get('type') in _COMPONENT_TYPES, f'unknown component type: {component.get("type")}' + assert isinstance(component.get('name'), str), 'a component needs a name' + assert component['name'], 'a component name must not be empty' + + if 'purl' in component: + assert _PURL.match(component['purl']), f'malformed purl: {component["purl"]}' + + _assert_hashes(component.get('hashes', [])) + _assert_properties(component.get('properties', [])) + + +def assert_valid_cyclonedx(bom: dict) -> None: + """Assert the document is a well-formed CycloneDX 1.4 BOM with an internally consistent dependency graph.""" + assert bom.get('bomFormat') == 'CycloneDX' + assert bom.get('specVersion') == '1.4' + assert isinstance(bom.get('version'), int), 'BOM version must be an integer' + assert bom['version'] >= 1, 'BOM version starts at 1' + + metadata = bom.get('metadata', {}) + assert _TIMESTAMP.match(metadata['timestamp']), f'malformed timestamp: {metadata.get("timestamp")}' + assert metadata['tools'], 'a generated BOM should say what generated it' + _assert_properties(metadata.get('properties', [])) + + root_component = metadata['component'] + _assert_component(root_component) + + declared_refs = {root_component['bom-ref']} + for component in bom.get('components', []): + _assert_component(component) + + ref = component['bom-ref'] + assert ref not in declared_refs, f'duplicate bom-ref: {ref}' + declared_refs.add(ref) + + seen_refs = set() + for dependency in bom.get('dependencies', []): + ref = dependency['ref'] + assert ref in declared_refs, f'dependency references an undeclared component: {ref}' + assert ref not in seen_refs, f'duplicate dependency entry: {ref}' + seen_refs.add(ref) + + for target in dependency.get('dependsOn', []): + assert target in declared_refs, f'dependsOn references an undeclared component: {target}' + assert target != ref, f'component depends on itself: {ref}' diff --git a/tests/cli/files_collector/binary/fixtures.py b/tests/cli/files_collector/binary/fixtures.py new file mode 100644 index 00000000..41020c6c --- /dev/null +++ b/tests/cli/files_collector/binary/fixtures.py @@ -0,0 +1,194 @@ +"""Archive builders for the binary-extraction suite. + +Every fixture is *built* at test time into ``tmp_path``. Nothing binary is committed: CI runs with no network egress +so a downloaded artifact would fail there even where it passes locally, and committing hostile archives into a +security product's own repository is a problem of its own. The upside is that each hostile case reads as code. +""" + +import io +import zipfile +from typing import Optional, Union + +# ZipInfo.external_attr carries the unix mode in its high 16 bits +REGULAR_FILE_ATTR = 0o100644 << 16 +SYMLINK_ATTR = 0o120777 << 16 +DIRECTORY_ATTR = 0o040755 << 16 +FIFO_ATTR = 0o010644 << 16 + +# an archive written by MS-DOS tooling records no unix mode at all +MSDOS_ATTR = 0 + +_Contents = dict[str, Union[bytes, str]] + + +def archive_bytes( + files: Optional[_Contents] = None, + symlinks: Optional[dict[str, str]] = None, + external_attrs: Optional[dict[str, int]] = None, + compress: bool = True, +) -> bytes: + """Build an archive in memory. + + ``files`` maps entry name to content. ``symlinks`` maps entry name to link target, written with the unix mode + bits that mark it a symlink. ``external_attrs`` overrides the mode for a named entry. + """ + files = files or {} + symlinks = symlinks or {} + external_attrs = external_attrs or {} + compress_type = zipfile.ZIP_DEFLATED if compress else zipfile.ZIP_STORED + + buffer = io.BytesIO() + with zipfile.ZipFile(buffer, 'w', compress_type) as archive: + for name, content in files.items(): + info = zipfile.ZipInfo(name) + info.compress_type = compress_type + info.external_attr = external_attrs.get(name, REGULAR_FILE_ATTR) + archive.writestr(info, content) + + for name, target in symlinks.items(): + info = zipfile.ZipInfo(name) + info.compress_type = compress_type + info.external_attr = SYMLINK_ATTR + archive.writestr(info, target) + + return buffer.getvalue() + + +def pom_properties(group: str, artifact: str, version: str) -> str: + return f'#Generated by Maven\ngroupId={group}\nartifactId={artifact}\nversion={version}\n' + + +def pom_properties_entry_name(group: str, artifact: str) -> str: + return f'META-INF/maven/{group}/{artifact}/pom.properties' + + +def manifest(**attributes: str) -> str: + lines = ['Manifest-Version: 1.0'] + lines.extend(f'{key.replace("_", "-")}: {value}' for key, value in attributes.items()) + return '\n'.join(lines) + '\n' + + +def library_jar(group: str, artifact: str, version: str) -> bytes: + """A jar that identifies itself the way a Maven-built artifact does.""" + return archive_bytes( + files={ + 'META-INF/MANIFEST.MF': manifest(Implementation_Title=artifact, Implementation_Version=version), + pom_properties_entry_name(group, artifact): pom_properties(group, artifact, version), + f'{group.replace(".", "/")}/{artifact}/Main.class': b'\xca\xfe\xba\xbe fake class bytes', + } + ) + + +def war_bytes(libraries: Optional[dict[str, bytes]] = None) -> bytes: + """A WAR with its libraries under ``WEB-INF/lib``.""" + files: _Contents = { + 'WEB-INF/web.xml': '', + 'index.jsp': '', + } + for name, content in (libraries or {}).items(): + files[f'WEB-INF/lib/{name}'] = content + + return archive_bytes(files=files) + + +def boot_jar_bytes(libraries: Optional[dict[str, bytes]] = None) -> bytes: + """A Spring Boot fat JAR, whose libraries live under ``BOOT-INF/lib``.""" + files: _Contents = { + 'META-INF/MANIFEST.MF': manifest(Main_Class='org.springframework.boot.loader.JarLauncher'), + 'BOOT-INF/classes/com/acme/App.class': b'\xca\xfe\xba\xbe', + } + for name, content in (libraries or {}).items(): + files[f'BOOT-INF/lib/{name}'] = content + + return archive_bytes(files=files) + + +def ear_bytes(modules: Optional[dict[str, bytes]] = None, libraries: Optional[dict[str, bytes]] = None) -> bytes: + """An EAR whose modules sit at its root, as ``application.xml`` would declare them.""" + files: _Contents = {'META-INF/application.xml': ''} + files.update(modules or {}) + for name, content in (libraries or {}).items(): + files[f'APP-INF/lib/{name}'] = content + + return archive_bytes(files=files) + + +# --- hostile corpus --------------------------------------------------------------------------------------------- + + +def zip_slip_bytes(name: str = '../../evil.txt') -> bytes: + return archive_bytes(files={'harmless.txt': 'ok', name: 'owned'}) + + +def absolute_path_bytes() -> bytes: + return archive_bytes(files={'/etc/passwd': 'root:x:0:0:'}) + + +def drive_letter_bytes() -> bytes: + return archive_bytes(files={'C:\\Windows\\System32\\evil.dll': 'owned'}) + + +def unc_path_bytes() -> bytes: + return archive_bytes(files={'\\\\host\\share\\evil.txt': 'owned'}) + + +def reserved_name_bytes(name: str = 'CON') -> bytes: + return archive_bytes(files={name: 'owned'}) + + +def symlink_bytes() -> bytes: + return archive_bytes(files={'real.txt': 'ok'}, symlinks={'escape.txt': '/etc/passwd'}) + + +def non_regular_entry_bytes() -> bytes: + return archive_bytes(files={'fifo': '', 'real.txt': 'ok'}, external_attrs={'fifo': FIFO_ATTR}) + + +def msdos_created_bytes() -> bytes: + """An archive with no unix mode recorded at all, as produced by Windows tooling. Must still be readable.""" + return archive_bytes(files={'real.txt': 'ok'}, external_attrs={'real.txt': MSDOS_ATTR}) + + +def ratio_bomb_bytes(uncompressed_size: int = 4 * 1024 * 1024) -> bytes: + """Highly compressible payload: a few kilobytes on disk, megabytes once read.""" + return archive_bytes(files={'bomb.bin': b'\x00' * uncompressed_size}) + + +def size_bomb_bytes(uncompressed_size: int = 64 * 1024) -> bytes: + """A single entry larger than a deliberately small per-entry cap.""" + return archive_bytes(files={'big.bin': b'A' * uncompressed_size}, compress=False) + + +def total_size_bomb_bytes(entry_count: int = 8, entry_size: int = 16 * 1024) -> bytes: + """Several modest entries that only breach the cap in aggregate.""" + return archive_bytes( + files={f'part-{index}.bin': b'B' * entry_size for index in range(entry_count)}, + compress=False, + ) + + +def count_bomb_bytes(entry_count: int) -> bytes: + return archive_bytes(files={f'entry-{index}.txt': '' for index in range(entry_count)}) + + +def empty_archive_bytes() -> bytes: + return archive_bytes() + + +def not_a_zip_bytes() -> bytes: + return b'this is not a zip file, it merely has the name of one' * 8 + + +def truncated_central_directory_bytes() -> bytes: + """A valid archive with its tail lopped off, so the central directory can never be located.""" + valid = archive_bytes(files={'a.txt': 'a' * 512, 'b.txt': 'b' * 512}) + return valid[: len(valid) // 2] + + +def malformed_local_header_bytes() -> bytes: + """Central directory intact, local file header corrupted: opens cleanly, fails on read.""" + valid = bytearray(archive_bytes(files={'a.txt': 'a' * 512})) + signature_offset = valid.find(b'PK\x03\x04') + assert signature_offset != -1 + valid[signature_offset : signature_offset + 4] = b'XXXX' + return bytes(valid) diff --git a/tests/cli/files_collector/binary/test_collector.py b/tests/cli/files_collector/binary/test_collector.py new file mode 100644 index 00000000..dd6cd5db --- /dev/null +++ b/tests/cli/files_collector/binary/test_collector.py @@ -0,0 +1,342 @@ +import json +import os +import xml.etree.ElementTree as ET +from pathlib import Path +from unittest.mock import MagicMock, patch + +import pytest +import typer + +from cycode.cli.apps.scan.binary.identity import ( + IDENTITY_FROM_FILENAME, + IDENTITY_FROM_GIT_REMOTE, + IDENTITY_FROM_PROJECT_NAME, + assert_monitor_has_an_explicit_identity, + resolve_platform_identity, +) +from cycode.cli.exceptions.custom_exceptions import MalformedArchiveError +from cycode.cli.files_collector.binary.base_extractor import ExtractionResult +from cycode.cli.files_collector.binary.collector import ( + BinaryCollectionResult, + build_document_path, + build_manifest_path, + build_synthetic_manifest, + collect_binary_documents, + find_supported_artifacts, + get_resolver, +) +from cycode.cli.files_collector.binary.maven_central import MavenCentralDigestResolver +from cycode.cli.files_collector.binary.resolver import NullDigestResolver +from cycode.cli.utils.path_utils import get_path_by_os +from tests.cli.files_collector.binary import fixtures + +_COLLECTOR_MODULE = 'cycode.cli.files_collector.binary.collector' +_IDENTITY_MODULE = 'cycode.cli.apps.scan.binary.identity' + +_GUAVA = ('com.google.guava', 'guava', '31.1-jre') +_LOG4J = ('org.apache.logging.log4j', 'log4j-core', '2.14.1') + + +@pytest.fixture +def mock_ctx() -> typer.Context: + ctx = MagicMock(spec=typer.Context) + ctx.obj = {'progress_bar': None, 'binary_max_depth': 3, 'keep_bom': False, 'monitor': False} + return ctx + + +def _os_path(*parts: str) -> str: + """Document paths use the platform separator, so expectations must too. CI runs Windows.""" + return get_path_by_os('/'.join(parts)) + + +def _bom_document(collection: object) -> object: + return next(document for document in collection.documents if document.path.endswith('bom.json')) + + +def _war(tmp_path: Path, name: str = 'payments.war') -> Path: + path = tmp_path / name + path.write_bytes( + fixtures.war_bytes( + libraries={ + 'guava.jar': fixtures.library_jar(*_GUAVA), + 'log4j-core.jar': fixtures.library_jar(*_LOG4J), + } + ) + ) + return path + + +class TestFindSupportedArtifacts: + def test_a_single_file(self, tmp_path: Path) -> None: + path = _war(tmp_path) + + assert find_supported_artifacts((str(path),)) == [str(path)] + + def test_a_directory_is_walked(self, tmp_path: Path) -> None: + _war(tmp_path, 'a.war') + (tmp_path / 'nested').mkdir() + _war(tmp_path / 'nested', 'b.jar') + (tmp_path / 'README.md').write_text('not an artifact') + + found = [os.path.basename(path) for path in find_supported_artifacts((str(tmp_path),))] + + assert sorted(found) == ['a.war', 'b.jar'] + + def test_unsupported_files_are_ignored(self, tmp_path: Path) -> None: + (tmp_path / 'app.zip').write_bytes(fixtures.archive_bytes(files={'a.txt': 'a'})) + (tmp_path / 'pom.xml').write_text('') + + assert find_supported_artifacts((str(tmp_path),)) == [] + + +class TestDocumentPath: + def test_a_relative_path_is_kept_for_provenance(self) -> None: + artifact = os.path.join('dist', 'payments.war') + + # mirrors the cyclonedx-maven-plugin layout: /pom.xml and /target/bom.json + assert build_document_path(artifact) == _os_path('dist', 'payments.war', 'target', 'bom.json') + assert build_manifest_path(artifact) == _os_path('dist', 'payments.war', 'pom.xml') + + def test_an_absolute_path_under_the_working_directory_is_relativised(self, tmp_path: Path) -> None: + artifact = tmp_path / 'dist' / 'payments.war' + + with patch(f'{_COLLECTOR_MODULE}.os.getcwd', return_value=str(tmp_path)): + assert build_document_path(str(artifact)) == _os_path('dist', 'payments.war', 'target', 'bom.json') + + def test_a_path_outside_the_working_directory_falls_back_to_the_filename(self, tmp_path: Path) -> None: + with patch(f'{_COLLECTOR_MODULE}.os.getcwd', return_value=str(tmp_path / 'somewhere' / 'else')): + assert build_document_path(str(tmp_path / 'payments.war')) == _os_path('payments.war', 'target', 'bom.json') + + def test_it_always_ends_with_bom_json(self, tmp_path: Path) -> None: + # the existing SCA routing recognises the document by this name + assert os.path.basename(build_document_path(str(_war(tmp_path)))) == 'bom.json' + + +class TestSyntheticManifest: + """Phase 0 established the engine will not route a lone bom.json. This manifest is what makes it route.""" + + def test_it_is_well_formed_xml_declaring_no_dependencies(self) -> None: + manifest = build_synthetic_manifest('payments.war') + root = ET.fromstring(manifest) # noqa: S314 - the document under test is one we generated + + assert root.tag.endswith('project') + # every finding must still come from the BOM, so the manifest must never contribute dependencies of its own + assert root.find('.//{*}dependencies') is None + assert root.find('.//{*}dependency') is None + + def test_it_names_the_artifact(self) -> None: + manifest = build_synthetic_manifest('payments.war') + + assert 'payments.war' in manifest + assert 'payments.war' in manifest + + @pytest.mark.parametrize( + ('artifact_name', 'expected_id'), + [ + ('payments.war', 'payments.war'), + ('my app (1).jar', 'my-app--1-.jar'), + ('caf\u00e9.jar', 'caf-.jar'), + ], + ) + def test_unusual_filenames_produce_a_usable_artifact_id(self, artifact_name: str, expected_id: str) -> None: + manifest = build_synthetic_manifest(artifact_name) + + assert f'{expected_id}' in manifest + ET.fromstring(manifest) # noqa: S314 - the document under test is one we generated + + def test_xml_metacharacters_in_a_filename_cannot_break_the_document(self) -> None: + # a filename is attacker-influenced in a sweep over an artifact repository + manifest = build_synthetic_manifest('evil&.jar') + + root = ET.fromstring(manifest) # noqa: S314 - the document under test is one we generated + assert root.find('.//{*}dependencies') is None + assert root.find('{*}name').text == 'evil&.jar' + + +class TestCollectBinaryDocuments: + def test_a_bom_and_a_manifest_per_artifact(self, mock_ctx: typer.Context, tmp_path: Path) -> None: + _war(tmp_path, 'a.war') + _war(tmp_path, 'b.war') + + collection = collect_binary_documents(mock_ctx, (str(tmp_path),)) + + assert sorted(document.path for document in collection.documents) == sorted( + [ + _os_path('a.war', 'pom.xml'), + _os_path('a.war', 'target', 'bom.json'), + _os_path('b.war', 'pom.xml'), + _os_path('b.war', 'target', 'bom.json'), + ] + ) + + def test_the_document_content_is_the_synthesised_bom(self, mock_ctx: typer.Context, tmp_path: Path) -> None: + path = _war(tmp_path) + + collection = collect_binary_documents(mock_ctx, (str(path),)) + bom = json.loads(_bom_document(collection).content) + + assert bom['bomFormat'] == 'CycloneDX' + assert bom['metadata']['component']['name'] == 'payments.war' + assert {component['purl'] for component in bom['components']} == { + 'pkg:maven/com.google.guava/guava@31.1-jre', + 'pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1', + } + + def test_the_archive_itself_never_becomes_a_document(self, mock_ctx: typer.Context, tmp_path: Path) -> None: + # this is what keeps the binary filter in file_excluder correct and untouched + path = _war(tmp_path) + + collection = collect_binary_documents(mock_ctx, (str(path),)) + + assert all(not document.path.endswith('.war') for document in collection.documents) + assert {os.path.basename(document.path) for document in collection.documents} == {'bom.json', 'pom.xml'} + + def test_counters_are_aggregated_across_artifacts(self, mock_ctx: typer.Context, tmp_path: Path) -> None: + _war(tmp_path, 'a.war') + anonymous = fixtures.archive_bytes(files={'com/acme/Shim.class': b'\xca\xfe\xba\xbe'}) + (tmp_path / 'b.war').write_bytes(fixtures.war_bytes(libraries={'shim.jar': anonymous})) + + collection = collect_binary_documents(mock_ctx, (str(tmp_path),)) + + assert collection.identified_count == 2 + assert collection.unidentified_count == 1 + assert collection.resolver_available is False + + def test_the_resolver_reason_is_the_last_snapshot(self) -> None: + # one resolver serves every artifact and its failure count grows; the first artifact's snapshot is stale + collection = BinaryCollectionResult() + collection.results_by_artifact['a.jar'] = ExtractionResult( + resolver_available=False, resolver_unavailability_reason='failed for 1 of 1 digests' + ) + collection.results_by_artifact['b.jar'] = ExtractionResult( + resolver_available=False, resolver_unavailability_reason='failed for 2 of 5 digests' + ) + + assert collection.resolver_unavailability_reason == 'failed for 2 of 5 digests' + + def test_the_resolver_is_opt_in(self, mock_ctx: typer.Context) -> None: + assert isinstance(get_resolver(mock_ctx), NullDigestResolver) + + mock_ctx.obj['maven_central'] = True + + assert isinstance(get_resolver(mock_ctx), MavenCentralDigestResolver) + + def test_max_depth_is_taken_from_the_context(self, mock_ctx: typer.Context, tmp_path: Path) -> None: + inner = fixtures.war_bytes(libraries={'guava.jar': fixtures.library_jar(*_GUAVA)}) + (tmp_path / 'app.ear').write_bytes(fixtures.ear_bytes(modules={'web.war': inner})) + mock_ctx.obj['binary_max_depth'] = 1 + + collection = collect_binary_documents(mock_ctx, (str(tmp_path),)) + bom = json.loads(_bom_document(collection).content) + + # the war was reported but never opened, so guava inside it was never seen + assert bom['components'] == [] + + def test_one_unreadable_artifact_does_not_stop_the_others(self, mock_ctx: typer.Context, tmp_path: Path) -> None: + _war(tmp_path, 'good.war') + (tmp_path / 'broken.jar').write_bytes(fixtures.not_a_zip_bytes()) + + collection = collect_binary_documents(mock_ctx, (str(tmp_path),)) + + assert sorted(document.path for document in collection.documents) == sorted( + [_os_path('good.war', 'pom.xml'), _os_path('good.war', 'target', 'bom.json')] + ) + assert list(collection.failures) == [str(tmp_path / 'broken.jar')] + + def test_stop_on_error_propagates(self, mock_ctx: typer.Context, tmp_path: Path) -> None: + (tmp_path / 'broken.jar').write_bytes(fixtures.not_a_zip_bytes()) + + with pytest.raises(MalformedArchiveError): + collect_binary_documents(mock_ctx, (str(tmp_path),), stop_on_error=True) + + def test_nothing_to_scan_is_not_an_error(self, mock_ctx: typer.Context, tmp_path: Path) -> None: + (tmp_path / 'README.md').write_text('no artifacts here') + + collection = collect_binary_documents(mock_ctx, (str(tmp_path),)) + + assert collection.documents == [] + assert collection.failures == {} + + def test_keep_bom_writes_the_document_beside_the_artifact(self, mock_ctx: typer.Context, tmp_path: Path) -> None: + path = _war(tmp_path) + mock_ctx.obj['keep_bom'] = True + + collect_binary_documents(mock_ctx, (str(path),)) + + written = tmp_path / 'payments.war.bom.json' + assert written.exists() + assert json.loads(written.read_text())['bomFormat'] == 'CycloneDX' + + def test_keep_bom_is_off_by_default(self, mock_ctx: typer.Context, tmp_path: Path) -> None: + path = _war(tmp_path) + + collect_binary_documents(mock_ctx, (str(path),)) + + assert not (tmp_path / 'payments.war.bom.json').exists() + + def test_the_progress_bar_reflects_the_archive_count(self, mock_ctx: typer.Context, tmp_path: Path) -> None: + _war(tmp_path, 'a.war') + _war(tmp_path, 'b.war') + progress_bar = MagicMock() + mock_ctx.obj['progress_bar'] = progress_bar + + collect_binary_documents(mock_ctx, (str(tmp_path),)) + + assert progress_bar.set_section_length.call_args[0][1] == 2 + assert progress_bar.update.call_count == 2 + + +class TestPlatformIdentity: + def test_project_name_wins(self, mock_ctx: typer.Context, tmp_path: Path) -> None: + mock_ctx.obj['project_name'] = 'payments-service' + path = _war(tmp_path) + + with patch(f'{_IDENTITY_MODULE}.get_remote_url_scan_parameter', return_value='https://git/acme/repo.git'): + identity = resolve_platform_identity(mock_ctx, (str(path),)) + + assert identity.value == 'payments-service' + assert identity.source == IDENTITY_FROM_PROJECT_NAME + + def test_a_git_remote_is_used_when_there_is_no_override(self, mock_ctx: typer.Context, tmp_path: Path) -> None: + path = _war(tmp_path) + + with patch(f'{_IDENTITY_MODULE}.get_remote_url_scan_parameter', return_value='https://git/acme/repo.git'): + identity = resolve_platform_identity(mock_ctx, (str(path),)) + + assert identity.value == 'https://git/acme/repo.git' + assert identity.source == IDENTITY_FROM_GIT_REMOTE + + def test_a_detached_artifact_falls_back_to_its_filename(self, mock_ctx: typer.Context, tmp_path: Path) -> None: + path = _war(tmp_path) + + with patch(f'{_IDENTITY_MODULE}.get_remote_url_scan_parameter', return_value=None): + identity = resolve_platform_identity(mock_ctx, (str(path),)) + + assert identity.value == 'payments.war' + assert identity.source == IDENTITY_FROM_FILENAME + assert identity.is_explicit is False + + +class TestMonitorGuard: + def test_a_filename_identity_is_refused(self, mock_ctx: typer.Context, tmp_path: Path) -> None: + path = _war(tmp_path) + + with patch(f'{_IDENTITY_MODULE}.get_remote_url_scan_parameter', return_value=None): + identity = resolve_platform_identity(mock_ctx, (str(path),)) + + with pytest.raises(typer.BadParameter) as error: + assert_monitor_has_an_explicit_identity(identity) + + # the message has to name the fix, not just the problem + assert '--project-name' in str(error.value) + assert 'payments.war' in str(error.value) + + @pytest.mark.parametrize('remote_url', ['https://git/acme/repo.git', None]) + def test_an_explicit_identity_is_permitted(self, mock_ctx: typer.Context, tmp_path: Path, remote_url: str) -> None: + mock_ctx.obj['project_name'] = None if remote_url else 'payments-service' + path = _war(tmp_path) + + with patch(f'{_IDENTITY_MODULE}.get_remote_url_scan_parameter', return_value=remote_url): + identity = resolve_platform_identity(mock_ctx, (str(path),)) + + assert_monitor_has_an_explicit_identity(identity) diff --git a/tests/cli/files_collector/binary/test_cyclonedx_builder.py b/tests/cli/files_collector/binary/test_cyclonedx_builder.py new file mode 100644 index 00000000..29d0ddb4 --- /dev/null +++ b/tests/cli/files_collector/binary/test_cyclonedx_builder.py @@ -0,0 +1,414 @@ +import copy +import json +from pathlib import Path +from typing import ClassVar + +import pytest + +from cycode.cli.files_collector.binary import cyclonedx_builder +from cycode.cli.files_collector.binary.base_extractor import ( + CONFIDENCE_AMBIGUOUS, + CONFIDENCE_EXACT, + EVIDENCE_MANIFEST, + EVIDENCE_POM_PROPERTIES, + ExtractionResult, + IdentifiedComponent, + UnidentifiedArtifact, +) +from cycode.cli.files_collector.binary.java_extractor import JavaArchiveExtractor +from tests.cli.files_collector.binary import fixtures +from tests.cli.files_collector.binary.cyclonedx_assertions import assert_valid_cyclonedx + +_FIXED_TIMESTAMP = '2026-08-26T12:00:00Z' + +_GUAVA = ('com.google.guava', 'guava', '31.1-jre') +_LOG4J = ('org.apache.logging.log4j', 'log4j-core', '2.14.1') +_SLF4J = ('org.slf4j', 'slf4j-api', '1.7.36') + + +def _component( + group: str, + artifact: str, + version: str, + logical_path: str = 'app.war > WEB-INF/lib/x.jar', + parent: str = 'app.war', + evidence: str = EVIDENCE_POM_PROPERTIES, + confidence: str = CONFIDENCE_EXACT, +) -> IdentifiedComponent: + return IdentifiedComponent( + group=group, + artifact=artifact, + version=version, + sha1='a' * 40, + sha256='b' * 64, + logical_path=logical_path, + parent=parent, + evidence=evidence, + confidence=confidence, + ) + + +def _build(result: ExtractionResult, artifact_name: str = 'app.war') -> dict: + return cyclonedx_builder.build_bom(artifact_name, result, timestamp=_FIXED_TIMESTAMP) + + +def _normalise(bom: dict) -> dict: + """Blank out digests so a golden document survives zlib differences across the CI matrix. + + Compressed bytes vary with the zlib build, so the digest of a generated fixture archive is not stable across + six Python versions and three operating systems. The digests are asserted separately, against what the + extractor actually computed. + """ + normalised = copy.deepcopy(bom) + for component in normalised.get('components', []): + for digest in component.get('hashes', []): + digest['content'] = f'<{digest["alg"]}>' + + normalised['metadata']['tools'] = [''] + return normalised + + +class TestDocumentStructure: + def test_an_empty_result_is_still_a_valid_document(self) -> None: + bom = _build(ExtractionResult()) + + assert_valid_cyclonedx(bom) + assert bom['components'] == [] + assert bom['metadata']['component']['name'] == 'app.war' + assert bom['metadata']['component']['type'] == 'application' + + def test_metadata_records_provenance(self) -> None: + result = ExtractionResult( + components=[_component(*_GUAVA)], + unidentified=[UnidentifiedArtifact('app.war > WEB-INF/lib/shim.jar', 'c' * 40, 44)], + ) + + properties = {entry['name']: entry['value'] for entry in _build(result)['metadata']['properties']} + + assert properties['cycode:source'] == 'binary-extraction' + assert properties['cycode:coverage'] == '1/2' + assert properties['cycode:graph'] == 'containment' + + def test_graph_kind_reflects_recovered_edges(self) -> None: + result = ExtractionResult(components=[_component(*_GUAVA)], has_real_edges=True) + + properties = {entry['name']: entry['value'] for entry in _build(result)['metadata']['properties']} + + assert properties['cycode:graph'] == 'containment+partial' + + def test_output_is_deterministic(self) -> None: + result = ExtractionResult(components=[_component(*_GUAVA), _component(*_LOG4J)]) + + first = cyclonedx_builder.build_bom_json('app.war', result, timestamp=_FIXED_TIMESTAMP) + second = cyclonedx_builder.build_bom_json('app.war', result, timestamp=_FIXED_TIMESTAMP) + + assert first == second + assert json.loads(first)['specVersion'] == '1.4' + + +class TestComponents: + def test_a_component_carries_its_coordinates_and_digests(self) -> None: + bom = _build(ExtractionResult(components=[_component(*_GUAVA)])) + component = bom['components'][0] + + assert component['bom-ref'] == 'pkg:maven/com.google.guava/guava@31.1-jre' + assert component['purl'] == 'pkg:maven/com.google.guava/guava@31.1-jre' + assert component['group'] == 'com.google.guava' + assert component['name'] == 'guava' + assert component['version'] == '31.1-jre' + assert component['type'] == 'library' + assert component['hashes'] == [ + {'alg': 'SHA-1', 'content': 'a' * 40}, + {'alg': 'SHA-256', 'content': 'b' * 64}, + ] + + def test_evidence_and_confidence_are_recorded(self) -> None: + result = ExtractionResult( + components=[_component('', 'widget', '1.0', evidence=EVIDENCE_MANIFEST, confidence=CONFIDENCE_AMBIGUOUS)] + ) + + properties = {entry['name']: entry['value'] for entry in _build(result)['components'][0]['properties']} + + assert properties['cycode:evidence'] == 'manifest.mf' + assert properties['cycode:confidence'] == 'ambiguous' + + def test_a_component_with_no_group_omits_it_rather_than_inventing_one(self) -> None: + bom = _build(ExtractionResult(components=[_component('', 'widget', '1.0')])) + component = bom['components'][0] + + assert 'group' not in component + assert component['purl'] == 'pkg:maven/widget@1.0' + assert_valid_cyclonedx(bom) + + def test_the_same_coordinate_shipped_twice_is_one_component_with_both_paths(self) -> None: + result = ExtractionResult( + components=[ + _component(*_GUAVA, logical_path='app.war > WEB-INF/lib/guava.jar'), + _component(*_GUAVA, logical_path='app.war > WEB-INF/lib/guava-shadow.jar'), + ] + ) + + bom = _build(result) + + assert len(bom['components']) == 1 + paths = {entry['name']: entry['value'] for entry in bom['components'][0]['properties']}['cycode:path'] + assert paths == 'app.war > WEB-INF/lib/guava.jar, app.war > WEB-INF/lib/guava-shadow.jar' + assert_valid_cyclonedx(bom) + + def test_no_digest_means_no_sha256_entry(self) -> None: + component = IdentifiedComponent( + group='com.acme', + artifact='widget', + version='1.0', + sha1='a' * 40, + logical_path='app.war > WEB-INF/lib/widget.jar', + parent='app.war', + evidence=EVIDENCE_POM_PROPERTIES, + confidence=CONFIDENCE_EXACT, + ) + + bom = _build(ExtractionResult(components=[component])) + + assert bom['components'][0]['hashes'] == [{'alg': 'SHA-1', 'content': 'a' * 40}] + assert_valid_cyclonedx(bom) + + +class TestDependencyGraph: + def test_every_component_appears_even_with_no_edges(self) -> None: + # an absent entry reads as missing data; an empty dependsOn reads as "depends on nothing" + result = ExtractionResult(components=[_component(*_GUAVA), _component(*_LOG4J)]) + + refs = [entry['ref'] for entry in _build(result)['dependencies']] + + assert refs == [ + 'app.war', + 'pkg:maven/com.google.guava/guava@31.1-jre', + 'pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1', + ] + + def test_edges_are_emitted_and_sorted(self) -> None: + result = ExtractionResult( + components=[_component(*_GUAVA), _component(*_LOG4J)], + dependency_edges={ + 'app.war': [ + 'pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1', + 'pkg:maven/com.google.guava/guava@31.1-jre', + ] + }, + ) + + bom = _build(result) + root_entry = next(entry for entry in bom['dependencies'] if entry['ref'] == 'app.war') + + assert root_entry['dependsOn'] == [ + 'pkg:maven/com.google.guava/guava@31.1-jre', + 'pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1', + ] + assert_valid_cyclonedx(bom) + + def test_an_edge_to_an_unknown_component_is_dropped(self) -> None: + # a pom can name a dependency that was never actually shipped inside the artifact + result = ExtractionResult( + components=[_component(*_GUAVA)], + dependency_edges={'app.war': ['pkg:maven/never/shipped@1.0']}, + ) + + bom = _build(result) + + assert bom['dependencies'][0]['dependsOn'] == [] + assert_valid_cyclonedx(bom) + + +class TestAgainstRealArtifacts: + def _extract(self, tmp_path: Path, name: str, content: bytes) -> tuple[dict, ExtractionResult]: + path = tmp_path / name + path.write_bytes(content) + + extractor = JavaArchiveExtractor() + result = extractor.identify(extractor.extract(str(path))) + return cyclonedx_builder.build_bom(name, result, timestamp=_FIXED_TIMESTAMP), result + + def test_a_war_produces_a_valid_document(self, tmp_path: Path) -> None: + war = fixtures.war_bytes( + libraries={ + 'guava-31.1-jre.jar': fixtures.library_jar(*_GUAVA), + 'log4j-core-2.14.1.jar': fixtures.library_jar(*_LOG4J), + } + ) + + bom, result = self._extract(tmp_path, 'payments.war', war) + + assert_valid_cyclonedx(bom) + assert [component['purl'] for component in bom['components']] == [ + 'pkg:maven/com.google.guava/guava@31.1-jre', + 'pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1', + ] + # the digests in the document are the ones the extractor actually computed + emitted = {component['hashes'][0]['content'] for component in bom['components']} + assert emitted == {component.sha1 for component in result.components} + + def test_an_ear_nests_containment_edges(self, tmp_path: Path) -> None: + inner_war = fixtures.war_bytes(libraries={'log4j-core.jar': fixtures.library_jar(*_LOG4J)}) + ear = fixtures.ear_bytes( + modules={'web.war': inner_war}, + libraries={'guava.jar': fixtures.library_jar(*_GUAVA)}, + ) + + bom, _ = self._extract(tmp_path, 'payments.ear', ear) + edges = {entry['ref']: entry['dependsOn'] for entry in bom['dependencies']} + + assert_valid_cyclonedx(bom) + # the war is not a Maven artifact here, so the jar inside it attaches to the nearest thing we can name + assert 'pkg:maven/com.google.guava/guava@31.1-jre' in edges['payments.ear'] + assert 'pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1' in edges['payments.ear'] + + def test_a_real_pom_edge_replaces_the_containment_edge(self, tmp_path: Path) -> None: + # guava declares slf4j, so slf4j hangs off guava rather than off the application + pom = b""" + + com.google.guava + guava + 31.1-jre + + + org.slf4j + slf4j-api + 1.7.36 + + + """ + + guava = fixtures.archive_bytes( + files={ + fixtures.pom_properties_entry_name(*_GUAVA[:2]): fixtures.pom_properties(*_GUAVA), + f'META-INF/maven/{_GUAVA[0]}/{_GUAVA[1]}/pom.xml': pom, + } + ) + war = fixtures.war_bytes(libraries={'guava.jar': guava, 'slf4j-api.jar': fixtures.library_jar(*_SLF4J)}) + + bom, result = self._extract(tmp_path, 'payments.war', war) + edges = {entry['ref']: entry['dependsOn'] for entry in bom['dependencies']} + + assert_valid_cyclonedx(bom) + assert result.has_real_edges is True + assert edges['pkg:maven/com.google.guava/guava@31.1-jre'] == ['pkg:maven/org.slf4j/slf4j-api@1.7.36'] + # the containment edge was dropped in favour of the real one + assert edges['payments.war'] == ['pkg:maven/com.google.guava/guava@31.1-jre'] + + def test_a_shaded_jar_yields_one_component_per_aggregated_project(self, tmp_path: Path) -> None: + shaded = fixtures.archive_bytes( + files={ + fixtures.pom_properties_entry_name(*_GUAVA[:2]): fixtures.pom_properties(*_GUAVA), + fixtures.pom_properties_entry_name(*_SLF4J[:2]): fixtures.pom_properties(*_SLF4J), + } + ) + + bom, _ = self._extract(tmp_path, 'uber.jar', fixtures.war_bytes(libraries={'shaded.jar': shaded})) + + assert_valid_cyclonedx(bom) + assert {component['purl'] for component in bom['components']} == { + 'pkg:maven/com.google.guava/guava@31.1-jre', + 'pkg:maven/org.slf4j/slf4j-api@1.7.36', + } + + def test_a_tier_three_component_is_marked_ambiguous(self, tmp_path: Path) -> None: + manifest_only = fixtures.archive_bytes( + files={ + 'META-INF/MANIFEST.MF': fixtures.manifest( + Implementation_Title='mystery-lib', + Implementation_Version='4.2', + Implementation_Vendor_Id='com.acme', + ) + } + ) + + bom, _ = self._extract(tmp_path, 'app.war', fixtures.war_bytes(libraries={'mystery.jar': manifest_only})) + properties = {entry['name']: entry['value'] for entry in bom['components'][0]['properties']} + + assert_valid_cyclonedx(bom) + assert properties['cycode:confidence'] == 'ambiguous' + assert properties['cycode:evidence'] == 'manifest.mf' + assert bom['components'][0]['purl'] == 'pkg:maven/com.acme/mystery-lib@4.2' + + +class TestGoldenDocument: + """Catches silent regressions in ordering, deduplication and edge construction.""" + + _EXPECTED: ClassVar[dict] = { + 'bomFormat': 'CycloneDX', + 'specVersion': '1.4', + 'version': 1, + 'metadata': { + 'timestamp': _FIXED_TIMESTAMP, + 'tools': [''], + 'component': {'bom-ref': 'payments.war', 'type': 'application', 'name': 'payments.war'}, + 'properties': [ + {'name': 'cycode:source', 'value': 'binary-extraction'}, + {'name': 'cycode:graph', 'value': 'containment'}, + {'name': 'cycode:coverage', 'value': '2/3'}, + ], + }, + 'components': [ + { + 'bom-ref': 'pkg:maven/com.google.guava/guava@31.1-jre', + 'type': 'library', + 'name': 'guava', + 'version': '31.1-jre', + 'purl': 'pkg:maven/com.google.guava/guava@31.1-jre', + 'hashes': [{'alg': 'SHA-1', 'content': ''}, {'alg': 'SHA-256', 'content': ''}], + 'properties': [ + {'name': 'cycode:evidence', 'value': 'pom.properties'}, + {'name': 'cycode:confidence', 'value': 'exact'}, + {'name': 'cycode:path', 'value': 'payments.war > WEB-INF/lib/guava-31.1-jre.jar'}, + ], + 'group': 'com.google.guava', + }, + { + 'bom-ref': 'pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1', + 'type': 'library', + 'name': 'log4j-core', + 'version': '2.14.1', + 'purl': 'pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1', + 'hashes': [{'alg': 'SHA-1', 'content': ''}, {'alg': 'SHA-256', 'content': ''}], + 'properties': [ + {'name': 'cycode:evidence', 'value': 'pom.properties'}, + {'name': 'cycode:confidence', 'value': 'exact'}, + {'name': 'cycode:path', 'value': 'payments.war > WEB-INF/lib/log4j-core-2.14.1.jar'}, + ], + 'group': 'org.apache.logging.log4j', + }, + ], + 'dependencies': [ + { + 'ref': 'payments.war', + 'dependsOn': [ + 'pkg:maven/com.google.guava/guava@31.1-jre', + 'pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1', + ], + }, + {'ref': 'pkg:maven/com.google.guava/guava@31.1-jre', 'dependsOn': []}, + {'ref': 'pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1', 'dependsOn': []}, + ], + } + + @pytest.fixture + def bom(self, tmp_path: Path) -> dict: + war = fixtures.war_bytes( + libraries={ + 'guava-31.1-jre.jar': fixtures.library_jar(*_GUAVA), + 'log4j-core-2.14.1.jar': fixtures.library_jar(*_LOG4J), + 'internal-shim.jar': fixtures.archive_bytes(files={'com/acme/Shim.class': b'\xca\xfe\xba\xbe'}), + } + ) + path = tmp_path / 'payments.war' + path.write_bytes(war) + + extractor = JavaArchiveExtractor() + result = extractor.identify(extractor.extract(str(path))) + return cyclonedx_builder.build_bom('payments.war', result, timestamp=_FIXED_TIMESTAMP) + + def test_matches_the_golden_document(self, bom: dict) -> None: + assert _normalise(bom) == self._EXPECTED + + def test_the_golden_document_is_valid(self, bom: dict) -> None: + assert_valid_cyclonedx(bom) diff --git a/tests/cli/files_collector/binary/test_hostile_output.py b/tests/cli/files_collector/binary/test_hostile_output.py new file mode 100644 index 00000000..02af13ad --- /dev/null +++ b/tests/cli/files_collector/binary/test_hostile_output.py @@ -0,0 +1,223 @@ +"""Regression tests for the findings raised by the pre-merge security review. + +Each one is a concrete attack that worked against an earlier revision of this feature. They are grouped here +rather than spread across the suite so a reviewer can see the whole class of "the archive author controls what we +render and what we read" problems in one place. +""" + +import hashlib +import io +import os +import zipfile +from pathlib import Path +from unittest.mock import MagicMock + +import pytest +import typer +from rich.console import Console + +from cycode.cli.files_collector.binary.collector import collect_binary_documents +from cycode.cli.files_collector.binary.identifiers import pom_xml +from cycode.cli.files_collector.binary.identifiers.pom_xml import UnsafeXmlError +from cycode.cli.files_collector.binary.java_extractor import JavaArchiveExtractor +from cycode.cli.files_collector.binary.safe_zip import SafeZip +from cycode.cli.printers.utils import binary_report +from tests.cli.files_collector.binary import fixtures + +_LOG4J_VULNERABLE = ('org.apache.logging.log4j', 'log4j-core', '2.14.1') +_LOG4J_PATCHED = ('org.apache.logging.log4j', 'log4j-core', '2.17.1') + + +def _maven_jar(group: str, artifact: str, version: str) -> bytes: + return fixtures.archive_bytes( + files={fixtures.pom_properties_entry_name(group, artifact): fixtures.pom_properties(group, artifact, version)} + ) + + +def _write(tmp_path: Path, name: str, content: bytes) -> str: + path = tmp_path / name + path.write_bytes(content) + return str(path) + + +@pytest.mark.filterwarnings('ignore:Duplicate name:UserWarning') +class TestDuplicateEntryNames: + """A crafted archive must not be able to hide a vulnerable jar behind a patched one of the same name. + + ``ZipFile.open(name)`` resolves through a dict of name -> ZipInfo in which the *last* duplicate wins, so + reading by name let an artifact author decide which of two same-named entries we actually digested. The + vulnerable copy was never read, and the scan came back clean. + """ + + def _evasion_war(self) -> bytes: + vulnerable = _maven_jar(*_LOG4J_VULNERABLE) + patched = _maven_jar(*_LOG4J_PATCHED) + + buffer = io.BytesIO() + with zipfile.ZipFile(buffer, 'w', zipfile.ZIP_DEFLATED) as archive: + archive.writestr('WEB-INF/web.xml', '') + archive.writestr('WEB-INF/lib/log4j-core.jar', vulnerable) + archive.writestr('WEB-INF/lib/log4j-core.jar', patched) + + return buffer.getvalue() + + def test_both_duplicates_are_read_from_their_own_records(self, tmp_path: Path) -> None: + vulnerable = _maven_jar(*_LOG4J_VULNERABLE) + patched = _maven_jar(*_LOG4J_PATCHED) + path = _write(tmp_path, 'evasion.war', self._evasion_war()) + + extractor = JavaArchiveExtractor() + result = extractor.identify(extractor.extract(path)) + + digests = {component.sha1 for component in result.components} + assert hashlib.sha1(vulnerable, usedforsecurity=False).hexdigest() in digests + assert hashlib.sha1(patched, usedforsecurity=False).hexdigest() in digests + + def test_the_vulnerable_version_is_still_reported(self, tmp_path: Path) -> None: + path = _write(tmp_path, 'evasion.war', self._evasion_war()) + + extractor = JavaArchiveExtractor() + result = extractor.identify(extractor.extract(path)) + versions = {component.version for component in result.components} + + # the whole point: a duplicate name must not be able to suppress the vulnerable copy + assert '2.14.1' in versions + assert '2.17.1' in versions + + def test_each_entry_reports_its_own_digest(self, tmp_path: Path) -> None: + path = _write(tmp_path, 'evasion.war', self._evasion_war()) + + with SafeZip.open(path) as archive: + entries = [entry for entry in archive.entries() if entry.name.endswith('.jar')] + digests = [archive.read(entry).sha1 for entry in entries] + + assert len(entries) == 2 + assert digests[0] != digests[1], 'both duplicates resolved to the same bytes' + + +class TestUntrustedTextIsSanitisedForDisplay: + """Entry names, manifest values and coordinates are authored by whoever built the artifact.""" + + def test_rich_markup_is_neutralised(self) -> None: + hostile = 'app.jar > BOOT-INF/lib/[link=javascript:alert(1)]lib[/link].jar' + + rendered = binary_report.for_display(hostile) + + # rich neutralises a tag by backslash-escaping its opening bracket + assert r'\[link=' in rendered + assert r'\[/link]' in rendered + + def test_a_hostile_name_survives_an_html_export_without_becoming_a_link(self) -> None: + # --export-type html writes this to a file a security engineer opens, or CI publishes as an artifact + hostile = 'lib/[link=javascript:alert(document.domain)]click[/link].jar' + console = Console(file=io.StringIO(), record=True, width=200, force_terminal=False) + + console.print(f' {binary_report.for_display(hostile)}') + html = console.export_html() + + assert 'javascript:' not in html + assert ' None: + rendered = binary_report.for_display(f'lib/evil{control}name.jar') + + assert control not in rendered + + def test_a_forged_coverage_line_cannot_be_injected(self) -> None: + # the coverage line is this feature's core claim; an artifact must not be able to forge one + hostile = 'x.jar\n\n99 identified | 0 unidentified | 0 vulnerabilities\n' + + rendered = binary_report.for_display(hostile) + + assert '\n' not in rendered + + def test_an_absurdly_long_name_is_truncated(self) -> None: + rendered = binary_report.for_display('a' * 5000) + + assert len(rendered) < 250 + + def test_an_ordinary_name_is_left_readable(self) -> None: + ordinary = 'payments.war > WEB-INF/lib/log4j-core-2.14.1.jar' + + assert binary_report.for_display(ordinary) == ordinary + + def test_the_bom_keeps_the_true_name(self, tmp_path: Path) -> None: + # sanitisation is presentation only; the uploaded document must describe what is really there + hostile_name = 'WEB-INF/lib/[weird]name.jar' + buffer = io.BytesIO() + with zipfile.ZipFile(buffer, 'w', zipfile.ZIP_DEFLATED) as archive: + archive.writestr('WEB-INF/web.xml', '') + archive.writestr(hostile_name, _maven_jar(*_LOG4J_VULNERABLE)) + + path = _write(tmp_path, 'app.war', buffer.getvalue()) + extractor = JavaArchiveExtractor() + result = extractor.identify(extractor.extract(path)) + + assert any('[weird]name.jar' in component.logical_path for component in result.components) + + +class TestKeepBomDoesNotFollowSymlinks: + """The scanned tree is untrusted: an unpacked vendor drop can plant a symlink at the output path.""" + + @pytest.fixture + def mock_ctx(self) -> typer.Context: + ctx = MagicMock(spec=typer.Context) + ctx.obj = {'progress_bar': None, 'binary_max_depth': 3, 'keep_bom': True, 'monitor': False} + return ctx + + @pytest.mark.skipif(os.name == 'nt', reason='symlink semantics and O_NOFOLLOW differ on Windows') + def test_a_symlinked_output_path_is_refused(self, mock_ctx: typer.Context, tmp_path: Path) -> None: + drop = tmp_path / 'drop' + drop.mkdir() + artifact = drop / 'app.war' + artifact.write_bytes(fixtures.war_bytes(libraries={'guava.jar': _maven_jar('com.google.guava', 'g', '1')})) + + victim = tmp_path / 'precious.txt' + victim.write_text('do not overwrite me') + (drop / 'app.war.bom.json').symlink_to(victim) + + collect_binary_documents(mock_ctx, (str(drop),)) + + # the scan still succeeds; the planted target is untouched + assert victim.read_text() == 'do not overwrite me' + + def test_an_ordinary_path_is_still_written(self, mock_ctx: typer.Context, tmp_path: Path) -> None: + artifact = tmp_path / 'app.war' + artifact.write_bytes(fixtures.war_bytes(libraries={'guava.jar': _maven_jar('com.google.guava', 'g', '1')})) + + collect_binary_documents(mock_ctx, (str(artifact),)) + + assert (tmp_path / 'app.war.bom.json').exists() + + +class TestXmlEncodingBypass: + """A byte-level DOCTYPE guard is bypassable; the guard now runs on decoded text.""" + + _BILLION_LAUGHS = ( + '' + ']>' + '&lol2;' + ) + + def test_a_utf16_encoded_declaration_is_refused(self) -> None: + # expat sniffs the byte-order mark and would parse the DTD that a bytes-level pattern never matched + payload = self._BILLION_LAUGHS.encode('utf-16') + + with pytest.raises(UnsafeXmlError, match=r'not valid UTF-8|document type or entity declaration'): + pom_xml.parse_xml(payload) + + def test_a_utf16_hostile_pom_yields_no_edges(self) -> None: + assert pom_xml.parse_dependencies(self._BILLION_LAUGHS.encode('utf-16')) == [] + + def test_a_utf8_bom_is_still_accepted(self) -> None: + pom = ( + '' + 'ab' + '' + ) + + assert [d.coordinate_key for d in pom_xml.parse_dependencies(pom.encode('utf-8-sig'))] == ['a:b'] diff --git a/tests/cli/files_collector/binary/test_identifiers.py b/tests/cli/files_collector/binary/test_identifiers.py new file mode 100644 index 00000000..6f699c8e --- /dev/null +++ b/tests/cli/files_collector/binary/test_identifiers.py @@ -0,0 +1,318 @@ +import pytest + +from cycode.cli.files_collector.binary.identifiers import manifest_mf, pom_properties, pom_xml +from cycode.cli.files_collector.binary.identifiers.pom_xml import UnsafeXmlError + +_POM_NAMESPACE = 'http://maven.apache.org/POM/4.0.0' + + +class TestPomProperties: + def test_a_maven_written_file(self) -> None: + payload = ( + b'#Generated by Maven\n' + b'#Mon Aug 26 12:00:00 UTC 2026\n' + b'groupId=com.google.guava\n' + b'artifactId=guava\n' + b'version=31.1-jre\n' + ) + + coordinates = pom_properties.identify(payload) + + assert coordinates.group == 'com.google.guava' + assert coordinates.artifact == 'guava' + assert coordinates.version == '31.1-jre' + + def test_colon_separator_and_surrounding_whitespace(self) -> None: + payload = b'groupId : com.acme \nartifactId: widget\nversion:\t2.0\n' + + coordinates = pom_properties.identify(payload) + + assert (coordinates.group, coordinates.artifact, coordinates.version) == ('com.acme', 'widget', '2.0') + + def test_a_byte_order_mark_is_tolerated(self) -> None: + payload = 'groupId=com.acme\nartifactId=widget\nversion=1.0\n'.encode() + + assert pom_properties.identify(payload).group == 'com.acme' + + @pytest.mark.parametrize( + 'payload', + [ + b'groupId=com.acme\nartifactId=widget\n', # no version + b'artifactId=widget\nversion=1.0\n', # no group + b'groupId=com.acme\nversion=1.0\n', # no artifact + b'', + b'# nothing but a comment\n', + b'! bang comments are comments too\n', + ], + ) + def test_an_incomplete_set_identifies_nothing(self, payload: bytes) -> None: + assert pom_properties.identify(payload) is None + + def test_undecodable_bytes_do_not_raise(self) -> None: + # one malformed file must not fail the scan of an entire deployable + assert pom_properties.identify(b'\xff\xfe\x00garbage') is None + + +class TestManifest: + def test_implementation_attributes(self) -> None: + payload = ( + b'Manifest-Version: 1.0\n' + b'Implementation-Title: log4j-core\n' + b'Implementation-Version: 2.14.1\n' + b'Implementation-Vendor-Id: org.apache.logging.log4j\n' + ) + + identity = manifest_mf.identify(payload) + + assert identity.coordinates.artifact == 'log4j-core' + assert identity.coordinates.version == '2.14.1' + # a declared vendor id is a declared group id, not an inference + assert identity.coordinates.group == 'org.apache.logging.log4j' + assert identity.source_attribute == manifest_mf.IMPLEMENTATION_TITLE + + def test_a_line_wrapped_at_seventy_two_bytes_is_rejoined(self) -> None: + # the format wraps at 72 bytes and continues with a leading single space + payload = ( + b'Manifest-Version: 1.0\n' + b'Implementation-Title: org.eclipse.jetty.util.some.very.lon\n' + b' g.artifact.name\n' + b'Implementation-Version: 9.4.44\n' + b'Implementation-Vendor-Id: org.eclipse.jetty\n' + ) + + identity = manifest_mf.identify(payload) + + assert identity.coordinates.artifact == 'org.eclipse.jetty.util.some.very.long.artifact.name' + + def test_a_multibyte_character_split_across_the_wrap(self) -> None: + # the wrap counts bytes, not characters, so continuation must be joined before decoding + value = 'café-parser' + encoded = value.encode('utf-8') + split_at = encoded.index(b'\xc3') + 1 # mid-character + payload = ( + b'Manifest-Version: 1.0\n' + b'Implementation-Title: ' + encoded[:split_at] + b'\n ' + encoded[split_at:] + b'\n' + b'Implementation-Version: 1.0\n' + b'Implementation-Vendor-Id: com.acme\n' + ) + + attributes = manifest_mf.parse_manifest(payload) + + assert attributes[manifest_mf.IMPLEMENTATION_TITLE] == value + + def test_carriage_returns_are_handled(self) -> None: + payload = ( + b'Manifest-Version: 1.0\r\nImplementation-Title: widget\r\nImplementation-Version: 1.0\r\n' + b'Implementation-Vendor-Id: com.acme\r\n' + ) + + assert manifest_mf.identify(payload).coordinates.artifact == 'widget' + + def test_only_the_main_section_is_read(self) -> None: + # per-entry sections describe individual files, not the artifact + payload = ( + b'Manifest-Version: 1.0\n' + b'Implementation-Title: real-artifact\n' + b'Implementation-Version: 1.0\n' + b'Implementation-Vendor-Id: com.acme\n' + b'\n' + b'Name: com/acme/Other.class\n' + b'Implementation-Title: not-the-artifact\n' + b'Implementation-Version: 9.9\n' + ) + + assert manifest_mf.identify(payload).coordinates.artifact == 'real-artifact' + + def test_osgi_bundle_attributes_are_the_fallback(self) -> None: + payload = ( + b'Manifest-Version: 1.0\nBundle-SymbolicName: com.acme.thing;singleton:=true\nBundle-Version: 3.1.0\n' + b'Implementation-Vendor-Id: com.acme\n' + ) + + identity = manifest_mf.identify(payload) + + # directives are not part of the name + assert identity.coordinates.artifact == 'com.acme.thing' + assert identity.coordinates.version == '3.1.0' + assert identity.source_attribute == manifest_mf.BUNDLE_SYMBOLIC_NAME + + def test_automatic_module_name_is_the_last_resort(self) -> None: + payload = ( + b'Manifest-Version: 1.0\nAutomatic-Module-Name: com.acme.widget\nImplementation-Version: 4.2\n' + b'Implementation-Vendor-Id: com.acme\n' + ) + + identity = manifest_mf.identify(payload) + + assert identity.coordinates.artifact == 'com.acme.widget' + assert identity.source_attribute == manifest_mf.AUTOMATIC_MODULE_NAME + + def test_no_group_means_no_coordinate(self) -> None: + # a purl with no namespace matches nothing, so it is not a low-confidence answer, it is noise + payload = b'Manifest-Version: 1.0\nImplementation-Title: widget\nImplementation-Version: 1.0\n' + + assert manifest_mf.identify(payload) is None + + def test_a_product_name_is_not_an_artifact_id_and_falls_through(self) -> None: + # seen in the wild: activation.jar titles itself after the server it shipped with + payload = ( + b'Manifest-Version: 1.0\n' + b'Implementation-Title: Sun Java System Application Server\n' + b'Implementation-Version: 1.1\n' + b'Implementation-Vendor-Id: com.sun\n' + b'Bundle-SymbolicName: javax.activation\n' + b'Bundle-Version: 1.1.0\n' + ) + + identity = manifest_mf.identify(payload) + + assert identity.coordinates.artifact == 'javax.activation' + assert identity.source_attribute == manifest_mf.BUNDLE_SYMBOLIC_NAME + + def test_a_build_banner_is_not_a_version(self) -> None: + # seen in the wild: a vendor connector stamps its build time into Implementation-Version + payload = ( + b'Manifest-Version: 1.0\n' + b'Implementation-Title: com.sap.conn.jco\n' + b'Implementation-Version: 20100905 1938 [3.0.6 (2010-08-24)]\n' + b'Implementation-Vendor-Id: com.sap\n' + ) + + assert manifest_mf.identify(payload) is None + + @pytest.mark.parametrize('version', ['2.14.1', '1.0M10', '9.4.44.v20210927', '3.0.0-SNAPSHOT', '1.13.0', '2']) + def test_real_version_shapes_are_accepted(self, version: str) -> None: + assert manifest_mf.is_version_shaped(version) + + @pytest.mark.parametrize('version', ['', 'v1.0', '20100905 1938 [3.0.6]', 'Build 2024-03-01', '1.0 beta']) + def test_things_that_are_not_versions(self, version: str) -> None: + assert not manifest_mf.is_version_shaped(version) + + @pytest.mark.parametrize( + 'payload', + [ + b'Manifest-Version: 1.0\nImplementation-Title: widget\nImplementation-Vendor-Id: a.b\n', # no version + b'Manifest-Version: 1.0\nImplementation-Version: 1.0\nImplementation-Vendor-Id: a.b\n', # no name + b'Manifest-Version: 1.0\nMain-Class: com.acme.App\n', + b'Manifest-Version: 1.0\nImplementation-Title: w\nImplementation-Version: 1\nImplementation-Vendor-Id: A B\n', # noqa: E501 + b'', + ], + ) + def test_a_manifest_that_says_nothing_usable(self, payload: bytes) -> None: + assert manifest_mf.identify(payload) is None + + +class TestPomXml: + def test_direct_dependencies(self) -> None: + payload = b""" + + + + com.google.guava + guava + 31.1-jre + + + org.slf4j + slf4j-api + + + """ + + dependencies = pom_xml.parse_dependencies(payload) + + assert [d.coordinate_key for d in dependencies] == ['com.google.guava:guava', 'org.slf4j:slf4j-api'] + assert dependencies[0].version == '31.1-jre' + assert dependencies[1].version is None + + def test_the_maven_namespace_is_handled(self) -> None: + payload = f""" + + + com.acmewidget + + """.encode() + + assert [d.coordinate_key for d in pom_xml.parse_dependencies(payload)] == ['com.acme:widget'] + + @pytest.mark.parametrize('scope', ['test', 'provided', 'system', 'TEST']) + def test_scopes_that_do_not_ship_are_excluded(self, scope: str) -> None: + payload = f""" + junitjunit{scope} + """.encode() + + assert pom_xml.parse_dependencies(payload) == [] + + def test_runtime_and_compile_scopes_are_kept(self) -> None: + payload = b""" + abruntime + cdcompile + """ + + assert [d.coordinate_key for d in pom_xml.parse_dependencies(payload)] == ['a:b', 'c:d'] + + def test_dependency_management_is_not_an_edge(self) -> None: + # these declare versions for modules that may never be depended on + payload = b""" + + + managedonly1.0 + + + + realdep + + """ + + assert [d.coordinate_key for d in pom_xml.parse_dependencies(payload)] == ['real:dep'] + + def test_a_pom_with_no_dependencies(self) -> None: + assert pom_xml.parse_dependencies(b'lonely') == [] + + +class TestXmlSafety: + _BILLION_LAUGHS = b""" + + + + ]> + &lol3;""" + + _EXTERNAL_ENTITY = b""" + ]> + &xxe;""" + + def test_billion_laughs_is_refused_before_parsing(self) -> None: + with pytest.raises(UnsafeXmlError, match='document type or entity declaration'): + pom_xml.parse_xml(self._BILLION_LAUGHS) + + def test_external_entity_is_refused_before_parsing(self) -> None: + with pytest.raises(UnsafeXmlError, match='document type or entity declaration'): + pom_xml.parse_xml(self._EXTERNAL_ENTITY) + + @pytest.mark.parametrize( + 'payload', + [ + b'', # lowercase + b'', # spaced + b'', + ], + ) + def test_declarations_are_refused_however_they_are_written(self, payload: bytes) -> None: + with pytest.raises(UnsafeXmlError): + pom_xml.parse_xml(payload) + + def test_a_hostile_pom_yields_no_edges_rather_than_raising(self) -> None: + # one bad pom inside a deployable must not fail the whole scan + assert pom_xml.parse_dependencies(self._BILLION_LAUGHS) == [] + + def test_malformed_xml_yields_no_edges(self) -> None: + assert pom_xml.parse_dependencies(b'') == [] + + def test_predefined_entities_still_work(self) -> None: + payload = b'' + payload += b'a&bc' + payload += b'' + + assert pom_xml.parse_dependencies(payload)[0].group == 'a&b' diff --git a/tests/cli/files_collector/binary/test_java_extractor.py b/tests/cli/files_collector/binary/test_java_extractor.py new file mode 100644 index 00000000..13d934d6 --- /dev/null +++ b/tests/cli/files_collector/binary/test_java_extractor.py @@ -0,0 +1,413 @@ +import hashlib +from pathlib import Path + +import pytest + +from cycode.cli.exceptions.custom_exceptions import BinaryExtractionError, UnsafeArchiveEntryError +from cycode.cli.files_collector.binary.base_extractor import ArchiveEntry +from cycode.cli.files_collector.binary.java_extractor import ( + JavaArchiveExtractor, + is_library_entry, + is_metadata_entry, +) +from tests.cli.files_collector.binary import fixtures + +_GUAVA = ('com.google.guava', 'guava', '31.1-jre') +_LOG4J = ('org.apache.logging.log4j', 'log4j-core', '2.14.1') + + +@pytest.fixture +def extractor() -> JavaArchiveExtractor: + return JavaArchiveExtractor() + + +def _write(tmp_path: Path, name: str, content: bytes) -> str: + path = tmp_path / name + path.write_bytes(content) + return str(path) + + +def _by_logical_path(entries: list[ArchiveEntry]) -> dict[str, ArchiveEntry]: + return {entry.logical_path: entry for entry in entries} + + +class TestHandles: + @pytest.mark.parametrize('name', ['app.jar', 'app.war', 'app.ear', 'DIST/App.WAR', '/abs/path/x.jar']) + def test_java_archives_are_claimed(self, extractor: JavaArchiveExtractor, name: str) -> None: + assert extractor.handles(name) is True + + @pytest.mark.parametrize('name', ['app.zip', 'pom.xml', 'app.tar.gz', 'jar', 'app.jarfile']) + def test_everything_else_is_declined(self, extractor: JavaArchiveExtractor, name: str) -> None: + assert extractor.handles(name) is False + + +class TestLayoutRecognition: + @pytest.mark.parametrize( + 'name', + [ + 'WEB-INF/lib/guava.jar', + 'BOOT-INF/lib/guava.jar', + 'APP-INF/lib/guava.jar', + 'lib/guava.jar', + ], + ) + def test_library_directories(self, name: str) -> None: + assert is_library_entry(name, '.war') is True + + def test_ear_modules_are_recognised_anywhere(self) -> None: + assert is_library_entry('web.war', '.ear') is True + assert is_library_entry('modules/ejb.jar', '.ear') is True + + def test_a_stray_jar_outside_a_library_directory_is_not_a_module(self) -> None: + assert is_library_entry('docs/samples/example.jar', '.war') is False + + def test_non_archives_are_never_libraries(self) -> None: + assert is_library_entry('WEB-INF/lib/readme.txt', '.war') is False + + @pytest.mark.parametrize( + 'name', + [ + 'META-INF/MANIFEST.MF', + 'META-INF/manifest.mf', + 'META-INF/maven/com.google.guava/guava/pom.properties', + 'META-INF/maven/com.google.guava/guava/pom.xml', + ], + ) + def test_metadata_entries(self, name: str) -> None: + assert is_metadata_entry(name) is True + + @pytest.mark.parametrize( + 'name', + [ + 'com/google/common/Thing.class', + 'META-INF/maven/pom.properties', + 'META-INF/maven/g/a/extra/pom.xml', + 'META-INF/LICENSE', + ], + ) + def test_non_metadata_entries(self, name: str) -> None: + assert is_metadata_entry(name) is False + + +class TestPlainJar: + def test_root_entry_describes_the_artifact(self, extractor: JavaArchiveExtractor, tmp_path: Path) -> None: + content = fixtures.library_jar(*_GUAVA) + path = _write(tmp_path, 'guava.jar', content) + + entries = extractor.extract(path) + root = entries[0] + + assert root.logical_path == 'guava.jar' + assert root.depth == 0 + assert root.parent is None + assert root.is_archive is True + assert root.sha1 == hashlib.sha1(content, usedforsecurity=False).hexdigest() + assert root.size == len(content) + + def test_metadata_is_carried_but_class_files_are_not(self, extractor: JavaArchiveExtractor, tmp_path: Path) -> None: + path = _write(tmp_path, 'guava.jar', fixtures.library_jar(*_GUAVA)) + + entries = extractor.extract(path) + logical_paths = [entry.logical_path for entry in entries] + + assert 'guava.jar > META-INF/MANIFEST.MF' in logical_paths + assert 'guava.jar > META-INF/maven/com.google.guava/guava/pom.properties' in logical_paths + assert not any('.class' in path for path in logical_paths) + + def test_metadata_payload_is_readable(self, extractor: JavaArchiveExtractor, tmp_path: Path) -> None: + path = _write(tmp_path, 'guava.jar', fixtures.library_jar(*_GUAVA)) + + entries = _by_logical_path(extractor.extract(path)) + pom = entries['guava.jar > META-INF/maven/com.google.guava/guava/pom.properties'] + + assert pom.payload is not None + assert b'version=31.1-jre' in pom.payload + assert pom.is_archive is False + assert pom.parent == 'guava.jar' + assert pom.depth == 1 + + +class TestWar: + def test_web_inf_lib_jars_are_found_and_opened(self, extractor: JavaArchiveExtractor, tmp_path: Path) -> None: + guava = fixtures.library_jar(*_GUAVA) + war = fixtures.war_bytes(libraries={'guava-31.1-jre.jar': guava}) + path = _write(tmp_path, 'payments.war', war) + + entries = _by_logical_path(extractor.extract(path)) + jar_path = 'payments.war > WEB-INF/lib/guava-31.1-jre.jar' + + assert jar_path in entries + assert entries[jar_path].is_archive is True + assert entries[jar_path].depth == 1 + assert entries[jar_path].parent == 'payments.war' + assert entries[jar_path].sha1 == hashlib.sha1(guava, usedforsecurity=False).hexdigest() + + # opened, so its own metadata came back with it + assert f'{jar_path} > META-INF/maven/com.google.guava/guava/pom.properties' in entries + + def test_nested_jar_bytes_are_not_retained(self, extractor: JavaArchiveExtractor, tmp_path: Path) -> None: + war = fixtures.war_bytes(libraries={'guava.jar': fixtures.library_jar(*_GUAVA)}) + path = _write(tmp_path, 'payments.war', war) + + entries = _by_logical_path(extractor.extract(path)) + + # holding every nested jar would mean holding the whole deployable in memory + assert entries['payments.war > WEB-INF/lib/guava.jar'].payload is None + + def test_several_libraries(self, extractor: JavaArchiveExtractor, tmp_path: Path) -> None: + war = fixtures.war_bytes( + libraries={ + 'guava.jar': fixtures.library_jar(*_GUAVA), + 'log4j-core.jar': fixtures.library_jar(*_LOG4J), + } + ) + path = _write(tmp_path, 'payments.war', war) + + entries = extractor.extract(path) + archives = [entry.logical_path for entry in entries if entry.is_archive] + + assert archives == [ + 'payments.war', + 'payments.war > WEB-INF/lib/guava.jar', + 'payments.war > WEB-INF/lib/log4j-core.jar', + ] + + +class TestSpringBoot: + def test_boot_inf_lib_is_recognised(self, extractor: JavaArchiveExtractor, tmp_path: Path) -> None: + boot = fixtures.boot_jar_bytes(libraries={'log4j-core-2.14.1.jar': fixtures.library_jar(*_LOG4J)}) + path = _write(tmp_path, 'app.jar', boot) + + entries = _by_logical_path(extractor.extract(path)) + jar_path = 'app.jar > BOOT-INF/lib/log4j-core-2.14.1.jar' + + assert jar_path in entries + assert f'{jar_path} > META-INF/maven/org.apache.logging.log4j/log4j-core/pom.properties' in entries + + +class TestEar: + def test_modules_and_app_inf_lib(self, extractor: JavaArchiveExtractor, tmp_path: Path) -> None: + war = fixtures.war_bytes(libraries={'log4j-core.jar': fixtures.library_jar(*_LOG4J)}) + ear = fixtures.ear_bytes(modules={'web.war': war}, libraries={'guava.jar': fixtures.library_jar(*_GUAVA)}) + path = _write(tmp_path, 'payments.ear', ear) + + entries = _by_logical_path(extractor.extract(path)) + + assert 'payments.ear > web.war' in entries + assert 'payments.ear > APP-INF/lib/guava.jar' in entries + assert 'payments.ear > web.war > WEB-INF/lib/log4j-core.jar' in entries + + def test_three_level_containment_reports_a_readable_chain( + self, extractor: JavaArchiveExtractor, tmp_path: Path + ) -> None: + war = fixtures.war_bytes(libraries={'log4j-core-2.14.1.jar': fixtures.library_jar(*_LOG4J)}) + ear = fixtures.ear_bytes(modules={'web.war': war}) + path = _write(tmp_path, 'payments.ear', ear) + + entries = _by_logical_path(extractor.extract(path)) + deep = entries['payments.ear > web.war > WEB-INF/lib/log4j-core-2.14.1.jar'] + + assert deep.depth == 2 + assert deep.parent == 'payments.ear > web.war' + assert deep.name == 'log4j-core-2.14.1.jar' + + +class TestRecursionSafety: + def _four_level_nest(self) -> bytes: + level4 = fixtures.library_jar('com.acme', 'deepest', '1.0') + level3 = fixtures.archive_bytes(files={'lib/level4.jar': level4}) + level2 = fixtures.archive_bytes(files={'lib/level3.jar': level3}) + return fixtures.war_bytes(libraries={'level2.jar': level2}) + + def test_depth_cap_stops_the_fourth_level_being_opened( + self, extractor: JavaArchiveExtractor, tmp_path: Path + ) -> None: + path = _write(tmp_path, 'app.war', self._four_level_nest()) + + entries = extractor.extract(path, max_depth=3) + logical_paths = [entry.logical_path for entry in entries] + deepest = 'app.war > WEB-INF/lib/level2.jar > lib/level3.jar > lib/level4.jar' + + # the fourth archive is reported, but never opened, so nothing from inside it comes back + assert deepest in logical_paths + assert max(entry.depth for entry in entries) == 3 + assert not any(path.startswith(f'{deepest} > ') for path in logical_paths) + + def test_a_lower_depth_cap_stops_sooner(self, extractor: JavaArchiveExtractor, tmp_path: Path) -> None: + path = _write(tmp_path, 'app.war', self._four_level_nest()) + + entries = extractor.extract(path, max_depth=1) + + assert max(entry.depth for entry in entries) == 1 + assert [entry.logical_path for entry in entries if entry.is_archive] == [ + 'app.war', + 'app.war > WEB-INF/lib/level2.jar', + ] + + def test_repeated_digest_is_opened_only_once(self, extractor: JavaArchiveExtractor, tmp_path: Path) -> None: + guava = fixtures.library_jar(*_GUAVA) + war = fixtures.war_bytes(libraries={'guava.jar': guava, 'guava-shadow.jar': guava}) + path = _write(tmp_path, 'app.war', war) + + entries = extractor.extract(path) + logical_paths = [entry.logical_path for entry in entries] + + # both copies are reported as shipped components... + assert 'app.war > WEB-INF/lib/guava.jar' in logical_paths + assert 'app.war > WEB-INF/lib/guava-shadow.jar' in logical_paths + # ...but identical bytes are only walked once, which is what makes a self-containing archive terminate + assert sum(1 for path in logical_paths if path.endswith('pom.properties')) == 1 + + def test_an_archive_containing_its_own_parent_terminates( + self, extractor: JavaArchiveExtractor, tmp_path: Path + ) -> None: + inner = fixtures.library_jar('com.acme', 'inner', '1.0') + outer = fixtures.war_bytes(libraries={'inner.jar': inner, 'inner-again.jar': inner}) + path = _write(tmp_path, 'app.war', outer) + + entries = extractor.extract(path, max_depth=10) + + assert len(entries) < 20 + + +class TestFailureModes: + def test_a_directory_is_refused(self, extractor: JavaArchiveExtractor, tmp_path: Path) -> None: + with pytest.raises(BinaryExtractionError, match='is not a file'): + extractor.extract(str(tmp_path)) + + def test_a_missing_file_is_refused(self, extractor: JavaArchiveExtractor, tmp_path: Path) -> None: + with pytest.raises(BinaryExtractionError, match='is not a file'): + extractor.extract(str(tmp_path / 'absent.jar')) + + def test_non_zip_bytes_named_jar(self, extractor: JavaArchiveExtractor, tmp_path: Path) -> None: + path = _write(tmp_path, 'app.jar', fixtures.not_a_zip_bytes()) + + with pytest.raises(BinaryExtractionError, match='not a readable archive'): + extractor.extract(path) + + def test_a_hostile_entry_inside_a_nested_jar_aborts_the_scan( + self, extractor: JavaArchiveExtractor, tmp_path: Path + ) -> None: + hostile = fixtures.zip_slip_bytes() + war = fixtures.war_bytes(libraries={'hostile.jar': hostile}) + path = _write(tmp_path, 'app.war', war) + + with pytest.raises(UnsafeArchiveEntryError): + extractor.extract(path) + + +class TestIdentify: + def test_shipped_libraries_are_identified_from_embedded_maven_metadata( + self, extractor: JavaArchiveExtractor, tmp_path: Path + ) -> None: + war = fixtures.war_bytes( + libraries={ + 'guava.jar': fixtures.library_jar(*_GUAVA), + 'log4j-core.jar': fixtures.library_jar(*_LOG4J), + } + ) + path = _write(tmp_path, 'payments.war', war) + + result = extractor.identify(extractor.extract(path)) + + assert [component.purl for component in result.components] == [ + 'pkg:maven/com.google.guava/guava@31.1-jre', + 'pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1', + ] + assert all(component.evidence == 'pom.properties' for component in result.components) + assert all(component.confidence == 'exact' for component in result.components) + assert result.unidentified == [] + # the artifact we were pointed at is not one of its own components + assert all(component.logical_path != 'payments.war' for component in result.components) + + def test_a_library_without_metadata_is_reported_unidentified( + self, extractor: JavaArchiveExtractor, tmp_path: Path + ) -> None: + anonymous = fixtures.archive_bytes(files={'com/acme/Shim.class': b'\xca\xfe\xba\xbe'}) + war = fixtures.war_bytes(libraries={'internal-shim.jar': anonymous}) + path = _write(tmp_path, 'payments.war', war) + + result = extractor.identify(extractor.extract(path)) + + assert result.components == [] + assert [item.logical_path for item in result.unidentified] == ['payments.war > WEB-INF/lib/internal-shim.jar'] + # we do not guess a coordinate from the filename + assert result.unidentified[0].sha1 == hashlib.sha1(anonymous, usedforsecurity=False).hexdigest() + + def test_a_standalone_library_jar_identifies_itself(self, extractor: JavaArchiveExtractor, tmp_path: Path) -> None: + # the vendor-artifact case: a jar that ships nothing is itself the thing being assessed + path = _write(tmp_path, 'guava.jar', fixtures.library_jar(*_GUAVA)) + + result = extractor.identify(extractor.extract(path)) + + assert [component.purl for component in result.components] == ['pkg:maven/com.google.guava/guava@31.1-jre'] + + def test_a_standalone_jar_without_metadata_is_reported_unidentified( + self, extractor: JavaArchiveExtractor, tmp_path: Path + ) -> None: + # the archive we were pointed at is the only thing under assessment; failing to name it is the whole + # coverage gap, and dropping it would let the scan claim full coverage of nothing + anonymous = fixtures.archive_bytes(files={'com/acme/Shim.class': b'\xca\xfe\xba\xbe'}) + path = _write(tmp_path, 'internal-shim.jar', anonymous) + + result = extractor.identify(extractor.extract(path)) + + assert result.components == [] + assert [item.logical_path for item in result.unidentified] == ['internal-shim.jar'] + assert result.unidentified[0].sha1 == hashlib.sha1(anonymous, usedforsecurity=False).hexdigest() + + def test_a_deployable_without_libraries_is_not_its_own_component( + self, extractor: JavaArchiveExtractor, tmp_path: Path + ) -> None: + # a war is an application, not a library, even when maven wrote its coordinates into it + files = { + 'WEB-INF/web.xml': '', + fixtures.pom_properties_entry_name('com.acme', 'payments'): fixtures.pom_properties( + 'com.acme', 'payments', '1.0' + ), + } + path = _write(tmp_path, 'payments.war', fixtures.archive_bytes(files=files)) + + result = extractor.identify(extractor.extract(path)) + + assert result.components == [] + assert result.unidentified == [] + + def test_counters_describe_the_walk(self, extractor: JavaArchiveExtractor, tmp_path: Path) -> None: + war = fixtures.war_bytes(libraries={'log4j-core.jar': fixtures.library_jar(*_LOG4J)}) + ear = fixtures.ear_bytes(modules={'web.war': war}) + path = _write(tmp_path, 'payments.ear', ear) + + result = extractor.identify(extractor.extract(path)) + + # the ear, the war and the jar were each opened + assert result.archives_opened == 3 + assert result.max_depth_reached == 3 + assert result.resolver_available is False + + def test_an_artifact_with_no_libraries_yields_nothing_unidentified( + self, extractor: JavaArchiveExtractor, tmp_path: Path + ) -> None: + path = _write(tmp_path, 'app.war', fixtures.war_bytes()) + + result = extractor.identify(extractor.extract(path)) + + assert result.unidentified == [] + assert result.components == [] + assert result.archives_opened == 1 + + def test_an_archive_left_unopened_at_the_depth_cap_is_not_counted_as_opened( + self, extractor: JavaArchiveExtractor, tmp_path: Path + ) -> None: + level3 = fixtures.library_jar('com.acme', 'deepest', '1.0') + level2 = fixtures.archive_bytes(files={'lib/level3.jar': level3}) + path = _write(tmp_path, 'app.war', fixtures.war_bytes(libraries={'level2.jar': level2})) + + result = extractor.identify(extractor.extract(path, max_depth=2)) + + # the war and level2 were walked; level3 was reported but never opened + assert result.archives_opened == 2 + assert result.max_depth_reached == 3 + # level2 carries no metadata of its own; level3 was never opened so its metadata was never read + assert len(result.components) == 0 + assert len(result.unidentified) == 2 diff --git a/tests/cli/files_collector/binary/test_maven_central.py b/tests/cli/files_collector/binary/test_maven_central.py new file mode 100644 index 00000000..888fb430 --- /dev/null +++ b/tests/cli/files_collector/binary/test_maven_central.py @@ -0,0 +1,238 @@ +"""``MavenCentralDigestResolver`` against a mocked search.maven.org. No request here ever leaves the process.""" + +import json +from pathlib import Path + +import pytest +import requests +import responses + +from cycode.cli import consts +from cycode.cli.files_collector.binary.java_extractor import JavaArchiveExtractor +from cycode.cli.files_collector.binary.maven_central import MavenCentralDigestResolver +from tests.cli.files_collector.binary import fixtures + +_GUAVA_SHA1 = 'bd41a290787b5301e63929676d792c507bbc00ae' +_UNKNOWN_SHA1 = '0' * 40 +_ACTIVATION_SHA1 = 'a' * 40 + + +def _search_body(*documents: tuple[str, str, str]) -> dict: + docs = [{'id': f'{g}:{a}:{v}', 'g': g, 'a': a, 'v': v, 'p': 'jar'} for g, a, v in documents] + return {'responseHeader': {'status': 0}, 'response': {'numFound': len(docs), 'start': 0, 'docs': docs}} + + +def _register(digest: str, body: dict, status: int = 200) -> None: + responses.add( + responses.GET, + consts.BINARY_MAVEN_CENTRAL_SEARCH_URL, + match=[responses.matchers.query_param_matcher({'q': f'1:"{digest}"', 'rows': '5', 'wt': 'json'})], + json=body, + status=status, + ) + + +class TestResolve: + @responses.activate + def test_a_hit_is_an_exact_purl(self) -> None: + _register(_GUAVA_SHA1, _search_body(('com.google.guava', 'guava', '27.0.1-jre'))) + resolver = MavenCentralDigestResolver(session=requests.Session()) + + assert resolver.resolve([_GUAVA_SHA1]) == {_GUAVA_SHA1: 'pkg:maven/com.google.guava/guava@27.0.1-jre'} + assert resolver.available is True + + @responses.activate + def test_a_miss_is_left_out_rather_than_guessed(self) -> None: + _register(_UNKNOWN_SHA1, _search_body()) + resolver = MavenCentralDigestResolver(session=requests.Session()) + + assert resolver.resolve([_UNKNOWN_SHA1]) == {} + # nothing went wrong: the archive is simply not on Maven Central, and that is not a partial result + assert resolver.available is True + + @responses.activate + def test_the_digest_is_sent_in_lower_case(self) -> None: + _register(_GUAVA_SHA1, _search_body(('com.google.guava', 'guava', '27.0.1-jre'))) + resolver = MavenCentralDigestResolver(session=requests.Session()) + + resolved = resolver.resolve([_GUAVA_SHA1.upper()]) + + # the caller's key is preserved so it can be mapped back to the archive it came from + assert list(resolved) == [_GUAVA_SHA1.upper()] + + @responses.activate + def test_a_relocated_artifact_takes_the_index_ranking(self) -> None: + _register( + _ACTIVATION_SHA1, + _search_body( + ('com.sun.activation', 'javax.activation', '1.2.0'), ('javax.activation', 'activation', '1.2.0') + ), + ) + resolver = MavenCentralDigestResolver(session=requests.Session()) + + assert resolver.resolve([_ACTIVATION_SHA1]) == { + _ACTIVATION_SHA1: 'pkg:maven/com.sun.activation/javax.activation@1.2.0' + } + + @responses.activate + def test_a_document_missing_a_coordinate_is_skipped(self) -> None: + responses.add( + responses.GET, + consts.BINARY_MAVEN_CENTRAL_SEARCH_URL, + json={'response': {'docs': [{'g': 'x', 'a': 'y'}, {'g': 'com.acme', 'a': 'lib', 'v': '1.0'}]}}, + ) + resolver = MavenCentralDigestResolver(session=requests.Session()) + + assert resolver.resolve([_GUAVA_SHA1]) == {_GUAVA_SHA1: 'pkg:maven/com.acme/lib@1.0'} + + @responses.activate + def test_the_user_agent_identifies_the_cli(self) -> None: + _register(_GUAVA_SHA1, _search_body()) + MavenCentralDigestResolver(session=requests.Session()).resolve([_GUAVA_SHA1]) + + assert responses.calls[0].request.headers['User-Agent'].startswith('CycodeCLI/') + + +class TestFailure: + @responses.activate + def test_a_connection_error_stops_the_run_and_makes_results_partial(self) -> None: + _register(_GUAVA_SHA1, _search_body(('com.google.guava', 'guava', '27.0.1-jre'))) + responses.add( + responses.GET, + consts.BINARY_MAVEN_CENTRAL_SEARCH_URL, + match=[responses.matchers.query_param_matcher({'q': f'1:"{_UNKNOWN_SHA1}"', 'rows': '5', 'wt': 'json'})], + body=requests.ConnectionError('name resolution failed'), + ) + resolver = MavenCentralDigestResolver(session=requests.Session()) + + resolved = resolver.resolve([_GUAVA_SHA1, _UNKNOWN_SHA1, _ACTIVATION_SHA1]) + + # what was resolved before the failure is kept; the digest after it was never asked for + assert resolved == {_GUAVA_SHA1: 'pkg:maven/com.google.guava/guava@27.0.1-jre'} + assert len(responses.calls) == 2 + assert resolver.available is False + assert 'failed for 2 of 3 digests' in resolver.unavailability_reason + assert 'name resolution failed' in resolver.unavailability_reason + + @responses.activate + def test_a_timeout_is_retried_once_and_the_run_continues(self) -> None: + # the index is intermittently slow: one slow digest must not cost the two hundred behind it + responses.add( + responses.GET, + consts.BINARY_MAVEN_CENTRAL_SEARCH_URL, + match=[responses.matchers.query_param_matcher({'q': f'1:"{_UNKNOWN_SHA1}"', 'rows': '5', 'wt': 'json'})], + body=requests.Timeout('slow'), + ) + _register(_GUAVA_SHA1, _search_body(('com.google.guava', 'guava', '27.0.1-jre'))) + resolver = MavenCentralDigestResolver(session=requests.Session()) + + resolved = resolver.resolve([_UNKNOWN_SHA1, _GUAVA_SHA1]) + + assert resolved == {_GUAVA_SHA1: 'pkg:maven/com.google.guava/guava@27.0.1-jre'} + assert len(responses.calls) == 3 # two attempts for the slow digest, one for the hit + assert resolver.available is False + assert 'failed for 1 of 2 digests' in resolver.unavailability_reason + + @responses.activate + def test_a_timeout_that_succeeds_on_retry_is_not_a_failure(self) -> None: + responses.add(responses.GET, consts.BINARY_MAVEN_CENTRAL_SEARCH_URL, body=requests.Timeout('slow')) + responses.add( + responses.GET, + consts.BINARY_MAVEN_CENTRAL_SEARCH_URL, + json=_search_body(('com.google.guava', 'guava', '27.0.1-jre')), + ) + resolver = MavenCentralDigestResolver(session=requests.Session()) + + resolved = resolver.resolve([_GUAVA_SHA1]) + + assert resolved == {_GUAVA_SHA1: 'pkg:maven/com.google.guava/guava@27.0.1-jre'} + assert resolver.available is True + + @responses.activate + def test_a_server_error_counts_against_the_digest_without_a_retry(self) -> None: + _register(_GUAVA_SHA1, {}, status=503) + resolver = MavenCentralDigestResolver(session=requests.Session()) + + assert resolver.resolve([_GUAVA_SHA1]) == {} + assert len(responses.calls) == 1 + assert resolver.available is False + + @responses.activate + def test_a_non_json_body_is_a_failure(self) -> None: + # a captive portal or a proxy error page answers 200 with HTML + responses.add(responses.GET, consts.BINARY_MAVEN_CENTRAL_SEARCH_URL, body='sign in') + resolver = MavenCentralDigestResolver(session=requests.Session()) + + assert resolver.resolve([_GUAVA_SHA1]) == {} + assert resolver.available is False + + @responses.activate + def test_a_later_call_after_a_connection_error_does_not_retry(self) -> None: + responses.add(responses.GET, consts.BINARY_MAVEN_CENTRAL_SEARCH_URL, body=requests.ConnectionError('refused')) + resolver = MavenCentralDigestResolver(session=requests.Session()) + + resolver.resolve([_GUAVA_SHA1]) + resolver.resolve([_UNKNOWN_SHA1]) + + assert len(responses.calls) == 1 + assert 'failed for 2 of 2 digests' in resolver.unavailability_reason + + +class TestThroughTheLadder: + def _write(self, tmp_path: Path, name: str, content: bytes) -> str: + path = tmp_path / name + path.write_bytes(content) + return str(path) + + @responses.activate + def test_an_anonymous_jar_on_maven_central_is_identified_exactly(self, tmp_path: Path) -> None: + anonymous = fixtures.archive_bytes(files={'com/google/common/Foo.class': b'\xca\xfe\xba\xbe'}) + responses.add( + responses.GET, + consts.BINARY_MAVEN_CENTRAL_SEARCH_URL, + json=_search_body(('com.google.guava', 'guava', '27.0.1-jre')), + ) + extractor = JavaArchiveExtractor(resolver=MavenCentralDigestResolver(session=requests.Session())) + path = self._write(tmp_path, 'app.war', fixtures.war_bytes(libraries={'mystery.jar': anonymous})) + + result = extractor.identify(extractor.extract(path)) + + assert [c.purl for c in result.components] == ['pkg:maven/com.google.guava/guava@27.0.1-jre'] + assert result.components[0].evidence == 'digest' + assert result.components[0].confidence == 'exact' + assert result.unidentified == [] + assert result.resolver_available is True + assert result.resolver_unavailability_reason is None + + @responses.activate + def test_a_failure_is_carried_on_the_result(self, tmp_path: Path) -> None: + anonymous = fixtures.archive_bytes(files={'com/acme/Foo.class': b'\xca\xfe\xba\xbe'}) + responses.add(responses.GET, consts.BINARY_MAVEN_CENTRAL_SEARCH_URL, body=requests.ConnectionError('offline')) + extractor = JavaArchiveExtractor(resolver=MavenCentralDigestResolver(session=requests.Session())) + path = self._write(tmp_path, 'app.war', fixtures.war_bytes(libraries={'mystery.jar': anonymous})) + + result = extractor.identify(extractor.extract(path)) + + assert len(result.unidentified) == 1 + assert result.resolver_available is False + assert 'offline' in result.resolver_unavailability_reason + + @responses.activate + def test_metadata_wins_and_maven_central_is_not_asked(self, tmp_path: Path) -> None: + extractor = JavaArchiveExtractor(resolver=MavenCentralDigestResolver(session=requests.Session())) + path = self._write(tmp_path, 'guava.jar', fixtures.library_jar('com.google.guava', 'guava', '31.1-jre')) + + result = extractor.identify(extractor.extract(path)) + + assert result.components[0].evidence == 'pom.properties' + assert len(responses.calls) == 0 + + +@pytest.mark.parametrize('body', [json.dumps([]), json.dumps({'response': 'nope'})]) +@responses.activate +def test_an_unexpected_json_shape_is_a_miss_not_a_crash(body: str) -> None: + responses.add(responses.GET, consts.BINARY_MAVEN_CENTRAL_SEARCH_URL, body=body, content_type='application/json') + resolver = MavenCentralDigestResolver(session=requests.Session()) + + assert resolver.resolve([_GUAVA_SHA1]) == {} + assert resolver.available is True diff --git a/tests/cli/files_collector/binary/test_resolver.py b/tests/cli/files_collector/binary/test_resolver.py new file mode 100644 index 00000000..653d7571 --- /dev/null +++ b/tests/cli/files_collector/binary/test_resolver.py @@ -0,0 +1,193 @@ +"""Tier 2 wiring and the ordering of the identification ladder. + +No requests are made here: the shipped resolver is a no-op and the endpoint behind tier 2 does not exist yet, so +these tests drive the seam with a stub. When ``CycodeDigestResolver`` lands in phase 6 it is the stub that gets +replaced with ``responses``, not this ordering. +""" + +import hashlib +from pathlib import Path + +import pytest + +from cycode.cli.files_collector.binary.java_extractor import JavaArchiveExtractor, parse_maven_purl +from cycode.cli.files_collector.binary.resolver import DigestResolver, NullDigestResolver +from tests.cli.files_collector.binary import fixtures + +_GUAVA = ('com.google.guava', 'guava', '31.1-jre') + + +class StubResolver(DigestResolver): + """Stands in for the backend index. Records what it was asked, so batching can be asserted.""" + + def __init__(self, answers: dict, available: bool = True) -> None: + self._answers = answers + self._available = available + self.calls: list[list[str]] = [] + + def resolve(self, digests: list[str]) -> dict: + self.calls.append(list(digests)) + if not self._available: + return {} + + return {digest: self._answers[digest] for digest in digests if digest in self._answers} + + @property + def available(self) -> bool: + return self._available + + +def _write(tmp_path: Path, name: str, content: bytes) -> str: + path = tmp_path / name + path.write_bytes(content) + return str(path) + + +def _sha1(payload: bytes) -> str: + return hashlib.sha1(payload, usedforsecurity=False).hexdigest() + + +class TestNullDigestResolver: + def test_it_resolves_nothing_and_says_so(self) -> None: + resolver = NullDigestResolver() + + assert resolver.resolve(['a' * 40]) == {} + assert resolver.available is False + + def test_it_is_the_default(self, tmp_path: Path) -> None: + path = _write(tmp_path, 'app.war', fixtures.war_bytes()) + extractor = JavaArchiveExtractor() + + result = extractor.identify(extractor.extract(path)) + + # this is what drives the degradation warning in the printers + assert result.resolver_available is False + + +class TestParseMavenPurl: + def test_a_namespaced_purl(self) -> None: + coordinates = parse_maven_purl('pkg:maven/com.google.guava/guava@31.1-jre') + + assert (coordinates.group, coordinates.artifact, coordinates.version) == _GUAVA + + def test_a_purl_with_no_namespace(self) -> None: + coordinates = parse_maven_purl('pkg:maven/widget@1.0') + + assert (coordinates.group, coordinates.artifact, coordinates.version) == ('', 'widget', '1.0') + + def test_qualifiers_and_subpaths_are_ignored(self) -> None: + coordinates = parse_maven_purl('pkg:maven/com.acme/widget@1.0?type=jar#sub') + + assert coordinates.version == '1.0' + + @pytest.mark.parametrize( + 'purl', + [None, '', 'pkg:npm/left-pad@1.0.0', 'pkg:maven/com.acme/widget', 'pkg:maven/@1.0', 'not-a-purl'], + ) + def test_anything_unusable_yields_nothing(self, purl: str) -> None: + assert parse_maven_purl(purl) is None + + +class TestTierOrdering: + def test_a_digest_hit_identifies_a_jar_with_no_metadata(self, tmp_path: Path) -> None: + anonymous = fixtures.archive_bytes(files={'com/acme/Shim.class': b'\xca\xfe\xba\xbe'}) + war = fixtures.war_bytes(libraries={'stripped.jar': anonymous}) + resolver = StubResolver({_sha1(anonymous): 'pkg:maven/com.google.guava/guava@31.1-jre'}) + + extractor = JavaArchiveExtractor(resolver=resolver) + result = extractor.identify(extractor.extract(_write(tmp_path, 'app.war', war))) + + assert [component.purl for component in result.components] == ['pkg:maven/com.google.guava/guava@31.1-jre'] + assert result.components[0].evidence == 'digest' + assert result.components[0].confidence == 'exact' + assert result.unidentified == [] + + def test_embedded_metadata_beats_a_digest_hit(self, tmp_path: Path) -> None: + guava = fixtures.library_jar(*_GUAVA) + war = fixtures.war_bytes(libraries={'guava.jar': guava}) + resolver = StubResolver({_sha1(guava): 'pkg:maven/wrong/answer@9.9'}) + + extractor = JavaArchiveExtractor(resolver=resolver) + result = extractor.identify(extractor.extract(_write(tmp_path, 'app.war', war))) + + assert result.components[0].purl == 'pkg:maven/com.google.guava/guava@31.1-jre' + assert result.components[0].evidence == 'pom.properties' + # tier 1 answered, so the digest was never offered for resolution + assert resolver.calls == [] + + def test_a_digest_hit_beats_the_manifest(self, tmp_path: Path) -> None: + manifest_only = fixtures.archive_bytes( + files={'META-INF/MANIFEST.MF': fixtures.manifest(Implementation_Title='guessy', Implementation_Version='1')} + ) + war = fixtures.war_bytes(libraries={'lib.jar': manifest_only}) + resolver = StubResolver({_sha1(manifest_only): 'pkg:maven/com.google.guava/guava@31.1-jre'}) + + extractor = JavaArchiveExtractor(resolver=resolver) + result = extractor.identify(extractor.extract(_write(tmp_path, 'app.war', war))) + + assert result.components[0].evidence == 'digest' + assert result.components[0].confidence == 'exact' + + def test_the_manifest_is_the_fallback_when_the_digest_is_unknown(self, tmp_path: Path) -> None: + manifest_only = fixtures.archive_bytes( + files={ + 'META-INF/MANIFEST.MF': fixtures.manifest( + Implementation_Title='guessy', Implementation_Version='1', Implementation_Vendor_Id='com.acme' + ) + } + ) + war = fixtures.war_bytes(libraries={'lib.jar': manifest_only}) + + extractor = JavaArchiveExtractor(resolver=StubResolver({})) + result = extractor.identify(extractor.extract(_write(tmp_path, 'app.war', war))) + + assert result.components[0].evidence == 'manifest.mf' + assert result.components[0].confidence == 'ambiguous' + + def test_one_component_per_digest_across_tiers(self, tmp_path: Path) -> None: + # a jar carrying both pom.properties and a manifest must not be reported twice + both = fixtures.library_jar(*_GUAVA) + war = fixtures.war_bytes(libraries={'guava.jar': both}) + + extractor = JavaArchiveExtractor() + result = extractor.identify(extractor.extract(_write(tmp_path, 'app.war', war))) + + assert len(result.components) == 1 + + +class TestBatching: + def test_every_unresolved_digest_goes_in_one_request(self, tmp_path: Path) -> None: + anonymous = [fixtures.archive_bytes(files={f'com/acme/{index}.class': b'x'}) for index in range(3)] + war = fixtures.war_bytes(libraries={f'lib-{index}.jar': content for index, content in enumerate(anonymous)}) + resolver = StubResolver({}) + + extractor = JavaArchiveExtractor(resolver=resolver) + extractor.identify(extractor.extract(_write(tmp_path, 'app.war', war))) + + assert len(resolver.calls) == 1 + assert sorted(resolver.calls[0]) == sorted(_sha1(content) for content in anonymous) + + def test_an_unavailable_resolver_is_still_asked_and_the_result_says_so(self, tmp_path: Path) -> None: + # the resolver keeps its own state: a real one that failed on artifact 1 must still be offered artifact 2, + # otherwise one slow lookup silently switches tier 2 off for the rest of the run + anonymous = fixtures.archive_bytes(files={'com/acme/Shim.class': b'x'}) + war = fixtures.war_bytes(libraries={'lib.jar': anonymous}) + resolver = StubResolver({_sha1(anonymous): 'pkg:maven/com.acme/thing@1.0'}, available=False) + + extractor = JavaArchiveExtractor(resolver=resolver) + result = extractor.identify(extractor.extract(_write(tmp_path, 'app.war', war))) + + assert resolver.calls == [[_sha1(anonymous)]] + assert result.resolver_available is False + # degradation, not failure: the jar is surfaced rather than dropped + assert len(result.unidentified) == 1 + + def test_nothing_is_sent_when_local_identification_covered_everything(self, tmp_path: Path) -> None: + # digests of proprietary jars only leave the machine when they have to + war = fixtures.war_bytes(libraries={'guava.jar': fixtures.library_jar(*_GUAVA)}) + resolver = StubResolver({}) + + extractor = JavaArchiveExtractor(resolver=resolver) + extractor.identify(extractor.extract(_write(tmp_path, 'app.war', war))) + + assert resolver.calls == [] diff --git a/tests/cli/files_collector/binary/test_safe_zip.py b/tests/cli/files_collector/binary/test_safe_zip.py new file mode 100644 index 00000000..12b5486f --- /dev/null +++ b/tests/cli/files_collector/binary/test_safe_zip.py @@ -0,0 +1,304 @@ +import hashlib +import io +from pathlib import Path + +import pytest + +from cycode.cli import consts +from cycode.cli.exceptions.custom_exceptions import ( + ArchiveCompressionRatioLimitError, + ArchiveEntryCountLimitError, + ArchiveEntrySizeLimitError, + ArchiveTotalSizeLimitError, + BinaryExtractionError, + MalformedArchiveError, + UnsafeArchiveEntryError, +) +from cycode.cli.files_collector.binary.safe_zip import ( + ArchiveBudget, + ArchiveLimits, + SafeZip, + validate_entry_name, +) +from tests.cli.files_collector.binary import fixtures + + +def _entry_names(archive: SafeZip) -> list[str]: + return [entry.name for entry in archive.entries()] + + +class TestEntryNameValidation: + @pytest.mark.parametrize( + 'name', + [ + '../../evil.txt', + 'WEB-INF/lib/../../../evil.jar', + '..', + 'a/../../b', + ], + ) + def test_path_escape_is_refused(self, name: str) -> None: + with pytest.raises(UnsafeArchiveEntryError, match='escapes the archive root'): + validate_entry_name(name) + + @pytest.mark.parametrize('name', ['/etc/passwd', '/', '/var/spool/cron/evil']) + def test_absolute_path_is_refused(self, name: str) -> None: + with pytest.raises(UnsafeArchiveEntryError, match='absolute path'): + validate_entry_name(name) + + @pytest.mark.parametrize('name', ['C:\\Windows\\evil.dll', 'c:/windows/evil.dll', 'Z:relative']) + def test_drive_letter_is_refused(self, name: str) -> None: + with pytest.raises(UnsafeArchiveEntryError, match='drive letter'): + validate_entry_name(name) + + @pytest.mark.parametrize('name', ['\\\\host\\share\\evil.txt', '//host/share/evil.txt']) + def test_unc_path_is_refused(self, name: str) -> None: + with pytest.raises(UnsafeArchiveEntryError, match='UNC path'): + validate_entry_name(name) + + @pytest.mark.parametrize( + 'name', + ['CON', 'nul', 'COM1', 'LPT9', 'CON.txt', 'aux.tar.gz', 'WEB-INF/lib/PRN.jar'], + ) + def test_windows_reserved_name_is_refused(self, name: str) -> None: + with pytest.raises(UnsafeArchiveEntryError, match='reserved device name'): + validate_entry_name(name) + + def test_null_byte_is_refused(self) -> None: + with pytest.raises(UnsafeArchiveEntryError, match='null byte'): + validate_entry_name('evil\x00.txt') + + def test_empty_name_is_refused(self) -> None: + with pytest.raises(UnsafeArchiveEntryError, match='empty name'): + validate_entry_name('') + + @pytest.mark.parametrize( + 'name', + [ + 'WEB-INF/lib/guava-31.1-jre.jar', + 'META-INF/maven/com.google.guava/guava/pom.properties', + 'a/./b/c.txt', + 'CONTENTS.txt', + 'COMMON/thing.jar', + 'lpt.txt', + ], + ) + def test_ordinary_names_are_accepted(self, name: str) -> None: + validate_entry_name(name) + + def test_every_refusal_is_a_binary_extraction_error(self) -> None: + # phase 3 registers one error class in the scan error map; every refusal here must be caught by it + with pytest.raises(BinaryExtractionError): + validate_entry_name('../evil') + + +class TestUnsafeArchivesAreRefusedOnIteration: + def test_zip_slip(self) -> None: + with SafeZip.open(fixtures.zip_slip_bytes()) as archive, pytest.raises(UnsafeArchiveEntryError): + _entry_names(archive) + + def test_absolute_path(self) -> None: + with SafeZip.open(fixtures.absolute_path_bytes()) as archive, pytest.raises(UnsafeArchiveEntryError): + _entry_names(archive) + + def test_drive_letter(self) -> None: + with SafeZip.open(fixtures.drive_letter_bytes()) as archive, pytest.raises(UnsafeArchiveEntryError): + _entry_names(archive) + + def test_unc_path(self) -> None: + with SafeZip.open(fixtures.unc_path_bytes()) as archive, pytest.raises(UnsafeArchiveEntryError): + _entry_names(archive) + + def test_reserved_device_name(self) -> None: + with SafeZip.open(fixtures.reserved_name_bytes()) as archive, pytest.raises(UnsafeArchiveEntryError): + _entry_names(archive) + + +class TestNonRegularEntries: + def test_symlink_entry_is_skipped_not_followed(self) -> None: + with SafeZip.open(fixtures.symlink_bytes()) as archive: + names = _entry_names(archive) + + assert names == ['real.txt'] + + def test_fifo_entry_is_skipped(self) -> None: + with SafeZip.open(fixtures.non_regular_entry_bytes()) as archive: + names = _entry_names(archive) + + assert names == ['real.txt'] + + def test_msdos_created_entry_is_still_read(self) -> None: + # windows tooling records no unix mode; treating "no opinion" as "not a regular file" would drop everything + with SafeZip.open(fixtures.msdos_created_bytes()) as archive: + names = _entry_names(archive) + + assert names == ['real.txt'] + + def test_directory_entries_are_skipped(self) -> None: + content = fixtures.archive_bytes( + files={'WEB-INF/': '', 'WEB-INF/web.xml': ''}, + external_attrs={'WEB-INF/': fixtures.DIRECTORY_ATTR}, + ) + with SafeZip.open(content) as archive: + names = _entry_names(archive) + + assert names == ['WEB-INF/web.xml'] + + +class TestReading: + def test_digests_match_the_content(self) -> None: + payload = b'the quick brown fox' * 100 + with SafeZip.open(fixtures.archive_bytes(files={'a.bin': payload})) as archive: + entry = next(iter(archive.entries())) + content = archive.read(entry) + + assert content.data == payload + assert content.size == len(payload) + assert content.sha1 == hashlib.sha1(payload, usedforsecurity=False).hexdigest() + assert content.sha256 == hashlib.sha256(payload).hexdigest() + + def test_digest_only_read_holds_no_bytes(self) -> None: + payload = b'x' * 4096 + with SafeZip.open(fixtures.archive_bytes(files={'a.bin': payload})) as archive: + entry = next(iter(archive.entries())) + content = archive.read(entry, buffer=False) + + assert content.data is None + assert content.size == len(payload) + assert content.sha1 == hashlib.sha1(payload, usedforsecurity=False).hexdigest() + + def test_empty_archive_yields_nothing(self) -> None: + with SafeZip.open(fixtures.empty_archive_bytes()) as archive: + assert _entry_names(archive) == [] + + def test_opens_from_a_path(self, tmp_path: Path) -> None: + path = tmp_path / 'app.jar' + path.write_bytes(fixtures.archive_bytes(files={'a.txt': 'a'})) + + with SafeZip.open(str(path)) as archive: + assert _entry_names(archive) == ['a.txt'] + assert archive.source_name == 'app.jar' + + def test_opens_from_a_stream(self) -> None: + stream = io.BytesIO(fixtures.archive_bytes(files={'a.txt': 'a'})) + + with SafeZip.open(stream, source_name='streamed.jar') as archive: + assert _entry_names(archive) == ['a.txt'] + assert archive.source_name == 'streamed.jar' + + +class TestMalformedArchives: + def test_bytes_that_are_not_a_zip(self) -> None: + with pytest.raises(MalformedArchiveError, match='not a readable archive'): + SafeZip.open(fixtures.not_a_zip_bytes(), source_name='app.jar') + + def test_truncated_central_directory(self) -> None: + with pytest.raises(MalformedArchiveError): + SafeZip.open(fixtures.truncated_central_directory_bytes(), source_name='app.jar') + + def test_malformed_local_header_fails_on_read_not_on_open(self) -> None: + with SafeZip.open(fixtures.malformed_local_header_bytes(), source_name='app.jar') as archive: + entry = next(iter(archive.entries())) + with pytest.raises(MalformedArchiveError, match='could not be read'): + archive.read(entry) + + def test_missing_file_is_a_clean_error_not_a_traceback(self, tmp_path: Path) -> None: + with pytest.raises(MalformedArchiveError): + SafeZip.open(str(tmp_path / 'absent.jar')) + + +class TestLimits: + def test_shipped_limits_are_the_documented_ones(self) -> None: + limits = ArchiveLimits() + + assert limits.max_entry_count == consts.BINARY_MAX_ENTRY_COUNT == 100_000 + assert limits.max_entry_size_in_bytes == consts.BINARY_MAX_ENTRY_SIZE_IN_BYTES == 512 * 1024 * 1024 + assert limits.max_total_size_in_bytes == consts.BINARY_MAX_TOTAL_SIZE_IN_BYTES == 2 * 1024 * 1024 * 1024 + assert limits.max_compression_ratio == consts.BINARY_MAX_COMPRESSION_RATIO == 200 + + def test_entry_count_ceiling(self) -> None: + budget = ArchiveBudget(ArchiveLimits(max_entry_count=5)) + + with pytest.raises(ArchiveEntryCountLimitError, match='past the limit of 5'): + SafeZip.open(fixtures.count_bomb_bytes(6), budget=budget) + + def test_entry_count_at_the_ceiling_is_allowed(self) -> None: + budget = ArchiveBudget(ArchiveLimits(max_entry_count=5)) + + with SafeZip.open(fixtures.count_bomb_bytes(5), budget=budget) as archive: + assert len(_entry_names(archive)) == 5 + + def test_declared_entry_size_is_refused_before_any_read(self) -> None: + budget = ArchiveBudget(ArchiveLimits(max_entry_size_in_bytes=1024)) + + with SafeZip.open(fixtures.size_bomb_bytes(), budget=budget) as archive: + entry = next(iter(archive.entries())) + with pytest.raises(ArchiveEntrySizeLimitError, match='declares'): + archive.read(entry) + + assert budget.consumed_bytes == 0 + + def test_entry_size_is_enforced_mid_stream_when_the_header_lies(self) -> None: + budget = ArchiveBudget(ArchiveLimits(max_entry_size_in_bytes=1024)) + content = fixtures.size_bomb_bytes(uncompressed_size=64 * 1024) + + with SafeZip.open(content, budget=budget) as archive: + entry = next(iter(archive.entries())) + # forge a header that understates the payload, exactly as a hostile archive would + liar = type(entry)(name=entry.name, size=16, compress_size=entry.compress_size) + with pytest.raises(ArchiveEntrySizeLimitError, match='expands past'): + archive.read(liar) + + # the abort happened while streaming rather than after the whole entry was in memory + assert budget.consumed_bytes <= 1024 + 64 * 1024 + + def test_total_size_ceiling_across_entries(self) -> None: + budget = ArchiveBudget(ArchiveLimits(max_total_size_in_bytes=48 * 1024)) + content = fixtures.total_size_bomb_bytes(entry_count=8, entry_size=16 * 1024) + + with SafeZip.open(content, budget=budget) as archive, pytest.raises(ArchiveTotalSizeLimitError): + for entry in archive.entries(): + archive.read(entry) + + def test_compression_ratio_ceiling_at_shipped_limits(self) -> None: + with SafeZip.open(fixtures.ratio_bomb_bytes()) as archive: + entry = next(iter(archive.entries())) + with pytest.raises(ArchiveCompressionRatioLimitError, match='decompression bomb'): + archive.read(entry) + + def test_ratio_ceiling_ignores_small_highly_compressible_files(self) -> None: + # a few kilobytes of whitespace in a legitimate pom easily beats 200:1; only sustained expansion is a bomb + payload = b' ' * 8192 + with SafeZip.open(fixtures.archive_bytes(files={'pom.xml': payload})) as archive: + entry = next(iter(archive.entries())) + assert archive.read(entry).size == len(payload) + + def test_ratio_abort_happens_before_full_expansion(self) -> None: + uncompressed_size = 8 * 1024 * 1024 + budget = ArchiveBudget() + + with SafeZip.open(fixtures.ratio_bomb_bytes(uncompressed_size), budget=budget) as archive: + entry = next(iter(archive.entries())) + with pytest.raises(ArchiveCompressionRatioLimitError): + archive.read(entry) + + assert budget.consumed_bytes < uncompressed_size + + +class TestBudgetSharing: + def test_a_shared_budget_accumulates_across_archives(self) -> None: + budget = ArchiveBudget(ArchiveLimits(max_total_size_in_bytes=48 * 1024)) + payload = b'C' * (16 * 1024) + content = fixtures.archive_bytes(files={'a.bin': payload}, compress=False) + + for _ in range(3): + with SafeZip.open(content, budget=budget) as archive: + entry = next(iter(archive.entries())) + archive.read(entry) + + assert budget.consumed_bytes == 48 * 1024 + + with SafeZip.open(content, budget=budget) as archive: + entry = next(iter(archive.entries())) + with pytest.raises(ArchiveTotalSizeLimitError): + archive.read(entry) diff --git a/tests/cli/printers/utils/test_binary_report.py b/tests/cli/printers/utils/test_binary_report.py new file mode 100644 index 00000000..7103a51b --- /dev/null +++ b/tests/cli/printers/utils/test_binary_report.py @@ -0,0 +1,283 @@ +from typing import Optional +from unittest.mock import MagicMock + +import pytest +import typer + +from cycode.cli.files_collector.binary.base_extractor import ( + CONFIDENCE_AMBIGUOUS, + CONFIDENCE_EXACT, + EVIDENCE_MANIFEST, + EVIDENCE_POM_PROPERTIES, + ExtractionResult, + IdentifiedComponent, + UnidentifiedArtifact, +) +from cycode.cli.files_collector.binary.collector import BinaryCollectionResult +from cycode.cli.models import DocumentDetections, LocalScanResult +from cycode.cli.printers.utils import binary_report +from cycode.cyclient.models import Detection + +_LOG4J = ('org.apache.logging.log4j', 'log4j-core', '2.14.1') +_MYSTERY = ('', 'mystery-lib', '4.2') + + +def _component( + group: str, + artifact: str, + version: str, + logical_path: str = 'app.war > WEB-INF/lib/x.jar', + evidence: str = EVIDENCE_POM_PROPERTIES, + confidence: str = CONFIDENCE_EXACT, +) -> IdentifiedComponent: + return IdentifiedComponent( + group=group, + artifact=artifact, + version=version, + sha1='a' * 40, + logical_path=logical_path, + parent='app.war', + evidence=evidence, + confidence=confidence, + ) + + +def _collection( + components: Optional[list] = None, + unidentified: Optional[list] = None, + resolver_available: bool = False, +) -> BinaryCollectionResult: + collection = BinaryCollectionResult() + collection.results_by_artifact['app.war'] = ExtractionResult( + components=components or [], + unidentified=unidentified or [], + resolver_available=resolver_available, + ) + return collection + + +def _ctx(collection: Optional[BinaryCollectionResult] = None, offline: bool = False) -> typer.Context: + ctx = MagicMock(spec=typer.Context) + ctx.obj = {'offline': offline} + if collection is not None: + ctx.obj['binary_result'] = collection + return ctx + + +def _detection(package_name: str, package_version: str) -> Detection: + return Detection( + detection_type_id='id', + type='sca', + message='msg', + detection_details={'package_name': package_name, 'package_version': package_version}, + detection_rule_id='rule', + severity='Critical', + ) + + +def _scan_results(*detections: Detection) -> list: + document = MagicMock() + return [ + LocalScanResult( + scan_id='scan', + report_url=None, + document_detections=[DocumentDetections(document=document, detections=list(detections))], + issue_detected=bool(detections), + detections_count=len(detections), + relevant_detections_count=len(detections), + ) + ] + + +class TestGetBinaryCollection: + def test_absent_for_a_normal_scan(self) -> None: + assert binary_report.get_binary_collection(_ctx()) is None + + def test_present_for_a_binary_scan(self) -> None: + collection = _collection() + + assert binary_report.get_binary_collection(_ctx(collection)) is collection + + +class TestDetectionEvidence: + def test_a_finding_is_matched_back_to_its_component(self) -> None: + collection = _collection(components=[_component(*_LOG4J, logical_path='app.war > WEB-INF/lib/log4j.jar')]) + detection = _detection('org.apache.logging.log4j:log4j-core', '2.14.1') + + evidence = binary_report.get_detection_evidence(_ctx(collection), detection) + + assert evidence.logical_path == 'app.war > WEB-INF/lib/log4j.jar' + assert evidence.evidence == EVIDENCE_POM_PROPERTIES + assert evidence.is_ambiguous is False + + def test_a_component_with_no_group_is_matched_on_its_name_alone(self) -> None: + collection = _collection(components=[_component(*_MYSTERY, evidence=EVIDENCE_MANIFEST)]) + detection = _detection('mystery-lib', '4.2') + + assert binary_report.get_detection_evidence(_ctx(collection), detection) is not None + + def test_a_version_mismatch_does_not_match(self) -> None: + collection = _collection(components=[_component(*_LOG4J)]) + detection = _detection('org.apache.logging.log4j:log4j-core', '2.17.1') + + assert binary_report.get_detection_evidence(_ctx(collection), detection) is None + + def test_nothing_is_matched_on_a_normal_scan(self) -> None: + assert binary_report.get_detection_evidence(_ctx(), _detection('a:b', '1')) is None + + +class TestExitCodeGating: + def test_an_exact_finding_gates_the_build(self) -> None: + collection = _collection(components=[_component(*_LOG4J)]) + results = _scan_results(_detection('org.apache.logging.log4j:log4j-core', '2.14.1')) + + assert binary_report.has_gating_detections(_ctx(collection), results) is True + + def test_a_tier_three_finding_does_not_gate_the_build(self) -> None: + # a fabricated CVE from a guessed coordinate must never break someone's release + collection = _collection( + components=[_component(*_MYSTERY, evidence=EVIDENCE_MANIFEST, confidence=CONFIDENCE_AMBIGUOUS)] + ) + results = _scan_results(_detection('mystery-lib', '4.2')) + + assert binary_report.has_gating_detections(_ctx(collection), results) is False + assert binary_report.count_low_confidence(_ctx(collection), results) == 1 + + def test_one_exact_finding_among_ambiguous_ones_still_gates(self) -> None: + collection = _collection( + components=[ + _component(*_LOG4J), + _component(*_MYSTERY, evidence=EVIDENCE_MANIFEST, confidence=CONFIDENCE_AMBIGUOUS), + ] + ) + results = _scan_results( + _detection('mystery-lib', '4.2'), + _detection('org.apache.logging.log4j:log4j-core', '2.14.1'), + ) + + assert binary_report.has_gating_detections(_ctx(collection), results) is True + + def test_a_finding_we_cannot_match_is_treated_as_gating(self) -> None: + # failing open here would silently drop real findings; low confidence has to be proven, not assumed + collection = _collection(components=[_component(*_LOG4J)]) + results = _scan_results(_detection('something:unmatched', '9.9')) + + assert binary_report.has_gating_detections(_ctx(collection), results) is True + + +class TestDegradationWarning: + def test_warns_when_resolution_was_unavailable_and_something_went_unidentified(self) -> None: + collection = _collection( + unidentified=[UnidentifiedArtifact('app.war > WEB-INF/lib/shim.jar', 'b' * 40, 44)], + resolver_available=False, + ) + + assert binary_report.should_warn_about_degradation(_ctx(collection), collection) is True + + def test_does_not_warn_when_everything_was_identified(self) -> None: + collection = _collection(components=[_component(*_LOG4J)], resolver_available=False) + + assert binary_report.should_warn_about_degradation(_ctx(collection), collection) is False + + def test_offline_silences_the_warning(self) -> None: + collection = _collection( + unidentified=[UnidentifiedArtifact('app.war > WEB-INF/lib/shim.jar', 'b' * 40, 44)], + resolver_available=False, + ) + + assert binary_report.should_warn_about_degradation(_ctx(collection, offline=True), collection) is False + + def test_offline_does_not_change_the_counts(self) -> None: + # acknowledging the trade-off silences the warning; it must not make the numbers less true + collection = _collection( + components=[_component(*_LOG4J)], + unidentified=[UnidentifiedArtifact('app.war > WEB-INF/lib/shim.jar', 'b' * 40, 44)], + ) + + assert binary_report.get_coverage_summary(collection, 3) == '1 identified | 1 unidentified | 3 vulnerabilities' + + def test_manifest_only_matches_are_called_out_in_the_coverage_line(self) -> None: + # "2 identified" with one of them guessed from a manifest is a different result from two exact matches + collection = _collection( + components=[ + _component(*_LOG4J), + _component(*_MYSTERY, evidence=EVIDENCE_MANIFEST, confidence=CONFIDENCE_AMBIGUOUS), + ], + unidentified=[UnidentifiedArtifact('app.war > WEB-INF/lib/shim.jar', 'b' * 40, 44)], + ) + + assert collection.low_confidence_count == 1 + assert ( + binary_report.get_coverage_summary(collection, 0) + == '2 identified (1 low confidence) | 1 unidentified | 0 vulnerabilities' + ) + + +class TestUnidentified: + def test_entries_are_sorted_for_a_stable_diff(self) -> None: + collection = _collection( + unidentified=[ + UnidentifiedArtifact('app.war > WEB-INF/lib/z.jar', 'c' * 40, 10), + UnidentifiedArtifact('app.war > WEB-INF/lib/a.jar', 'd' * 40, 20), + ] + ) + + assert [entry.logical_path for entry in binary_report.get_unidentified(collection)] == [ + 'app.war > WEB-INF/lib/a.jar', + 'app.war > WEB-INF/lib/z.jar', + ] + + +class TestFormatSize: + @pytest.mark.parametrize( + ('size', 'expected'), + [(0, '0 B'), (512, '512 B'), (1024, '1 KB'), (44000, '43 KB'), (5 * 1024 * 1024, '5.0 MB')], + ) + def test_human_readable_sizes(self, size: int, expected: str) -> None: + assert binary_report.format_size(size) == expected + + +class TestDegradationWording: + """The warning must not read like a transient outage: tier 2 has not shipped, it has not gone down.""" + + def _lines(self) -> list: + collection = _collection( + components=[_component(*_LOG4J)], + unidentified=[UnidentifiedArtifact('app.war > WEB-INF/lib/shim.jar', 'b' * 40, 44)], + ) + return binary_report.get_degradation_lines(collection) + + def test_it_leads_with_the_coverage_gap(self) -> None: + assert self._lines()[0].startswith('1 of 2 components could not be identified') + + def test_it_says_partial(self) -> None: + assert 'PARTIAL' in ' '.join(self._lines()) + + def test_it_does_not_imply_an_outage(self) -> None: + text = ' '.join(self._lines()) + + # "unavailable" reads as "it broke, try later"; this tier has simply not been built yet + assert 'not available in this release' in text + assert 'lookup unavailable' not in text + + def test_it_names_the_way_out(self) -> None: + text = ' '.join(self._lines()) + + assert '--offline' in text + assert '--maven-central' in text + + def test_a_lookup_that_failed_says_so_instead(self) -> None: + collection = _collection( + components=[_component(*_LOG4J)], + unidentified=[UnidentifiedArtifact('app.war > WEB-INF/lib/shim.jar', 'b' * 40, 44)], + ) + collection.results_by_artifact[ + 'app.war' + ].resolver_unavailability_reason = ( + 'Maven Central lookup failed (timed out); digests after that point were not resolved.' + ) + + text = ' '.join(binary_report.get_degradation_lines(collection)) + + assert 'Maven Central lookup failed (timed out)' in text + assert 'not available in this release' not in text