security: separate logical and host path contracts

docs/repo-sentinel-baseline-review.md

@@ -213,14 +213,23 @@ variables and global/system Git config are excluded; lazy fetching is disabled.
 Archive attributes, textconv, checkout filters, and executable file modes do
 not transform or execute the returned bytes.
 
-Initial admission is intentionally narrow: regular files with mode `100644`
-or `100755`, ASCII path components containing letters, digits, spaces, dots,
-underscores or hyphens, at most 255 bytes per component. Dot components, `.git`,
-Windows device names (including spaces before an extension), trailing
-dots/spaces, and case-folding collisions are
-refused. Symlinks, gitlinks and other modes refuse the entire snapshot. This
-portable subset excludes legitimate names, including non-ASCII names; refusal
-must remain visible rather than silently dropping files.
+Path admission now represents logical Git-tree identity rather than host
+filesystem portability. Each raw component must decode as strict UTF-8,
+re-encode to the original bytes, and remain within the 255 raw-byte component
+bound. Empty components, raw `/`, exact `.` or `..`, C0/C1 controls, DEL,
+malformed framing, exact duplicate paths, and exact file/directory namespace
+conflicts refuse the complete snapshot. Symlinks, gitlinks and other modes
+remain unsupported.
+
+The strict UTF-8 rule is a deliberate compatibility boundary for this reader,
+not a claim that arbitrary non-NUL, non-slash Git path bytes are invalid Git.
+Decoded names are not normalized, case-folded, transliterated, or repaired.
+Logical identity is exact Python `str` equality, so case variants and NFC/NFD
+variants remain distinct and exact-string sorting stays deterministic. Host-only
+concerns such as `.git`, backslash, colon, Windows device names, wildcards, and
+trailing dots or spaces are intentionally left to the materializer. Ordinary
+Unicode and inert punctuation such as `&`, apostrophes, `!`, `~`, and `$` are
+admitted without filename-specific exceptions.
 
 Defaults bound the complete read to 4,096 files, 8,192 object reads, 16 MiB of
 cumulative raw object bodies, 2 MiB per blob, 32 directory levels and 30 seconds.
@@ -234,12 +243,13 @@ Real bare-repository tests exercise both object formats, raw binary content,
 archive attributes, replacement refs, type/mode mismatch, unsafe paths,
 collisions, malformed trees and file/object/byte/depth budgets. A stalled child
 checks the deadline, and injected transport corruption checks independent
-identity validation. Run `python -m unittest tests.test_repo_sentinel_reader`.
+identity validation. Run
+`python -m unittest discover -s tests -p 'test_repo_sentinel_reader.py'`.
 
-This API is not wired into the scanner or workflow yet. PR-ref acquisition,
-filesystem materialization, producer/head binding and enforcement remain
-separate work. Existing scan and baseline behavior is unchanged. Rollback is
-removal of the reader and its tests; there is no persisted state or migration.
+These APIs are not wired into the scanner or workflow yet. PR-ref acquisition,
+producer/head binding, and enforcement remain separate work. Existing scan and
+baseline behavior is unchanged. Rollback restores the prior reader/materializer
+contract; there is no persisted state or migration.
 
 ### Temporary Snapshot Materialization
 
@@ -250,6 +260,34 @@ directory below an existing, caller-owned scratch root. The scratch root itself
 must be a real directory, not a symlink or Windows reparse point; its ancestors
 and stability are caller trust requirements.
 
+The empty private staging container is created first so root-dependent path
+limits can be evaluated. Before any snapshot subdirectory or file is populated,
+the materializer revalidates every `SnapshotFile.path`, derives all implicit
+directories, and completes indexed collision and type-prefix checks across the
+whole tree. This independent preflight also protects against caller-constructed
+`Snapshot` values that did not originate in the reader.
+
+Canonical materialization rejects Windows separators, drive/UNC/device forms,
+ADS colons, reserved punctuation and device names, including the Windows-defined
+superscript-digit `COM`/`LPT` aliases, trailing dots/spaces, and
+ASCII-case-insensitive `.git` components on every supported host. Portable-v1
+case aliases use an explicit ASCII-only `A-Z` mapping independent of the runner
+OS. NFC and NFD keys detect Unicode portability aliases separately without
+changing logical identity. This deterministic policy does not claim to emulate
+every evolving filesystem-specific Unicode casing rule. Full Unicode lowercasing
+or case folding is not used, so names such as `ẞ`/`ß`, `ß`/`SS`, and `İ`/`i`
+remain distinct. Logical names can therefore be reader-valid but
+materializer-invalid by design.
+
+The preflight retains the reader's 255-byte component and depth bounds. It also
+checks UTF-16 component and conservative `MAX_PATH` length against the actual
+private staging root on Windows, and runtime `pathconf` component/full-path
+limits on POSIX. Unavailable or indeterminate POSIX limits refuse the operation;
+the absolute limit is consequently a documented runtime-root precondition
+rather than an emulation of a foreign filesystem. Validated component tuples
+are joined with `root.joinpath(*components)`; unvalidated logical path strings
+never reach host path joining.
+
 Files are created exclusively and read back with exact byte comparison before
 the context yields `MaterializedSnapshot(snapshot, root)`. Existing files and
 symlinks are never overwritten. Directory entries are checked before use;
@@ -269,16 +307,19 @@ The caller must prevent concurrent mutation by other processes or the consumer.
 These checks do not defend against a hostile local actor changing ancestors or
 files during use. Reader byte/count limits bound the selected data, but its
 deadline does not bound filesystem I/O, cleanup or consumer execution. Host
-path-length or disk failures refuse materialization rather than shortening paths
-or returning fewer files. Empty Git directories are not part of the reader's
-file snapshot and are not reconstructed.
-
-Run `python -m unittest tests.test_repo_sentinel_materialize` for real-object
-fixtures covering bytes, executable-mode metadata, empty files/trees, lifecycle,
-reader refusal, collisions, symlinks, same-length corruption, partial setup failures
-and cleanup failures. Scanner execution, PR-ref acquisition and workflow
-activation remain unwired. Rollback removes this additive helper/tests/docs;
-there is no persisted output format or repository-setting migration.
+path-length or disk failures refuse materialization rather than shortening
+paths or returning fewer files. Empty Git directories are not part of the
+reader's file snapshot and are not reconstructed.
+
+Run
+`python -m unittest discover -s tests -p 'test_repo_sentinel_materialize.py'`
+for real-object and direct-`Snapshot` fixtures covering bytes, executable-mode
+metadata, empty files/trees, lifecycle, layered path refusal, aliases,
+normalization and type collisions, path limits, preflight atomicity, symlinks,
+same-length corruption, partial setup failures, and cleanup failures. Scanner
+execution, PR-ref acquisition, and workflow activation remain unwired. Rollback
+restores the prior reader/materializer contract; there is no persisted output
+format or repository-setting migration.
 
 ## Relationship To Issue #5
 

scripts/repo_sentinel_materialize.py

@@ -2,15 +2,17 @@
 
 from __future__ import annotations
 
+import ntpath
 import os
 import stat
+import unicodedata
 from collections.abc import Iterator
 from contextlib import contextmanager
 from dataclasses import dataclass, field
 from pathlib import Path
 from tempfile import TemporaryDirectory
 
-from repo_sentinel_reader import ReaderLimits, Snapshot, read_snapshot
+from repo_sentinel_reader import ReaderLimits, Snapshot, SnapshotFile, read_snapshot
 
 
 class MaterializationRefused(ValueError):
@@ -23,6 +25,141 @@ class MaterializedSnapshot:
     root: Path = field(repr=False)
 
 
+@dataclass(frozen=True)
+class _MaterializationPlan:
+    files: tuple[tuple[SnapshotFile, tuple[str, ...]], ...]
+    directories: tuple[tuple[str, ...], ...]
+
+
+_WINDOWS_FORBIDDEN = frozenset('<>:"\\|?*')
+_WINDOWS_DEVICES = frozenset({"CON", "PRN", "AUX", "NUL", "CONIN$", "CONOUT$"})
+_WINDOWS_PORT_DEVICE_SUFFIXES = frozenset("123456789¹²³")
+_ASCII_CASE_TRANSLATION = {
+    codepoint: codepoint + (ord("a") - ord("A"))
+    for codepoint in range(ord("A"), ord("Z") + 1)
+}
+
+
+def _has_control_characters(value: str) -> bool:
+    return any(ord(character) < 0x20 or 0x7f <= ord(character) <= 0x9f
+               for character in value)
+
+
+def _logical_components(path: str, limits: ReaderLimits) -> tuple[str, ...]:
+    if type(path) is not str or ntpath.splitdrive(path)[0] or ntpath.isabs(path):
+        raise MaterializationRefused("unsupported_host_path")
+    components = tuple(path.split("/"))
+    if (not components or len(components) > limits.max_depth + 1
+            or any(not component or component in (".", "..")
+                   or _has_control_characters(component) for component in components)):
+        raise MaterializationRefused("unsupported_host_path")
+    for component in components:
+        try:
+            encoded = component.encode("utf-8", errors="strict")
+        except UnicodeEncodeError:
+            raise MaterializationRefused("unsupported_host_path") from None
+        if len(encoded) > 255:
+            raise MaterializationRefused("path_limit")
+    return components
+
+
+def _validate_portable_component(component: str) -> None:
+    stem = component.split(".", 1)[0].rstrip(" ").upper()
+    if (any(character in _WINDOWS_FORBIDDEN for character in component)
+            or component.endswith((".", " "))
+            or (component.isascii() and component.lower() == ".git")
+            or stem in _WINDOWS_DEVICES
+            or (len(stem) == 4 and stem[:3] in ("COM", "LPT")
+                and stem[3] in _WINDOWS_PORT_DEVICE_SUFFIXES)):
+        raise MaterializationRefused("unsupported_host_path")
+
+
+def _path_key(
+    components: tuple[str, ...], normalization: str | None
+) -> tuple[str, ...]:
+    if normalization is not None:
+        return tuple(
+            unicodedata.normalize(normalization, component) for component in components
+        )
+    return tuple(component.translate(_ASCII_CASE_TRANSLATION) for component in components)
+
+
+def _pathconf(root: Path, name: str) -> int | None:
+    try:
+        value = os.pathconf(root, name)
+    except (AttributeError, OSError, ValueError):
+        return None
+    return value if type(value) is int and value > 0 else None
+
+
+def _validate_host_lengths(root: Path, paths: tuple[tuple[str, ...], ...]) -> None:
+    if os.name == "nt":
+        for components in paths:
+            for component in components:
+                if len(component.encode("utf-16-le")) // 2 > 255:
+                    raise MaterializationRefused("path_limit")
+            target = root.joinpath(*components)
+            if len(str(target).encode("utf-16-le")) // 2 >= 260:
+                raise MaterializationRefused("path_limit")
+        return
+    if os.name != "posix":
+        raise MaterializationRefused("unsupported_host_path")
+    name_limit = _pathconf(root, "PC_NAME_MAX")
+    path_limit = _pathconf(root, "PC_PATH_MAX")
+    if name_limit is None or path_limit is None:
+        raise MaterializationRefused("unsupported_host_path")
+    for components in paths:
+        try:
+            if name_limit is not None and any(
+                len(os.fsencode(component)) > name_limit for component in components
+            ):
+                raise MaterializationRefused("path_limit")
+            target = root.joinpath(*components)
+            if path_limit is not None and len(os.fsencode(target)) >= path_limit:
+                raise MaterializationRefused("path_limit")
+        except UnicodeEncodeError:
+            raise MaterializationRefused("unsupported_host_path") from None
+
+
+def _preflight_snapshot(
+    snapshot: Snapshot, root: Path, limits: ReaderLimits
+) -> _MaterializationPlan:
+    files: list[tuple[SnapshotFile, tuple[str, ...]]] = []
+    logical_nodes: dict[tuple[str, ...], str] = {}
+    for item in snapshot.files:
+        components = _logical_components(item.path, limits)
+        for component in components:
+            _validate_portable_component(component)
+        for depth in range(1, len(components)):
+            directory = components[:depth]
+            if logical_nodes.get(directory) == "file":
+                raise MaterializationRefused("path_collision")
+            logical_nodes.setdefault(directory, "directory")
+        if components in logical_nodes:
+            raise MaterializationRefused("path_collision")
+        logical_nodes[components] = "file"
+        files.append((item, components))
+
+    for normalization in (None, "NFC", "NFD"):
+        aliases: dict[tuple[str, ...], tuple[str, ...]] = {}
+        for logical_path in logical_nodes:
+            key = _path_key(logical_path, normalization)
+            previous = aliases.get(key)
+            if previous is not None and previous != logical_path:
+                raise MaterializationRefused("host_path_collision")
+            aliases[key] = logical_path
+
+    all_paths = tuple(logical_nodes)
+    _validate_host_lengths(root, all_paths)
+    directories = tuple(
+        sorted(
+            (path for path, kind in logical_nodes.items() if kind == "directory"),
+            key=lambda path: (len(path), path),
+        )
+    )
+    return _MaterializationPlan(tuple(files), directories)
+
+
 def _regular_path(path: Path, *, directory: bool = False) -> None:
     info = path.lstat()
     reparse = getattr(info, "st_file_attributes", 0) & getattr(
@@ -76,17 +213,15 @@ def materialized_snapshot(
                                            dir=scratch_root.resolve())
             root = Path(temporary.name) / "data"
             root.mkdir(mode=0o700)
-            directories = {root}
-            for item in snapshot.files:
-                directory = root
-                for component in item.path.split("/")[:-1]:
-                    directory = directory / component
-                    if directory not in directories:
-                        directory.mkdir(mode=0o700)
-                        directories.add(directory)
-                    _regular_path(directory, directory=True)
-                _regular_path(root, directory=True)
-                _write_verified(root / item.path, item.data)
+            _regular_path(root, directory=True)
+            plan = _preflight_snapshot(snapshot, root, limits)
+            for components in plan.directories:
+                directory = root.joinpath(*components)
+                directory.mkdir(mode=0o700)
+                _regular_path(directory, directory=True)
+            _regular_path(root, directory=True)
+            for item, components in plan.files:
+                _write_verified(root.joinpath(*components), item.data)
         except OSError:
             raise MaterializationRefused("filesystem_io_failed") from None
         yield MaterializedSnapshot(snapshot, root)

scripts/repo_sentinel_reader.py

@@ -90,16 +90,18 @@ def _git(repository: Path, arguments: list[str], cap: int, deadline: float) -> b
 
 
 def _component(raw: bytes) -> str:
-    # A deliberately portable subset, before any future filesystem writes.
-    if not re.fullmatch(rb"[A-Za-z0-9._ -]{1,255}", raw):
-        raise ReaderRefused("unsupported_path")
-    name = raw.decode("ascii")
-    stem = name.split(".", 1)[0].rstrip(" ").upper()
-    if (name in (".", "..") or name.lower() == ".git"
-            or name.endswith((".", " "))
-            or stem in {"CON", "PRN", "AUX", "NUL"}
-            or re.fullmatch(r"(?:COM|LPT)[1-9]", stem)):
-        raise ReaderRefused("unsupported_path")
+    if len(raw) > 255:
+        raise ReaderRefused("path_limit")
+    try:
+        name = raw.decode("utf-8", errors="strict")
+    except UnicodeDecodeError:
+        raise ReaderRefused("unsupported_path_encoding") from None
+    if name.encode("utf-8") != raw:
+        raise ReaderRefused("unsupported_path_encoding")
+    if (not name or b"/" in raw or name in (".", "..")
+            or any(ord(character) < 0x20 or 0x7f <= ord(character) <= 0x9f
+                   for character in name)):
+        raise ReaderRefused("invalid_logical_path")
     return name
 
 
@@ -161,9 +163,9 @@ def walk(oid: str, parent: str, depth: int) -> None:
             child = tree[nul + 1:nul + 1 + oid_bytes].hex()
             offset = nul + 1 + oid_bytes
             path = parent + name
-            if path.casefold() in paths:
+            if path in paths:
                 raise ReaderRefused("path_collision")
-            paths.add(path.casefold())
+            paths.add(path)
             if mode == b"40000":
                 walk(child, path + "/", depth + 1)
             elif mode in (b"100644", b"100755"):