feat: retain and merge original source values

[SkyeAv]

Always retain original_* edge fields and deterministically aggregate distinct values during collision merges. Render values as sorted pipe-delimited strings and remove the obsolete --no-original API and CLI option.\n\n### Original-value retention\n- Deterministic aggregation: Collect distinct non-empty original_* scalar values independently, sort by UTF-8 bytes, and render A, A|B, or A|B|C.\n- Existing merge semantics: Preserve first-wins scalar handling and conflict counting for non-original fields.\n- Parity: Keep the Rust merge implementation and Python reference behavior aligned, including already pipe-joined values and empty values.\n\n### CLI and API\n- Removal: Delete the build-kg --no-original option and related API parameters rather than retaining a no-op.\n- Documentation: Update the CLI documentation and changelog to describe always-retained original fields.\n\n### Testing\n-
running 112 tests
test fullmap::tests::clean_strips_matching_and_duplicate_quotes ... ok
test fullmap::tests::increase_nofile_limit_reaches_hard_limit ... ok
test fullmap::tests::is_dead_term_classifies_banned_empty_and_numeric ... ok
test fullmap::tests::emit_term_inserts_normalized_forms_and_accumulates ... ok
test fullmap::tests::level_normalization_matches_fullmap_shape ... ok
test fullmap::tests::flush_shard_batch_cursor_falls_back_on_duplicate_hash ... ok
test fullmap::tests::lookup_fullmap_terms_wraps_every_fetch_in_detach ... ok
test fullmap::tests::build_fullmap_db_rejects_empty_synonym_list ... ok
test fullmap::tests::equiv_index_multi_run_merge_and_lookup ... ok
test fullmap::tests::cached_handle_serves_old_snapshot_while_path_absent ... ok
test fullmap::tests::merge_heap_merges_overlapping_runs_sorted_and_deduped ... ok
test fullmap::tests::lookup_rejects_v1_schema ... ok
test fullmap::tests::lookup_rejects_v3_schema_as_outdated ... ok
test fullmap::tests::cached_shard_follows_replacement_generation ... ok
test fullmap::tests::cached_primary_follows_replacement_generation ... ok
test fullmap::tests::lookup_rejects_v2_and_unsupported_schemas ... ok
test fullmap::tests::bundle_open_all_absent_serves_old_snapshot ... ok
test fullmap::tests::read_only_open_maps_outdated_file_format_to_rebuild_hint ... ok
test fullmap::tests::read_only_open_maps_recovery_required_to_rebuild_hint ... ok
test fullmap::tests::round_down_pow2_rounds_to_nearest_lower_power_of_two ... ok
test fullmap::tests::curie_run_roundtrip_spans_buffer_boundary ... ok
test fullmap::tests::legacy_four_shard_db_reads_only_four_shards ... ok
test fullmap::tests::runtime_shard_count_builds_and_reads_fewer_shards ... ok
test fullmap::tests::split_bucket_partitions_without_loss ... ok
test fullmap::tests::split_counts_balances_surplus_workers_deterministically ... ok
test fullmap::tests::term_run_writer_reader_roundtrip ... ok
test fullmap::tests::term_shard_is_deterministic_and_balanced ... ok
test fullmap::tests::token_qc_rejects_banned_tokens ... ok
test fullmap::tests::build_fullmap_db_deduplicates_curie_rows ... ok
test fullmap::tests::two_read_only_handles_coexist_on_one_file ... ok
test fullmap::tests::shard_count_of_rounds_non_pow2_meta_down ... ok
test fullmap::tests::curie_rows_spill_to_disk_and_stream_back ... ok
test json::tests::strip_nulls_keeps_emptied_nested_dict_and_list_scalars ... ok
test json::tests::strip_nulls_keeps_zero_and_false ... ok
test json::tests::strip_nulls_removes_absent_and_null_like_values ... ok
test fullmap::tests::build_fullmap_db_writes_sharded_layout ... ok
test fullmap::tests::writer_blocks_read_only_open_until_dropped ... ok
test ndjson::merge_state_desync::merge_state_desync_is_a_structured_error_not_silent_truncation ... ok
test ndjson::tests::declared_uuid_fields_hold_the_id_still_across_attribute_edits ... ok
test fullmap::tests::bundle_open_all_absent_rejects_diverged_cached_build_ids ... ok
test ndjson::tests::dedup_ndjson_deduplicates_nodes ... ok
test ndjson::tests::dedup_ndjson_duplicate_edges_get_stable_single_id ... ok
test ndjson::tests::dedup_ndjson_empty_object_only_writes_empty_output ... ok
test ndjson::tests::dedup_ndjson_keeps_distinct_records ... ok
test ndjson::tests::dedup_ndjson_labels_edges ... ok
test fullmap::tests::bundle_open_primary_absent_shards_current_exhausts_then_follows_primary ... ok
test ndjson::tests::dedup_ndjson_skips_blank_lines ... ok
test ndjson::tests::differing_keys_reports_only_real_differences_and_never_the_id ... ok
test ndjson::tests::format_original_values_uses_sorted_pipe_joining ... ok
test ndjson::tests::dedup_ndjson_strips_nested_null_like_values ... ok
test ndjson::tests::explicit_error_mode_still_aborts_on_divergent_edges ... ok
test ndjson::tests::identical_edges_still_collapse_under_declared_uuid_fields ... ok
test ndjson::tests::key_order_alone_no_longer_ships_a_duplicate_id ... ok
test ndjson::tests::merge_mode_aggregates_original_scalars_across_all_records ... ok
test ndjson::tests::merge_mode_dedups_list_objects_by_canonical_bytes ... ok
test ndjson::tests::merge_mode_case_count_is_order_independent ... ok
test ndjson::tests::merge_mode_folds_divergent_edges_into_one ... ok
test ndjson::tests::merge_mode_keeps_fields_only_the_second_record_carries ... ok
test ndjson::tests::merge_mode_one_sided_case_ids_still_recompute_the_count ... ok
test ndjson::tests::merge_mode_ignores_empty_originals_and_is_order_independent ... ok
test ndjson::tests::merge_mode_recomputes_number_of_cases_from_case_id_union ... ok
test ndjson::tests::merge_mode_output_is_order_independent ... ok
test ndjson::tests::merge_mode_sorts_only_lists_that_went_through_a_union ... ok
test ndjson::tests::merge_mode_suppresses_exact_repeats_without_remerging ... ok
test ndjson::tests::merge_mode_without_case_ids_keeps_first_wins_number_of_cases ... ok
test ndjson::tests::record_if_new_suppresses_only_exact_byte_duplicates ... ok
test ndjson::tests::merge_mode_writes_edges_in_first_seen_order ... ok
test ndjson::tests::streaming_edges_never_ship_supporting_case_ids ... ok
test ndjson::tests::the_domain_separates_identical_edges_across_graphs ... ok
test ndjson::tests::undeclared_uuid_fields_let_the_id_drift ... ok
test ndjson::tests::unknown_on_collision_is_rejected ... ok
test tests::xxh64_file_matches_known_digests ... ok
test ndjson::tests::uuid_fields_that_are_not_a_key_abort_the_build ... ok
test tests::xxh64_file_reports_io_failures_as_os_error ... ok
test tests::xxh64_matches_known_digests ... ok
test uuid::tests::a_missing_declared_field_contributes_nothing ... ok
test uuid::tests::declared_field_order_does_not_matter ... ok
test uuid::tests::declared_fields_still_track_declared_changes ... ok
test uuid::tests::distinct_domains_separate_identical_records ... ok
test uuid::tests::declared_fields_ignore_undeclared_changes ... ok
test uuid::tests::false_is_hashed_rather_than_dropped ... ok
test uuid::tests::nested_array_order_still_changes_the_uuid ... ok
test uuid::tests::golden_vectors_pin_the_encoding ... ok
test uuid::tests::uuid_for_json_object_distinguishes_distinct_keys ... ok
test uuid::tests::nested_object_key_order_does_not_change_the_uuid ... ok
test uuid::tests::uuid_for_json_object_rejects_non_objects ... ok
test uuid::tests::uuid_for_json_object_has_no_key_value_boundary_collision ... ok
test fullmap::tests::build_fullmap_db_cleans_quoted_names_before_indexing ... ok
test uuid::tests::uuid_for_json_object_is_independent_of_key_order ... ok
test uuid::tests::uuid_for_json_object_returns_uuid_shape ... ok
test fullmap::tests::builds_and_reads_redb_records ... ok
test uuid::tests::uuid_from_parts_is_deterministic ... ok
test uuid::tests::uuid_from_parts_is_injective_across_part_boundaries ... ok
test fullmap::tests::builds_records_from_alias_fields ... ok
test fullmap::tests::build_fullmap_db_does_not_index_banned_name_tokens ... ok
test fullmap::tests::evict_cached_path_removes_primary_and_all_shards ... ok
test fullmap::tests::parallel_shard_fanout_merges_in_input_order ... ok
test fullmap::tests::dead_term_filter_drops_numeric_synonyms ... ok
test fullmap::tests::prefix_exclusion_drops_matching_rows ... ok
test ndjson::merge_fold_reference::merge_fold_matches_reference_on_fuzz ... ok
test fullmap::tests::repeated_lookups_reuse_cached_database ... ok
test fullmap::tests::per_shard_spill_runs_and_parallel_merges_match_reference ... ok
test fullmap::tests::bundle_open_mixed_window_exhausts_then_follows_new_generation ... ok
test fullmap::tests::public_build_ignores_shards_env_var ... ok
test fullmap::tests::large_batch_lookup_defaults_to_parallel ... ok
test json::tests::canonical_json_bytes_matches_reference_on_fuzz ... ok
test fullmap::tests::workers_above_shard_count_still_match_serial ... ok
test fullmap::tests::lookup_after_rebuild_reads_one_consistent_generation ... ok
test fullmap::tests::intra_file_parallelism_single_file_many_workers ... ok
test fullmap::tests::spill_merge_matches_single_run_build ... ok
test fullmap::tests::parallel_writers_match_single_writer_build ... ok
test ndjson::merge_fold_speedup::merge_fold_speedup_bound_vs_reference ... ok

test result: ok. 112 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.62s

running 11 tests
test regenerate_golden ... ignored
test null_preferred_name_falls_back_to_curie ... ok
test dimension_tables_are_complete_and_consistent ... ok
test automatic_parallelism_matches_golden ... ok
test schema_and_shard_count_are_pinned ... ok
test golden_output_is_pinned ... ok
test class_row_without_equivalents_builds ... ok
test synonym_row_with_no_names_indexes_only_curie ... ok
test empty_synonym_file_builds_empty_db ... ok
test deterministic_across_rebuilds ... ok
test gz_input_matches_plain_input ... ok

test result: ok. 10 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.47s

running 14 tests
test missing_archive_is_a_not_found_error ... ok
test path_traversal_entry_is_rejected_and_writes_nothing_outside ... ok
test corrupt_archive_is_rejected_and_leaves_nothing ... ok
test symlink_entry_is_rejected ... ok
test pax_sparse_archive_is_rejected ... ok
test archive_without_primary_is_rejected ... ok
test extra_shard_is_rejected_and_named ... ok
test outdated_schema_is_rejected_and_demands_rebuild ... ok
test non_redb_primary_is_rejected ... ok
test missing_shard_is_rejected_and_named ... ok
test multiple_unnamed_primaries_are_rejected_and_listed ... ok
test progress_callback_details_are_pinned ... ok
test named_fullmap_primary_is_preferred_over_strays ... ok
test extract_prebuilt_matches_force_build ... ok

test result: ok. 14 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.40s

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s — passed.\n- — passed.\n- bringing up nodes...
bringing up nodes...

........................................................................ [ 22%]
........................................................................ [ 45%]
........................................................................ [ 68%]
........................................................................ [ 91%]
........................F. [100%]
=================================== FAILURES ===================================
________ test_compile_subgraph_threads_fine_phases_into_resolve_and_qc _________
[gw15] linux -- Python 3.13.13 /home/skyeav/Code/ISB/Tablassert-original-values/.venv/bin/python

monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7287fbf98f30>
tmp_path = PosixPath('/tmp/pytest-of-skyeav/pytest-4700/popen-gw15/test_compile_subgraph_threads_0')

def test_compile_subgraph_threads_fine_phases_into_resolve_and_qc(monkeypatch: Any, tmp_path: Path) -> None:
    """compile_subgraph forwards on_phase into resolve_batch/fullmap_audit so fine sub-phases fire in order."""
    rows: dict[str, list[dict[str, object]]] = {
        "brca1": [fake_fullmap_row("brca1", "HGNC:1100", "BRCA1", "Gene", 9606)],
        "tp53": [fake_fullmap_row("tp53", "HGNC:11998", "TP53", "Gene", 9606)],
    }
    install_fake_fullmap(monkeypatch, rows)

    # Safety net: this data passes QC at the exact stage, so SapBERT must never run (keeps the test offline).
    class DummySapBERT:
        def encode(self, values: list[str]) -> object:
            raise AssertionError("Stage 4 (SapBERT) must not run for exact-match QC data")

    monkeypatch.setattr("tablassert.qc.get_sapbert", lambda: DummySapBERT())

    table_path, _ = write_text_section(
        tmp_path,
        "fine_phases",
        {
            "statement": {"subject": {"method": "value", "encoding": "BRCA1"}, "object": {"method": "value", "encoding": "TP53"}},
            "provenance": {"repo": "PMC", "publication": "PMC0000000"},
        },
        ["ignored"],
    )
    data: Any = from_yaml(table_path)
    store: Path = tmp_path / "fine_phases.parquet"
    tcode_model: Tcode = Tcode.model_validate({**data, "config": table_path, "store": store, "qc": True})  # pyright: ignore

    phases: list[str] = []

  result_path: Path = lib.compile_subgraph(tcode_model.collect(tmp_path / "fullmap.redb"), on_phase=phases.append)  # pyright: ignore

                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

tests/test_lib.py:3591:

---

src/tablassert/lib.py:1488: in compile_subgraph
acc = fn(acc, *args, on_phase=on_phase) if acc is not None else fn(*args, on_phase=on_phase) # pyright: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/tablassert/qc.py:169: in fullmap_audit
extras.require("qc", required_by="the QC audit")

---

extra = 'qc'

def require(extra: str, *, required_by: str | None = None) -> None:
    """Fail loudly unless ``extra`` is fully installed.

    Preflight guard: call it at the point the user's intent is known (a ``--qc`` flag, the
    ``agent`` command) rather than where the import happens, so the error arrives before
    the work instead of after it.

    Args:
        extra: A detectable extra (``qc``, ``agent``, ``optimize`` or ``log``).
        required_by: Feature name for the message, phrased to follow "required by".
            Defaults to the extra's entry in :data:`FEATURES`.

    Raises:
        QcRuntimeMissingError: If ``extra`` is ``qc`` and any of its packages is absent
            (kept distinct so the documented ``qc-runtime-missing`` code still fires).
        MissingExtraError: If any other extra is incomplete.
    """
    absent: tuple[str, ...] = missing(extra)
    if not absent:
        return
    if extra == "qc":

      raise QcRuntimeMissingError(absent)

E tablassert.errors.QcRuntimeMissingError: Missing optional dependencies 'scikit-learn', 'sentence-transformers' — required by the QC audit. Install the [qc] extra: pip install "tablassert[qc]" (uv: uv tool install "tablassert[qc]")
E
E For further information visit https://tablassert.readthedocs.io/errors/qc-runtime-missing

src/tablassert/extras.py:115: QcRuntimeMissingError
=============================== warnings summary ===============================
tests/test_lib.py::test_compile_subgraph_e2e_column_cleanup_and_numeric_annotations
/home/skyeav/Code/ISB/Tablassert-original-values/.venv/lib/python3.13/site-packages/pydantic/main.py:732: BiolinkRelocationWarning: sample size (coerced to study_size) is study-level metadata, so its value is carried on the inlined supporting study as Study.study_size rather than emitted on the edge itself (biolink-model 4.4.4 replaced the deprecated supporting_study_* association slots with Study node properties). Use a slot a Biolink association declares (e.g. p_value, adjusted_p_value) if you need it on the edge itself.
return cls.pydantic_validator.validate_python(

tests/test_lib.py::test_compile_subgraph_e2e_column_cleanup_and_numeric_annotations
/home/skyeav/Code/ISB/Tablassert-original-values/.venv/lib/python3.13/site-packages/pydantic/main.py:732: BiolinkRelocationWarning: miscellaneous_notes is not a Biolink association slot, so it is folded into supporting_text as a "miscellaneous_notes: " string rather than emitted as its own edge field.
return cls.pydantic_validator.validate_python(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________

Name Stmts Miss Cover Missing

src/tablassert/init.py 4 0 100%
src/tablassert/_lazy.py 26 8 69% 39-49, 57-58
src/tablassert/agent.py 2225 2225 0% 15-5183
src/tablassert/biolink.py 286 26 91% 131, 153, 338, 353, 382, 673, 676, 710, 712, 715, 723, 744-745, 800-801, 843, 851-855, 866, 868, 899, 905, 907
src/tablassert/cli.py 590 216 63% 94, 102-105, 175-176, 278-279, 325-326, 343-351, 367-393, 426-445, 449-457, 475-487, 713, 755-758, 774-796, 882-1066, 1091-1101
src/tablassert/coerce.py 225 0 100%
src/tablassert/distill.py 49 49 0% 15-108
src/tablassert/enums.py 83 0 100%
src/tablassert/errors.py 62 16 74% 134, 151-156, 180, 202-209
src/tablassert/extras.py 37 15 59% 113, 116, 133-138, 152-163
src/tablassert/fullmap.py 241 86 64% 48-49, 57-62, 78, 103-118, 151-153, 166-178, 192, 199, 206-214, 220-259, 338, 372, 376-377, 382-383, 385-386, 388, 391-392, 423, 438-445, 516, 619, 629, 632, 641-642, 670, 715
src/tablassert/graph_target.py 71 71 0% 8-135
src/tablassert/ingests.py 36 12 67% 27-36, 39-42
src/tablassert/lib.py 548 46 92% 311-312, 314, 319, 421, 425-427, 439, 444, 688-695, 784, 838-839, 843-844, 853-854, 875, 919-926, 944-951, 968-970, 1009-1010, 1136, 1354, 1620, 1899, 1903
src/tablassert/log.py 70 13 81% 47-48, 58-61, 77-80, 100, 103, 115-122
src/tablassert/models.py 514 104 80% 80-83, 105, 118-133, 154, 185-186, 197-198, 233, 245, 268-269, 312-328, 371, 387, 408-415, 429, 486, 499, 551, 558, 570, 574, 627, 630, 640, 645, 648, 650, 663, 688, 693, 695, 722, 738-747, 773, 828-850, 890, 894, 936, 953-957, 1027, 1093, 1109, 1159, 1164-1189, 1203-1204, 1215, 1228
src/tablassert/nlp.py 11 0 100%
src/tablassert/progress.py 139 33 76% 67-69, 103-106, 110-113, 205-233, 238
src/tablassert/qc.py 127 106 17% 43-44, 61-85, 102-113, 172-288
src/tablassert/rig.py 278 43 85% 101, 104, 135, 138, 152-159, 196-199, 215, 275, 311, 319-320, 338, 396, 411, 423, 425, 427, 431-434, 444, 487, 616-623, 680-681
src/tablassert/study.py 132 132 0% 1-267
src/tablassert/utils.py 26 7 73% 31, 33, 36-37, 50-52

TOTAL 5780 3208 44%
=========================== short test summary info ============================
FAILED tests/test_lib.py::test_compile_subgraph_threads_fine_phases_into_resolve_and_qc
1 failed, 313 passed, 2 warnings in 16.13s — 314 passed, 2 warnings.\n- All checks passed! — passed.\n- /home/skyeav/Code/ISB/Tablassert-original-values/src/tablassert/cli.py
/home/skyeav/Code/ISB/Tablassert-original-values/src/tablassert/cli.py:449:44 - error: "logger" is unknown import symbol (reportAttributeAccessIssue)
/home/skyeav/Code/ISB/Tablassert-original-values/src/tablassert/log.py
/home/skyeav/Code/ISB/Tablassert-original-values/src/tablassert/log.py:21:10 - error: Import "loguru" could not be resolved (reportMissingImports)
/home/skyeav/Code/ISB/Tablassert-original-values/src/tablassert/log.py:107:10 - error: Import "loguru" could not be resolved (reportMissingImports)
/home/skyeav/Code/ISB/Tablassert-original-values/src/tablassert/qc.py
/home/skyeav/Code/ISB/Tablassert-original-values/src/tablassert/qc.py:12:12 - error: Import "sentence_transformers" could not be resolved (reportMissingImports)
/home/skyeav/Code/ISB/Tablassert-original-values/src/tablassert/qc.py:174:10 - error: Import "sklearn.metrics.pairwise" could not be resolved (reportMissingImports)
/home/skyeav/Code/ISB/Tablassert-original-values/src/tablassert/qc.py:212:61 - error: Argument of type "Unknown | object" cannot be assigned to parameter "values" of type "ArrayLike | None" in function "init"
  Type "Unknown | object" is not assignable to type "ArrayLike | None"
    Type "object" is not assignable to type "ArrayLike | None"
      "object" is incompatible with protocol "Iterable[Any]"
        "iter" is not present
      "object" is not assignable to "Series"
      "object" is incompatible with protocol "PyArrowArray"
        "buffers" is not present
        "tolist" is not present
... (reportArgumentType)
/home/skyeav/Code/ISB/Tablassert-original-values/src/tablassert/qc.py:212:102 - error: Argument of type "Unknown | object" cannot be assigned to parameter "values" of type "ArrayLike | None" in function "init"
  Type "Unknown | object" is not assignable to type "ArrayLike | None"
    Type "object" is not assignable to type "ArrayLike | None"
      "object" is incompatible with protocol "Iterable[Any]"
        "iter" is not present
      "object" is not assignable to "Series"
      "object" is incompatible with protocol "PyArrowArray"
        "buffers" is not present
        "tolist" is not present
... (reportArgumentType)
/home/skyeav/Code/ISB/Tablassert-original-values/src/tablassert/qc.py:214:43 - error: Operator ">=" not supported for types "Unknown | object" and "Literal[70]"
  Operator ">=" not supported for types "object" and "Literal[70]" (reportOperatorIssue)
/home/skyeav/Code/ISB/Tablassert-original-values/src/tablassert/qc.py:214:66 - error: Operator ">=" not supported for types "Unknown | object" and "Literal[80]"
  Operator ">=" not supported for types "object" and "Literal[80]" (reportOperatorIssue)
/home/skyeav/Code/ISB/Tablassert-original-values/tests/test_qc.py
/home/skyeav/Code/ISB/Tablassert-original-values/tests/test_qc.py:5:8 - error: Import "numpy" could not be resolved (reportMissingImports)
/home/skyeav/Code/ISB/Tablassert-original-values/tests/test_qc.py:9:6 - error: Import "sklearn.metrics" could not be resolved (reportMissingImports)
11 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to latest — passed.\n- — passed.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 4b7ae671-e607-47ff-9397-04bf65fa0761

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}