Skip to content

feat(sql-insight): creator + generate_qualified_name for SqlInsightFilter and SqlInsightBusinessQuestion - #1030

Open
shubhambatlan wants to merge 3 commits into
mainfrom
feat/sql-insight-filter-question-creators
Open

feat(sql-insight): creator + generate_qualified_name for SqlInsightFilter and SqlInsightBusinessQuestion#1030
shubhambatlan wants to merge 3 commits into
mainfrom
feat/sql-insight-filter-question-creators

Conversation

@shubhambatlan

@shubhambatlan shubhambatlan commented Sep 1, 2026

Copy link
Copy Markdown

✨ Description

SqlInsightJoin ships creator() and generate_qualified_name(). The other two
SqlInsight types had neither, so the only way to write a filter or a business
question was to hand-build the entity and hand-compute its qualifiedName.

That matters more than the missing convenience. A SQL insight's qualifiedName ends in
an md5 of its own content, and the SQL-Intelligence miner derives it the same way — so
a caller who gets the identity wrong by one byte creates a duplicate of the mined
row rather than converging on it. SqlInsightJoin.generate_qualified_name exists so
the SDK, the miner and every other caller share one implementation; this extends that
to the remaining two types.

Formulas, matching the miner's sql_intelligence DAG and the UI's authoring path:

filter    <columnQN>/filter/md5(operator)
question  <datasetQN>/question/md5(questionText)

Both are single-argument md5 with no ordering, so neither carries the composite-key
ordering subtlety the join formula does.

Jira link: n/a — raised from field work on the talk-to-data write-back path


🧩 Type of change

  • 🚀 New feature (non-breaking change that adds functionality)

Purely additive: two new methods per type. No existing signature or behaviour changes.


✅ How has this been tested?

23 new unit tests, all passing. tests/unit/model is otherwise unaffected —
connection_test.py::test_creator_async_rejects_invalid_connector_type_value fails on
main as well and predates this branch.

The tests are written against the identity contract, not the implementation:

  • qualifiedNames are pinned as literals, never recomputed from the formula under
    test. The contract is byte-identity with a system this SDK does not control, so an
    expectation derived from the same code would assert nothing and would follow a wrong
    formula without complaint.
  • an RFC 1321 md5 vector is asserted separately, so a broken digest or encoding
    fails on its own rather than taking every golden down with it.
  • convergence and discrimination: the same insight yields one qualifiedName; a
    different operator or a reworded question yields a different one. A formula that
    ignored its inputs would pass a convergence-only test.
  • the segment literal and the 32-hex lowercase digest are asserted directly,
    because that is exactly how this identity has been got wrong before — a business
    question once shipped as /businessQuestion/ plus a truncated sha256.
  • both anchorings, because a row carrying only one is half-visible: the asset
    page's Usage & Intelligence tab finds these by the dataset attribute, while the
    relationship is what renders the row on the asset itself.
  • query_count / unique_users are 0 — a human-declared insight has no query
    history and must never report invented popularity.

Cross-checked independently: the qualifiedNames these creators produce are
byte-identical to those written by an existing REST client whose filter and
business-question writes were confirmed against a live tenant.

✅ Regenerated and confirmed

The generated assets are committed, so this PR includes them. I have now run the real
generator (create_typedefs_file.pyclass_generator.py) against a live internal
sandbox in a throwaway worktree and compared the output:

All four method blocks — generate_qualified_name and creator, on both types —
are byte-identical to what is committed here.

Two notes for a reviewer:

  • The sandbox's typedefs have drifted well past the 11.0.0 baseline (a full
    regeneration touched 541 files), so only the two sql_insight_* files were compared
    and everything else was discarded. A regeneration against the typedefs this release
    is cut from would be the authoritative check; this one confirms the templates render
    as expected.
  • The generator emits the full global import block and leaves pruning to the
    formatting step. The committed files carry the pruned form, matching the existing
    sql_insight_join.py convention exactly.

Formatting is ruff throughout (ruff check, ruff --select I --fix, ruff-format)
ruff check passes on all five files. Where ruff-format reshaped a line in the
generated output, the same shape was applied to the template, so the next regeneration
does not churn the file back.


📋 Checklist

  • My code follows the project's style guidelines (black + isort + flake8 clean)
  • I've performed a self-review of my code
  • I've added comments in tricky or complex areas
  • I've updated the documentation as needed (docstrings on all four methods)
  • There are no new warnings from my changes
  • I've added tests to cover my changes
  • All new and existing tests pass locally

HISTORY.md is untouched — it looks curated at release time rather than per PR. Happy
to add an entry if that is the convention.

Security review

No new dependencies, no network calls, no credential handling, no logging. hashlib.md5
is used solely to reproduce an existing identity string and is marked # noqa: S324
with that rationale, matching the existing SqlInsightJoin usage.


🔭 Related, not fixed here

While cross-checking the formulas I found that atlan-frontend's
sqlInsightIdentity.ts sorts join column pairs by the formatted SOURCE=JOINED
string, whereas the miner and this SDK both order by the source column alone. The
two diverge on composite keys where one source column is a prefix of another followed
by a digit (ORDER_ID / ORDER_ID2), because = is 0x3D and digits are
0x30-0x39 — so a UI-authored composite join can duplicate a mined row. Single-pair
joins are unaffected. Nothing in this PR is affected; flagging it for whoever owns SQL
Intelligence.

shubhambatlan and others added 3 commits September 1, 2026 15:05
…usiness question

SqlInsightJoin has both; the other two SqlInsight types had neither, so the only
way to write a filter or a business question was to hand-build the entity and
hand-compute its qualifiedName. That is how a caller ends up duplicating the
miner's row instead of converging on it: the identity has to be byte-identical
and there was no shared implementation to reach for.

Formulas, matching the SQL-Intelligence miner and the UI's authoring path:

    filter    <columnQN>/filter/md5(operator)
    question  <datasetQN>/question/md5(questionText)

Both are single-argument md5 with no ordering, so neither has the composite-key
ordering subtlety the join formula carries.

Verified: the rendered methods compile in place; creator() round-trips both
entities with the right attributes and relationship edges; identical inputs
produce identical qualifiedNames; missing column / operator / question text are
rejected; and the qualifiedNames are byte-identical to those written by an
existing REST client whose filter and business-question writes were confirmed
against a live tenant.

Notes on the shape, following the join precedent:
- Filters hang off the COLUMN, so the dataset qualifiedName is derived from the
  column's parent rather than accepted as a separate input that could disagree.
- Both the dataset ATTRIBUTE and the column/dataset RELATIONSHIP are written. The
  asset page's Usage & Intelligence tab finds filters by the attribute, while the
  relationship is what renders the row on the asset; one without the other is
  half-visible.
- query_count and unique_users are set to 0: a human-declared insight has no
  observed usage and must never claim any.
…erated output

The generated assets are committed, so the templates alone would not have shipped
anything; this adds the rendered methods and the tests that hold them honest.

Tests are written against the identity CONTRACT rather than the implementation:

- qualifiedNames are pinned as LITERALS, never recomputed from the formula in the
  test. The contract is byte-identity with the SQL-Intelligence miner and the UI's
  authoring path, so an expectation derived from the code under test asserts
  nothing and would follow a wrong formula without complaint.
- an RFC 1321 vector is asserted separately, so "md5 is broken" fails on its own
  rather than taking every golden down with it and hiding the cause.
- convergence AND discrimination: the same insight yields one qualifiedName, a
  different operator or a reworded question yields a different one. A formula that
  ignored its inputs would pass a convergence-only test.
- the segment literal and the 32-hex lowercase digest are asserted directly,
  because those are exactly how this identity was got wrong before (a business
  question once shipped as `/businessQuestion/` plus a truncated sha256).
- both anchorings, because a row carrying only one is half-visible: the Usage &
  Intelligence tab finds these by the dataset attribute, the relationship is what
  renders the row on the asset.
- query_count/unique_users are 0, so a human-declared insight never reports
  invented popularity.

Not covered: updater / trim_to_required, which these types do not implement (nor
does SqlInsightJoin).

23 tests pass. The rest of tests/unit/model is unaffected; connection_test's async
connector-type failure predates this branch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… output

The repo formats with ruff (pre-commit: ruff, ruff --select I --fix, ruff-format),
not black/isort. ruff-format joins the filter's default-name expression onto one
line; applying that to the generated file alone would have been undone by the next
regeneration, so the template carries the same shape.

ruff check passes on all five files; 23 tests still pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant