feat(v9): opt-in lenient decoding for str-typed slots

[cmgrote]

Important

Draft — do not merge as-is. The root fix for the motivating case is
atlanhq/models#2041, which retypes Entity.custom_attributes to
Union[Dict[str, Any], None, UnsetType] at
toolkit/src/main/pkl/PythonMsgspecRenderer.pkl:2244. Measured against
application-sdk's canonical SQL transformed fixtures, that one-line renderer
change alone takes decoding from 1/105 to 105/105 records — the same result
this PR achieves, but by preserving values (1 stays 1, round-tripping
back out unchanged) instead of stringifying them, and with no hand-written
change to this repo beyond the regen.

This PR stays open because leniency still covers one case #2041 does not: a
typedef attribute whose declared map<string,string> the producer violates —
e.g. TableauDatasourceField.upstreamColumns carrying nested objects. But
silently coercing that suppresses exactly the finding a validator exists to
surface. Before merge it should coerce and report — surfacing the paths it
coerced — so a caller can count them as findings instead of having them vanish.

feat(v9): opt-in lenient decoding for str-typed slots

Problem

The v9 decoder is stricter than the Atlas server it models.

Atlas's string type accepts any value and stores its stringification —
AtlasBuiltInTypes.AtlasStringType in atlas-metastore:

public boolean isValidValue(Object obj) { return true; }
public String getNormalizedValue(Object obj) {
    if (obj != null) { return obj.toString(); }
    return null;
}

The generated models mirror the declared Atlas type faithfully — a typedef
attribute of array<map<string,string>> becomes List[Dict[str, str]], and
AtlasEntity.customAttributes (Map<String, String>) becomes
Dict[str, str] — but they cannot mirror that leniency, because msgspec
enforces str exactly.

So a producer that emits {"ordinal_position": 1} in customAttributes has
its entire record rejected here, while Atlas accepts the same record and
stores "1".

There is no existing seam for this:

Candidate	Why it doesn't work
msgspec.convert(..., strict=False) (already passed)	Lax mode coerces str → int, never int → str.
dec_hook=	Fires only for types msgspec cannot natively handle; a str/int mismatch raises before any hook runs.
Widening the annotation to Dict[str, Any]	Would fix it, but it is a typing-level breaking change for every consumer, and it belongs in atlanhq/models (PythonMsgspecRenderer.pkl:255 for the typedef mapping, :2244 for the hardcoded Entity.custom_attributes line).

Blast radius of the shape: the Atlas model set carries 45 map<string,string>
and 40 array<map<string,string>> attributes, which land as 64 distinct
Dict[str, str] field names across pyatlan_v9/model/assets/.

Fix

from_atlas_format(data, *, lenient_strings=False) and the same keyword on
from_atlas_json. When enabled, values bound for a str-typed slot are
coerced the way Atlas's string type would coerce them.

Default is off, so decoding an Atlas API response is byte-for-byte
unchanged. Turn it on when decoding a payload that has not been through
Atlas's normalisation — connector output, fixtures, a local validator — where
an int in a Dict[str, str] is a producer quirk Atlas would have stringified
rather than a contract breach.

The coercion plan is derived from the model itself via msgspec.inspect, so it
covers every str slot — plain str, Dict[str, str],
List[Dict[str, str]], List[str], and the same shapes on nested Related*
structs — with no hand-maintained list of field names to fall behind. A new
map<string,string> typedef attribute is covered the moment the models are
regenerated. Plans are cached per class; the cache entry is published before
the class is walked, so the cyclic Related* type graph terminates.

Policy, and the reasoning for each choice:

• Scalars become their compact JSON text. This matches
  getNormalizedValue exactly: 1 → "1", true → "true" (JSON, not
  Python's "True").
• Containers in a string slot also become compact JSON text. Atlas would
  store Java's Object.toString() ({a=b}); this stores {"a":"b"}. A
  documented divergence — JSON is the honest serialisation of a JSON value and
  it round trips.
• None is dropped from a slot that cannot hold it (Dict[str, str]
  values, and non-optional fields) and preserved where the model declares
  Optional. Dropping rather than stringifying is deliberate: a literal
  "null" is a value the producer never wrote.
• Non-str slots stay strict. A wrong-typed int or nested object
  elsewhere still raises, so genuine drift stays visible.

Logic lives in a new hand-maintained module, pyatlan_v9/model/string_coercion.py,
rather than inside transform.py — transform.py is stamped
Auto-generated by PythonMsgspecRenderer.pkl - DO NOT EDIT, so keeping the
logic out of it means a future regeneration costs the two-line wiring, not the
implementation.

Tests

tests_v9/unit/test_string_coercion.py — 29 tests:

• Every customAttributes shape: already-str, int, float, bool, null value,
  mixed, empty, nested object, nested array, explicit null map, absent,
  non-mapping.
• Every upstreamColumns shape, including the nested-object case and a ragged
  list.
• Plain str slots: scalar coerced; None preserved where the field is
  Optional.
• A nested RelatedTableauDatasourceField.upstream_columns reached through a
  relationship attribute — the shape a name-keyed fixup would miss.
• The flag off by default still raises, on both entry points.
• Non-str slots still raise with the flag on; malformed JSON still raises.
• The plan is model-derived (asserts upstreamColumns, upstreamFields,
  customAttributes, qualifiedName are all covered for one type) and cached
  per class.

7219 passed, 5 skipped across tests_v9/unit. ruff format --check and
ruff check clean. mypy clean on the new module (transform.py keeps its two
pre-existing register_asset errors, untouched by this change).

Performance

50k decodes of a realistic mysql column (24 customAttributes, most non-str):

	rec/s
lenient_strings=False (default path)	~205,000
lenient_strings=True, clean payload	~159,000
lenient_strings=True, dirty payload	~152,000

The default path is unaffected — the only added work is one if. Callers that
opt in pay ~26%.

Follow-ups (not in this PR)

1. atlanhq/models — PythonMsgspecRenderer.pkl:2244 hardcodes
   custom_attributes: Union[Dict[str, str], UnsetType] on the Entity base
   template, while the same renderer emits Any for the *Nested variant
   (referenceable.py:303). Worth making consistent regardless of this PR.
2. application-sdk can drop its private key-registry walk in
   application_sdk/validation/assets.py and pass lenient_strings=True
   instead — no inlined copy of from_atlas_json internals, and coverage of
   all 64 string-map fields rather than 2.

[linear]

FND-802

[Aryamanz29]

Thanks for the detailed writeup @cmgrote — the msgspec.inspect-derived plan (no hand-maintained field list, auto-covers new map<string,string> fields on regen), the cycle handling, and the off-by-default design are all solid.

A few thoughts, mostly agreeing with your own draft notes:

On v11 / the motivating case: I don't think we need this to land the customAttributes fix. models#2041 is merged, and my model-regen PR (#1024) already carries it — pyatlan_v9/model/assets/entity.py now has custom_attributes: Union[Dict[str, Any], None, UnsetType]. So the connector case is covered for v11 by the regen, and — as you note — by preserving values (1 stays 1, round-trips) rather than stringifying them. So no rush to squeeze #1022 in.

Prefer widening over coercing where the field is genuinely free-form. #2041's approach keeps the value; coercion mutates 1 → "1", which is lossy for any downstream consumer. For the remaining ~64 map<string,string> fields (upstreamColumns et al.), the same one-level-up fix — the renderer typedef mapping (PythonMsgspecRenderer.pkl:255, your follow-up #1) — preserves values, needs no per-call work, and no hand-maintained module. Worth doing that before reaching for runtime coercion.

The wiring lands in a generated file. transform.py is stamped DO NOT EDIT, so the two-line hook will be dropped on the next regen (which is exactly the flow #1024 runs). If we keep this path, the wiring should be emitted by the renderer, or lenient_strings should be a thin wrapper around from_atlas_format rather than an edit inside it — so the implementation survives regeneration cleanly.

Validator semantics are the crux — and you flag it yourself. Silent coercion is correct for "scalar in a string slot" (Atlas accepts it, so a validator shouldn't flag it). But for a genuine violation (nested object in a declared map<string,string>), silently coercing suppresses the exact finding the validator exists to produce. So "coerce and report" isn't a nicety here — for a validator, the report is the product. I'd lean toward that being a separate validation pass rather than a side effect of the decoder.

Minor: containers stringify to compact JSON ({"a":"b"}), but Atlas stores Java's toString() ({a=b}), so the "matches what Atlas stores" claim doesn't hold for containers — fine if documented, just flagging for anyone who relies on the coerced value round-tripping to Atlas's stored form.

Net: the two PRs aren't competing. #1024 = the root fix (via #2041) for the case that's actually hurting, in time for v11. #1022 = a broader validator-leniency feature worth maturing separately — with the wiring moved out of generated code and the reporting surface added. Happy to help on either.