[bot] Merge master/48281984 into rel/dev - #1787
Merged
Merged
Conversation
An alert can be narrowed two ways: a date entry in `filters`, or a date
group-by in `attributes`. Only the first was checked, and the second is
worse -- a group-by makes the alert fire per period value instead of on
the latest one, which is a different alert from the one the fixture
describes. An alert with `filters: []` and `attributes: [order_date.month]`
therefore scored 7/7 clean; a recorded GPT-5.2 run carried exactly that
invented month group-by and passed.
This activates coverage the dataset already carries. Item
e2a1e22a-2020-4a24-b3fe-2350315f7a73 ("...for any individual product
brand...") stores `"Attributes": [{"using": "label/product_brand"}]`, and
that key was dropped on the floor. It is the regression gate for this
change, not just the new unit tests.
The two sides cannot be deep-compared as raw dicts. Fixtures author the AAC
tool-input form, `{"using": "label/x"}`; `create_metric_alert` receives the
resolved AFM form, `{"localIdentifier": "a0", "label": {"identifier":
{"id": "x"}}}`, forwarded verbatim from `prepare_metric_alert_proposal`. So
`_attribute_label_ids` canonicalises both sides to a bare label id and the
comparison is a sorted multiset: order-insensitive, and the converter's
`localIdentifier` stops mattering without a strip step. Identity is all
that is compared -- `showAllValues` is the agent's to choose.
The asymmetry is specific to attributes. `Filters` in the same fixture is
already AFM-shaped, which is why `_check_filters` gets away with a raw
`_deep_subset` and is left alone.
Malformed input is split structurally rather than by side:
- Not a list of groupings at all -> False. The agent answered wrongly, and a
wrong answer is a FAIL. Raising would record an ERROR, which
`json_report.py` excludes from the failure count, so a malformed answer
would rank above a merely wrong one.
- A list holding an unreadable entry -> raises. Entries are typed
`AttributeItem` at the tool boundary, so the plausible cause is the wire
format moving, and that has to be unmissable rather than read as every
agent regressing at once. The same rule covers a `label/x` versus
`attribute/x` mix-up in a fixture.
The expectation side needs no guard beyond that: `_normalize_expected_output`
rejects a non-list before the run spends an API call, and `from_dict`, the
other way into the field, has no callers.
Deliberately not done: `generate_simulated_alert_response` is unchanged. An
`Attributes: []` expectation needs no support -- the simulated user does not
invent a grouping and the check verifies it did not. A non-empty expectation
does have a gap, but rule 3's "check ALL of these" list would have to grow
too for a symmetric rule to be honest, perturbing all 18 items to serve the
one that already requests its grouping in its own question. So: a non-empty
`Attributes` expectation requires the item's question to ask for that
grouping, and that requirement is documented on the normalizer.
Note for dashboards: `quality_score` for alert items moves from /7 to /8,
since both Langfuse sinks derive it as the fraction of true booleans in the
detail dict. Scores are not comparable across this commit, and
`attributes_correct` is a new score name.
The other half of GDAI-2175 -- whether the agent asked before proposing --
is not observable from final tool arguments and stays with
`verify_alert_asks_for_date` in gdc-nas.
JIRA: GDAI-2175
risk: low
…utes fix(gooddata-eval): assert alert group-by attributes
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## rel/dev #1787 +/- ##
===========================================
+ Coverage 81.70% 81.76% +0.05%
===========================================
Files 275 275
Lines 19848 19903 +55
===========================================
+ Hits 16217 16273 +56
+ Misses 3631 3630 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚀 Automated PR to perform merge from master into rel/dev with changes up to 4828198 (created by https://github.com/gooddata/gooddata-python-sdk/actions/runs/34223841560).