Add description to embeddings for aggregation helper

[shixiao-coder]

This PR separate the add descriptions changes between aggregation-helper and ingestion-helper from #732

This PR optimize the aggregation-helper to embed descriptions and optimize related tests

[codacy-production]

Not up to standards ⛔

🔴 Issues 5 high · 3 medium · 2 minor

Alerts:
⚠ 10 issues (≤ 0 issues of at least minor severity)

Results:
10 new issues

Category	Results
Compatibility	5 high
Documentation	1 minor
Security	3 medium
CodeStyle	1 minor

View in Codacy

🟢 Metrics 0 complexity

Metric	Results
Complexity	0

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}

[gemini-code-assist]

Code Review

This pull request refactors the embedding generation process by migrating from a BigQuery-native approach to a Spanner-first extraction method. It introduces a new _generate_spanner_query method to construct GQL queries, streams results from Spanner to BigQuery using pandas, and updates the EmbeddingSpec to support a dictionary-based node_types structure. The review feedback highlights critical issues including an incorrect Spanner SQL syntax for JSON_OBJECT, a potential resource leak due to missing input validation, a duplicate test assertion, and a requirement to handle the reconstruction of JSON properties during the streaming phase.

[gemini-code-assist]

To support the Spanner-compatible array of structs format for properties, we should convert the list of key-value structs into a dictionary in Python before loading the data into BigQuery. Ensure that the final data is serialized back to a JSON string, as database clients expect JSON-serialized strings by design.

                emb_content = row[2]
                if isinstance(emb_content, str):
                    try:
                        data = json.loads(emb_content)
                        if isinstance(data, dict) and "properties" in data:
                            props = data["properties"]
                            if isinstance(props, list):
                                data["properties"] = {p["key"]: p["value"] for p in props if p and "key" in p}
                        emb_content = json.dumps(data)
                    except Exception:
                        pass

References

1. When passing JSON data to Spanner (or other database clients) that expects JSON-serialized strings by design, do not replace the serialized strings with Python dictionaries or OrderedDict objects, as the serialized format is required for proper database ingestion.