finops: cloud billing via agent package, cost_daily facts, AWS billing daily collector + analysis playbook - #34
Open
geisbruch wants to merge 77 commits into
Open
finops: cloud billing via agent package, cost_daily facts, AWS billing daily collector + analysis playbook#34geisbruch wants to merge 77 commits into
geisbruch wants to merge 77 commits into
Conversation
…ts, AWS billing daily collector Suite for total cost allocation, iterated on kwik-e-mart: - packages/cloud-query: np package (simple) running AWS SDK calls with the pod's IAM role; async result via the engine callback (np-package-call plugin, engine PR #180), 300 KB cap, callback host allow-list. - tool-cloud-query.yaml: reusable child (agent tags + calls → results). - wf1-aws-billing-daily.yaml: Cost Explorer by service / usage type / per EC2 resource + inventory → cost_daily facts: cloud services (Σ = day), buckets, tagged scopes, EKS clusters composed (nodes, control plane, LBs, networking, storage, other) with blended rates per core-h / GiB-h, databases mapped to null services by host. Amortized cost basis. - wf-cost-fact-upsert.yaml + specs/cost_daily.spec.json + setup scripts. - docs: kwik-e-mart analysis (2026-09-10) and the mapping playbook to repeat it per account and service; design spec and phase-0 plan. Verified live in dry-run on kwik (116 facts/day, Σ = 37.316 amortized). 9 E2E tests (plugins stubbed) + 19 runner unit tests. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…ildx push, digest in release.json Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…t-plugins/workflows/aws-cost-explorer (+ :latest); --latest flag Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…ce (no platform package registration); agent run without pin via NP_ALLOWED_REGISTRIES Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…nd the tool; playbook §4b (pod identity via agent patch, cross-account trust, read-only policy) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…ge version (pin → service account), AssumeRole, expected_account guard; docs Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…readItem) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…ker (single and multi-account) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
… to listed accounts + ExternalId, Athena/Glue to workgroup and CUR tables Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…e account) 02-aws-worker-identity.sh creates the read-only worker role (docs/iam trust + policy), the ServiceAccount in the agent's worker namespace and the Pod Identity association, then prints the NP_WORKER_RULES entry the agent needs. For nullplatform's own runtime the same thing is IaC (iac-null-runtime PR #124, IRSA). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…count)
The control plane only resolves an agent selector inside the given NRN subtree:
nullplatform's runtime agent lives at organization=4:account=17 and is NOT found
from the organization root ("No agents found … [account:null]"). The tool takes
agent_nrn and sends {nrn, tags}; wf1 takes agent_nrn/org_nrn (org_nrn drives the
null service listing for the DB-host mapping); wf0 targets carry both.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…tool embeds the image artifact; wf1 has no cron
- setup/publish.ts replaces publish-local.ts: --base, NP_TOKEN bearer, --vars
(per-org variable initialValues), --alias, --no-activate, --update (new revision).
- setup/vars.nullplatform.json: agent {cluster: runtime} @ organization=4:account=17,
org_nrn organization=4, dispatcher target for account 283477532906.
- tool-cloud-query: when `image` is set, the oci_image artifact also goes inside
action_context.notification.package, so engines whose np-package-call predates
the `image` input (prod today) still run the image without a package registration.
- wf1: cron removed; wf0 (dispatcher) is the scheduler, wf1 is the per-account
collector (manual or sub-workflow).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…ns a receipt in callback mode wf1 §4: cluster nodes fold into the cluster's `nodes` component (no per-node row); instances that are neither nodes nor null scopes (Karpenter churn terminated before collection, untagged VMs) collapse into ONE `bucket` row per day (`ec2-instances-unattributed`, quantity = instance count, usage_hours, instance_types). On nullplatform's account 2026-09-09 produced 554 ec2:instance rows and a 1.1 MB build_facts output (> the 1 MB sandbox limit); now it is tens of rows. Spec: usage_hours, instances_seen, instance_types. cloud-query 0.0.2 (public.ecr.aws …@sha256:c92e6b54…): in callback mode the command completion is a receipt (per-call status, sizes, identity, callback outcome) — the platform drops completions above ~400 KB and re-delivers the command. publish.ts: alias re-point is PUT /aliases/:alias. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…irst-day findings Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…Watch Logs (`logs`) clients Needed for the per-database split of a shared RDS cluster (PI db.load grouped by db.name / db.user / db.application — CloudWatch itself has no per-database metrics) and for the CloudWatch cost split by log group (names carry the app). Policy adds pi:GetResourceMetrics, pi:DescribeDimensionKeys, logs:DescribeLogGroups (all reads). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…account Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…idence, shared resources split by metric Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…dence, specs, seed rules - specs/cost_mapping_rule + cost_mapping_suggestion; cost_daily gains day (filterable copy of date), tags, host (evidence), category, share, rule_id, source_fact_id, rollup maps. - wf2-allocate-daily: raw facts of a day + active rules → allocated facts per source fact × owner (categorized) + rollup per application/bucket/unallocated. Leaves per cloud service (resource rows, cluster components, remainder) so Σ allocated = Σ services. Rule engine: scope + match predicates (equals/regex with captures/in/exists), targets literal/capture/ split/map, priority, first match; defaults for null dims, null services and cluster components. - wf-suggest-mappings: unallocated leaves × (null services by host, application parameters) → proposed rules with evidence and recovered USD. - wf1: day on every fact, tags/host on scope/resource/service rows, cluster components carry their cloud service (networking split VPC vs EC2-Other). - wf-cost-fact-upsert: catalog_slug input (rules, suggestions). - setup: rules.nullplatform.json (security → platform, CloudWatch remainder → platform, shared RDS clusters split among the consumers found in parameters), 03-mapping-rules.sh, publish order + vars. 16 E2E tests. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…llable + admin grant from token; prod rev 4 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…, wf2 reads raw by stage (catalog filters on late-added fields do not match), shared-apis rule scope rds:db Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
… allocated rows (source finops_allocator, methods, subject_type unallocated) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…rmance Insights / CloudWatch inputs) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…per RDS subject, wf2 splits by them; approvals Aurora rule by database load (15 DBs → apps) with equal-split fallback Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…ons from DB_NAME parameters and slug naming propose map entries Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…tance type (Cost Explorer INSTANCE_TYPE); RDS cost per cluster from Cost Explorer by owner tag instead of an even split Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…/shared-apis owners from parameters (CNAMEs) + Performance Insights Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…ng applications; application_cost_daily entity - wf3-k8s-consumption-daily: scopes from the lake → collector on the agent (day mode, the cost tracker's script) → per-scope chargeable/used core-h & GiB-h priced with the day's blended rates → raw-k8s-scope-* rows + cluster row metric_shares/metric_owners/k8s_overhead_usd. Replaces the metadata cost_tracking write. - wf2: default:cluster-consumption (by_metric_parent) splits every cluster component by the shares on the parent cluster row (category kubernetes, scope_id on the row); the uncovered share is kubernetes_overhead. New application_cost_daily rows (total, direct/kubernetes, by category, by cloud service, items with rule_id, kubernetes consumption summary). - wf0: k8s step between collection and allocation (k8s_clusters var/input). - specs: application_cost_daily; cost_daily + metric_owners, core_h_*/gb_h_*, pods_avg, k8s_overhead_usd. 18 E2E tests. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…ok lessons - specs: restore the placeholder principal 732189543 (an org-specific user id had leaked into rule/suggestion specs → 400 unknown user_id in another org); 01-catalog-spec.sh now rewrites every `type: user` grant to the token's user - setup/vars.itti.json: itti-tuti (account 1041301647, AWS 985539773184 us-east-1), no Kubernetes clusters yet - runbook: map null accounts → AWS accounts via runtime_configuration, agent version requirement (package-exec ≥ 0.9.0, 0.11.1 recommended) and the 'Command failed to start after all retry attempts' symptom, curl probe, NP_API_KEY config entry, cron off until the agent is ready, dashboard before data Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
- wf2: rule method `spread` — platform cost shared by every application of the day, weighted by attributed cost (default) or equal; resolved after the main pass; summary.platform_spread_usd; exact reconciliation (remainder on last part) - wf1: DescribeLogGroups (phase 1) + GetMetricData IncomingBytes per log group (phase 2) → metric_shares on the CloudWatch ingestion/storage usage-type buckets (cloudwatch.IncomingBytes / cloudwatch.StoredBytes) - spec: allocation_method enum + spread - setup/rules.itti.json: security/compliance, CloudWatch remainder and VPC as platform spread (log-group by_metric rule pending the real group names) - tests: 19/19 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…cs, metric_owners - wf1: applications + namespaces of the agent's account (np_apps, np_namespaces); <namespace>.<application>[.suffix] log groups resolve to their owner (metric_owners); EMF custom metrics / metric stream shares by the *_agg log groups' IncomingBytes per application (cloudwatch.EmfBytes) - wf2: usage-type buckets carrying metric_shares are allocation leaves; by_metric falls back to fact.metric_owners[key] when the rule has no entry - rules.itti.json: by_metric for logs and EMF metrics, spread for the rest - tests 20/20 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…usters Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…t, wf3 New Relic mode, DB rules from parameters - wf1: accounts without CE resource-level data price the cluster from the INSTANCE_TYPE rows (running nodes → cluster by type, else the sole cluster); DocumentDB service split among its clusters; DB facts carry their own service - wf3: collector_mode agent|newrelic (decider); newrelic = one NerdGraph NRQL per cluster-day FACET label.scope_id × hour → same per-scope shape as the agent collector; inputs collector_mode / nr_account_id - rules.itti.json: Aurora by PI db.load (DB_NAME from parameters), DocumentDB by <app>db, ElastiCache equal split among REDIS_HOST consumers - vars.itti.json: cluster eks-1-tuti-null-use1-dev, newrelic mode - runbook: discovery rows for resource-level CE, pod metrics source, databases, CloudWatch - tests 21/21 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…st timeout as string Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…roject scope_dims An org with thousands of scopes dragged ~0.5 MB of lake rows and a 5-minute API walk through wf3 for a 29-scope cluster (itti). The lake query and the API fetch now stay inside the subtree the cluster serves (the org, or one account). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
… into cost_daily - specs/scope_usage_daily.spec.json (+ setup script): one row per scope-day with per-hour used vs requested, pods, chargeable core-h/GiB-h, utilization, waste - wf3 writes the usage rows first, then the priced facts (usage_id → row) and the cluster shares; summary.usage_rows / source - cost_daily spec: usage_id - tests 21/21 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…'); report: Kubernetes usage section (scope_usage_daily) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…ted daily charts (top-N from the Lake), usage section Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…acts without a scope/service of their own Application-level charges (spread, EMF metrics, database shares) had no environment in itti: the whole AWS account maps to one dimension value, so wf0 targets carry `dimensions` and wf1 stamps them as the fact's default dimensions. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…edged the dispatcher Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…cate test Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…d as services (service_kind cloud); wf1 lists services under the agent's account Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…ped the owner bucket of shared rows Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…nly); the summary step no longer receives the batches (1 MB sandbox input cap) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…ccount environment) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
… hidden charts as 1px lines) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…ndered as outlines); document FE quirks Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…t has no day filter and returned every day (2.3 MB at 5 days, over the 2 MB step budget) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…, dashboard quirks, tuti reference Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
… itti and null): allocation on its own schedule, not as a long dispatcher child Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…urrent PATCH by a retry or a second allocator) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
New
finops/suite — total cost allocation, iterated on kwik-e-mart and built to land on itti:packages/cloud-query— a nullplatform package (simple) that runs AWS SDK calls (Cost Explorer, EC2, ELB, RDS, tagging API, CloudWatch, STS) on the customer's agent with the pod's IAM role, no credentials in workflows. Pagination, 300 KB result cap, result pushed to the engine callback with a host allow-list.tool-cloud-query.yaml— reusable child around the engine pluginnp-package-call(async two-phase + callback; nullplatform/governance-workflow-system#180).wf1-aws-billing-daily.yaml— one day of billing →cost_dailyfacts: cloud services (Σ = amortized day), usage-type buckets, EC2 scopes by tag, EKS clusters composed (nodes, control plane, LBs by tag, VPC networking, EBS by attachment, other) with blended rates per core-hour / GiB-hour, databases mapped to null services by host.dry_runinput.wf-cost-fact-upsert.yaml,specs/cost_daily.spec.json(one row per subject per day, every id part is a field), setup scripts.docs/superpowers/.Decisions worth knowing
attributes.hostequals the RDS endpoint (transactions→ appdemo-services),allocation_method: service_owner.Verification
npx vitest finops, plugins stubbed) + 19 runner unit tests (bun test).cloud_service= 37.316 = AWS amortized total; clusterdevelopent27.74/day, 624 core-h, 1,440 GiB-h.specification: read-only grants and cannot be patched by its creator; recreate with the grants now in the JSON.Next
allocatedstage: one row per application per resource per day, categorized, cut by environment / namespace / account; then Kubernetes consumption.🤖 Generated with Claude Code
https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf