Skip to content

finops: cloud billing via agent package, cost_daily facts, AWS billing daily collector + analysis playbook - #34

Open
geisbruch wants to merge 77 commits into
mainfrom
feat/finops
Open

finops: cloud billing via agent package, cost_daily facts, AWS billing daily collector + analysis playbook#34
geisbruch wants to merge 77 commits into
mainfrom
feat/finops

Conversation

@geisbruch

Copy link
Copy Markdown
Contributor

What

New finops/ suite — total cost allocation, iterated on kwik-e-mart and built to land on itti:

  • packages/cloud-query — a nullplatform package (simple) that runs AWS SDK calls (Cost Explorer, EC2, ELB, RDS, tagging API, CloudWatch, STS) on the customer's agent with the pod's IAM role, no credentials in workflows. Pagination, 300 KB result cap, result pushed to the engine callback with a host allow-list.
  • tool-cloud-query.yaml — reusable child around the engine plugin np-package-call (async two-phase + callback; nullplatform/governance-workflow-system#180).
  • wf1-aws-billing-daily.yaml — one day of billing → cost_daily facts: cloud services (Σ = amortized day), usage-type buckets, EC2 scopes by tag, EKS clusters composed (nodes, control plane, LBs by tag, VPC networking, EBS by attachment, other) with blended rates per core-hour / GiB-hour, databases mapped to null services by host. dry_run input.
  • wf-cost-fact-upsert.yaml, specs/cost_daily.spec.json (one row per subject per day, every id part is a field), setup scripts.
  • Docs: analysis of kwik-e-mart (2026-09-10) and the mapping playbook to repeat it per account and per service (what attributes directly, what goes to the cluster, what stays shared, and which identifiers join cloud resources to null). Design spec + phase-0 plan under docs/superpowers/.

Decisions worth knowing

  • Amortized cost is the chargeback basis (kwik's EC2 is Savings-Plan covered: unblended shows 0.001/day + a -22.85 negation line).
  • The cluster is not split per scope from billing; it carries blended rates so the allocator charges by consumption (phase 3, in-cluster package).
  • Databases map to the null service whose attributes.host equals the RDS endpoint (transactions → app demo-services), allocation_method: service_owner.

Verification

  • 9 E2E tests (npx vitest finops, plugins stubbed) + 19 runner unit tests (bun test).
  • Live on a local engine + local docker agent against kwik: 116 facts for 2026-09-09, Σ cloud_service = 37.316 = AWS amortized total; cluster developent 27.74/day, 624 core-h, 1,440 GiB-h.
  • Writing to the kwik catalog is pending: the spec created today has specification: read-only grants and cannot be patched by its creator; recreate with the grants now in the JSON.

Next

allocated stage: one row per application per resource per day, categorized, cut by environment / namespace / account; then Kubernetes consumption.

🤖 Generated with Claude Code

https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf

geisbruch and others added 30 commits September 10, 2026 13:59
…ts, AWS billing daily collector

Suite for total cost allocation, iterated on kwik-e-mart:
- packages/cloud-query: np package (simple) running AWS SDK calls with the
  pod's IAM role; async result via the engine callback (np-package-call
  plugin, engine PR #180), 300 KB cap, callback host allow-list.
- tool-cloud-query.yaml: reusable child (agent tags + calls → results).
- wf1-aws-billing-daily.yaml: Cost Explorer by service / usage type / per
  EC2 resource + inventory → cost_daily facts: cloud services (Σ = day),
  buckets, tagged scopes, EKS clusters composed (nodes, control plane, LBs,
  networking, storage, other) with blended rates per core-h / GiB-h,
  databases mapped to null services by host. Amortized cost basis.
- wf-cost-fact-upsert.yaml + specs/cost_daily.spec.json + setup scripts.
- docs: kwik-e-mart analysis (2026-09-10) and the mapping playbook to
  repeat it per account and service; design spec and phase-0 plan.
Verified live in dry-run on kwik (116 facts/day, Σ = 37.316 amortized).
9 E2E tests (plugins stubbed) + 19 runner unit tests.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…ildx push, digest in release.json

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…t-plugins/workflows/aws-cost-explorer (+ :latest); --latest flag

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…ce (no platform package registration); agent run without pin via NP_ALLOWED_REGISTRIES

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…nd the tool; playbook §4b (pod identity via agent patch, cross-account trust, read-only policy)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…ge version (pin → service account), AssumeRole, expected_account guard; docs

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…readItem)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…ker (single and multi-account)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
… to listed accounts + ExternalId, Athena/Glue to workgroup and CUR tables

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…e account)

02-aws-worker-identity.sh creates the read-only worker role (docs/iam trust +
policy), the ServiceAccount in the agent's worker namespace and the Pod Identity
association, then prints the NP_WORKER_RULES entry the agent needs. For
nullplatform's own runtime the same thing is IaC (iac-null-runtime PR #124, IRSA).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…count)

The control plane only resolves an agent selector inside the given NRN subtree:
nullplatform's runtime agent lives at organization=4:account=17 and is NOT found
from the organization root ("No agents found … [account:null]"). The tool takes
agent_nrn and sends {nrn, tags}; wf1 takes agent_nrn/org_nrn (org_nrn drives the
null service listing for the DB-host mapping); wf0 targets carry both.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…tool embeds the image artifact; wf1 has no cron

- setup/publish.ts replaces publish-local.ts: --base, NP_TOKEN bearer, --vars
  (per-org variable initialValues), --alias, --no-activate, --update (new revision).
- setup/vars.nullplatform.json: agent {cluster: runtime} @ organization=4:account=17,
  org_nrn organization=4, dispatcher target for account 283477532906.
- tool-cloud-query: when `image` is set, the oci_image artifact also goes inside
  action_context.notification.package, so engines whose np-package-call predates
  the `image` input (prod today) still run the image without a package registration.
- wf1: cron removed; wf0 (dispatcher) is the scheduler, wf1 is the per-account
  collector (manual or sub-workflow).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…ns a receipt in callback mode

wf1 §4: cluster nodes fold into the cluster's `nodes` component (no per-node row);
instances that are neither nodes nor null scopes (Karpenter churn terminated before
collection, untagged VMs) collapse into ONE `bucket` row per day
(`ec2-instances-unattributed`, quantity = instance count, usage_hours, instance_types).
On nullplatform's account 2026-09-09 produced 554 ec2:instance rows and a 1.1 MB
build_facts output (> the 1 MB sandbox limit); now it is tens of rows.
Spec: usage_hours, instances_seen, instance_types.

cloud-query 0.0.2 (public.ecr.aws …@sha256:c92e6b54…): in callback mode the command
completion is a receipt (per-call status, sizes, identity, callback outcome) — the
platform drops completions above ~400 KB and re-delivers the command.

publish.ts: alias re-point is PUT /aliases/:alias.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…irst-day findings

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…Watch Logs (`logs`) clients

Needed for the per-database split of a shared RDS cluster (PI db.load grouped by
db.name / db.user / db.application — CloudWatch itself has no per-database metrics)
and for the CloudWatch cost split by log group (names carry the app). Policy adds
pi:GetResourceMetrics, pi:DescribeDimensionKeys, logs:DescribeLogGroups (all reads).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…idence, shared resources split by metric

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…dence, specs, seed rules

- specs/cost_mapping_rule + cost_mapping_suggestion; cost_daily gains day (filterable copy of
  date), tags, host (evidence), category, share, rule_id, source_fact_id, rollup maps.
- wf2-allocate-daily: raw facts of a day + active rules → allocated facts per source fact × owner
  (categorized) + rollup per application/bucket/unallocated. Leaves per cloud service
  (resource rows, cluster components, remainder) so Σ allocated = Σ services. Rule engine:
  scope + match predicates (equals/regex with captures/in/exists), targets literal/capture/
  split/map, priority, first match; defaults for null dims, null services and cluster components.
- wf-suggest-mappings: unallocated leaves × (null services by host, application parameters)
  → proposed rules with evidence and recovered USD.
- wf1: day on every fact, tags/host on scope/resource/service rows, cluster components carry
  their cloud service (networking split VPC vs EC2-Other).
- wf-cost-fact-upsert: catalog_slug input (rules, suggestions).
- setup: rules.nullplatform.json (security → platform, CloudWatch remainder → platform,
  shared RDS clusters split among the consumers found in parameters), 03-mapping-rules.sh,
  publish order + vars. 16 E2E tests.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…llable + admin grant from token; prod rev 4

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…, wf2 reads raw by stage (catalog filters on late-added fields do not match), shared-apis rule scope rds:db

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
… allocated rows (source finops_allocator, methods, subject_type unallocated)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…rmance Insights / CloudWatch inputs)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…per RDS subject, wf2 splits by them; approvals Aurora rule by database load (15 DBs → apps) with equal-split fallback

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…ons from DB_NAME parameters and slug naming propose map entries

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…tance type (Cost Explorer INSTANCE_TYPE); RDS cost per cluster from Cost Explorer by owner tag instead of an even split

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…/shared-apis owners from parameters (CNAMEs) + Performance Insights

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…ng applications; application_cost_daily entity

- wf3-k8s-consumption-daily: scopes from the lake → collector on the agent (day mode, the cost
  tracker's script) → per-scope chargeable/used core-h & GiB-h priced with the day's blended rates
  → raw-k8s-scope-* rows + cluster row metric_shares/metric_owners/k8s_overhead_usd. Replaces the
  metadata cost_tracking write.
- wf2: default:cluster-consumption (by_metric_parent) splits every cluster component by the
  shares on the parent cluster row (category kubernetes, scope_id on the row); the uncovered share
  is kubernetes_overhead. New application_cost_daily rows (total, direct/kubernetes, by category,
  by cloud service, items with rule_id, kubernetes consumption summary).
- wf0: k8s step between collection and allocation (k8s_clusters var/input).
- specs: application_cost_daily; cost_daily + metric_owners, core_h_*/gb_h_*, pods_avg,
  k8s_overhead_usd. 18 E2E tests.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
geisbruch and others added 30 commits September 11, 2026 12:39
…ok lessons

- specs: restore the placeholder principal 732189543 (an org-specific user id had
  leaked into rule/suggestion specs → 400 unknown user_id in another org);
  01-catalog-spec.sh now rewrites every `type: user` grant to the token's user
- setup/vars.itti.json: itti-tuti (account 1041301647, AWS 985539773184 us-east-1),
  no Kubernetes clusters yet
- runbook: map null accounts → AWS accounts via runtime_configuration, agent
  version requirement (package-exec ≥ 0.9.0, 0.11.1 recommended) and the
  'Command failed to start after all retry attempts' symptom, curl probe,
  NP_API_KEY config entry, cron off until the agent is ready, dashboard before data

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
- wf2: rule method `spread` — platform cost shared by every application of the
  day, weighted by attributed cost (default) or equal; resolved after the main
  pass; summary.platform_spread_usd; exact reconciliation (remainder on last part)
- wf1: DescribeLogGroups (phase 1) + GetMetricData IncomingBytes per log group
  (phase 2) → metric_shares on the CloudWatch ingestion/storage usage-type
  buckets (cloudwatch.IncomingBytes / cloudwatch.StoredBytes)
- spec: allocation_method enum + spread
- setup/rules.itti.json: security/compliance, CloudWatch remainder and VPC as
  platform spread (log-group by_metric rule pending the real group names)
- tests: 19/19

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…cs, metric_owners

- wf1: applications + namespaces of the agent's account (np_apps, np_namespaces);
  <namespace>.<application>[.suffix] log groups resolve to their owner
  (metric_owners); EMF custom metrics / metric stream shares by the *_agg log
  groups' IncomingBytes per application (cloudwatch.EmfBytes)
- wf2: usage-type buckets carrying metric_shares are allocation leaves;
  by_metric falls back to fact.metric_owners[key] when the rule has no entry
- rules.itti.json: by_metric for logs and EMF metrics, spread for the rest
- tests 20/20

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…t, wf3 New Relic mode, DB rules from parameters

- wf1: accounts without CE resource-level data price the cluster from the
  INSTANCE_TYPE rows (running nodes → cluster by type, else the sole cluster);
  DocumentDB service split among its clusters; DB facts carry their own service
- wf3: collector_mode agent|newrelic (decider); newrelic = one NerdGraph NRQL per
  cluster-day FACET label.scope_id × hour → same per-scope shape as the agent
  collector; inputs collector_mode / nr_account_id
- rules.itti.json: Aurora by PI db.load (DB_NAME from parameters), DocumentDB by
  <app>db, ElastiCache equal split among REDIS_HOST consumers
- vars.itti.json: cluster eks-1-tuti-null-use1-dev, newrelic mode
- runbook: discovery rows for resource-level CE, pod metrics source, databases, CloudWatch
- tests 21/21

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…st timeout as string

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…roject scope_dims

An org with thousands of scopes dragged ~0.5 MB of lake rows and a 5-minute API
walk through wf3 for a 29-scope cluster (itti). The lake query and the API fetch
now stay inside the subtree the cluster serves (the org, or one account).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
… into cost_daily

- specs/scope_usage_daily.spec.json (+ setup script): one row per scope-day with
  per-hour used vs requested, pods, chargeable core-h/GiB-h, utilization, waste
- wf3 writes the usage rows first, then the priced facts (usage_id → row) and
  the cluster shares; summary.usage_rows / source
- cost_daily spec: usage_id
- tests 21/21

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…'); report: Kubernetes usage section (scope_usage_daily)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…ted daily charts (top-N from the Lake), usage section

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…acts without a scope/service of their own

Application-level charges (spread, EMF metrics, database shares) had no environment
in itti: the whole AWS account maps to one dimension value, so wf0 targets carry
`dimensions` and wf1 stamps them as the fact's default dimensions.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…edged the dispatcher

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…cate test

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…d as services (service_kind cloud); wf1 lists services under the agent's account

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…ped the owner bucket of shared rows

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…nly); the summary step no longer receives the batches (1 MB sandbox input cap)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…ccount environment)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
… hidden charts as 1px lines)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…ndered as outlines); document FE quirks

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…t has no day filter and returned every day (2.3 MB at 5 days, over the 2 MB step budget)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…, dashboard quirks, tuti reference

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
… itti and null): allocation on its own schedule, not as a long dispatcher child

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
…urrent PATCH by a retry or a second allocator)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017X2B2sJEPwBUCUKs5eWQNf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant