Skip to content

feat(eval): add Native Agent observation product #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jgoneit wants to merge 1 commit into from
Closed

feat(eval): add Native Agent observation product #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions .codex-plugin/plugin.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
{
"name": "eval",
"version": "0.2.0-dev.0+codex.20260825144155",
"description": "Record private post-task observations and produce deterministic Eval analyses.",
"author": {
"name": "Jaegon Seo",
"email": "jgoneit@gmail.com",
"url": "https://github.com/jgoneit"
},
"homepage": "https://github.com/jgoneit/eval",
"repository": "https://github.com/jgoneit/eval",
"license": "MIT",
"keywords": [
"agent",
"evaluation",
"observability"
],
"skills": "./skills/",
"interface": {
"displayName": "Eval",
"shortDescription": "Record and analyze post-task observations",
"longDescription": "Eval provides a thin Native Agent surface for private, best-effort post-task observations and deterministic validation, summary, and comparison reports.",
"developerName": "Jaegon Seo",
"category": "Developer Tools",
"capabilities": [
"Read",
"Write"
],
"websiteURL": "https://github.com/jgoneit/eval",
"defaultPrompt": [
"Use $eval to validate the private observation store.",
"Use $eval to summarize observations through today.",
"Use $eval to compare a module by usage."
]
}
}
90 changes: 58 additions & 32 deletions CHARTER.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,59 +3,85 @@
## Responsibility

Eval measures whether each Agent Toolkit module creates more value than cost on
real tasks.
real tasks. It records bounded post-task facts and produces deterministic,
observational aggregates for human review.

It provides a bounded post-task observation and reporting contract for module
effects, defects, cost, and user friction.
Eval owns a thin execution surface: the independent `evalctl` CLI and the
`eval` Plugin Skill. It does not delegate execution to Harness or any other
central runtime.

## Position

The Native Agent, user, or CI owns task execution, module choice, composition,
and Eval invocation. Eval begins only after that caller identifies a terminal
task outcome and decides measurement is useful.
The Native Agent, user, or CI owns the primary task, module selection,
composition, terminal outcome, and optional Eval selection. Eval begins only
after that caller has identified the task as completed, failed, or abandoned.

Eval's one intended operational side effect is an authorized append to the
external private observation log. It does not mutate the user's completed task
or any Toolkit module state.
Eval's only intended mutating product operation is an authorized, atomic append
to the external private v2 observation log. Validation and analysis are
read-only. Eval does not mutate the completed task or any Toolkit module state.

A caller may select Eval explicitly or under a standing Agent or Host policy.
That caller-owned selection is not self-activation: Eval has no task-completion
listener, scheduler, hook, daemon, or background process.

## Invariants

- A task remains executable and completable without Eval.
- The Native Agent, user, or CI explicitly invokes Eval; Eval never
self-activates.
- Observation requires a terminal outcome and Host write authority.
- Eval never invokes Spec, Ward, Seal, an Agent, or CI.
- Eval owns no workflow transition, execution order, retry, repair, or release
gate.
- No module automatically invokes Eval, and Eval does not automatically invoke
another module.
- Task effects remain separate from used-only module metrics so used and unused
cohorts can be compared without inventing module behavior.
- Unknown and unassessed values remain `null`.
- Observation and report contracts are versioned and provider-neutral.
- A task remains executable, completable, and reportable without Eval.
- Eval runs only after a terminal outcome; it never changes that outcome.
- An implicit Native Agent attempt occurs at most once and only under existing
Host authority. It asks for no additional approval and does not retry.
- Observation failure, including an unavailable CLI, denied write, invalid
data, or lock failure, cannot fail or revise the primary task.
- Eval never invokes Spec, Ward, Seal, another module, an Agent, Harness, or CI.
- Eval owns no workflow transition, execution order, retry, repair, release
gate, or module selection.
- Observation never changes how the primary Agent gathers evidence, chooses
tools, or completes its work.
- Observation v1 is read-only compatibility data. New writes are v2 and v1/v2
rows cannot share a correction chain.
- Module absence means unassessed. Only explicit `used: false` enters an unused
cohort.
- A used module with an unavailable exact public version remains a used
observation and is excluded only from version comparison.
- Metrics and task effects use closed, versioned, locally registered
extensions. Unknown or unassessed facts remain absent or `null`.
- Core validates row shape, cross-value semantics, and revision chains before
writing or analyzing observations.
- The private writer permits one exclusive writer at a time and replaces the
complete log atomically; it never exposes a partial append.
- Raw observations remain outside source repositories under current-user-only
access.
- One observation file has one active writer.
- Reports disclose sample sizes, missing values, and observational limitations.
- Product and release decisions remain human-owned.
- Analysis is deterministic, discloses sample and exclusion counts, and keeps
measured, bounded, and unavailable values distinct.
- Analysis never states causality or emits a product, retention, promotion,
removal, or release decision.
- Publication requires manual privacy review, and final product and release
decisions remain human-owned.

## Fixed non-goals

Eval is not:

- Agent execution or orchestration;
- module execution, retry, or repair;
- a Plugin, Skill, Hook, or automatic task-completion listener;
- a central Toolkit runtime or shared lifecycle state;
- automatic release approval or promotion;
- module execution, version probing, retry, or repair;
- a central Toolkit runtime, Harness lifecycle, or shared workflow state;
- a task-completion listener, autonomous scheduler, or self-triggering service;
- automatic release approval, promotion, retention, or removal;
- a real-time telemetry platform or dashboard-first product;
- repository context, Agent Memory, RAG, or prompt injection;
- prompt, transcript, Chain-of-Thought, command, source-code, secret, or raw
per-event user behavior collection;
- automatic publication, retention, deletion, synchronization, or transmission
of private observations; or
- proof that an observational difference is causal.

## Maturity boundary

Contract verification proves only that the scaffold is internally consistent.
A cumulative report proves only that the evidence workflow operated. Neither
state alone proves that a module is valuable or ready for release.
Eval `0.2.0-dev.0` is a development product. Product verification can establish
that the CLI, contracts, writer, analysis, Plugin, and Skill follow their
defined boundaries. It cannot establish module value.

This repository contains no real-task observation set and no cumulative
decision report. A future privacy-reviewed cumulative report would show only
that the evidence workflow operated; it would not prove causality or authorize
a release.
129 changes: 90 additions & 39 deletions PRIVACY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,73 +1,124 @@
# Eval Privacy Contract

Eval collects only bounded structured facts needed to measure module value and
cost. The observation contract has no free-text field.
cost. Observation v2 and every registered extension are closed contracts with
no free-text field.

## Prohibited observation data

Never store any of the following in an observation:
Never store any of the following in an observation or extension:

- repository, organization, customer, product, project, or internal system
names;
- actual file or directory paths;
- task descriptions, business content, source code, patches, or artifacts;
- prompts, transcripts, Chain-of-Thought, or raw Agent output;
- commands, command arguments, URLs, hostnames, or infrastructure identifiers;
- credentials, tokens, keys, secrets, personal data, or raw/per-event user
behavior traces;
- credentials, tokens, keys, secrets, personal data, or raw or per-event user
behavior traces; or
- free-text notes, explanations, rationales, copied errors, or task narratives.

Unknown properties are rejected. `task_id` and each `observation_id` are
independently generated random UUIDv4 values and must never be derived from
work content. `supersedes` is either `null` for revision 1 or the immediately
preceding `observation_id`; it is never generated from work content. Repository
identity is not recorded, even as a hash.
Unknown properties and unregistered extension payloads are rejected.
`task_id` and each `observation_id` are independently generated random UUIDv4
values and must never be derived from work content. `supersedes` identifies
only the immediately preceding random observation ID. Repository identity is
not recorded, even as a hash.

Module versions and model identifiers contain only bounded public identifiers.
The schema constrains their syntax; it does not prove that an identifier is
public. A model identifier is `null` unless public availability is known, and
whenever the value is unavailable or sensitive. A used module requires its
exact public version; if that version cannot be recorded safely, do not append
the observation and do not misstate the module as unused.
public. Use `model: null` whenever public availability is unknown or the value
is sensitive.

A used module whose exact public version is known uses version state
`known-public`. When the exact public version cannot be learned from existing
task context or cannot be recorded safely, keep `used: true` and use
`unavailable` with `value: null`. Do not discard the observation, probe the
module, or misstate the module as unused. Such a record remains in the usage
cohort and is excluded only from version comparison.

Module absence means usage is unassessed. `used: false` is recorded only when
non-use is known and requires `not-applicable`, `value: null`, and
`metrics: null`. Unknown metrics use `metrics: null`; an unknown task-effect
key is absent. Unknown values inside an assessed extension remain `null`; they
are never guessed as false or zero.

## External private storage

Raw observations stay outside every source checkout at the path defined in
[protocol.md](protocol.md). A set `XDG_STATE_HOME` must be absolute. Only an
unset `XDG_STATE_HOME` falls back to `$HOME/.local/state`, and `HOME` must be
absolute. Resolve the path physically and reject any location inside a Git
worktree.
New raw observations stay outside every source checkout at:

- `$XDG_STATE_HOME/jgoneit/eval/v2/observations.jsonl` when
`XDG_STATE_HOME` is set and absolute; or
- `$HOME/.local/state/jgoneit/eval/v2/observations.jsonl` only when
`XDG_STATE_HOME` is unset and `HOME` is absolute.

The corresponding v1 file remains a read-only compatibility source. Eval does
not append v1 rows or move raw rows between versions. `--state-root ABS`
selects another absolute base with the same `jgoneit/eval/v1` and
`jgoneit/eval/v2` separation.

Relative roots, filesystem roots, paths inside a Git worktree,
caller-controlled symlinks, and non-regular data or lock files are rejected.
On Darwin, an immutable root-owned top-level alias such as `/tmp` may be
resolved to its physical root-owned sticky directory before the private child
checks run. On POSIX-like hosts, an existing XDG or HOME state root must be
owned by the current user and must not be group- or other-writable; mode
`0755` is therefore permitted for that root.
Eval-managed descendants are owned by the current user with mode `0700`, and
the data, lock, and temporary files are owned by the current user with mode
`0600`. Other hosts require equivalent current-user-only access for
Eval-managed state through Host ACLs. In `0.2.0-dev.0`, Windows Store access
fails closed with a permission error because owner and DACL verification is
not yet implemented; the Windows build does not infer privacy from opaque
`FileMode` metadata.

The caller must already hold Host write authority. The Native Agent Skill does
not request additional permission, retry, or change the task result when that
authority is absent. `observe --best-effort` returns only a bounded, path-free
skip reason; the implicit Skill suppresses that output. A successful implicit
observation reports no identifier or raw value.

## Writer boundary

Core inspects private path metadata, takes a kernel-backed exclusive lock, and
rechecks caller-controlled paths while holding the lock. It validates the
complete existing v2 log and prospective row before replacement. It then
writes the existing bytes plus one canonical row to a same-directory temporary
file, syncs the file, atomically replaces the live file, and syncs the
directory.

On POSIX-like hosts, the state directory has mode `0700` and
`observations.jsonl` has mode `0600`. Other hosts use equivalent
current-user-only access controls. The file must be a regular, non-symlink file
owned by the current user where the Host exposes those concepts.
This is a crash-atomic logical append: after a failure, the live path contains
either the previous complete log or the complete prospective log, never a
partial row. The lock serializes writers; it is not a daemon, network lock,
merge protocol, or shared Toolkit runtime.

The caller must hold Host write authority and be designated as the sole active
writer for that file by external Host policy or coordination. Multiple allowed
invocation owners do not imply simultaneous writers. If sole-writer status
cannot be established, do not append. Eval v0.1 does not lock, encrypt,
transmit, retain, delete, synchronize, or merge the file automatically. The
user owns backup and retention policy.
Eval does not encrypt, transmit, upload, synchronize, back up, retain, redact,
delete, or merge the observation store automatically. The user owns storage,
backup, retention, and deletion policy. Filesystem privacy is not a substitute
for full-disk encryption or correct Host account security.

## Reports
## Analysis and reports

Private reports remain outside source checkouts. They may contain aggregate
counts but never raw rows, task IDs, observation IDs, task narratives, or other
prohibited content.
Validation may identify row and chain failures but must not print raw rows in a
public response. Summaries and comparisons contain aggregate facts, exclusion
counts, and bounded cohort labels; they contain no task IDs, observation IDs,
task narratives, or source paths.

A report copied into the repository or otherwise published must:
Private reports remain outside source checkouts. A report copied into the
repository or otherwise published must:

1. disclose real and excluded synthetic counts, superseded rows, missing values,
and cohort sizes;
2. omit raw observations, individual rows, and private identifiers;
1. disclose real and excluded synthetic counts, superseded rows, invalid rows
or chains, missing values, and cohort sizes;
2. omit raw observations, individual rows, private identifiers, and source
paths;
3. suppress every public cohort cell with `n < 5` using `<5`;
4. suppress complementary cells when another displayed value could reveal a
hidden count;
5. use only aggregate, non-sensitive language;
6. label sampling, missing-data, measurement, and causal limitations;
6. label sampling, missing-data, measurement, and causal limitations; and
7. confirm that it records a human decision and triggers no workflow or release
action.

Publication requires an explicit manual privacy review. Schema or repository
verification is necessary but does not constitute that review.
Publication always requires an explicit manual privacy review. Schema,
semantic, repository, Plugin, Skill, or E2E verification does not constitute
that review. Retention, promotion, removal, and release decisions remain human
actions.
Loading