An open-source tool that turns one scenario document into staged demo videos and screenshots by driving a real app deterministically and recording the result. Designed to be driven by AI coding agents (Claude Code, Codex, Cursor, …) as commands and skills — while every stage stays an independent, agent-free CLI.
screen-use (su for short) treats a demo like a film shoot: plan the
narrative in conversation, write it down as a scenario, rehearse it (dry-run),
shoot it (record), and edit it (render). Every artifact is a plain file and
every stage is an independent CLI, so anything can be re-run at any time —
with or without an agent.
Real production output, one set per driver.
Video (Android, adb driver) — the demo set for
Seed Note, an Android block editor,
filmed on a real device with su. Each GIF below is one screenplay.yaml: the
gestures (including the long-press drags) were injected deterministically, the
take was verified and auto-retaken until clean, and the captions and pacing
were rendered as post-processing overlays. To change the wording of a caption,
you re-render; to re-shoot after a UI change, you re-run the same scenario.
Three scenarios, three videos — paste capture · table-cell editing · drag-and-drop organizing
Stills (Obsidian, CDP driver) — the README screenshot set for Hilo, an Obsidian highlight plugin: the scenario drives the app into each state and takes a crisp screenshot at every anchor point, so the whole set regenerates with one command after a UI change.
Demo-video tools are usually locked inside a GUI editor, or macro-style recorders whose output is a black box. su goes the other way.
| su | typical demo recorders | |
|---|---|---|
| Planning | narrative designed in conversation with an LLM | click and type by hand, again and again |
| Reproducibility | deterministic replay/re-record from a scenario file | re-demo by hand every time |
| Artifacts | screenplay, timeline, raw video, render config — all plain files | editor-internal state (black box) |
| Direction | cursor, zoom, captions, titles are post-processing overlays (re-render only) | re-shoot to change anything |
| Output | both video and stills from the same scenario | usually one or the other |
| Where it runs | inside Claude Code / Codex as commands & skills | a separate app |
- Video — play deterministically while recording, then render cursor, click ripples, zoom, captions, and titles as overlays.
- Stills — take a crisp screenshot at each anchor point of the scenario, for app-store shots, documentation figures, or step-by-step guides.
- Both share the same scenario, driver, and
timeline.jsoncontract; only the final stage differs.
- One
screenplay.yamlholdsmeta(target app & output),setup(unfilmed preparation),scenes(filmed steps), anddirection(staging cues). - Pacing is a first-class semantic — scene
pace,wait, typingcadenceproduce the feel of a human operating the app.
- Execution leaves nothing but precise timestamps in
timeline.json. - Zoom, speed-ups, captions, cursor, ripples, and titles are
direction:cues layered on in post-processing. Changing the staging means a re-render, not a re-shoot.
- pre-flight / post-take verification — before filming, assert the app is
frontmost on the expected screen (
preflight); after filming, assert the end state and recording integrity (verify). Bad takes (launcher footage, truncated video) are discarded and re-shot automatically. Nothing is checked during filming, so the footage stays smooth. - Log assertions (
verify.log) — after the take, device logs from the filming window are checked against present/absent patterns, catching mid-take failures that the final screen cannot reveal. - Probe & freshness gate — a probe pass resolves selectors into a coordinates sidecar ahead of time; the capture pass consumes baked coordinates only, and rejects them if the scenario, app build, or device geometry changed since the probe.
/su:campaign— manage a project's manifest of visual assets/su:scenario— author one asset's scenario/su:record— produce a video/su:capture— produce stills- Commands are thin wrappers; procedure and judgment live in the matching
skills, with the shared discipline in a single
su-flowskill.
- One
su initplants the commands and skills into Claude Code, Codex, Cursor, Gemini CLI, Windsurf, and 20+ other agent platforms, handling each platform's paths and formats automatically.
- Node.js 20+, with
ffmpeg/ffprobeon PATH. - Video recording on Windows (Obsidian target): .NET SDK 8 to build
tools/recorder-win/(the WGC screen-recorder CLI). - Polish-quality renders:
npm installinsidetools/remotion/. - Android targets:
adbandscrcpyon PATH, plus a connected device or emulator. Smooth physical long-press drags (drag+longPress) need the appium-uiautomator2-server APKs on the device (plain tap/swipe/type do not).
npm ci
npm run build
npm link # registers the global `su` commandRun once in the project you want to make demos for:
su init # auto-detect installed agents (defaults to claude)
su init --tools claude,codex # target specific platforms
su init --tools all # every supported platform
su init --dry-run # preview what would be installedThe typical flow: open a campaign → write a scenario → produce, driven as commands inside your agent.
/su:campaign init # shared context for all assets
/su:campaign add hero-demo --video
/su:scenario hero-demo # plan in conversation → screenplay.yaml → dry-run
/su:record hero-demo # record → render (quick/polish) → verify
Artifacts (recording.mp4, timeline.json, final.mp4, stills) land inside your project's campaign folder — never the tool's install path:
screen-use/<campaign>/
├─ CAMPAIGN.md # manifest (version-controlled)
├─ <asset>.screenplay.yaml # scenario (version-controlled)
└─ output/<asset>/ # generated (gitignored)
If only direction: cues changed (zoom, captions, titles), re-render —
re-record only when scenes or setup change.
screenplay.yaml has four sections (version: "0"):
- meta — target (
obsidian/android) and window/output geometry. - setup — unfilmed preparation steps establishing a deterministic start state.
- scenes — filmed deterministic steps:
move/click/drag/type/key/wait/scroll. Selectors are baked into coordinates before filming. Android narrows the vocabulary to what a touchscreen can do (the schema enforces it), and physical long-press drags route through UiAutomator2 for smooth, human-like eased motion. - direction — staging cues anchored to steps:
zoom/speed/captions/title. - preflight / verify (optional) — seam assertions that reject bad takes for automatic re-shoots.
The full field definitions live in src/schema/screenplay.ts (zod, with
readable validation errors).
- Obsidian (CDP, Electron) — full video pipeline, production-proven.
- Android (adb + scrcpy) — control, capture, and both renderers (quick/polish), verified on emulators and real Galaxy-class devices.
- Other targets plug in behind the
Driverinterface (src/runner/driver.ts) — inspect·play·record per platform; everything else (screenplay, timeline, renderers) is reused as-is.
screenplay.yaml (scenes/steps + direction anchors + pacing)
→ driver.play() → raw video + timeline.json + manifest
→ ffmpeg stages / Remotion bridge → final video
→ (or) anchor screenshots → still set
Module headers under src/ document each piece's contract and the design
decisions behind it.
screen-use was scaffolded with Scaff — a
lightweight, markdown-based AI development harness. Every iteration — the CDP
and adb drivers, the render pipeline, the reliability guards hardened through
field-defect rounds, release prep — was tracked as plain-file GOAL.md /
DESIGN.md / CONTEXT.md documents with per-goal archives, so the project's
decision history lives in reviewable markdown rather than tool state.
AGPL-3.0-only. Portions of the render pipeline reimplement mechanisms from MIT-licensed work by MobAI — see NOTICE. Contributions: see CONTRIBUTING.md (short version: issues welcome, pull requests are not accepted, forking is what the license is for).