workshop

An agent pipeline where every action that reaches the outside world stops and waits for a person.

The control model is the point. Real provider calls are blocked by a configuration guard that fails at startup rather than falling back to live mode, every external operation is idempotency-keyed against a database-level unique constraint, and publishing is exclusively human-triggered. The default configuration is fully mocked; touching anything real requires opting in per integration.

The domain happens to be ecommerce. Three of the nine stages below are human gates (design approval, listing copy approval, manual publish); nothing advances past them on its own:

trend discovery -> optional research report -> design -> design approval -> listing copy approval -> Printful product -> Etsy draft -> manual publish -> monitoring

Python 3.12, FastAPI, Postgres with Alembic, Streamlit, Docker Compose.

Etsy supports authenticated draft-listing workflows. Printful supports real catalog reads and an operator-driven Workshop for curation, mockups, and sync-product creation; all real writes require separate safeguards. OpenAI image generation and web-grounded research can be enabled independently.

The architecture rationale and the decisions behind it are documented in ARCHITECTURE_REVIEW.md.

Quick Start

1. Install Docker Desktop or another Docker Compose implementation.
2. Copy .env.example to .env (or run make env).
3. Keep INTEGRATIONS_ENABLED=false for a fully mocked stack, or explicitly enable only OPENAI_MODE=real.
4. Start the stack:

docker compose up --build

Open:

• Dashboard: http://localhost:8501
• FastAPI docs: http://localhost:8000/docs
• Health check: http://localhost:8000/api/health

The stack is four services: Postgres, a one-shot migrate job, the API (which also runs the background job worker and the monitoring loop), and the Streamlit dashboard.

Common commands are available through the Makefile:

make help
make up
make health
make lint
make typecheck
make test
make test-integration
make down

make down preserves local data. The explicitly named make reset-local-data CONFIRM=reset target deletes the local Postgres and generated-asset volumes.

See TODO.md for the pilot milestone checklist and the production backlog.

For development without Docker (requires uv):

make install          # uv sync --all-extras into .venv
alembic upgrade head  # needs a reachable Postgres (see DATABASE_URL)
.venv/bin/uvicorn ecommerce_agent.api.main:app --reload

make install also enables the committed pre-commit hook. In an existing checkout, run make install-hooks once. The hook runs the same Ruff lint and format checks as CI.

Run Streamlit separately:

API_BASE_URL=http://localhost:8000 .venv/bin/streamlit run src/ecommerce_agent/dashboard/app.py

Architecture

flowchart LR
    UI["Streamlit admin"] --> API["FastAPI"]
    API --> Pipeline["pipeline/: step functions + transition map"]
    API --> Jobs[("jobs table")]
    Worker["in-process worker task"] --> Jobs
    Worker --> Pipeline
    Monitor["in-process monitoring loop"] --> Pipeline
    Pipeline --> DB[("Postgres: single source of truth")]
    Pipeline --> Providers["provider protocols"]
    Providers --> Mocks["deterministic mocks"]

Loading

Workflow state lives in exactly one place: a status column on each entity, guarded by an explicit legal-transition map (pipeline/transitions.py). Each pipeline step (generate_design, generate_copy, create_listings) is an idempotent function: it validates the expected status, does its work keyed on a natural unique constraint, advances the status, and commits.

Human approvals are plain API calls (POST /api/runs/{id}/decisions). A decision records an ApprovalEvent, advances the status, and enqueues the follow-up step — all in one transaction (a transactional outbox). The worker claims jobs with FOR UPDATE SKIP LOCKED; stale claims requeue after a timeout, and idempotent steps make re-runs safe, so an approval reliably causes its follow-up step exactly once, surviving restarts.

Human Approval Guarantees

• Trends cannot enter design production without an approval event.
• Rejected or revision-requested designs cannot advance to listing creation.
• Listing copy must be approved before mock Printful/Etsy setup.
• The Etsy gateway exposes draft creation but no publishing operation.
• Publishing is recorded only after a human confirms it occurred in Etsy.
• Printful order confirmation or any other charged operation is not exposed.
• Duplicate approval and publication idempotency keys return HTTP 409.
• Provider-specific modes default to mock and require the top-level INTEGRATIONS_ENABLED kill switch.

The MVP is local-only and has no authentication. Do not expose ports remotely. Add OIDC or an authenticated reverse proxy before deployment.

Data Model

Core tables created by Alembic include:

Table	Holds
trend_discovery_runs	Durable manual scans for emerging apparel trends
trends	Trend candidates with evidence, apparel scores, confidence, and risk warnings
research_runs	In-depth research tasks, structured output, and rendered Markdown
research_source_configs	Selected optional research sources such as GA4
etsy_stats_imports	Deduplicated manual imports of Etsy shopper search terms
products	The full concept → design → copy → listings → publish lifecycle of one product
design_assets	Every generated design revision with its brief, checksum, and review outcome
approval_events	Append-only human decisions with unique idempotency keys
external_operations	Append-only provider-call audit with unique idempotency keys
analytics_snapshots	Append-only listing metrics captured by monitoring
pipeline_events	Append-only step/decision activity log per subject
jobs	Durable queue for background steps
oauth_credentials	Encrypted Etsy access/refresh tokens and expiry metadata
image_generations	Durable standalone image prompts, assets, usage, cost estimates, and outcomes
merch_designs	Immutable reusable designs promoted from Design Studio or approved workflow assets
printful_categories	Cached Printful category hierarchy
printful_catalog_products	Cached products, variants, prices, curation, and Workshop score
printful_product_drafts	Design/product combinations, validation, mockups, and sync-product submissions

Generated files are written through the AssetStore protocol to the generated_assets volume and served through FastAPI; only metadata and durable asset references are stored in Postgres. An S3-compatible store is needed before deploying multiple application instances.

Image Playground

To enable real image generation while leaving Etsy and Printful mocked:

INTEGRATIONS_ENABLED=true
ETSY_MODE=mock
PRINTFUL_MODE=mock
OPENAI_MODE=real
OPENAI_API_KEY=...
GEMINI_API_KEY=...
IMAGE_GENERATION_MODEL=gpt-image-2
OPENAI_IMAGE_MODEL=gpt-image-2
OPENAI_IMAGE_QUALITY=medium
OPENAI_IMAGE_WIDTH=1024
OPENAI_IMAGE_HEIGHT=1024
OPENAI_IMAGE_MAX_COST_USD=0.50

The Design Studio can generate with OpenAI gpt-image-2 or Google Gemini image models: gemini-3.1-flash-image, gemini-3-pro-image, and gemini-2.5-flash-image. GOOGLE_API_KEY is also accepted and takes precedence over GEMINI_API_KEY. Gemini sizes are limited to documented API sizes and are rejected before generation when unsupported. For now, Gemini REST generation is limited to 1024x1024 because the live API rejects the optional sizing config shown in the current docs.

Create a Gemini key from Google AI Studio: https://aistudio.google.com/app/apikey

The Design Studio stores every prompt and outcome in Postgres. A request is rejected before generation when its configured output-cost estimate exceeds the per-image limit. Dollar amounts are estimates because provider responses report usage rather than the final billed amount.

Trend Research And Reports

The Trend Inbox starts manual discovery scans. In real OpenAI mode, scans and reports use Responses API web search in background mode, store verified source URLs, and continue through the durable job queue while the dashboard is closed. Mock mode provides deterministic local results.

Each trend includes a short explanation, 1–5 apparel score, confidence, evidence, and separate trademark, copyright, cultural, and marketplace-policy warnings. Selecting Research opens an editable topic form. Manual topics use the same form. Completed reports render consistently in the dashboard and download as Markdown.

Research context can include Etsy API listing and transaction signals, manually imported Etsy Stats search terms, an optional read-only GA4 property, and cached Printful products, prices, placements, and techniques.

To connect GA4, create a Google OAuth web client, enable the Google Analytics Admin and Data APIs, add the callback as an authorized redirect URI, and set:

GOOGLE_CLIENT_ID=...
GOOGLE_CLIENT_SECRET=...
GOOGLE_TOKEN_ENCRYPTION_KEY=...  # Fernet key
GOOGLE_ANALYTICS_REDIRECT_URI=https://api.workshop.example.com/api/research-sources/google-analytics/oauth/callback

Then connect Google Analytics and select a property from Research Reports. GA4 is a first-party validation signal, not broad market discovery.

Etsy shopper search terms remain in Shop Manager Stats rather than the current public API. Research Reports explains how to open Shop Manager → Stats → How shoppers found you → Etsy search, download Workshop's CSV template, and upload the copied terms and visit counts. Google Trends is disabled because its official API remains limited-access alpha.

Tests

make test        # unit + smoke (SQLite, sub-second)
make lint
make typecheck   # mypy strict on src/
make test-integration  # Alembic migration + job-queue locking against Postgres

CI (GitHub Actions) runs lint, format check, mypy, unit/smoke tests, the Postgres integration suite, and the Docker build on every push and pull request. Dependencies are locked with uv.lock; make lock re-resolves them.

Real Integration Checklist

1. Etsy: complete the initial OAuth 2.0 PKCE authorization flow, then connect the real gateway to the implemented encrypted token manager; add rate-limit handling, image upload, inventory, receipts, and transactions.
2. Printful: implement bearer auth, store selection, catalog validation, webhooks, and explicit safeguards around order confirmation.
3. OpenAI: add daily budgets, print-readiness checks, and promotion from the standalone Design Studio into the product design queue.
4. Notifications (Slack first): signed callbacks and Block Kit approval messages that submit through the same /api/runs/{id}/decisions endpoint used by Streamlit.
5. Replace local asset storage, add OIDC/roles, add secret management, and configure observability.
6. Add contract tests against sandbox/test shops before enabling any real client.

Official references: Printful API, Etsy listings, Etsy API reference, Etsy webhooks, and Slack interactivity.

Etsy Token Refresh

Etsy access tokens expire after one hour. The EtsyTokenManager refreshes them automatically five minutes before expiry and atomically stores Etsy's returned access and refresh tokens in Postgres. Tokens are encrypted at rest with Fernet, and a row lock prevents concurrent application requests from performing duplicate refreshes. The EtsyAuthenticatedHttpClient obtains a managed token for every real Etsy request.

You still need to authorize the Etsy app once. Until the in-app PKCE callback is implemented, put the resulting refresh token in ETSY_REFRESH_TOKEN for one-time bootstrap. Also set ETSY_API_KEY, ETSY_SHOP_ID, and a stable encryption key:

.venv/bin/python -c \
  "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

Store that output in ETSY_TOKEN_ENCRYPTION_KEY. Never rotate or lose this key without first reauthorizing Etsy, because existing database credentials would become undecryptable. After the first successful refresh, Postgres holds the rotated token; changing or removing the bootstrap refresh token does not affect normal operation. Production deployments should put the encryption key and bootstrap token in the cloud secret manager rather than a committed file.

Printful Workshop

The Printful Workshop page provides:

• catalog browsing by category with favorites, notes, a manual 1–5 rating, and a transparent 0–100 Workshop score;
• price scoring based on the cheapest in-stock US variant compared only with products in the same leaf category;
• promotion of Design Studio images and approved workflow designs into an independent merch-design library;
• Cartesian creation of design × product drafts with variant, placement, DPI, cost, margin, and Unlimited Color embroidery validation;
• read-only Product Template browsing with “Use as starting point”;
• mockup generation, durable idempotent sync-product submission, retry, and attachment to an existing Workshop product.

Real catalog reads:

INTEGRATIONS_ENABLED=true
PRINTFUL_MODE=real
PRINTFUL_ACCESS_TOKEN=...
PRINTFUL_STORE_ID=5208171
PRINTFUL_WRITES_ENABLED=false

PRINTFUL_MODE=mock is accepted only when APP_ENV=test; development and production cannot start with a synthetic Printful gateway.

Real mockups, file registration, and sync-product creation additionally require a publicly reachable design URL. By default, Workshop serves the locally stored design through an HMAC-signed, expiring endpoint:

PRINTFUL_WRITES_ENABLED=true
PRINTFUL_ASSET_MODE=local
PRINTFUL_ASSET_PUBLIC_BASE_URL=https://api.workshop.example.com
PRINTFUL_ASSET_SIGNING_KEY=...
PRINTFUL_ASSET_URL_TTL_SECONDS=3600

Printful receives a URL under /api/printful/assets/{token}. The token binds the local asset URI to an expiry timestamp, and invalid, modified, or expired URLs return 404. This endpoint must remain reachable by Printful even when the rest of the API is behind an identity-aware proxy; the signed token is the endpoint authorization.

S3-compatible storage remains an optional alternative for multi-instance deployments:

PRINTFUL_ASSET_MODE=s3
S3_BUCKET=...
S3_REGION=us-east-1
S3_ENDPOINT_URL=                  # optional for AWS S3
S3_ACCESS_KEY_ID=...
S3_SECRET_ACCESS_KEY=...

Printful Product Templates can be listed, read, and deleted through Printful's API, but the API does not expose template creation. Workshop therefore creates sync products and keeps its editable design/product recipes locally. Order creation and order confirmation are intentionally not exposed.