Skip to content

Route studio create --from URLs through Data Liberation - #3952

Draft
chubes4 wants to merge 50 commits into
trunkfrom
feat/site-artifact-import-cli
Draft

Route studio create --from URLs through Data Liberation#3952
chubes4 wants to merge 50 commits into
trunkfrom
feat/site-artifact-import-cli

Conversation

@chubes4

@chubes4 chubes4 commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Proposed Changes

studio create --from <url> routes through Data Liberation and materializes its portable artifact through Static Site Importer. The vendored engine is synchronized with DLA draft #119 at d9912441fa6a732e7c06df84923487d969fe573c, including the independently landed capture fixes for media URLs, comma-safe srcsets, hydrated video, bounded slideshows, and navigation capture.

This PR remains draft while the DLA destination-removal work and Studio integration are under review. Updating the vendor is not a claim of complete site parity or publication to Studio trunk.

Contributor Attribution

Aagam Shah (@aagam-shah) authored the transform-URL safety fix and regression in #4784 (2b9000c9a9687b551fc9dd95fc7caaad72a0bec7) and the comma-safe srcset fix and regression in #4785 (cdd4723ad00ad275d4abc4e9358ac0e7b4763320). Their original commits and Claude co-author trailers are preserved in this branch history. The fixes were also ported into Automattic/data-liberation-agent#170 and Automattic/data-liberation-agent#171. The upstream port descriptions now explicitly credit Aagam; the port commits themselves did not preserve his authorship, which was an attribution mistake.

Verification

  • npm run cli:build passed locally.
  • npm run typecheck passed locally.
  • npm test -- apps/cli/commands/site/tests/create.test.ts passed: 80 tests.
  • Final CI and packaged runtime verification remain required. In particular, the CLI runtime artifact contract must be checked separately from a successful CLI build.

AI Assistance

GPT-5.6 Terra via OpenCode, orchestrated by GPT-6 Astra via OpenCode, synchronized the vendored package, refreshed dependencies, and ran the listed verification. GPT-6 Astra reviewed and corrected contributor attribution. Aagam Shah authored the two original fixes credited above; the porting and vendoring work does not replace that credit.

@chubes4

chubes4 commented Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

CI follow-up pushed in 7d7f45d8c:

  • Unit lane now runs npm run test -- --tagsFilter='!e2e', keeping tagged CLI integration suites in the dedicated serialized CLI E2E lane.
  • Filtered Unit verification passed: 211 files, 2747 passed, 60 skipped.
  • Focused start-stop.e2e.test.ts passed: 2/2.
  • The full local CLI E2E lane remains machine-sensitive under concurrent Playground/PHP load; isolated lifecycle coverage passes and the existing dedicated lane remains serialized.

AI assistance: openai/gpt-5.6-sol via OpenCode diagnosed the CI lane overlap, drafted the one-line test-selection change, and ran verification. Chris Huber remains responsible for the change.

@chubes4

chubes4 commented Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

1,145-route scale validation

The latest branch head (ca6191a59) is pushed and the end-to-end Richlynn run completed successfully through this Studio CLI path.

  • Source: https://www.richlynngroup.com/
  • Routes: 1,145/1,145 completed
  • Batches: 124/124 completed
  • Final state: zero failed or pending batches
  • WordPress routes: 1,148 unique page URIs, no duplicate paths
  • Cache: 5,364 network requests avoided
  • Recovery: resumed across compiler interruption, encoded Unicode paths, transient HTML classification, and canonical route collisions

This proves the Studio orchestration and dependent SSI batching path can handle the target scale. It does not mark the generated site as fidelity-complete: Richlynn still has 13,055 core/html blocks, 1,235 reported fallbacks, 396 retained external assets, and no exact visual/editor parity proof. The generated site remains evidence under the zero-fallback gate.

Dependency PRs:

Validation on the pushed Studio head: npx vitest run apps/cli/commands/site/tests/create.test.ts apps/cli/lib/tests/run-wp-cli-command.test.ts (66/66 passed).

AI assistance: openai/gpt-5.6-sol via OpenCode was used for scale-run diagnosis, implementation support, verification, and drafting this evidence update. Chris Huber remains responsible for the submitted changes.

@chubes4 chubes4 changed the title Experiment: create Studio sites from static artifacts Add studio create --from static source imports Aug 3, 2026
@borkweb

borkweb commented Aug 3, 2026

Copy link
Copy Markdown
Member

The /liberate command already imports static sites into Studio via the blocks-engine 🕺

You can do so via: /liberate ~/path/to/static/site

That being said, switching out liberate's tooling to a different set of direct calls would make sense if the php-transformer has caught up to the JS approach! I also like the direct CLI command that you have for it.

@chubes4

chubes4 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

@borkweb I have not compared the outputs side by side, but I was hoping the approaches could coexist initially. This PR intentionally avoids disrupting existing behavior.

Longer term, I think Data Liberation could own the browser-dependent stages: client-side rendering, runtime behavior, and the visual feedback loop, and use the Studio CLI to materialize the site.

The shared contract would be a clean HTML-based website artifact, regardless of the original source.

I think that refactoring can be explored separately from this PR. Redirecting /liberate is not necessary for this primitive to exist.

Comment thread packages/data-liberation-agent/src/lib/screenshot/screenshotter.ts Fixed
Comment thread packages/data-liberation-agent/src/lib/screenshot/screenshotter.ts Fixed
Comment thread packages/data-liberation-agent/src/lib/screenshot/screenshotter.ts Fixed
@chubes4

chubes4 commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

Synced the DLA iPhone mobile capture identity repair into the existing draft, resolving the generated-bundle-only rebase conflict by rebuilding from the rebased vendor source. Verification: vendor targeted test and full vendor suite passed (87 files, 907 tests); Studio CLI build, lint, and typecheck passed. Full Studio test has 14 environment failures after the fresh dependency setup because the bundled SQLite integration was initially absent; postinstall then completed the runtime asset setup. AI assistance: OpenAI gpt-5.6-terra via OpenCode performed source-level vendoring, regeneration, rebase, and verification.

const text = $(el).text().trim();
if (!text || seen.has(href)) return;
seen.add(href);
if (!text || rawHref === '#' || rawHref.startsWith('javascript:')) return;
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants