diff --git a/.env.example b/.env.example index f5b04dc..9f2eb13 100644 --- a/.env.example +++ b/.env.example @@ -33,6 +33,19 @@ GITHUB_REPO=TypeType-Video/TypeType GITHUB_ISSUE_TEMPLATE=bug_report_backend.md DOWNLOADER_S3_ACCESS_KEY=SET_ME_ACCESS_KEY DOWNLOADER_S3_SECRET_KEY=SET_ME_SECRET_KEY +DOWNLOADER_SERVICE_URL=http://typetype-downloader:18093 +SUBTITLE_SERVICE_URL=http://typetype-token:8081 +YOUTUBE_REMOTE_LOGIN_SERVICE_URL=http://typetype-token:8081 +YOUTUBE_REMOTE_LOGIN_CALLBACK_BASE_URL=http://typetype-server:8080 +TYPETYPE_API_BASE=http://typetype-server:8080 +S3_ENDPOINT=http://garage:3900 +S3_PUBLIC_ENDPOINT=http://garage:3900 +TYPETYPE_SERVER_HOST=typetype-server +TYPETYPE_SERVER_PORT=8080 +TYPETYPE_TOKEN_HOST=typetype-token +TYPETYPE_TOKEN_PORT=8081 +TYPETYPE_DOWNLOADER_HOST=typetype-downloader +TYPETYPE_DOWNLOADER_PORT=18093 GARAGE_RPC_SECRET=SET_ME_GARAGE_RPC_SECRET YOUTUBE_REMOTE_LOGIN_ENABLED=false YOUTUBE_OUTBOUND_PROXY_URL= diff --git a/README.md b/README.md index a9fb6dd..7530633 100644 --- a/README.md +++ b/README.md @@ -29,20 +29,36 @@ multi-service discovery, personal libraries, downloads, administration, and SABR playback without installing a client.

- TypeType playing a YouTube video + TypeType playing a YouTube video

### Search across three services

- Switching a TypeType search between YouTube, NicoNico, and BiliBili + Browsing search results from YouTube, NicoNico, and BiliBili +

+ +### Personal library + +

+ Browsing playlists and personal video libraries in TypeType +

+ +### Web playback + +

+ Browsing a video page and its player in TypeType

### Mobile web -| Home | Playback | Audio only | +| Home | Search | Playback | | --- | --- | --- | -| ![TypeType mobile home](assets/screenshots/07-mobile-home.png) | ![TypeType mobile playback](assets/screenshots/11-mobile-watch.png) | ![TypeType mobile audio-only mode](assets/screenshots/12-mobile-audio-only.png) | +| ![TypeType mobile home](https://raw.githubusercontent.com/TypeType-Video/TypeType/7f46bbb/assets/screenshots/07-mobile-home.png) | ![TypeType mobile search](https://raw.githubusercontent.com/TypeType-Video/TypeType/7f46bbb/assets/screenshots/08-mobile-search.png) | ![TypeType mobile playback](https://raw.githubusercontent.com/TypeType-Video/TypeType/7f46bbb/assets/screenshots/11-mobile-watch.png) | + +

+ Browsing TypeType search results on a mobile screen +

## Native Android app @@ -69,6 +85,23 @@ and Android clients both connect to the same TypeType instance. The installer creates `~/typetype-stack`, generates installation-specific secrets, and asks before starting the stack. +The first startup runs one short-lived `typetype-init` task. It creates the +Downloader database, prepares the Garage configuration, and stores the private +YouTube session secrets in named volumes before the application services start. +Compose removes its container automatically after a successful setup. + +For an existing installation, update the Compose files and run: + +```sh +./scripts/run-stack-init.sh +docker compose up -d --remove-orphans --wait +``` + +The migration keeps the existing `postgres_data`, `typetype_secrets`, and +Garage volumes. Do not use `docker compose down -v` during an update: that +would delete the account database, generated secrets, and stored download +artifacts. + - [Quick start](https://typetype-video.github.io/Docs-TypeType/self-hosting/quick-start) - [Manual Docker Compose setup](https://typetype-video.github.io/Docs-TypeType/self-hosting/docker-compose#manual-setup) - [Configuration](https://typetype-video.github.io/Docs-TypeType/self-hosting/configuration) @@ -102,17 +135,26 @@ Read the [localization guide](https://github.com/TypeType-Video/TypeType-Fronten ## Thanks to sponsors +### Recurring sponsor +

@Toastienergy +

+ +Thank you to [@Toastienergy](https://github.com/Toastienergy) for supporting +TypeType through a recurring GitHub Sponsors contribution. It helps me cover the +infrastructure and spend more time improving the project. + +### One-time donor + +

@filippobaroni

-A big thx to [@Toastienergy](https://github.com/Toastienergy) and -[@filippobaroni](https://github.com/filippobaroni) for supporting TypeType through -GitHub Sponsors. Their support helps me cover the infrastructure and spend more -time improving the project. +I also thank [@filippobaroni](https://github.com/filippobaroni) for the one-time +GitHub Sponsors donation that helped support TypeType. -If u want to support TypeType financially, u can do so through +If you want to support TypeType financially, you can do so through [GitHub Sponsors](https://github.com/sponsors/Priveetee). ## For developers diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 12f68a4..848aee0 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -1,3 +1,93 @@ +# TypeType 1.8.1 + +TypeType 1.8.1 focuses on making playback more dependable across YouTube, BiliBili and NicoNico, improving mobile controls, strengthening profiles and notifications, making recommendations service-aware, and simplifying self-hosting. + +## Reddit and Lemmy + +We are very sad to announce that Reddit has banned the TypeType community! We are still trying to understand why. + +It is frustrating to lose a place where people shared feedback, ideas, translations, bug reports and support for one another. We are truly grateful to everyone who helped make that community useful and welcoming. + +We are moving the TypeType community to Lemmy: + +[Join the TypeType community on Lemmy](https://blorp.lemmy.zip/home/c/TypeType@lemmy.zip) + +We hope Lemmy gives TypeType a much better welcome! + +Lemmy matters because federation keeps a community connected across instances instead of tying it to a single platform. We want TypeType to have a more independent, open and durable community space. + +Please join us, contribute and share. Every test, translation, report, conversation and contribution helps keep this project alive. We hope to see you there! + +## Playback and Providers + +- Recover stalled and evicted SABR segments instead of leaving playback stuck. +- Improve paused Firefox seeks and isolate WebKit media transitions. +- Preserve the exact playback position across provider changes and source reloads. +- Isolate HLS request generations so stale Chromium requests cannot overwrite current playback. +- Align HLS buffering with the MSE player and release provider resources between videos. +- Keep mobile and compact player controls stable, including horizontal volume sliders. +- Restore volume and cinema-mode state reliably after provider initialization. +- Support custom audio SponsorBlock sliders correctly. +- Fix BiliBili playback failures reported in [#262](https://github.com/TypeType-Video/TypeType/issues/262). +- Improve BiliBili audio selection, bullet comments, media handles, range caching and connection reuse. +- Improve NicoNico playback reloads, media-handle routing and segment caching. +- Keep provider media behind expiring opaque handles. +- Preserve request cancellation during provider playback. +- Update TypeType-Player to MSE `0.1.60`. +- Refresh the PipePipeExtractor integration. + +## Profiles, Search and Notifications + +- Add account profiles and profile switching, addressing the workflow tracked in [#234](https://github.com/TypeType-Video/TypeType/issues/234). +- Add avatar support and isolated profile data. +- Bind authentication and service notifications to the active profile. +- Add service-aware search and recommendations so each provider receives relevant content. +- Add a panoramic search experience on desktop and mobile. +- Improve playback-progress loading with batch lookups. +- Add RSS video thumbnails. +- Improve notification delivery and preserve notification state during feed refreshes. + +## Watch Experience and Localization + +- Improve the floating and compact players with fullscreen and return controls. +- Keep player state stable while profile data refreshes. +- Improve mobile layouts and translated-label handling. +- Continue improving the English, French and German interface. +- Add a multilingual Lemmy announcement with a permanent dismissal option. + +## Performance and Self-Hosting + +- Bound playback, retry and server cache memory. +- Reduce unnecessary BiliBili lookups and reuse range connections. +- Parallelize provider media mapping while preserving cancellation. +- Avoid redundant settings writes and defer player initialization until playback is needed. +- Improve Downloader shutdown, job snapshots and artifact persistence. +- Update Downloader to `1.8.1`. +- Consolidate stack initialization into one short-lived `typetype-init` service, advancing the work tracked in [#254](https://github.com/TypeType-Video/TypeType/issues/254). +- Automate secret generation, Garage configuration and Downloader database setup. +- Make service URLs configurable, continuing the work tracked in [#261](https://github.com/TypeType-Video/TypeType/issues/261). +- Fix hardcoded Nginx service routing from [#251](https://github.com/TypeType-Video/TypeType/issues/251). +- Update Garage to `2.4.1`. + +## Issues Fixed in This Dev Line + +- [#262](https://github.com/TypeType-Video/TypeType/issues/262): BiliBili videos could not be played. +- [#251](https://github.com/TypeType-Video/TypeType/issues/251): Nginx used hardcoded `dockerdns` routing. + +## Thx + +Thx to everyone who tested TypeType and shared feedback during this release. + +Thx to @Toastienergy for translating the German interface and helping test the beta. + +A special thx to @Toastienergy for supporting TypeType. + +## Updating + +Follow the [update guide](https://typetype-video.github.io/Docs-TypeType/self-hosting/maintenance). + +If necessary, follow the [rollback guide](https://typetype-video.github.io/Docs-TypeType/self-hosting/rollback). + # TypeType 1.8.0 TypeType 1.8.0 improves playback reliability, the watch experience, diff --git a/TypeType-Downloader b/TypeType-Downloader index a79d46a..321a487 160000 --- a/TypeType-Downloader +++ b/TypeType-Downloader @@ -1 +1 @@ -Subproject commit a79d46a5d7371d6a7cc6fdef618158e6afeca504 +Subproject commit 321a4870673313643d531eee4faf44e1820d5d3e diff --git a/TypeType-Frontend b/TypeType-Frontend index 75014fd..f19386c 160000 --- a/TypeType-Frontend +++ b/TypeType-Frontend @@ -1 +1 @@ -Subproject commit 75014fd767eafacb2cacca59d4a7181c585adbde +Subproject commit f19386cb44f2cda189e66659afcd9428e76587e0 diff --git a/TypeType-Player b/TypeType-Player index 6d6c983..8eb0027 160000 --- a/TypeType-Player +++ b/TypeType-Player @@ -1 +1 @@ -Subproject commit 6d6c983afed1b08fc03196ac13546a30272fcd52 +Subproject commit 8eb0027f5270b78d9cb17657662868e66b71738d diff --git a/TypeType-Server b/TypeType-Server index e28953d..945d59a 160000 --- a/TypeType-Server +++ b/TypeType-Server @@ -1 +1 @@ -Subproject commit e28953da4550b8c35582df704ce6d87ca0fc89d8 +Subproject commit 945d59a25bb8c4ee9a979cd435c21cecb23af40c diff --git a/TypeType-Token b/TypeType-Token index bbe01b9..b91a2a7 160000 --- a/TypeType-Token +++ b/TypeType-Token @@ -1 +1 @@ -Subproject commit bbe01b95030f3c22dce035cffcebf9dff83387b8 +Subproject commit b91a2a7683a8d3839abf4f4aa4f4031189d5cb29 diff --git a/assets/gifs/01-multi-service-search-fluid.gif b/assets/gifs/01-multi-service-search-fluid.gif new file mode 100644 index 0000000..2b8194b Binary files /dev/null and b/assets/gifs/01-multi-service-search-fluid.gif differ diff --git a/assets/gifs/01-multi-service-search.gif b/assets/gifs/01-multi-service-search.gif deleted file mode 100644 index 1a378b2..0000000 Binary files a/assets/gifs/01-multi-service-search.gif and /dev/null differ diff --git a/assets/gifs/02-library-flow-fluid.gif b/assets/gifs/02-library-flow-fluid.gif new file mode 100644 index 0000000..d59250c Binary files /dev/null and b/assets/gifs/02-library-flow-fluid.gif differ diff --git a/assets/gifs/02-save-to-playlist.gif b/assets/gifs/02-save-to-playlist.gif deleted file mode 100644 index d40cadc..0000000 Binary files a/assets/gifs/02-save-to-playlist.gif and /dev/null differ diff --git a/assets/gifs/03-resume-from-history.gif b/assets/gifs/03-resume-from-history.gif deleted file mode 100644 index 5e997a6..0000000 Binary files a/assets/gifs/03-resume-from-history.gif and /dev/null differ diff --git a/assets/gifs/03-watch-flow-fluid.gif b/assets/gifs/03-watch-flow-fluid.gif new file mode 100644 index 0000000..7b6a526 Binary files /dev/null and b/assets/gifs/03-watch-flow-fluid.gif differ diff --git a/assets/gifs/04-mobile-flow-fluid.gif b/assets/gifs/04-mobile-flow-fluid.gif new file mode 100644 index 0000000..ca834bb Binary files /dev/null and b/assets/gifs/04-mobile-flow-fluid.gif differ diff --git a/assets/screenshots/01-hero-watch-desktop.png b/assets/screenshots/01-hero-watch-desktop.png index 9232da2..7bd7317 100644 Binary files a/assets/screenshots/01-hero-watch-desktop.png and b/assets/screenshots/01-hero-watch-desktop.png differ diff --git a/assets/screenshots/02-youtube-search-desktop.png b/assets/screenshots/02-youtube-search-desktop.png index c9325db..bce186a 100644 Binary files a/assets/screenshots/02-youtube-search-desktop.png and b/assets/screenshots/02-youtube-search-desktop.png differ diff --git a/assets/screenshots/03-niconico-search-desktop.png b/assets/screenshots/03-niconico-search-desktop.png index 18c2194..f196876 100644 Binary files a/assets/screenshots/03-niconico-search-desktop.png and b/assets/screenshots/03-niconico-search-desktop.png differ diff --git a/assets/screenshots/04-bilibili-search-desktop.png b/assets/screenshots/04-bilibili-search-desktop.png index 4c2c470..2445351 100644 Binary files a/assets/screenshots/04-bilibili-search-desktop.png and b/assets/screenshots/04-bilibili-search-desktop.png differ diff --git a/assets/screenshots/05-subscriptions-desktop.png b/assets/screenshots/05-subscriptions-desktop.png index 71c5500..3071681 100644 Binary files a/assets/screenshots/05-subscriptions-desktop.png and b/assets/screenshots/05-subscriptions-desktop.png differ diff --git a/assets/screenshots/06-history-desktop.png b/assets/screenshots/06-history-desktop.png index a882627..0ca3df9 100644 Binary files a/assets/screenshots/06-history-desktop.png and b/assets/screenshots/06-history-desktop.png differ diff --git a/assets/screenshots/07-mobile-home.png b/assets/screenshots/07-mobile-home.png index 681abf4..da9a5ea 100644 Binary files a/assets/screenshots/07-mobile-home.png and b/assets/screenshots/07-mobile-home.png differ diff --git a/assets/screenshots/08-mobile-search.png b/assets/screenshots/08-mobile-search.png index 470a87f..abc4505 100644 Binary files a/assets/screenshots/08-mobile-search.png and b/assets/screenshots/08-mobile-search.png differ diff --git a/assets/screenshots/09-mobile-history.png b/assets/screenshots/09-mobile-history.png index 060471f..d8cdd6b 100644 Binary files a/assets/screenshots/09-mobile-history.png and b/assets/screenshots/09-mobile-history.png differ diff --git a/assets/screenshots/10-mobile-playlists.png b/assets/screenshots/10-mobile-playlists.png index 81460c9..30bd613 100644 Binary files a/assets/screenshots/10-mobile-playlists.png and b/assets/screenshots/10-mobile-playlists.png differ diff --git a/assets/screenshots/11-mobile-watch.png b/assets/screenshots/11-mobile-watch.png index 8da9736..5478bca 100644 Binary files a/assets/screenshots/11-mobile-watch.png and b/assets/screenshots/11-mobile-watch.png differ diff --git a/assets/screenshots/12-mobile-audio-only.png b/assets/screenshots/12-mobile-audio-only.png index 6a0df79..5478bca 100644 Binary files a/assets/screenshots/12-mobile-audio-only.png and b/assets/screenshots/12-mobile-audio-only.png differ diff --git a/docker-compose.dev.yml b/docker-compose.dev.yml index 5b0fe25..7d9bf4f 100644 --- a/docker-compose.dev.yml +++ b/docker-compose.dev.yml @@ -1,6 +1,13 @@ services: typetype: image: ${TYPETYPE_WEB_BETA_IMAGE:-ghcr.io/typetype-video/typetype-beta:latest} + environment: + TYPETYPE_SERVER_HOST: ${TYPETYPE_SERVER_HOST:-typetype-beta-server} + TYPETYPE_SERVER_PORT: "${TYPETYPE_SERVER_PORT:-8080}" + TYPETYPE_TOKEN_HOST: ${TYPETYPE_TOKEN_HOST:-typetype-token} + TYPETYPE_TOKEN_PORT: "${TYPETYPE_TOKEN_PORT:-8081}" + TYPETYPE_DOWNLOADER_HOST: ${TYPETYPE_DOWNLOADER_HOST:-typetype-downloader} + TYPETYPE_DOWNLOADER_PORT: "${TYPETYPE_DOWNLOADER_PORT:-18093}" ports: - "${HOST_PORT_WEB_BETA:-18082}:80" depends_on: @@ -25,18 +32,26 @@ services: DATABASE_USER: ${DATABASE_USER:-${POSTGRES_USER:-typetype}} DATABASE_PASSWORD: ${DATABASE_PASSWORD:-${POSTGRES_PASSWORD:-typetype}} DRAGONFLY_URL: ${DRAGONFLY_URL:-redis://dragonfly:6379} - DOWNLOADER_SERVICE_URL: http://typetype-downloader:18093 + DOWNLOADER_SERVICE_URL: ${DOWNLOADER_SERVICE_URL:-http://typetype-downloader:18093} + SUBTITLE_SERVICE_URL: ${SUBTITLE_SERVICE_URL:-http://typetype-token:8081} YOUTUBE_OUTBOUND_PROXY_URL: ${YOUTUBE_OUTBOUND_PROXY_URL:-} YOUTUBE_REMOTE_LOGIN_ENABLED: ${YOUTUBE_REMOTE_LOGIN_ENABLED:-false} - YOUTUBE_REMOTE_LOGIN_SERVICE_URL: http://typetype-token:8081 - YOUTUBE_REMOTE_LOGIN_CALLBACK_BASE_URL: http://typetype-server:8080 - YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN: ${YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN:-} + YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN: "" + YOUTUBE_REMOTE_LOGIN_SERVICE_URL: ${YOUTUBE_REMOTE_LOGIN_SERVICE_URL:-http://typetype-token:8081} + YOUTUBE_REMOTE_LOGIN_CALLBACK_BASE_URL: ${YOUTUBE_REMOTE_LOGIN_CALLBACK_BASE_URL:-http://typetype-beta-server:8080} + YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN_FILE: /run/typetype-secrets/youtube_remote_login_internal_token + YOUTUBE_SESSION_ENCRYPTION_KEY: "" + YOUTUBE_SESSION_ENCRYPTION_KEY_FILE: /run/typetype-secrets/youtube_session_encryption_key YOUTUBE_REMOTE_LOGIN_TTL_MS: ${YOUTUBE_REMOTE_LOGIN_TTL_MS:-480000} YOUTUBE_REMOTE_LOGIN_MAX_SESSIONS: ${YOUTUBE_REMOTE_LOGIN_MAX_SESSIONS:-2} YOUTUBE_REMOTE_LOGIN_MAX_FRAME_BYTES: ${YOUTUBE_REMOTE_LOGIN_MAX_FRAME_BYTES:-524288} + networks: + default: + aliases: + - typetype-beta-server + volumes: + - typetype_secrets:/run/typetype-secrets:ro depends_on: - postgres-init: - condition: service_completed_successfully dragonfly: condition: service_healthy typetype-token: @@ -54,7 +69,7 @@ services: environment: HTTP_PORT: "18093" PUBLIC_BASE_URL: /api/downloader - TYPETYPE_API_BASE: http://typetype-server:8080 + TYPETYPE_API_BASE: ${TYPETYPE_API_BASE:-http://typetype-beta-server:8080} DB_URL: ${DB_URL:-postgres://${POSTGRES_USER:-typetype}:${POSTGRES_PASSWORD:-typetype}@postgres:5432/typetype_downloader?sslmode=disable} DB_USER: ${DB_USER:-${POSTGRES_USER:-typetype}} DB_PASSWORD: ${DB_PASSWORD:-${POSTGRES_PASSWORD:-typetype}} @@ -69,16 +84,14 @@ services: DOWNLOAD_RANGE_MODE: query MUXER: avformat STORAGE_BACKEND: s3 - S3_ENDPOINT: http://garage:3900 - S3_PUBLIC_ENDPOINT: http://garage:3900 + S3_ENDPOINT: ${S3_ENDPOINT:-http://garage:3900} + S3_PUBLIC_ENDPOINT: ${S3_PUBLIC_ENDPOINT:-http://garage:3900} S3_REGION: garage S3_BUCKET: typetype-downloads S3_ACCESS_KEY: ${DOWNLOADER_S3_ACCESS_KEY} S3_SECRET_KEY: ${DOWNLOADER_S3_SECRET_KEY} S3_ARTIFACT_TTL_SECONDS: "7200" depends_on: - postgres-init: - condition: service_completed_successfully dragonfly: condition: service_healthy garage: @@ -97,11 +110,14 @@ services: NODE_ENV: production YOUTUBE_OUTBOUND_PROXY_URL: ${YOUTUBE_OUTBOUND_PROXY_URL:-} YOUTUBE_REMOTE_LOGIN_ENABLED: ${YOUTUBE_REMOTE_LOGIN_ENABLED:-false} - YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN: ${YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN:-} - YOUTUBE_REMOTE_LOGIN_CALLBACK_ORIGIN: ${YOUTUBE_REMOTE_LOGIN_CALLBACK_ORIGIN:-http://typetype-server:8080} + YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN: "" + YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN_FILE: /run/typetype-secrets/youtube_remote_login_internal_token + YOUTUBE_REMOTE_LOGIN_CALLBACK_ORIGIN: ${YOUTUBE_REMOTE_LOGIN_CALLBACK_ORIGIN:-http://typetype-beta-server:8080} YOUTUBE_REMOTE_LOGIN_MAX_SESSIONS: ${YOUTUBE_REMOTE_LOGIN_MAX_SESSIONS:-2} YOUTUBE_REMOTE_LOGIN_FRAME_FPS: ${YOUTUBE_REMOTE_LOGIN_FRAME_FPS:-10} YOUTUBE_REMOTE_LOGIN_MAX_FRAME_BYTES: ${YOUTUBE_REMOTE_LOGIN_MAX_FRAME_BYTES:-524288} + volumes: + - typetype_secrets:/run/typetype-secrets:ro healthcheck: test: ["CMD-SHELL", "curl --fail --silent http://127.0.0.1:8081/health > /dev/null"] restart: unless-stopped @@ -120,24 +136,25 @@ services: retries: 30 restart: unless-stopped - postgres-init: + typetype-init: + profiles: ["init"] image: postgres:17 depends_on: postgres: condition: service_healthy environment: + POSTGRES_HOST: ${POSTGRES_HOST:-postgres} POSTGRES_DB: ${POSTGRES_DB:-typetype} POSTGRES_USER: ${POSTGRES_USER:-typetype} PGPASSWORD: ${POSTGRES_PASSWORD:-typetype} - command: - - /bin/sh - - -ec - - | - until pg_isready -h postgres -U "$$POSTGRES_USER" -d "$$POSTGRES_DB"; do sleep 1; done - EXISTS=$$(psql -h postgres -U "$$POSTGRES_USER" -d "$$POSTGRES_DB" -tAc "SELECT 1 FROM pg_database WHERE datname='typetype_downloader'") - if [ "$$EXISTS" != "1" ]; then - psql -h postgres -U "$$POSTGRES_USER" -d "$$POSTGRES_DB" -c "CREATE DATABASE typetype_downloader" - fi + YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN: ${YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN:-} + YOUTUBE_SESSION_ENCRYPTION_KEY: ${YOUTUBE_SESSION_ENCRYPTION_KEY:-} + command: ["/bin/sh", "/usr/local/bin/initialize-stack.sh"] + volumes: + - typetype_secrets:/run/typetype-secrets + - garage_config:/etc/garage + - ./.typetype-migration:/migration:ro + - ./scripts/initialize-stack.sh:/usr/local/bin/initialize-stack.sh:ro restart: "no" dragonfly: @@ -148,42 +165,8 @@ services: test: ["CMD", "redis-cli", "ping"] restart: unless-stopped - garage-config: - image: busybox:1.38 - command: - - /bin/sh - - -ec - - | - config=/etc/garage/garage.toml - if [ -s "$$config" ]; then - exit 0 - fi - umask 077 - if [ -s /migration/garage.toml ]; then - cp /migration/garage.toml "$$config" - exit 0 - fi - cat > "$$config" <<'EOF' - metadata_dir = "/var/lib/garage/meta" - data_dir = "/var/lib/garage/data" - db_engine = "sqlite" - - replication_factor = 1 - - rpc_bind_addr = "[::]:3901" - rpc_public_addr = "127.0.0.1:3901" - - [s3_api] - s3_region = "garage" - api_bind_addr = "[::]:3900" - EOF - volumes: - - garage_config:/etc/garage - - ./.typetype-migration:/migration:ro - restart: "no" - garage: - image: dxflrs/garage:v2.2.0 + image: dxflrs/garage:v2.4.1 ports: - "${HOST_BIND_GARAGE_S3_BETA:-127.0.0.1}:${HOST_PORT_GARAGE_S3_BETA:-3900}:3900" environment: @@ -193,9 +176,6 @@ services: - garage_config:/etc/garage - garage_meta:/var/lib/garage/meta - garage_data:/var/lib/garage/data - depends_on: - garage-config: - condition: service_completed_successfully healthcheck: test: ["CMD", "/garage", "status"] interval: 5s @@ -205,6 +185,7 @@ services: volumes: postgres_data: + typetype_secrets: garage_config: garage_meta: garage_data: diff --git a/docker-compose.yml b/docker-compose.yml index a942f3d..7f745e6 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,6 +1,13 @@ services: typetype: image: ${TYPETYPE_WEB_IMAGE:-ghcr.io/typetype-video/typetype:latest} + environment: + TYPETYPE_SERVER_HOST: ${TYPETYPE_SERVER_HOST:-typetype-server} + TYPETYPE_SERVER_PORT: "${TYPETYPE_SERVER_PORT:-8080}" + TYPETYPE_TOKEN_HOST: ${TYPETYPE_TOKEN_HOST:-typetype-token} + TYPETYPE_TOKEN_PORT: "${TYPETYPE_TOKEN_PORT:-8081}" + TYPETYPE_DOWNLOADER_HOST: ${TYPETYPE_DOWNLOADER_HOST:-typetype-downloader} + TYPETYPE_DOWNLOADER_PORT: "${TYPETYPE_DOWNLOADER_PORT:-18093}" ports: - "${HOST_PORT_WEB:-8082}:80" depends_on: @@ -26,12 +33,12 @@ services: DRAGONFLY_URL: ${DRAGONFLY_URL:-redis://dragonfly:6379} GITHUB_REPO: ${GITHUB_REPO:-TypeType-Video/TypeType} GITHUB_ISSUE_TEMPLATE: ${GITHUB_ISSUE_TEMPLATE:-bug_report_backend.md} - DOWNLOADER_SERVICE_URL: http://typetype-downloader:18093 + DOWNLOADER_SERVICE_URL: ${DOWNLOADER_SERVICE_URL:-http://typetype-downloader:18093} + SUBTITLE_SERVICE_URL: ${SUBTITLE_SERVICE_URL:-http://typetype-token:8081} YOUTUBE_OUTBOUND_PROXY_URL: ${YOUTUBE_OUTBOUND_PROXY_URL:-} YOUTUBE_REMOTE_LOGIN_ENABLED: ${YOUTUBE_REMOTE_LOGIN_ENABLED:-true} - YOUTUBE_REMOTE_LOGIN_SERVICE_URL: http://typetype-token:8081 - YOUTUBE_REMOTE_LOGIN_CALLBACK_BASE_URL: http://typetype-server:8080 - YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN: ${YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN:-} + YOUTUBE_REMOTE_LOGIN_SERVICE_URL: ${YOUTUBE_REMOTE_LOGIN_SERVICE_URL:-http://typetype-token:8081} + YOUTUBE_REMOTE_LOGIN_CALLBACK_BASE_URL: ${YOUTUBE_REMOTE_LOGIN_CALLBACK_BASE_URL:-http://typetype-server:8080} YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN_FILE: /run/typetype-secrets/youtube_remote_login_internal_token YOUTUBE_SESSION_ENCRYPTION_KEY_FILE: /run/typetype-secrets/youtube_session_encryption_key YOUTUBE_REMOTE_LOGIN_TTL_MS: ${YOUTUBE_REMOTE_LOGIN_TTL_MS:-480000} @@ -40,10 +47,6 @@ services: volumes: - typetype_secrets:/run/typetype-secrets:ro depends_on: - typetype-secrets: - condition: service_completed_successfully - postgres-init: - condition: service_completed_successfully dragonfly: condition: service_healthy typetype-token: @@ -54,43 +57,12 @@ services: test: ["CMD-SHELL", "wget -q -O /dev/null http://127.0.0.1:8080/health"] restart: unless-stopped - typetype-secrets: - image: busybox:1.38 - command: - - /bin/sh - - -ec - - | - generate_secret() { - dd if=/dev/urandom bs=48 count=1 2>/dev/null | base64 | tr '+/' '-_' | tr -d '=\n' - } - ensure_secret() { - file="$$1" - value="$$2" - placeholder="$$3" - legacy_placeholder="$$4" - if [ -n "$$value" ] && [ "$$value" != "$$placeholder" ] && [ "$$value" != "$$legacy_placeholder" ]; then - printf '%s' "$$value" > "$$file" - elif [ ! -s "$$file" ]; then - generate_secret > "$$file" - fi - chmod 0444 "$$file" - } - mkdir -p /run/typetype-secrets - ensure_secret /run/typetype-secrets/youtube_remote_login_internal_token "$$YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN" SET_ME_YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN SET_ME_SHARED_SECRET - ensure_secret /run/typetype-secrets/youtube_session_encryption_key "$$YOUTUBE_SESSION_ENCRYPTION_KEY" SET_ME_YOUTUBE_SESSION_ENCRYPTION_KEY SET_ME_SHARED_SECRET - environment: - YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN: ${YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN:-} - YOUTUBE_SESSION_ENCRYPTION_KEY: ${YOUTUBE_SESSION_ENCRYPTION_KEY:-} - volumes: - - typetype_secrets:/run/typetype-secrets - restart: "no" - typetype-downloader: image: ${TYPETYPE_DOWNLOADER_IMAGE:-ghcr.io/typetype-video/typetype-downloader:latest} environment: HTTP_PORT: "18093" PUBLIC_BASE_URL: /api/downloader - TYPETYPE_API_BASE: http://typetype-server:8080 + TYPETYPE_API_BASE: ${TYPETYPE_API_BASE:-http://typetype-server:8080} DB_URL: ${DB_URL:-postgres://${POSTGRES_USER:-typetype}:${POSTGRES_PASSWORD:-typetype}@postgres:5432/typetype_downloader?sslmode=disable} REDIS_HOST: dragonfly REDIS_PORT: "6379" @@ -103,16 +75,14 @@ services: DOWNLOAD_RANGE_MODE: query MUXER: avformat STORAGE_BACKEND: s3 - S3_ENDPOINT: http://garage:3900 - S3_PUBLIC_ENDPOINT: http://garage:3900 + S3_ENDPOINT: ${S3_ENDPOINT:-http://garage:3900} + S3_PUBLIC_ENDPOINT: ${S3_PUBLIC_ENDPOINT:-http://garage:3900} S3_REGION: garage S3_BUCKET: typetype-downloads S3_ACCESS_KEY: ${DOWNLOADER_S3_ACCESS_KEY:-} S3_SECRET_KEY: ${DOWNLOADER_S3_SECRET_KEY:-} S3_ARTIFACT_TTL_SECONDS: "7200" depends_on: - postgres-init: - condition: service_completed_successfully dragonfly: condition: service_healthy garage: @@ -131,7 +101,6 @@ services: NODE_ENV: production YOUTUBE_OUTBOUND_PROXY_URL: ${YOUTUBE_OUTBOUND_PROXY_URL:-} YOUTUBE_REMOTE_LOGIN_ENABLED: ${YOUTUBE_REMOTE_LOGIN_ENABLED:-true} - YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN: ${YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN:-} YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN_FILE: /run/typetype-secrets/youtube_remote_login_internal_token YOUTUBE_REMOTE_LOGIN_CALLBACK_ORIGIN: ${YOUTUBE_REMOTE_LOGIN_CALLBACK_ORIGIN:-http://typetype-server:8080} YOUTUBE_REMOTE_LOGIN_MAX_SESSIONS: ${YOUTUBE_REMOTE_LOGIN_MAX_SESSIONS:-2} @@ -139,9 +108,6 @@ services: YOUTUBE_REMOTE_LOGIN_MAX_FRAME_BYTES: ${YOUTUBE_REMOTE_LOGIN_MAX_FRAME_BYTES:-524288} volumes: - typetype_secrets:/run/typetype-secrets:ro - depends_on: - typetype-secrets: - condition: service_completed_successfully healthcheck: test: ["CMD-SHELL", "curl --fail --silent http://127.0.0.1:8081/health > /dev/null"] restart: unless-stopped @@ -160,24 +126,25 @@ services: retries: 30 restart: unless-stopped - postgres-init: + typetype-init: + profiles: ["init"] image: postgres:17 depends_on: postgres: condition: service_healthy environment: + POSTGRES_HOST: ${POSTGRES_HOST:-postgres} POSTGRES_DB: ${POSTGRES_DB:-typetype} POSTGRES_USER: ${POSTGRES_USER:-typetype} PGPASSWORD: ${POSTGRES_PASSWORD:-typetype} - command: - - /bin/sh - - -ec - - | - until pg_isready -h postgres -U "$$POSTGRES_USER" -d "$$POSTGRES_DB"; do sleep 1; done - EXISTS=$$(psql -h postgres -U "$$POSTGRES_USER" -d "$$POSTGRES_DB" -tAc "SELECT 1 FROM pg_database WHERE datname='typetype_downloader'") - if [ "$$EXISTS" != "1" ]; then - psql -h postgres -U "$$POSTGRES_USER" -d "$$POSTGRES_DB" -c "CREATE DATABASE typetype_downloader" - fi + YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN: ${YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN:-} + YOUTUBE_SESSION_ENCRYPTION_KEY: ${YOUTUBE_SESSION_ENCRYPTION_KEY:-} + command: ["/bin/sh", "/usr/local/bin/initialize-stack.sh"] + volumes: + - typetype_secrets:/run/typetype-secrets + - garage_config:/etc/garage + - ./.typetype-migration:/migration:ro + - ./scripts/initialize-stack.sh:/usr/local/bin/initialize-stack.sh:ro restart: "no" dragonfly: @@ -187,42 +154,8 @@ services: healthcheck: test: ["CMD", "redis-cli", "ping"] restart: unless-stopped - garage-config: - image: busybox:1.38 - command: - - /bin/sh - - -ec - - | - config=/etc/garage/garage.toml - if [ -s "$$config" ]; then - exit 0 - fi - umask 077 - if [ -s /migration/garage.toml ]; then - cp /migration/garage.toml "$$config" - exit 0 - fi - cat > "$$config" <<'EOF' - metadata_dir = "/var/lib/garage/meta" - data_dir = "/var/lib/garage/data" - db_engine = "sqlite" - - replication_factor = 1 - - rpc_bind_addr = "[::]:3901" - rpc_public_addr = "127.0.0.1:3901" - - [s3_api] - s3_region = "garage" - api_bind_addr = "[::]:3900" - EOF - volumes: - - garage_config:/etc/garage - - ./.typetype-migration:/migration:ro - restart: "no" - garage: - image: dxflrs/garage:v2.2.0 + image: dxflrs/garage:v2.4.1 ports: - "${HOST_BIND_GARAGE_S3:-127.0.0.1}:${HOST_PORT_GARAGE_S3:-3900}:3900" environment: @@ -232,9 +165,6 @@ services: - garage_config:/etc/garage - garage_meta:/var/lib/garage/meta - garage_data:/var/lib/garage/data - depends_on: - garage-config: - condition: service_completed_successfully healthcheck: test: ["CMD", "/garage", "status"] interval: 5s diff --git a/scripts/deploy-beta.sh b/scripts/deploy-beta.sh index 5c96da7..72ce7ca 100755 --- a/scripts/deploy-beta.sh +++ b/scripts/deploy-beta.sh @@ -99,17 +99,18 @@ managed_files=( docker-compose.dev.yml scripts/bootstrap-garage.sh scripts/check-youtube-egress.sh + scripts/initialize-stack.sh + scripts/run-stack-init.sh scripts/deploy-beta.sh ) services=( typetype typetype-server + typetype-init typetype-downloader typetype-token postgres - postgres-init dragonfly - garage-config garage ) @@ -201,6 +202,8 @@ install -m 644 "$source_root/.env.example" "$root/.env.example" install -m 644 "$source_root/docker-compose.dev.yml" "$root/docker-compose.dev.yml" install -m 755 "$source_root/scripts/bootstrap-garage.sh" "$root/scripts/bootstrap-garage.sh" install -m 755 "$source_root/scripts/check-youtube-egress.sh" "$root/scripts/check-youtube-egress.sh" +install -m 755 "$source_root/scripts/initialize-stack.sh" "$root/scripts/initialize-stack.sh" +install -m 755 "$source_root/scripts/run-stack-init.sh" "$root/scripts/run-stack-init.sh" install -m 755 "$source_root/scripts/deploy-beta.sh" "$root/scripts/deploy-beta.sh" install -d -m 700 "$root/.typetype-migration" if [[ -s "$root/garage.toml" ]]; then @@ -222,6 +225,7 @@ probe() { if [[ "$component" == all ]]; then stage=update-full-stack compose pull + COMPOSE_FILE="$root/docker-compose.dev.yml" ./scripts/run-stack-init.sh compose up -d --remove-orphans --wait --wait-timeout 180 ./scripts/bootstrap-garage.sh probe http://127.0.0.1:8080/health diff --git a/scripts/deploy-beta.test.sh b/scripts/deploy-beta.test.sh index b524165..7a2d50c 100755 --- a/scripts/deploy-beta.test.sh +++ b/scripts/deploy-beta.test.sh @@ -14,6 +14,8 @@ for file in \ docker-compose.dev.yml \ scripts/bootstrap-garage.sh \ scripts/check-youtube-egress.sh \ + scripts/initialize-stack.sh \ + scripts/run-stack-init.sh \ scripts/deploy-beta.sh; do mkdir -p "$stack/$(dirname "$file")" cp "$repository/$file" "$stack/$file" @@ -49,8 +51,8 @@ case "$1" in case "$*" in *"config --services"*) printf '%s\n' typetype typetype-server typetype-downloader \ - typetype-token postgres postgres-init \ - dragonfly garage-config garage + typetype-token postgres typetype-init \ + dragonfly garage ;; *"config --environment"*) echo "YOUTUBE_OUTBOUND_PROXY_URL=http://127.0.0.1:29083" diff --git a/scripts/deploy-stable.sh b/scripts/deploy-stable.sh index 71ec6b3..958c28d 100755 --- a/scripts/deploy-stable.sh +++ b/scripts/deploy-stable.sh @@ -2,10 +2,17 @@ set -euo pipefail source_root="${1:?deployment source is required}" -project=typetype-stack -server=$(docker ps -q \ - --filter "label=com.docker.compose.project=${project}" \ - --filter label=com.docker.compose.service=typetype-server) +project= +server= +for candidate in typetype-stack typetype; do + server=$(docker ps -q \ + --filter "label=com.docker.compose.project=${candidate}" \ + --filter label=com.docker.compose.service=typetype-server | head -n 1) + if [[ -n "$server" ]]; then + project="$candidate" + break + fi +done test -n "$server" root=$(docker inspect "$server" --format '{{index .Config.Labels "com.docker.compose.project.working_dir"}}') test -d "$root" @@ -24,11 +31,14 @@ managed_files=( docker-compose.arm64.yml scripts/bootstrap-env.sh scripts/bootstrap-garage.sh + scripts/initialize-stack.sh + scripts/run-stack-init.sh scripts/deploy-stable.sh ) services=( typetype typetype-server + typetype-init typetype-secrets typetype-downloader typetype-token @@ -55,7 +65,14 @@ for file in "${managed_files[@]}"; do done printf 'services:\n' > "$backup/rollback.yml" +declare -A current_services=() +while read -r current_service; do + current_services["$current_service"]=1 +done < <(compose config --services) for service in "${services[@]}"; do + if [[ -z "${current_services[$service]:-}" ]]; then + continue + fi container=$(compose ps -a -q "$service" 2>/dev/null | head -n 1 || true) if [[ -n "$container" ]]; then image=$(docker inspect "$container" --format '{{.Image}}') @@ -114,11 +131,14 @@ for file in "${managed_files[@]}"; do done chmod 755 "$root/scripts/bootstrap-env.sh" chmod 755 "$root/scripts/bootstrap-garage.sh" +chmod 755 "$root/scripts/initialize-stack.sh" +chmod 755 "$root/scripts/run-stack-init.sh" chmod 755 "$root/scripts/deploy-stable.sh" cd "$root" ./scripts/bootstrap-env.sh compose pull +./scripts/run-stack-init.sh compose up -d --remove-orphans --wait --wait-timeout 180 ./scripts/bootstrap-garage.sh diff --git a/scripts/initialize-stack.sh b/scripts/initialize-stack.sh new file mode 100644 index 0000000..627837f --- /dev/null +++ b/scripts/initialize-stack.sh @@ -0,0 +1,66 @@ +#!/bin/sh +set -eu + +generate_secret() { + openssl rand -base64 48 | tr '+/' '-_' | tr -d '=\n' +} + +ensure_secret() { + file="$1" + value="$2" + placeholder="$3" + legacy_placeholder="$4" + if [ -n "$value" ] && [ "$value" != "$placeholder" ] && [ "$value" != "$legacy_placeholder" ]; then + printf '%s' "$value" > "$file" + elif [ ! -s "$file" ]; then + generate_secret > "$file" + fi + chmod 0444 "$file" +} + +mkdir -p /run/typetype-secrets +ensure_secret \ + /run/typetype-secrets/youtube_remote_login_internal_token \ + "${YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN:-}" \ + SET_ME_YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN \ + SET_ME_SHARED_SECRET +ensure_secret \ + /run/typetype-secrets/youtube_session_encryption_key \ + "${YOUTUBE_SESSION_ENCRYPTION_KEY:-}" \ + SET_ME_YOUTUBE_SESSION_ENCRYPTION_KEY \ + SET_ME_SHARED_SECRET + +config=/etc/garage/garage.toml +if [ ! -s "$config" ]; then + umask 077 + if [ -s /migration/garage.toml ]; then + cp /migration/garage.toml "$config" + else + cat > "$config" <<'EOF' +metadata_dir = "/var/lib/garage/meta" +data_dir = "/var/lib/garage/data" +db_engine = "sqlite" + +replication_factor = 1 + +rpc_bind_addr = "[::]:3901" +rpc_public_addr = "127.0.0.1:3901" + +[s3_api] +s3_region = "garage" +api_bind_addr = "[::]:3900" +EOF + fi +fi + +postgres_host="${POSTGRES_HOST:-postgres}" +until pg_isready -h "$postgres_host" -U "$POSTGRES_USER" -d "$POSTGRES_DB"; do + sleep 1 +done + +exists="$(psql -h "$postgres_host" -U "$POSTGRES_USER" -d "$POSTGRES_DB" -tAc \ + "SELECT 1 FROM pg_database WHERE datname='typetype_downloader'")" +if [ "$exists" != "1" ]; then + psql -h "$postgres_host" -U "$POSTGRES_USER" -d "$POSTGRES_DB" \ + -c 'CREATE DATABASE typetype_downloader' +fi diff --git a/scripts/install-stack.sh b/scripts/install-stack.sh index f7a30ee..e346f12 100755 --- a/scripts/install-stack.sh +++ b/scripts/install-stack.sh @@ -22,6 +22,8 @@ DEFAULT_YOUTUBE_REMOTE_LOGIN_TTL_MS="480000" DEFAULT_YOUTUBE_REMOTE_LOGIN_MAX_SESSIONS="2" DEFAULT_YOUTUBE_REMOTE_LOGIN_FRAME_FPS="10" DEFAULT_YOUTUBE_REMOTE_LOGIN_MAX_FRAME_BYTES="524288" +DEFAULT_BETA_SERVER_HOST="typetype-beta-server" +DEFAULT_BETA_SERVER_URL="http://typetype-beta-server:8080" PLACEHOLDER_YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN="SET_ME_SHARED_SECRET" usage() { @@ -204,6 +206,32 @@ ensure_env_default() { fi } +ensure_beta_internal_server_env() { + local env_file="$1" + local current + + if [[ ${BETA_STACK} -ne 1 ]]; then + return 0 + fi + + current="$(get_env_var "${env_file}" "YOUTUBE_REMOTE_LOGIN_CALLBACK_ORIGIN")" + if [[ -z "${current}" || "${current}" == "http://typetype-server:8080" || "${current}" == "http://typetype-beta-stack-typetype-server-1:8080" ]]; then + set_env_var "${env_file}" "YOUTUBE_REMOTE_LOGIN_CALLBACK_ORIGIN" "${DEFAULT_BETA_SERVER_URL}" + fi + current="$(get_env_var "${env_file}" "YOUTUBE_REMOTE_LOGIN_CALLBACK_BASE_URL")" + if [[ -z "${current}" || "${current}" == "http://typetype-server:8080" || "${current}" == "http://typetype-beta-stack-typetype-server-1:8080" ]]; then + set_env_var "${env_file}" "YOUTUBE_REMOTE_LOGIN_CALLBACK_BASE_URL" "${DEFAULT_BETA_SERVER_URL}" + fi + current="$(get_env_var "${env_file}" "TYPETYPE_API_BASE")" + if [[ -z "${current}" || "${current}" == "http://typetype-server:8080" || "${current}" == "http://typetype-beta-stack-typetype-server-1:8080" ]]; then + set_env_var "${env_file}" "TYPETYPE_API_BASE" "${DEFAULT_BETA_SERVER_URL}" + fi + current="$(get_env_var "${env_file}" "TYPETYPE_SERVER_HOST")" + if [[ -z "${current}" || "${current}" == "typetype-server" || "${current}" == "typetype-beta-stack-typetype-server-1" ]]; then + set_env_var "${env_file}" "TYPETYPE_SERVER_HOST" "${DEFAULT_BETA_SERVER_HOST}" + fi +} + ensure_youtube_remote_login_env() { local env_file="$1" local current_token @@ -222,6 +250,24 @@ ensure_youtube_remote_login_env() { fi } +ensure_service_url_env() { + local env_file="$1" + + ensure_env_default "${env_file}" "DOWNLOADER_SERVICE_URL" "http://typetype-downloader:18093" + ensure_env_default "${env_file}" "SUBTITLE_SERVICE_URL" "http://typetype-token:8081" + ensure_env_default "${env_file}" "YOUTUBE_REMOTE_LOGIN_SERVICE_URL" "http://typetype-token:8081" + ensure_env_default "${env_file}" "YOUTUBE_REMOTE_LOGIN_CALLBACK_BASE_URL" "http://typetype-server:8080" + ensure_env_default "${env_file}" "TYPETYPE_API_BASE" "http://typetype-server:8080" + ensure_env_default "${env_file}" "S3_ENDPOINT" "http://garage:3900" + ensure_env_default "${env_file}" "S3_PUBLIC_ENDPOINT" "http://garage:3900" + ensure_env_default "${env_file}" "TYPETYPE_SERVER_HOST" "typetype-server" + ensure_env_default "${env_file}" "TYPETYPE_SERVER_PORT" "8080" + ensure_env_default "${env_file}" "TYPETYPE_TOKEN_HOST" "typetype-token" + ensure_env_default "${env_file}" "TYPETYPE_TOKEN_PORT" "8081" + ensure_env_default "${env_file}" "TYPETYPE_DOWNLOADER_HOST" "typetype-downloader" + ensure_env_default "${env_file}" "TYPETYPE_DOWNLOADER_PORT" "18093" +} + port_is_listening() { local port="$1" local port_hex @@ -498,12 +544,16 @@ fetch_file ".env.example" "${INSTALL_DIR}/.env.example" fetch_file "scripts/install-stack.sh" "${INSTALL_DIR}/scripts/install-stack.sh" fetch_file "scripts/bootstrap-env.sh" "${INSTALL_DIR}/scripts/bootstrap-env.sh" fetch_file "scripts/bootstrap-garage.sh" "${INSTALL_DIR}/scripts/bootstrap-garage.sh" +fetch_file "scripts/initialize-stack.sh" "${INSTALL_DIR}/scripts/initialize-stack.sh" +fetch_file "scripts/run-stack-init.sh" "${INSTALL_DIR}/scripts/run-stack-init.sh" fetch_file "scripts/setup-stack.sh" "${INSTALL_DIR}/scripts/setup-stack.sh" fetch_file "scripts/validate-stack.sh" "${INSTALL_DIR}/scripts/validate-stack.sh" chmod +x "${INSTALL_DIR}/scripts/install-stack.sh" chmod +x "${INSTALL_DIR}/scripts/bootstrap-env.sh" chmod +x "${INSTALL_DIR}/scripts/bootstrap-garage.sh" +chmod +x "${INSTALL_DIR}/scripts/initialize-stack.sh" +chmod +x "${INSTALL_DIR}/scripts/run-stack-init.sh" chmod +x "${INSTALL_DIR}/scripts/setup-stack.sh" chmod +x "${INSTALL_DIR}/scripts/validate-stack.sh" @@ -529,6 +579,8 @@ fi ensure_random_downloader_keys "${INSTALL_DIR}/.env" ensure_youtube_remote_login_env "${INSTALL_DIR}/.env" +ensure_service_url_env "${INSTALL_DIR}/.env" +ensure_beta_internal_server_env "${INSTALL_DIR}/.env" "${INSTALL_DIR}/scripts/bootstrap-env.sh" if [[ ${BETA_STACK} -eq 1 ]]; then @@ -574,14 +626,14 @@ if [[ ${START_STACK} -eq 0 ]]; then echo "[install] Download-only complete." [[ ! -s "${INSTALL_DIR}/garage.toml" ]] || echo "[install] The existing garage.toml will be imported on the next Compose startup." [[ -z "${BACKUP_DIR}" ]] || echo "[install] Rollback files: ${BACKUP_DIR}" - echo "[install] Next step: cd ${INSTALL_DIR} && $(compose_command_hint 'up -d')" + echo "[install] Next step: cd ${INSTALL_DIR} && ./scripts/run-stack-init.sh && $(compose_command_hint 'up -d')" exit 0 fi if [[ ${AUTO_APPROVE} -eq 0 ]] && ! confirm_tty "Proceed with Docker pull + startup in ${INSTALL_DIR}?"; then echo "[install] Stack files are ready in ${INSTALL_DIR}." echo "[install] Docker startup skipped." - echo "[install] Next step: cd ${INSTALL_DIR} && $(compose_command_hint 'up -d')" + echo "[install] Next step: cd ${INSTALL_DIR} && ./scripts/run-stack-init.sh && $(compose_command_hint 'up -d')" exit 0 fi @@ -589,7 +641,12 @@ echo "[install] Pulling Docker images..." docker compose "${COMPOSE_ARGS[@]}" --env-file "${INSTALL_DIR}/.env" pull echo "[install] Starting stack..." -docker compose "${COMPOSE_ARGS[@]}" --env-file "${INSTALL_DIR}/.env" up -d --wait --wait-timeout 180 +COMPOSE_FILE="${COMPOSE_FILE}" \ + COMPOSE_OVERRIDE_FILE="${COMPOSE_OVERRIDE_FILE}" \ + COMPOSE_CUSTOM_FILE="${CUSTOM_COMPOSE_FILE}" \ + "${INSTALL_DIR}/scripts/run-stack-init.sh" +docker compose "${COMPOSE_ARGS[@]}" --env-file "${INSTALL_DIR}/.env" \ + up -d --remove-orphans --wait --wait-timeout 180 echo "[install] Bootstrapping Garage..." ( diff --git a/scripts/install-stack.test.sh b/scripts/install-stack.test.sh index de8fe54..38243b5 100755 --- a/scripts/install-stack.test.sh +++ b/scripts/install-stack.test.sh @@ -43,6 +43,8 @@ for file in \ scripts/install-stack.sh \ scripts/bootstrap-env.sh \ scripts/bootstrap-garage.sh \ + scripts/initialize-stack.sh \ + scripts/run-stack-init.sh \ scripts/setup-stack.sh \ scripts/validate-stack.sh; do test -s "${install_dir}/${file}" @@ -59,3 +61,22 @@ remote_login_token="$(grep '^YOUTUBE_REMOTE_LOGIN_INTERNAL_TOKEN=' "${install_di [[ "$downloader_access_key" =~ ^GK[0-9a-f]{24}$ ]] [[ "$downloader_secret_key" =~ ^[0-9a-f]{64}$ ]] [[ "$remote_login_token" =~ ^[A-Za-z0-9_-]{64}$ ]] + +beta_install_dir="${temporary}/beta-stack" +PATH="${fake_bin}:${PATH}" \ +FAKE_DOCKER_LOG="$docker_log" \ +FAKE_PYTHON_LOG="$python_log" \ + bash "${repository}/scripts/install-stack.sh" \ + --source-dir "$repository" \ + --dir "$beta_install_dir" \ + --beta \ + --download-only \ + --yes > "${temporary}/beta-install.log" + +grep -Fxq 'YOUTUBE_REMOTE_LOGIN_CALLBACK_ORIGIN=http://typetype-beta-server:8080' \ + "$beta_install_dir/.env" +grep -Fxq 'YOUTUBE_REMOTE_LOGIN_CALLBACK_BASE_URL=http://typetype-beta-server:8080' \ + "$beta_install_dir/.env" +grep -Fxq 'TYPETYPE_API_BASE=http://typetype-beta-server:8080' \ + "$beta_install_dir/.env" +grep -Fxq 'TYPETYPE_SERVER_HOST=typetype-beta-server' "$beta_install_dir/.env" diff --git a/scripts/run-stack-init.sh b/scripts/run-stack-init.sh new file mode 100755 index 0000000..2efcb64 --- /dev/null +++ b/scripts/run-stack-init.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail + +root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +compose_file="${COMPOSE_FILE:-${root}/docker-compose.yml}" +compose_override_file="${COMPOSE_OVERRIDE_FILE:-}" +compose_custom_file="${COMPOSE_CUSTOM_FILE:-}" + +compose_args=(--project-directory "$root" --env-file "$root/.env" -f "$compose_file") +if [[ -n "$compose_override_file" ]]; then + compose_args+=(-f "$compose_override_file") +fi +if [[ -n "$compose_custom_file" ]]; then + compose_args+=(-f "$compose_custom_file") +fi + +compose() { + docker compose "${compose_args[@]}" "$@" +} + +# Remove the persistent container created by stacks predating one-shot init. +compose rm -f -s typetype-init >/dev/null 2>&1 || true +compose run --rm typetype-init diff --git a/scripts/run-stack-init.test.sh b/scripts/run-stack-init.test.sh new file mode 100755 index 0000000..d6deaf0 --- /dev/null +++ b/scripts/run-stack-init.test.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash +set -euo pipefail + +repository="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +temporary="$(mktemp -d)" +trap 'rm -rf "$temporary"' EXIT + +cat > "$temporary/docker" <<'EOF' +#!/usr/bin/env bash +set -euo pipefail +printf '%q ' "$@" >> "$FAKE_DOCKER_LOG" +printf '\n' >> "$FAKE_DOCKER_LOG" +EOF +chmod +x "$temporary/docker" + +PATH="$temporary:$PATH" \ +FAKE_DOCKER_LOG="$temporary/docker.log" \ + "$repository/scripts/run-stack-init.sh" + +grep -Fq 'rm -f -s typetype-init' "$temporary/docker.log" +grep -Fq 'run --rm typetype-init' "$temporary/docker.log" diff --git a/scripts/setup-stack.sh b/scripts/setup-stack.sh index 43cbefb..80128d1 100755 --- a/scripts/setup-stack.sh +++ b/scripts/setup-stack.sh @@ -112,6 +112,24 @@ ensure_youtube_remote_login_env() { fi } +ensure_service_url_env() { + local env_file="$1" + + ensure_env_default "${env_file}" "DOWNLOADER_SERVICE_URL" "http://typetype-downloader:18093" + ensure_env_default "${env_file}" "SUBTITLE_SERVICE_URL" "http://typetype-token:8081" + ensure_env_default "${env_file}" "YOUTUBE_REMOTE_LOGIN_SERVICE_URL" "http://typetype-token:8081" + ensure_env_default "${env_file}" "YOUTUBE_REMOTE_LOGIN_CALLBACK_BASE_URL" "http://typetype-server:8080" + ensure_env_default "${env_file}" "TYPETYPE_API_BASE" "http://typetype-server:8080" + ensure_env_default "${env_file}" "S3_ENDPOINT" "http://garage:3900" + ensure_env_default "${env_file}" "S3_PUBLIC_ENDPOINT" "http://garage:3900" + ensure_env_default "${env_file}" "TYPETYPE_SERVER_HOST" "typetype-server" + ensure_env_default "${env_file}" "TYPETYPE_SERVER_PORT" "8080" + ensure_env_default "${env_file}" "TYPETYPE_TOKEN_HOST" "typetype-token" + ensure_env_default "${env_file}" "TYPETYPE_TOKEN_PORT" "8081" + ensure_env_default "${env_file}" "TYPETYPE_DOWNLOADER_HOST" "typetype-downloader" + ensure_env_default "${env_file}" "TYPETYPE_DOWNLOADER_PORT" "18093" +} + get_env_var() { local env_file="$1" local key="$2" @@ -272,6 +290,7 @@ if [[ -s "${ROOT_DIR}/garage.toml" ]]; then fi ensure_youtube_remote_login_env "${ENV_FILE}" +ensure_service_url_env "${ENV_FILE}" "${ROOT_DIR}/scripts/bootstrap-env.sh" HOST_PORT_SERVER_RESOLVED="$(choose_stack_port "${ENV_FILE}" "HOST_PORT_SERVER" "${DEFAULT_HOST_PORT_SERVER}" "API")" @@ -296,7 +315,11 @@ echo "[setup] Pulling images..." docker compose "${COMPOSE_ARGS[@]}" pull echo "[setup] Starting services..." -docker compose "${COMPOSE_ARGS[@]}" up -d --wait --wait-timeout 180 +COMPOSE_FILE="${ROOT_DIR}/docker-compose.yml" \ + COMPOSE_OVERRIDE_FILE="${COMPOSE_OVERRIDE_FILE}" \ + COMPOSE_CUSTOM_FILE="${COMPOSE_CUSTOM_FILE}" \ + "${ROOT_DIR}/scripts/run-stack-init.sh" +docker compose "${COMPOSE_ARGS[@]}" up -d --remove-orphans --wait --wait-timeout 180 echo "[setup] Bootstrapping Garage for downloader..." COMPOSE_FILE="${ROOT_DIR}/docker-compose.yml" \ diff --git a/scripts/validate-stack.sh b/scripts/validate-stack.sh index 85dd5be..df50760 100755 --- a/scripts/validate-stack.sh +++ b/scripts/validate-stack.sh @@ -21,6 +21,7 @@ for script in scripts/*.sh; do done ./scripts/install-stack.test.sh ./scripts/deploy-beta.test.sh +./scripts/run-stack-init.test.sh docker compose --env-file .env.example -f docker-compose.yml config -q docker compose --env-file .env.example -f docker-compose.yml -f docker-compose.arm64.yml config -q @@ -28,9 +29,66 @@ docker compose --env-file .env.example -f docker-compose.dev.yml config -q YOUTUBE_OUTBOUND_PROXY_URL=http://127.0.0.1:29083 \ docker compose --env-file .env.example -f docker-compose.dev.yml config -q +custom_config="$(TYPETYPE_SERVER_HOST=custom-server \ + TYPETYPE_TOKEN_HOST=custom-token \ + TYPETYPE_DOWNLOADER_HOST=custom-downloader \ + DOWNLOADER_SERVICE_URL=http://custom-downloader:28093 \ + SUBTITLE_SERVICE_URL=http://custom-token:28081 \ + YOUTUBE_REMOTE_LOGIN_SERVICE_URL=http://custom-token:28081 \ + YOUTUBE_REMOTE_LOGIN_CALLBACK_BASE_URL=http://custom-server:28080 \ + TYPETYPE_API_BASE=http://custom-server:28080 \ + S3_ENDPOINT=http://custom-garage:23900 \ + S3_PUBLIC_ENDPOINT=http://public-garage:23900 \ + docker compose --env-file .env.example -f docker-compose.yml config)" +for expected in \ + 'TYPETYPE_SERVER_HOST: custom-server' \ + 'TYPETYPE_TOKEN_HOST: custom-token' \ + 'TYPETYPE_DOWNLOADER_HOST: custom-downloader' \ + 'DOWNLOADER_SERVICE_URL: http://custom-downloader:28093' \ + 'SUBTITLE_SERVICE_URL: http://custom-token:28081' \ + 'YOUTUBE_REMOTE_LOGIN_SERVICE_URL: http://custom-token:28081' \ + 'YOUTUBE_REMOTE_LOGIN_CALLBACK_BASE_URL: http://custom-server:28080' \ + 'TYPETYPE_API_BASE: http://custom-server:28080' \ + 'S3_ENDPOINT: http://custom-garage:23900' \ + 'S3_PUBLIC_ENDPOINT: http://public-garage:23900'; do + if ! grep -q "${expected}" <<<"${custom_config}"; then + echo "custom service URL was not propagated: ${expected}" >&2 + exit 1 + fi +done + stable_config="$(docker compose --env-file .env.example -f docker-compose.yml config)" dev_config="$(YOUTUBE_OUTBOUND_PROXY_URL=http://127.0.0.1:29083 \ docker compose --env-file .env.example -f docker-compose.dev.yml config)" +stable_services="$(docker compose --env-file .env.example -f docker-compose.yml config --services)" +dev_services="$(YOUTUBE_OUTBOUND_PROXY_URL=http://127.0.0.1:29083 \ + docker compose --env-file .env.example -f docker-compose.dev.yml config --services)" +for services in "$stable_services" "$dev_services"; do + if grep -Fxq typetype-init <<<"$services"; then + echo "default Compose startup must exclude the one-shot init service" >&2 + exit 1 + fi + for removed_service in typetype-secrets postgres-init garage-config; do + if grep -Fxq "$removed_service" <<<"$services"; then + echo "obsolete init service is still exposed: ${removed_service}" >&2 + exit 1 + fi + done +done +for compose_file in docker-compose.yml docker-compose.dev.yml; do + init_config="$(docker compose --profile init --env-file .env.example \ + -f "$compose_file" config)" + init_services="$(docker compose --profile init --env-file .env.example \ + -f "$compose_file" config --services)" + if ! grep -Fxq typetype-init <<<"$init_services"; then + echo "the init profile must expose the consolidated init service" >&2 + exit 1 + fi + if ! grep -q 'initialize-stack.sh' <<<"$init_config"; then + echo "the init profile must mount the shared initialization script" >&2 + exit 1 + fi +done if grep -q '/etc/nginx/conf.d/default.conf' <<<"${stable_config}${dev_config}"; then echo "default Compose must use the nginx configuration bundled in the web image" >&2 exit 1 @@ -61,3 +119,7 @@ for config in "$stable_config" "$dev_config"; do exit 1 fi done +if grep -q 'condition: service_completed_successfully' <<<"${stable_config}${dev_config}"; then + echo "runtime services must not depend on the one-shot init container" >&2 + exit 1 +fi