Esa Kataja
522827879b
A submission failed with HTTP 403 against a freshly published image. The cause was not Alpine, which carries the current yt-dlp: it was the apk layer being reused from an older build, so the image shipped a yt-dlp two months behind while /healthz reported today's tag. "A rebuild is the update" only holds if the layer is genuinely re-run, so the image target now passes --pull --no-cache. Slower, and the only thing that makes the release honest about what is inside it.
276 lines
12 KiB
Markdown
276 lines
12 KiB
Markdown
# Deployment
|
||
|
||
How Levyraati gets onto a server and how it is changed once it is there. Configuration variables are
|
||
tabulated in the [README](../README.md#configuration); this file is the procedures.
|
||
|
||
The whole deployment is **one container and one directory**. There is no database server, no
|
||
migration step to run by hand, and no build on the target machine.
|
||
|
||
---
|
||
|
||
## The server's compose file
|
||
|
||
The `docker-compose.yml` in the repository root **builds from source** — that is the development
|
||
one, and it is what you want on a machine that has the code checked out. A server has no source, so
|
||
it runs a published image instead. Keep this second file on the server; it is not in the repository
|
||
because it describes one particular deployment rather than the app.
|
||
|
||
```yaml
|
||
services:
|
||
app:
|
||
# Registry included. Pin a release tag, never :latest — a restart must not quietly change the
|
||
# running version. Kept in .env so this file carries no host of yours.
|
||
image: ${IMAGE:?set IMAGE in .env}
|
||
environment:
|
||
# Only read while the users table is empty: they create the first account and are ignored
|
||
# from then on. Safe to remove once that account exists.
|
||
ADMIN_EMAIL: ${ADMIN_EMAIL:-}
|
||
ADMIN_PASSWORD: ${ADMIN_PASSWORD:-}
|
||
ADMIN_NAME: ${ADMIN_NAME:-Ylläpito}
|
||
ADDR: ":8080"
|
||
SECURE_COOKIES: ${SECURE_COOKIES:-true}
|
||
PUBLIC_URL: ${PUBLIC_URL:-}
|
||
# The SQLite file sits in here beside the audio, so this one mount is the whole backup.
|
||
volumes:
|
||
- ./storage:/storage
|
||
ports:
|
||
- "8080:8080"
|
||
restart: unless-stopped
|
||
```
|
||
|
||
Two differences from the development file, and the reason for each:
|
||
|
||
| | Development | Server |
|
||
|---|---|---|
|
||
| Source of the binary | `build:` from the checkout | `image:` pulled from the registry |
|
||
| Version | `VERSION` build arg, `dev` by default | baked into the tagged image |
|
||
|
||
There is one port. `/admin` rides the member listener behind the same session cookie as everything
|
||
else, so there is nothing extra to publish, tunnel or firewall.
|
||
|
||
Alongside it, a `.env` — same variables as [.env.example](../.env.example), plus the image:
|
||
|
||
```sh
|
||
IMAGE=registry.example.com/owner/levyraati26-go:2026.08.02-1
|
||
ADMIN_EMAIL=… # first start only
|
||
ADMIN_PASSWORD=… # first start only
|
||
SECURE_COOKIES=true
|
||
PUBLIC_URL=https://levyraati.example.com
|
||
```
|
||
|
||
---
|
||
|
||
## What the server needs
|
||
|
||
- Docker with the Compose plugin, or Podman with `podman-compose`.
|
||
- Credentials for the registry holding the image (`docker login <registry>`), unless it is public.
|
||
- A reverse proxy terminating TLS in front of port 8080. Cookies are `Secure`, so the members' site
|
||
over plain HTTP will not keep anyone logged in.
|
||
- Outbound network access: yt-dlp reaches YouTube, and the lyrics lookup reaches LRCLIB. Neither is
|
||
fatal to lose — submissions fail with a visible message and lyrics stay empty.
|
||
|
||
Nothing else. No Go toolchain, no ffmpeg on the host — those live in the image.
|
||
|
||
---
|
||
|
||
## Building and publishing a release
|
||
|
||
Done from a checkout, not on the server. The version reaches the binary only through the build arg,
|
||
so it must match the tag or `/healthz` will lie about what is deployed:
|
||
|
||
```sh
|
||
git switch main && git merge dev
|
||
git tag 2026.08.02-1
|
||
podman build --build-arg VERSION=2026.08.02-1 \
|
||
-t registry.example.com/owner/levyraati26-go:2026.08.02-1 \
|
||
-t registry.example.com/owner/levyraati26-go:latest .
|
||
podman push registry.example.com/owner/levyraati26-go:2026.08.02-1
|
||
podman push registry.example.com/owner/levyraati26-go:latest
|
||
```
|
||
|
||
Check before pushing that the tag took: `podman run --rm -p 8099:8080 -e ADMIN_PASSWORD=x IMAGE`
|
||
then `curl localhost:8099/healthz` should answer `ok 2026.08.02-1`, not `ok dev`.
|
||
|
||
---
|
||
|
||
## First deployment
|
||
|
||
Two files go on the server — the compose file above and `.env`. **Not** a git clone; the source is
|
||
not needed to run this.
|
||
|
||
```sh
|
||
mkdir -p /srv/levyraati && cd /srv/levyraati
|
||
# put docker-compose.yml and .env here
|
||
chmod 600 .env # it holds the only admin credential there is
|
||
docker compose pull
|
||
docker compose up -d
|
||
docker compose logs -f app # watch the migrations apply
|
||
```
|
||
|
||
The first start creates `./storage` with `audio/`, `avatars/`, `tmp/` and `levyraati.db`, applies
|
||
every migration, and only then accepts connections. It creates **no users** — nobody can register
|
||
until you mint an invite.
|
||
|
||
Confirm it is alive, and that the version is the one you meant to deploy:
|
||
|
||
```sh
|
||
curl -s localhost:8080/healthz # -> ok 2026.08.02-1
|
||
```
|
||
|
||
### Reverse proxy
|
||
|
||
Proxy your public hostname to `127.0.0.1:8080`. Two things matter beyond the defaults:
|
||
|
||
- **Upload size.** Submissions are capped at 50 MB by the app; a proxy with a 1 MB default body
|
||
limit rejects them first, and the error is not the app's clear one. Raise it past 50 MB
|
||
(`client_max_body_size 64m` in nginx, `MaxRequestBodySize` in Caddy).
|
||
- **Response buffering off**, or at least generous timeouts, for `/audio/{id}` — it serves Range
|
||
requests so the player can seek.
|
||
|
||
### Admin access
|
||
|
||
Log in as your own account and open `/admin`. Nothing to tunnel, nothing extra to proxy: the page is
|
||
part of the site and is gated on the `is_admin` flag on your user row. A signed-in member without the
|
||
flag gets a 404 there, so the page does not advertise itself.
|
||
|
||
TLS at the proxy is what makes the invite *Kopioi* button work — the clipboard API needs a secure
|
||
context, and `https://` is one. Over plain HTTP on a real hostname the button will not fire.
|
||
|
||
From the page: mint invites, reset passwords, ban members, delete songs, read feedback.
|
||
|
||
**First start.** On an empty database the app creates one account from `ADMIN_EMAIL` /
|
||
`ADMIN_PASSWORD` and marks it admin. Once it exists those variables do nothing; drop them from
|
||
`.env` if you would rather not keep a password there.
|
||
|
||
**Lost the admin password?** There is no recovery endpoint and no recovery key. Set a new bcrypt
|
||
hash directly in the SQLite file — re-running the app with `ADMIN_PASSWORD` will not help, because
|
||
seeding only fires on an empty `users` table.
|
||
|
||
---
|
||
|
||
## Upgrading
|
||
|
||
```sh
|
||
cd /srv/levyraati
|
||
# back up first — see below; it takes a second and this is exactly when you want it
|
||
$EDITOR .env # point IMAGE at the new tag
|
||
docker compose pull
|
||
docker compose up -d
|
||
curl -s localhost:8080/healthz # confirm the new version is answering
|
||
```
|
||
|
||
Migrations run at startup, inside the new container, before it serves. There is no separate step.
|
||
|
||
**Expect a few seconds of downtime.** One container, one SQLite file, no rolling deploy — and a
|
||
restart deliberately fails every in-flight submission. Deploy when nobody is mid-review.
|
||
|
||
### Rolling back
|
||
|
||
Point `IMAGE` at the previous tag and `docker compose up -d`. **Only safe if the release you are
|
||
leaving added no migration** — migrations are forward-only and the old binary will not understand a
|
||
schema it has never seen. Check `migrations/` between the two tags first; if one landed, restore the
|
||
backup taken before the upgrade instead.
|
||
|
||
### What a restart does to work in progress
|
||
|
||
Conversions run as goroutines inside the process, so a restart kills them. This is handled, not
|
||
ignored: the startup sweep marks every submission still `queued`, `downloading` or `converting` as
|
||
`failed` with "interrupted by restart", so nothing is stuck saying "converting" forever. The
|
||
submitter sees the failure and can retry a URL submission or re-upload a file. Published songs and
|
||
reviews are untouched.
|
||
|
||
---
|
||
|
||
## Backups
|
||
|
||
`./storage` holds everything — audio, avatars, and `levyraati.db`.
|
||
|
||
**Do not just copy the database file while the app is running.** WAL mode means recent writes live
|
||
in `levyraati.db-wal`, and a bare copy can miss them or catch a torn state. Ask SQLite for a
|
||
consistent snapshot instead — it is safe against a live, writing database:
|
||
|
||
```sh
|
||
cd /srv/levyraati
|
||
docker compose exec app sqlite3 /storage/levyraati.db ".backup '/storage/tmp/backup.db'"
|
||
gzip -c storage/tmp/backup.db > /backups/levyraati-$(date +%F).db.gz
|
||
rm storage/tmp/backup.db
|
||
tar czf /backups/levyraati-audio-$(date +%F).tar.gz -C storage audio avatars
|
||
```
|
||
|
||
`storage/tmp/` is in-flight conversions and is safe to skip; it is cleared at startup anyway.
|
||
|
||
A daily cron of those four lines is a complete backup strategy for this app.
|
||
|
||
### Restoring
|
||
|
||
```sh
|
||
docker compose down
|
||
gunzip -c /backups/levyraati-2026-08-02.db.gz > storage/levyraati.db
|
||
rm -f storage/levyraati.db-wal storage/levyraati.db-shm # stale sidecars of the old file
|
||
tar xzf /backups/levyraati-audio-2026-08-02.tar.gz -C storage
|
||
docker compose up -d
|
||
```
|
||
|
||
Deleting the `-wal` and `-shm` files matters: left behind, they belong to the database you just
|
||
replaced, and SQLite will try to apply them to the restored one.
|
||
|
||
Audio and rows are backed up separately but must be restored together — a song row whose `.ogg` is
|
||
missing gives a broken player, and an orphan `.ogg` is invisible to everyone.
|
||
|
||
---
|
||
|
||
## Operations
|
||
|
||
### Logs
|
||
|
||
```sh
|
||
docker compose logs -f app
|
||
```
|
||
|
||
JSON to stdout, nothing else. Every line carries a `ctx` field (`startup`, `auth`, `songs`,
|
||
`submissions`, `invites`, `reports`) to filter on.
|
||
|
||
Two startup warnings are worth reading rather than skipping: `submissions interrupted by restart`
|
||
says the sweep cleaned up after a restart, and `failed submissions present` is often the first sign
|
||
that yt-dlp has gone stale.
|
||
|
||
### yt-dlp goes stale
|
||
|
||
yt-dlp rots against YouTube — routine maintenance, not an incident. It comes from Alpine's community
|
||
repository in the image, so **the fix is a rebuild**, which means publishing a new image rather than
|
||
anything on the server. Rebuild monthly. Failures show the yt-dlp error to the submitter, so members
|
||
usually notice before you read a log.
|
||
|
||
### Database shell
|
||
|
||
```sh
|
||
docker compose exec app sqlite3 /storage/levyraati.db
|
||
```
|
||
|
||
Writes here are unaudited and unvalidated — the schema holds the constraints, but the app's rules
|
||
(the review window, the reveal rule, the lock) are in Go. Prefer the admin panel.
|
||
|
||
### Disk
|
||
|
||
Audio is Opus at 96 kbps: roughly 2–3 MB per song, so a hundred songs is a few hundred megabytes.
|
||
`storage/tmp/` briefly holds a 50 MB upload plus its converted copy per in-flight submission,
|
||
bounded by the two conversion slots.
|
||
|
||
---
|
||
|
||
## Troubleshooting
|
||
|
||
| Symptom | Cause |
|
||
|---|---|
|
||
| Container exits immediately on a first start | `ADMIN_EMAIL` or `ADMIN_PASSWORD` unset on an empty database. The log says so; a site nobody can log into is worse than one that will not boot |
|
||
| `set IMAGE in .env` | Compose has no image to run; `IMAGE` is required and unset |
|
||
| `/healthz` says `ok dev` | The image was built without `--build-arg VERSION`, so what is deployed cannot be identified |
|
||
| Login never sticks | Plain HTTP with `SECURE_COOKIES=true`. Terminate TLS, or set it `false` for a local test |
|
||
| Uploads fail near 50 MB | The reverse proxy's body limit, not the app's |
|
||
| Invite links are relative | `PUBLIC_URL` unset |
|
||
| Everything 500s after a restore | `-wal`/`-shm` sidecars from the replaced database were left in place |
|
||
| Submissions all fail at download | yt-dlp is stale, or YouTube is refusing this server's IP. `docker compose logs app \| grep '"stage":"download"'` shows yt-dlp's own stderr under `detail`. A plain `HTTP Error 403` is the stale case — rebuild with `make image`, which forces `--no-cache` so the `apk add` layer is genuinely re-run. A `docker build` without it can ship a months-old yt-dlp from a cached layer |
|
||
| A submitter reports a *virhekoodi* | `docker compose logs app \| grep <code>` — one line, with the stage, the submission id, the URL and the tool's stderr |
|
||
| `/admin` returns 404 while logged in | That account has no `is_admin`. Set it in the database; nothing in the UI grants it |
|
||
| Setting `ADMIN_PASSWORD` again changes nothing | Seeding only fires on an empty `users` table. Reset the hash in the database instead |
|