Files
FahdShalhoub db50e015bf
Build, Push and Deploy Discovery / build-push-deploy (push) Successful in 5m2s
FEAT: Swicthed QUERY Method To GET
2026-08-30 10:56:51 +03:00

501 lines
29 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Repo layout
```
.
├── cms/ Go service: video ingestion/CMS (implemented)
├── discovery/ Go service: catalogue discovery/read side (skeleton)
├── infrastructure/ Pulumi (Go) program provisioning all AWS resources
├── tests/ Go module: Gherkin/gobdd scenarios run against a live cms
└── .gitea/workflows/ Gitea Actions CI/CD pipelines
```
`discovery` is the read side. It owns no ingestion: it learns what exists by
draining `discovery-catalogue-events`, the queue subscribed to the catalogue
topic `cms` announces ready videos on, and keeps its own copy in its own
database. Neither service reads the other's tables and nothing polls `cms`.
It serves `GET /health`, `GET /api/videos/{id}`, `GET /api/videos` (the
catalogue search — see below) and the Swagger UI. Migration `0001_init` still
creates nothing (it predates the domain and exists only because
`internal/db/migrate.go` embeds `migrations/*.sql`, which will not compile
against an empty directory); `0002_create_videos_table` is where the schema
actually starts, and `0003``0005` add what the search needs. It now has an `internal/services` (SQS and Redis) and an
`internal/consumers`, and a task role scoped to that one queue — receive,
delete, get-attributes, and nothing else. It cannot publish back onto the
topic: it is a subscriber, not a participant. The Redis client needs nothing
from that role: ElastiCache is reached over the Redis protocol on the private
network, so the security group is what grants access, not IAM.
## Commands
### cms (Go 1.25, module `thamanyah/cms/v2`)
```bash
cd cms
go run . # serves on :8081 (requires DB_*/S3_*/MEDIACONVERT_* env vars — see below)
go run . migrate # applies pending DB migrations, then exits (no HTTP server)
go build ./...
go vet ./...
```
### discovery (Go 1.25, module `thamanyah/discovery`)
Same commands as `cms`, and the same two entry paths (`runServer`,
`runMigrate`) in `discovery/main.go` — only the port and the dependencies
differ. Note the module path carries no `/v2`: unlike `cms`, this module has
never had a v1.
```bash
cd discovery
go run . # serves on :8080 (requires DB_* plus AWS_REGION/CATALOGUE_EVENTS_QUEUE_URL/REDIS_ADDR — no S3/MediaConvert)
go run . migrate # applies pending DB migrations, then exits (no HTTP server)
go build ./...
go vet ./...
```
Its OpenAPI spec is generated into `discovery/docs` by the same `swag init`
invocation as `cms`, run from `discovery/`. In `docker-compose.yml` it is a
service of its own on `127.0.0.1:8080`, wired to the `discovery` database and
role LocalStack provisions, plus the AWS_* vars its consumer needs and
`REDIS_ADDR` pointing at the `redis` container — and, like
`cms`, started in server mode only, so a freshly created local database needs
`docker exec discovery ./discovery migrate` once. Until that runs, the consumer
logs `relation "videos" does not exist` per announcement and leaves them on the
queue; they are picked up once the table exists.
`discovery`'s data model is one table, `videos`: the catalogue's copy of an
announced video, keyed by **the id `cms` issued** (no `DEFAULT` — the id
arrives in the announcement and is reused verbatim, so both services name the
same video the same way). `categories` is a denormalized `TEXT[]` of names:
`cms` owns the vocabulary and announces names, so there is nothing to join to.
`repositories.SaveVideo` is an **upsert** — the topic delivers at-least-once,
so the same announcement can arrive twice, and a plain `INSERT` would fail on
the second, wedging every message queued behind it. A later announcement wins.
Three more columns/indexes exist purely for the search (`0003``0005`):
`search_vector` is a **generated** `tsvector` column (`to_tsvector('arabic',
title)`, `STORED`) with a GIN index; `categories` has a GIN index for the `&&`
overlap; and `videos_recent_idx` is a btree on `(created_at DESC, id DESC)`.
The text search configuration is `'arabic'`, not `'english'` or `'simple'`: it
stems Arabic while leaving Latin-script tokens as written, which suits a mixed
catalogue. The cost is that English is **not** stemmed, so "documentaries" does
not find "documentary". It is named explicitly in the column *and* in every
query — a query built under a different configuration stems its terms
differently and silently matches nothing.
### Catalogue search (`GET /api/videos`)
Served on **GET**, not POST: a search is safe and idempotent, which POST is not,
and its parameters fit a query string. It shares the path with
`GET /api/videos/{id}`; `ServeMux` keeps them apart, the more specific pattern
winning. Two things to know before touching it:
- This endpoint was on **QUERY** until the method was changed to GET. The
parameters are now a query string, not a JSON body, so there is no
`api.SearchRequest` any more — `handlers.parseSearchQuery` reads
`r.URL.Query()` into an unexported `searchQuery`. Anything still sending a
JSON body gets the whole catalogue, since every parameter is optional and it
supplied none of them.
- It **does** appear in the OpenAPI spec now, as `@Router /api/videos [get]`
with one `@Param … query` per parameter (`categories` carries
`collectionFormat(multi)`). That is the reason the change is worth anything
beyond method purity: QUERY had no `PathItem` slot in Swagger 2.0 or
OpenAPI 3.x, so `swag` rejected the annotation with `invalid method: QUERY`
and the endpoint had to be described in prose instead.
`categories` is **repeated**, not comma-separated
(`?categories=news&categories=documentary`), so a category name containing a
comma stays one name; an empty value is dropped rather than treated as a
category named `""`. `limit` is the only parameter that can be malformed — a
non-numeric one is a 400 `Malformed Search`; zero or negative reads as "no
preference" and takes the default, as omitting it does.
Response `{videos, nextCursor}`.
`title` is matched lexically, ANDing the words with the **last one as a prefix**
(`desert & fal:*`) so a part-typed word still matches; `categories` narrows to
videos filed under **any** of the names (`&&`, not `@>`); `limit` defaults to 20
and is **capped at 100** rather than refused. Paging is **keyset**, not offset:
the opaque cursor carries the last row's `(rank, created_at, id)`, so pages stay
stable while the catalogue is being written to. The query asks for `limit+1`
rows and hands back a cursor only when the extra row appears, which is how the
last page reports itself as last.
`tsQueryFor` builds the tsquery by splitting input on everything that is not a
letter or digit. That drops every character tsquery gives a meaning to, which is
what makes it safe to interpolate — `websearch_to_tsquery` parses user syntax
but cannot express a prefix match.
A measured caveat: for a **ranked** search, deep paging is not a seek. `ts_rank`
is computed per row, so the keyset comparison lands as a filter rather than an
index condition and every matching row is still scanned. Keyset paging still
buys stability and avoids OFFSET's growing discard cost, but only the unranked
browse path (no title term) is a true index seek — that is what
`videos_recent_idx` is for, and it takes a 200k-row browse from ~15 ms to
~0.03 ms.
#### Search cache (Redis / ElastiCache)
Because of that caveat, the search reads through a Redis cache before it
touches Postgres — `handlers.cachedSearch`/`cacheSearch` around the repository
call, storing the `api.SearchResults` a page serialises to. `REDIS_ADDR` points
at the ElastiCache node `infrastructure/main.go` provisions (a `redis`
container under compose). Locally it takes a repeated ranked search from
~6.8 ms to ~0.5 ms.
- **The key is `discovery:search:v1:` + a canonical JSON encoding of
`{t: title, c: categories, l: limit, p: cursor}`.** The cursor *is* the page
identity — keyset paging has no page number to key on. `limit` is in the key
because the same title/categories/cursor at a different limit is a different
set of rows. JSON rather than pasted-together separators so a category name
containing `,` or `|` cannot be mistaken for a boundary. The title is
lowercased (`tsQueryFor` folds case anyway) but the **categories are not**:
`categories && $2` compares them verbatim, so `News` and `news` really are
different searches. Categories are sorted and deduped — `&&` means "any of
these", so order never changed the answer — and never nil, so `"categories":
[]` and an omitted member share an entry.
- **TTL is 60 s and nothing invalidates on write.** The consumer writes rows
continuously and would have to know which cached pages a new title belongs
on — for a ranked search, every page it outranks. Expiry is cheaper and
bounds staleness to roughly the announcement's own delivery lag.
- **Only 200s are cached**, so a malformed cursor still reaches the repository
and is still a 400.
- **Every cache failure is a miss.** A Get/Set error is logged and the search
is answered from Postgres; the service also *starts* with an unreachable
cache, warning rather than panicking as the DB and SQS assertions do — a slow
read side beats no read side. Each call is bounded by
`services.cacheOperationTimeout` (100 ms, retries off), because an unbounded
cache miss on a dead node costs the dial timeout *plus* the query it was
avoiding. A total cache outage adds ~200 ms per search (a failed read and a
failed write), not seconds.
`cms`, `discovery` and `infrastructure` hold no test files of their own. The
only tests in the repo are the black-box BDD scenarios in `tests/` — see below.
`cms` is a JSON API only — it serves no HTML and has no static assets. The
OpenAPI spec is generated from swaggo annotations on the handlers into
`cms/docs`, a committed, compiled-in Go package. If you add or change a
handler, its annotation comments, or a request/response struct, regenerate it:
```bash
go install github.com/swaggo/swag/cmd/swag@v1.16.6 # match go.mod; not installed by default
swag init --generalInfo main.go --dir ./ --parseInternal --output ./docs
```
`--parseInternal` is required — the handlers live under `internal/`, which
swag skips without it. Keep the `swaggo/swag` version in `go.mod` and the CLI
in lockstep: `http-swagger/v2` transitively pulls a much older `swag` whose
`swag.Spec` struct lacks the `LeftDelim`/`RightDelim` fields newer generators
emit, and the build breaks outright if the two drift. (The CLI's `--version`
misreports itself as v1.16.4; `go version -m $(go env GOPATH)/bin/swag` gives
the real one.)
### tests (Go, module `thamanyah/tests`)
```bash
cd tests
go test ./... # runs features/*.feature against CMS_BASE_URL (default http://localhost:8081)
go test -v ./... # -v prints the Gherkin: gobdd nests a subtest per feature/scenario/step
CMS_BASE_URL=… go test ./...
```
Black-box BDD covering the video upload feature: its own Go module importing
nothing from `cms/`, talking to a running service over HTTP only, so the same
scenarios run against compose and against a deployed environment. Skips (does
not fail) when nothing is serving. Uploading is a plain `PUT` to the presigned
URL, no AWS SDK. Scenarios can be tagged `@known-gap` to pin current behaviour
that differs from the documented contract — see `tests/README.md`.
Note compose starts `cms` in server mode only: the migrate container exists in
the ECS task definition, not in `docker-compose.yml`, so a freshly created
local database needs `docker exec cms ./cms migrate` once or every scenario
fails on `relation "categories" does not exist`.
### infrastructure (Go, Pulumi, module `thamanyah`)
```bash
cd infrastructure
pulumi preview # plan changes against stack "main"
pulumi up # apply — this touches real AWS resources, confirm with the user first
pulumi stack output # e.g. ecsClusterArn, cmsServiceArn
```
Deploys run in CI (`.gitea/workflows/infrastructure-deploy.yml`) on push to
`main` touching `infrastructure/**`. Treat local `pulumi up` as something to
confirm with the user, not a routine dev command — it mutates shared cloud
state and Pulumi state isn't safe to update concurrently with CI.
## Architecture
### cms package layout
Four `internal` packages, split by the kind of thing they hold — keep new code
on the same seams:
| package | holds |
|---|---|
| `internal/api` | the **wire contract**: request/response structs with their JSON + swaggo tags. No logic. |
| `internal/models` | the **domain types** the rest of the code passes around: `Video`, `Category`, `VideoStatus`. No JSON tags, no SQL. |
| `internal/handlers` | HTTP: decode `api.*`, validate, call repositories/services, encode `api.*`. |
| `internal/db` | the `*sql.DB` connection lifecycle and migrations; `internal/db/repositories` holds all SQL. |
| `internal/services` | **AWS only** — S3, MediaConvert and SQS. Nothing database-related lives here. |
`api` and `models` are deliberately separate types even where the fields look
alike: `api.CompleteResponse` is the published schema, `models.Video` is the
row. Handlers translate between them field by field, so renaming a `models`
field doesn't move the published API and vice versa.
### cms service
Plain `net/http` (Go 1.22+ pattern-based `ServeMux`), no framework. Entry
point `cms/main.go` builds the dependencies and assigns them to package-level
interface vars that the handlers call through:
- `services.S3Client` (`services.S3`) ← `*services.S3Concrete`
- `services.MediaConvertClient` (`services.MediaConvert`) ← `*services.MediaConvertConcrete`
- `repositories.VideoRepo` (`repositories.VideoRepository`) ← `*repositories.ConcreteVideoRepository`
- `repositories.CatagoriesRepo` (`repositories.CatagoriesRepository`) ← `*repositories.ConcreteCatagoriesRepository`
Each interface has exactly one implementation; the indirection is what makes
the handlers substitutable in tests, even though no tests exist yet. Note the
spelling: the categories repository is `Catagories`/`CatagoriesRepo` in
`repositories/catagories.go` (and `handlers.validateCatagoryIDS`) — the
misspelling is load-bearing for compilation, so match it rather than "fixing"
it piecemeal.
On boot, `S3Concrete.AssertSuccessfulConnection` proactively exercises
head-bucket/put/get/presign against the bucket (writing a throwaway
`.s3-connectivity-check` object), and `db.AssertSuccessfulConnection` pings
Postgres — both panic on failure rather than letting the service come up in a
broken state.
`cms/main.go` has two entry paths, dispatched on `os.Args[1]`: the default
path (`runServer`) boots the HTTP server; `./cms migrate` (`runMigrate`)
only opens the DB connection, applies pending migrations via
`cms/internal/db.Migrate` (golang-migrate, `iofs` source, SQL files embedded
from `cms/internal/db/migrations/*.sql`), and exits — it does not touch
S3/MediaConvert or start the server. This is run as its own ECS container
before the main container starts (see infrastructure below), so `runServer`
never runs migrations itself, only `AssertSuccessfulConnection`.
The DB connection is opened once for the process lifetime by
`db.CreateDBConnection(connString)` and closed by `db.CloseConnection` — both
plain functions over `*sql.DB` in `internal/db/client.go`, not methods on a
wrapper type. Repositories take that `*sql.DB` as their `SQLDB` field.
Routes (`cms/main.go`): `GET /health`, `GET /api/categories`,
`POST /api/videos/presign`, `POST /api/videos`, plus Swagger UI at
`GET /swagger/` (`/swagger/doc.json` serves the spec). The UI assets are
embedded in the binary by `swaggo/files`, so nothing is read from disk and
nothing is fetched from a CDN at runtime.
Handlers live in `cms/internal/handlers``handlers.go` holds `Health` and
the shared response writers, `videos.go` the categories and upload endpoints.
The wire structs they serve live in `cms/internal/api` (`api.go` for
`HealthResponse` and `ProblemDetails`, `videos.go` for `Category`,
`CategoriesResponse`, `PresignRequest`/`PresignResponse` and
`CompleteRequest`/`CompleteResponse`), referenced by the handlers' swaggo
annotations as `api.CompleteResponse` and so on — so the generated spec's
definition names track that package, and renaming a type there changes the
published schema names.
There is no view layer: the `internal/views` templ package, the `static/`
directory, and the htmx frontend were all removed when the service became a
JSON API, along with the `templ` dependency.
Error responses are RFC 9457 Problem Details objects
(`application/problem+json`), written by
`writeProblem(w, status, title, detail)`. `type` is always `"about:blank"`;
`title` is a short summary held identical across every occurrence of a given
problem, so clients can branch on it; `detail` is the only member that varies
with request data. Success responses go through `writeJSON`
(`application/json`). Both share `writeJSONContent`. Note this deviates
slightly from RFC 9457, which pairs an `about:blank` type with a title that is
just the HTTP status phrase — meaningful titles like these are supposed to
carry a real `type` URI. Adding per-problem type URIs is the conforming fix if
it ever matters.
### Data model (Postgres, `cms/internal/db/migrations/`, two migrations: `0001`, `0002`)
- `videos` — one row per uploaded video: `title`, `description`, `tags`
(free-text, comma-separated — not normalized), `file_name`, `storage_key`
(the S3 key, unique), `mediaconvert_job_id`, `status` (written once as
`"processing"` on insert — see Known gaps), `size_bytes`, timestamps.
`id` is a `UUID` filled by the column's own `DEFAULT gen_random_uuid()`
(v4) and read back through `RETURNING id` — the application does not
generate it. (An earlier revision generated UUIDv7 application-side; that
was reverted in `785154c`, so primary keys are random, not time-ordered.)
`status` is a plain `TEXT NOT NULL DEFAULT 'processing'` column with **no
CHECK constraint** — the closed set exists only in Go, as the
`models.VideoStatus` string type in `cms/internal/models/video.go`
(`processing`, `ready`, `failed`). Extending it means adding a constant
there and extending the `enums` annotation on `api.CompleteResponse.Status`
before regenerating the spec. (The doc comment on `VideoStatus` claims a
`videos_status_check` constraint enforces it in the database — that
constraint does not exist; nothing has ever created it.)
- `categories` — a small fixed lookup table (`documentary`, `news`,
`entertainment`, `podcast`, `other`), seeded by migration `0001`. Its
`SMALLSERIAL` ids are part of the public API: `GET /api/categories` returns
`{id, name}` pairs (ordered by name, not id) and `POST /api/videos` takes
`categoryIds`, so the seed order in migration `0001` is what fixes which id
means which name — never renumber it.
- `video_categories` — join table (`video_id`, `category_id`, composite PK,
`ON DELETE CASCADE`) added in migration `0002`: a video can belong to
*multiple* categories, not just one. `ConcreteVideoRepository.CreateVideo`
inserts the `videos` row and its `video_categories` links inside a single
transaction, taking the category ids as given — `handlers.validateCatagoryIDS`
is what checks them against `CatagoriesRepo.ListCategoriesIDs` before the
transcode job is queued, leaving the foreign key and the composite PK as
backstops (see Known gaps for how those failures surface).
### Video upload → transcode pipeline
1. Client calls `POST /api/videos/presign` → cms returns a presigned S3 `PUT`
URL for `raw-uploads-bucket`, key `videos/<random-hex>.<ext>`. The
submitted `contentType` (only `video/mp4` or `video/quicktime`) is signed
into the URL, so the client's `PUT` must send the identical header.
2. Client `PUT`s the file directly to S3 (from a browser this requires the
bucket's CORS rule, set up in `infrastructure/main.go`,
its allowed origin is now stale). The file never passes through cms.
3. Client calls `POST /api/videos` with the metadata (categories given as
`categoryIds` from `GET /api/categories`) + key → cms validates the
category ids, calls `MediaConvertClient.QueueEncodingJob(key)`, submitting
a MediaConvert job `s3://raw-uploads-bucket/<key>`
`s3://encoded-bucket/<key>` (H.264/AAC → MP4, QVBR rate control — QVBR
requires `MaxBitrate` to be set explicitly), then
`repositories.VideoRepo.CreateVideo` persists the `videos` row (status
`"processing"`) and its `video_categories` links.
4. Finished output lands in `encoded-bucket`, served via CloudFront.
5. MediaConvert reports the job's state changes to an SNS topic that fans out
to `cms-mediaconvert-events`, which `internal/consumers` reads for the
lifetime of the process: it records the outcome (`ready`/`failed`) and the
playback URL, rewriting the `s3://` playlist path onto `PLAYBACK_BASE_URL`.
A job that came out **ready** is then announced on the `catalogue-events`
SNS topic via `services.CatalogueClient``{videoId, title, playbackUrl,
categories}`, categories by **name** so a subscriber needs nothing from
cms to interpret them. Failed jobs are recorded but never announced. A
topic, not a queue, so the announcement fans out: `discovery-catalogue-events`
subscribes today (raw delivery, so there is no SNS envelope to unwrap), and
a second reader can subscribe its own queue without cms changing. Note a
topic only delivers to subscriptions that exist when it publishes, so a new
subscriber's queue has to be in place before the announcement, not after.
Delivery is at-least-once: a failed publish leaves the *job* event on the
consumer's queue, and the outcome update is idempotent, so a redelivery
re-announces and a subscriber can see a video twice.
Config wiring: cms reads `S3_BUCKET`, `MEDIACONVERT_INPUT_BUCKET`,
`MEDIACONVERT_OUTPUT_BUCKET`, `MEDIACONVERT_ROLE_ARN`, `AWS_REGION`,
`MEDIACONVERT_EVENTS_QUEUE_URL`, `CATALOGUE_EVENTS_TOPIC_ARN`,
`PLAYBACK_BASE_URL`,
`DB_HOST`, `DB_PORT`, `DB_NAME`, `DB_USER`, `DB_PASSWORD` from env vars
injected by the ECS task definition (`extraEnv` and the unconditional DB_*
vars in `deployFargateService`, `infrastructure/main.go`) — `main.go` panics
on boot if any required var is empty.
### infrastructure (`infrastructure/main.go`, single Pulumi Go program, region `us-east-1`, stack `main`)
- **Postgres**: one shared RDS instance (`db.t3.micro`, single-AZ, no
backups — intentionally minimal). Each app (`cms`, `discovery`) gets its
own login role and same-named database via the `postgresql` provider
(`newServiceDatabase`), so services never share DB credentials.
- **ElastiCache**: one `cache.t4g.micro` Redis node (`search-cache`, engine
7.1, single-AZ, no replica, no snapshots) that `discovery` answers repeated
catalogue searches from. Its contents are derivable from Postgres by
definition, so there is nothing to back up. Its security group admits 6379
from `ecs-service-sg` **only** — narrower than the database's, which also
admits the deployer's IP for the `postgresql` provider; there is nothing to
administer here from a laptop. The endpoint is the single node's address
(`CacheNodes[0]`, not `ConfigurationEndpoint` — that is a Memcached thing),
exported as `searchCacheAddress` and injected as `REDIS_ADDR`. Skipped under
LocalStack, where docker-compose runs a plain `redis:7-alpine` container
instead: ElastiCache picks its own endpoint, and compose needs a literal
`REDIS_ADDR` before anything is provisioned.
- **ECS Fargate**: one cluster (`app-cluster`), one ALB *per service* (each
gets its own DNS name rather than sharing a load balancer on different
ports). `deployFargateService(...)` is the shared helper building a
service's ECR repo, CloudWatch log group, task definition, ECS service,
and ALB. `taskRole` is optional (nil = no AWS identity beyond the shared
execution role); `extraEnv` appends container env vars beyond the DB_* set;
`runMigrations` (true for both services) adds a second,
non-essential `<name>-migrate` container to the task — same image,
`command: ["migrate"]` — with the main container's `dependsOn` set to
`condition: "COMPLETE"` on it. This is ECS's container-dependency
mechanism, the Fargate equivalent of a Kubernetes init container: ECS runs
the migrate container to completion (exit 0) before starting the main
container, so schema migrations always finish before the service accepts
traffic. No separate CI/Docker migration step exists — `Dockerfile`'s
`ENTRYPOINT ["./cms"]` plus the container's `command` override composes to
`./cms migrate`.
- **S3 + CloudFront**: `encoded-bucket` holds finished transcoded output,
served publicly via CloudFront using Origin Access Control (OAC) — the
bucket itself blocks all public access; only CloudFront's OAC principal
can read it. The distribution carries a **response headers policy** adding
permissive CORS headers (`*`, `GET`/`HEAD`/`OPTIONS`), and `OPTIONS` is in
the behaviour's allowed/cached methods so CloudFront answers preflights
itself: HLS is fetched by JavaScript, so a playlist or segment served
without `Access-Control-Allow-Origin` is discarded by the browser. The
bucket also carries its own equivalent CORS rule, which is what the
LocalStack stack relies on — there is no CloudFront there and
`PLAYBACK_BASE_URL` points the player straight at S3.
- **S3 raw uploads**: `raw-uploads-bucket` is a separate, private bucket for
pre-transcode uploads — deliberately kept apart from `encoded-bucket` so
raw source video is never reachable through the public CDN. CORS is
scoped to `PUT` only, from the `cms` ALB's own origin — correct back when
cms served the upload page itself, but stale now that it serves no UI (see
Known gaps).
- **IAM roles** — five distinct roles/users, each scoped narrowly, don't
conflate them:
- `ecs-task-execution-role` — shared by both services' ECS *agent* (image
pull, log write, Secrets Manager read for DB password). Not usable by
application code inside the container.
- `cms-task-role` — the `cms` container's own AWS identity: S3
`ListBucket`/`PutObject`/`GetObject` on `raw-uploads-bucket` only,
`mediaconvert:CreateJob`, `iam:PassRole` scoped to the MediaConvert
service role (`iam:PassedToService` condition), receive/delete on
`cms-mediaconvert-events`, and `sns:Publish` on the `catalogue-events`
topic — the only thing it writes to. It has no access to
`discovery-catalogue-events`, the queue subscribed to that topic: cms
publishes, it does not reach into a subscriber.
- `discovery-task-role` — the `discovery` container's own AWS identity, and
its only one: receive/delete/get-attributes on
`discovery-catalogue-events`. No S3, no MediaConvert, and no `sns:Publish`
— it consumes the catalogue, it does not add to it. The search cache is
absent from it on purpose: ElastiCache is reached over the Redis protocol
inside the VPC, so `search-cache-sg` is the grant, not IAM.
- `mediaconvert-service-role` — trusted by `mediaconvert.amazonaws.com`,
not by ECS; the role MediaConvert itself assumes (passed as
`CreateJobInput.Role`) to read `raw-uploads-bucket` and write
`encoded-bucket`. Distinct from `cms-task-role` by design: one is "cms
calling AWS", the other is "AWS calling AWS on cms's behalf".
- `gitea-ci-user` — an IAM **user** (static access keys, not OIDC — the
Gitea Actions runner doesn't support instance-profile auth) scoped to
just ECR push (`cms`/`discovery` repos) and
`ecs:UpdateService`/`ecs:DescribeServices` on the two ECS services.
Never broaden this to `ecr:*`/`ecs:*`.
### CI/CD (`.gitea/workflows/`)
- **`cms-deploy.yml`**: push to `main` touching `cms/**`. Builds/pushes the
Docker image to ECR using `gitea-ci-user`, installs the AWS CLI (not
preinstalled on the runner image — via AWS's official install script, not
a third-party action), then `aws ecs update-service --force-new-deployment`.
Cluster/service ARNs come from `pulumi stack output` and are set as repo
*variables* (not secrets — ARNs aren't sensitive).
- **`discovery-deploy.yml`**: the same pipeline for `discovery`, on pushes
touching `discovery/**`. One difference: the ECS service ARN is read from
the `DISCOVERY_ECS_SERVICE_ARN` repo variable rather than pinned in the
workflow, so it has to be set (from `pulumi stack output
discoveryServiceArn`) before the first deploy can succeed.
- **`infrastructure-deploy.yml`**: push to `main` touching
`infrastructure/**`. Runs `pulumi up` using a separate, broader AWS
credential (`PULUMI_AWS_ACCESS_KEY_ID`/`SECRET`) than `gitea-ci-user`,
since provisioning IAM/RDS/ECS/CloudFront needs wider permissions than
pushing images and forcing deployments. Guarded with a concurrency group
since Pulumi state isn't safe to update concurrently.
- The runner's `ubuntu-latest` label maps to a docker image configured on
the runner host (outside this repo) — currently minimal, lacking the AWS
CLI, hence the manual install step in `cms-deploy.yml`.