thamanyah/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Repo layout

.
├── cms/            Go service: video ingestion/CMS (implemented)
├── discovery/      Go service: catalogue discovery/read side (skeleton)
├── infrastructure/ Pulumi (Go) program provisioning all AWS resources
├── tests/          Go module: Gherkin/gobdd scenarios run against a live cms
└── .gitea/workflows/  Gitea Actions CI/CD pipelines

discovery is the read side. It owns no ingestion: it learns what exists by draining discovery-catalogue-events, the queue subscribed to the catalogue topic cms announces ready videos on, and keeps its own copy in its own database. Neither service reads the other's tables and nothing polls cms.

It serves GET /health, GET /api/videos/{id}, GET /api/videos (the catalogue search — see below) and the Swagger UI. Migration 0001_init still creates nothing (it predates the domain and exists only because internal/db/migrate.go embeds migrations/*.sql, which will not compile against an empty directory); 0002_create_videos_table is where the schema actually starts, and 0003–0005 add what the search needs. It now has an internal/services (SQS and Redis) and an internal/consumers, and a task role scoped to that one queue — receive, delete, get-attributes, and nothing else. It cannot publish back onto the topic: it is a subscriber, not a participant. The Redis client needs nothing from that role: ElastiCache is reached over the Redis protocol on the private network, so the security group is what grants access, not IAM.

Commands

cms (Go 1.25, module thamanyah/cms/v2)

cd cms
go run .                    # serves on :8081 (requires DB_*/S3_*/MEDIACONVERT_* env vars — see below)
go run . migrate            # applies pending DB migrations, then exits (no HTTP server)
go build ./...
go vet ./...

discovery (Go 1.25, module thamanyah/discovery)

Same commands as cms, and the same two entry paths (runServer, runMigrate) in discovery/main.go — only the port and the dependencies differ. Note the module path carries no /v2: unlike cms, this module has never had a v1.

cd discovery
go run .                    # serves on :8080 (requires DB_* plus AWS_REGION/CATALOGUE_EVENTS_QUEUE_URL/REDIS_ADDR — no S3/MediaConvert)
go run . migrate            # applies pending DB migrations, then exits (no HTTP server)
go build ./...
go vet ./...

Its OpenAPI spec is generated into discovery/docs by the same swag init invocation as cms, run from discovery/. In docker-compose.yml it is a service of its own on 127.0.0.1:8080, wired to the discovery database and role LocalStack provisions, plus the AWS_* vars its consumer needs and REDIS_ADDR pointing at the redis container — and, like cms, started in server mode only, so a freshly created local database needs docker exec discovery ./discovery migrate once. Until that runs, the consumer logs relation "videos" does not exist per announcement and leaves them on the queue; they are picked up once the table exists.

discovery's data model is one table, videos: the catalogue's copy of an announced video, keyed by the id cms issued (no DEFAULT — the id arrives in the announcement and is reused verbatim, so both services name the same video the same way). categories is a denormalized TEXT[] of names: cms owns the vocabulary and announces names, so there is nothing to join to. repositories.SaveVideo is an upsert — the topic delivers at-least-once, so the same announcement can arrive twice, and a plain INSERT would fail on the second, wedging every message queued behind it. A later announcement wins.

Three more columns/indexes exist purely for the search (0003–0005): search_vector is a generated tsvector column (to_tsvector('arabic', title), STORED) with a GIN index; categories has a GIN index for the && overlap; and videos_recent_idx is a btree on (created_at DESC, id DESC). The text search configuration is 'arabic', not 'english' or 'simple': it stems Arabic while leaving Latin-script tokens as written, which suits a mixed catalogue. The cost is that English is not stemmed, so "documentaries" does not find "documentary". It is named explicitly in the column and in every query — a query built under a different configuration stems its terms differently and silently matches nothing.

Catalogue search (GET /api/videos)

Served on GET, not POST: a search is safe and idempotent, which POST is not, and its parameters fit a query string. It shares the path with GET /api/videos/{id}; ServeMux keeps them apart, the more specific pattern winning. Two things to know before touching it:

• This endpoint was on QUERY until the method was changed to GET. The parameters are now a query string, not a JSON body, so there is no api.SearchRequest any more — handlers.parseSearchQuery reads r.URL.Query() into an unexported searchQuery. Anything still sending a JSON body gets the whole catalogue, since every parameter is optional and it supplied none of them.
• It does appear in the OpenAPI spec now, as @Router /api/videos [get] with one @Param … query per parameter (categories carries collectionFormat(multi)). That is the reason the change is worth anything beyond method purity: QUERY had no PathItem slot in Swagger 2.0 or OpenAPI 3.x, so swag rejected the annotation with invalid method: QUERY and the endpoint had to be described in prose instead.

categories is repeated, not comma-separated (?categories=news&categories=documentary), so a category name containing a comma stays one name; an empty value is dropped rather than treated as a category named "". limit is the only parameter that can be malformed — a non-numeric one is a 400 Malformed Search; zero or negative reads as "no preference" and takes the default, as omitting it does.

Response {videos, nextCursor}. title is matched lexically, ANDing the words with the last one as a prefix (desert & fal:*) so a part-typed word still matches; categories narrows to videos filed under any of the names (&&, not @>); limit defaults to 20 and is capped at 100 rather than refused. Paging is keyset, not offset: the opaque cursor carries the last row's (rank, created_at, id), so pages stay stable while the catalogue is being written to. The query asks for limit+1 rows and hands back a cursor only when the extra row appears, which is how the last page reports itself as last.

tsQueryFor builds the tsquery by splitting input on everything that is not a letter or digit. That drops every character tsquery gives a meaning to, which is what makes it safe to interpolate — websearch_to_tsquery parses user syntax but cannot express a prefix match.

A measured caveat: for a ranked search, deep paging is not a seek. ts_rank is computed per row, so the keyset comparison lands as a filter rather than an index condition and every matching row is still scanned. Keyset paging still buys stability and avoids OFFSET's growing discard cost, but only the unranked browse path (no title term) is a true index seek — that is what videos_recent_idx is for, and it takes a 200k-row browse from ~15 ms to ~0.03 ms.

Search cache (Redis / ElastiCache)

Because of that caveat, the search reads through a Redis cache before it touches Postgres — handlers.cachedSearch/cacheSearch around the repository call, storing the api.SearchResults a page serialises to. REDIS_ADDR points at the ElastiCache node infrastructure/main.go provisions (a redis container under compose). Locally it takes a repeated ranked search from ~6.8 ms to ~0.5 ms.

• The key is discovery:search:v1: + a canonical JSON encoding of {t: title, c: categories, l: limit, p: cursor}. The cursor is the page identity — keyset paging has no page number to key on. limit is in the key because the same title/categories/cursor at a different limit is a different set of rows. JSON rather than pasted-together separators so a category name containing , or | cannot be mistaken for a boundary. The title is lowercased (tsQueryFor folds case anyway) but the categories are not: categories && $2 compares them verbatim, so News and news really are different searches. Categories are sorted and deduped — && means "any of these", so order never changed the answer — and never nil, so "categories": [] and an omitted member share an entry.
• TTL is 60 s and nothing invalidates on write. The consumer writes rows continuously and would have to know which cached pages a new title belongs on — for a ranked search, every page it outranks. Expiry is cheaper and bounds staleness to roughly the announcement's own delivery lag.
• Only 200s are cached, so a malformed cursor still reaches the repository and is still a 400.
• Every cache failure is a miss. A Get/Set error is logged and the search is answered from Postgres; the service also starts with an unreachable cache, warning rather than panicking as the DB and SQS assertions do — a slow read side beats no read side. Each call is bounded by services.cacheOperationTimeout (100 ms, retries off), because an unbounded cache miss on a dead node costs the dial timeout plus the query it was avoiding. A total cache outage adds ~200 ms per search (a failed read and a failed write), not seconds.

cms, discovery and infrastructure hold no test files of their own. The only tests in the repo are the black-box BDD scenarios in tests/ — see below.

cms is a JSON API only — it serves no HTML and has no static assets. The OpenAPI spec is generated from swaggo annotations on the handlers into cms/docs, a committed, compiled-in Go package. If you add or change a handler, its annotation comments, or a request/response struct, regenerate it:

go install github.com/swaggo/swag/cmd/swag@v1.16.6   # match go.mod; not installed by default
swag init --generalInfo main.go --dir ./ --parseInternal --output ./docs

--parseInternal is required — the handlers live under internal/, which swag skips without it. Keep the swaggo/swag version in go.mod and the CLI in lockstep: http-swagger/v2 transitively pulls a much older swag whose swag.Spec struct lacks the LeftDelim/RightDelim fields newer generators emit, and the build breaks outright if the two drift. (The CLI's --version misreports itself as v1.16.4; go version -m $(go env GOPATH)/bin/swag gives the real one.)

tests (Go, module thamanyah/tests)

cd tests
go test ./...               # runs features/*.feature against CMS_BASE_URL (default http://localhost:8081)
go test -v ./...            # -v prints the Gherkin: gobdd nests a subtest per feature/scenario/step
CMS_BASE_URL=… go test ./...

Black-box BDD covering the video upload feature: its own Go module importing nothing from cms/, talking to a running service over HTTP only, so the same scenarios run against compose and against a deployed environment. Skips (does not fail) when nothing is serving. Uploading is a plain PUT to the presigned URL, no AWS SDK. Scenarios can be tagged @known-gap to pin current behaviour that differs from the documented contract — see tests/README.md.

Note compose starts cms in server mode only: the migrate container exists in the ECS task definition, not in docker-compose.yml, so a freshly created local database needs docker exec cms ./cms migrate once or every scenario fails on relation "categories" does not exist.

infrastructure (Go, Pulumi, module thamanyah)

cd infrastructure
pulumi preview               # plan changes against stack "main"
pulumi up                    # apply — this touches real AWS resources, confirm with the user first
pulumi stack output          # e.g. ecsClusterArn, cmsServiceArn

Deploys run in CI (.gitea/workflows/infrastructure-deploy.yml) on push to main touching infrastructure/**. Treat local pulumi up as something to confirm with the user, not a routine dev command — it mutates shared cloud state and Pulumi state isn't safe to update concurrently with CI.

Architecture

cms package layout

Four internal packages, split by the kind of thing they hold — keep new code on the same seams:

package	holds
internal/api	the wire contract: request/response structs with their JSON + swaggo tags. No logic.
internal/models	the domain types the rest of the code passes around: Video, Category, VideoStatus. No JSON tags, no SQL.
internal/handlers	HTTP: decode api.*, validate, call repositories/services, encode api.*.
internal/db	the *sql.DB connection lifecycle and migrations; internal/db/repositories holds all SQL.
internal/services	AWS only — S3, MediaConvert and SQS. Nothing database-related lives here.

api and models are deliberately separate types even where the fields look alike: api.CompleteResponse is the published schema, models.Video is the row. Handlers translate between them field by field, so renaming a models field doesn't move the published API and vice versa.

cms service

Plain net/http (Go 1.22+ pattern-based ServeMux), no framework. Entry point cms/main.go builds the dependencies and assigns them to package-level interface vars that the handlers call through:

• services.S3Client (services.S3) ← *services.S3Concrete
• services.MediaConvertClient (services.MediaConvert) ← *services.MediaConvertConcrete
• repositories.VideoRepo (repositories.VideoRepository) ← *repositories.ConcreteVideoRepository
• repositories.CatagoriesRepo (repositories.CatagoriesRepository) ← *repositories.ConcreteCatagoriesRepository

Each interface has exactly one implementation; the indirection is what makes the handlers substitutable in tests, even though no tests exist yet. Note the spelling: the categories repository is Catagories/CatagoriesRepo in repositories/catagories.go (and handlers.validateCatagoryIDS) — the misspelling is load-bearing for compilation, so match it rather than "fixing" it piecemeal.

On boot, S3Concrete.AssertSuccessfulConnection proactively exercises head-bucket/put/get/presign against the bucket (writing a throwaway .s3-connectivity-check object), and db.AssertSuccessfulConnection pings Postgres — both panic on failure rather than letting the service come up in a broken state.

cms/main.go has two entry paths, dispatched on os.Args[1]: the default path (runServer) boots the HTTP server; ./cms migrate (runMigrate) only opens the DB connection, applies pending migrations via cms/internal/db.Migrate (golang-migrate, iofs source, SQL files embedded from cms/internal/db/migrations/*.sql), and exits — it does not touch S3/MediaConvert or start the server. This is run as its own ECS container before the main container starts (see infrastructure below), so runServer never runs migrations itself, only AssertSuccessfulConnection.

The DB connection is opened once for the process lifetime by db.CreateDBConnection(connString) and closed by db.CloseConnection — both plain functions over *sql.DB in internal/db/client.go, not methods on a wrapper type. Repositories take that *sql.DB as their SQLDB field.

Routes (cms/main.go): GET /health, GET /api/categories, POST /api/videos/presign, POST /api/videos, plus Swagger UI at GET /swagger/ (/swagger/doc.json serves the spec). The UI assets are embedded in the binary by swaggo/files, so nothing is read from disk and nothing is fetched from a CDN at runtime.

Handlers live in cms/internal/handlers — handlers.go holds Health and the shared response writers, videos.go the categories and upload endpoints. The wire structs they serve live in cms/internal/api (api.go for HealthResponse and ProblemDetails, videos.go for Category, CategoriesResponse, PresignRequest/PresignResponse and CompleteRequest/CompleteResponse), referenced by the handlers' swaggo annotations as api.CompleteResponse and so on — so the generated spec's definition names track that package, and renaming a type there changes the published schema names.

There is no view layer: the internal/views templ package, the static/ directory, and the htmx frontend were all removed when the service became a JSON API, along with the templ dependency.

Error responses are RFC 9457 Problem Details objects (application/problem+json), written by writeProblem(w, status, title, detail). type is always "about:blank"; title is a short summary held identical across every occurrence of a given problem, so clients can branch on it; detail is the only member that varies with request data. Success responses go through writeJSON (application/json). Both share writeJSONContent. Note this deviates slightly from RFC 9457, which pairs an about:blank type with a title that is just the HTTP status phrase — meaningful titles like these are supposed to carry a real type URI. Adding per-problem type URIs is the conforming fix if it ever matters.

Data model (Postgres, cms/internal/db/migrations/, two migrations: 0001, 0002)

• videos — one row per uploaded video: title, description, tags (free-text, comma-separated — not normalized), file_name, storage_key (the S3 key, unique), mediaconvert_job_id, status (written once as "processing" on insert — see Known gaps), size_bytes, timestamps. id is a UUID filled by the column's own DEFAULT gen_random_uuid() (v4) and read back through RETURNING id — the application does not generate it. (An earlier revision generated UUIDv7 application-side; that was reverted in 785154c, so primary keys are random, not time-ordered.) status is a plain TEXT NOT NULL DEFAULT 'processing' column with no CHECK constraint — the closed set exists only in Go, as the models.VideoStatus string type in cms/internal/models/video.go (processing, ready, failed). Extending it means adding a constant there and extending the enums annotation on api.CompleteResponse.Status before regenerating the spec. (The doc comment on VideoStatus claims a videos_status_check constraint enforces it in the database — that constraint does not exist; nothing has ever created it.)
• categories — a small fixed lookup table (documentary, news, entertainment, podcast, other), seeded by migration 0001. Its SMALLSERIAL ids are part of the public API: GET /api/categories returns {id, name} pairs (ordered by name, not id) and POST /api/videos takes categoryIds, so the seed order in migration 0001 is what fixes which id means which name — never renumber it.
• video_categories — join table (video_id, category_id, composite PK, ON DELETE CASCADE) added in migration 0002: a video can belong to multiple categories, not just one. ConcreteVideoRepository.CreateVideo inserts the videos row and its video_categories links inside a single transaction, taking the category ids as given — handlers.validateCatagoryIDS is what checks them against CatagoriesRepo.ListCategoriesIDs before the transcode job is queued, leaving the foreign key and the composite PK as backstops (see Known gaps for how those failures surface).

Video upload → transcode pipeline

1. Client calls POST /api/videos/presign → cms returns a presigned S3 PUT URL for raw-uploads-bucket, key videos/<random-hex>.<ext>. The submitted contentType (only video/mp4 or video/quicktime) is signed into the URL, so the client's PUT must send the identical header.
2. Client PUTs the file directly to S3 (from a browser this requires the bucket's CORS rule, set up in infrastructure/main.go, its allowed origin is now stale). The file never passes through cms.
3. Client calls POST /api/videos with the metadata (categories given as categoryIds from GET /api/categories) + key → cms validates the category ids, calls MediaConvertClient.QueueEncodingJob(key), submitting a MediaConvert job s3://raw-uploads-bucket/<key> → s3://encoded-bucket/<key> (H.264/AAC → MP4, QVBR rate control — QVBR requires MaxBitrate to be set explicitly), then repositories.VideoRepo.CreateVideo persists the videos row (status "processing") and its video_categories links.
4. Finished output lands in encoded-bucket, served via CloudFront.
5. MediaConvert reports the job's state changes to an SNS topic that fans out to cms-mediaconvert-events, which internal/consumers reads for the lifetime of the process: it records the outcome (ready/failed) and the playback URL, rewriting the s3:// playlist path onto PLAYBACK_BASE_URL. A job that came out ready is then announced on the catalogue-events SNS topic via services.CatalogueClient — {videoId, title, playbackUrl, categories}, categories by name so a subscriber needs nothing from cms to interpret them. Failed jobs are recorded but never announced. A topic, not a queue, so the announcement fans out: discovery-catalogue-events subscribes today (raw delivery, so there is no SNS envelope to unwrap), and a second reader can subscribe its own queue without cms changing. Note a topic only delivers to subscriptions that exist when it publishes, so a new subscriber's queue has to be in place before the announcement, not after. Delivery is at-least-once: a failed publish leaves the job event on the consumer's queue, and the outcome update is idempotent, so a redelivery re-announces and a subscriber can see a video twice.

Config wiring: cms reads S3_BUCKET, MEDIACONVERT_INPUT_BUCKET, MEDIACONVERT_OUTPUT_BUCKET, MEDIACONVERT_ROLE_ARN, AWS_REGION, MEDIACONVERT_EVENTS_QUEUE_URL, CATALOGUE_EVENTS_TOPIC_ARN, PLAYBACK_BASE_URL, DB_HOST, DB_PORT, DB_NAME, DB_USER, DB_PASSWORD from env vars injected by the ECS task definition (extraEnv and the unconditional DB_* vars in deployFargateService, infrastructure/main.go) — main.go panics on boot if any required var is empty.

infrastructure (infrastructure/main.go, single Pulumi Go program, region us-east-1, stack main)

• Postgres: one shared RDS instance (db.t3.micro, single-AZ, no backups — intentionally minimal). Each app (cms, discovery) gets its own login role and same-named database via the postgresql provider (newServiceDatabase), so services never share DB credentials.
• ElastiCache: one cache.t4g.micro Redis node (search-cache, engine 7.1, single-AZ, no replica, no snapshots) that discovery answers repeated catalogue searches from. Its contents are derivable from Postgres by definition, so there is nothing to back up. Its security group admits 6379 from ecs-service-sg only — narrower than the database's, which also admits the deployer's IP for the postgresql provider; there is nothing to administer here from a laptop. The endpoint is the single node's address (CacheNodes[0], not ConfigurationEndpoint — that is a Memcached thing), exported as searchCacheAddress and injected as REDIS_ADDR. Skipped under LocalStack, where docker-compose runs a plain redis:7-alpine container instead: ElastiCache picks its own endpoint, and compose needs a literal REDIS_ADDR before anything is provisioned.
• ECS Fargate: one cluster (app-cluster), one ALB per service (each gets its own DNS name rather than sharing a load balancer on different ports). deployFargateService(...) is the shared helper building a service's ECR repo, CloudWatch log group, task definition, ECS service, and ALB. taskRole is optional (nil = no AWS identity beyond the shared execution role); extraEnv appends container env vars beyond the DB_* set; runMigrations (true for both services) adds a second, non-essential <name>-migrate container to the task — same image, command: ["migrate"] — with the main container's dependsOn set to condition: "COMPLETE" on it. This is ECS's container-dependency mechanism, the Fargate equivalent of a Kubernetes init container: ECS runs the migrate container to completion (exit 0) before starting the main container, so schema migrations always finish before the service accepts traffic. No separate CI/Docker migration step exists — Dockerfile's ENTRYPOINT ["./cms"] plus the container's command override composes to ./cms migrate.
• S3 + CloudFront: encoded-bucket holds finished transcoded output, served publicly via CloudFront using Origin Access Control (OAC) — the bucket itself blocks all public access; only CloudFront's OAC principal can read it. The distribution carries a response headers policy adding permissive CORS headers (*, GET/HEAD/OPTIONS), and OPTIONS is in the behaviour's allowed/cached methods so CloudFront answers preflights itself: HLS is fetched by JavaScript, so a playlist or segment served without Access-Control-Allow-Origin is discarded by the browser. The bucket also carries its own equivalent CORS rule, which is what the LocalStack stack relies on — there is no CloudFront there and PLAYBACK_BASE_URL points the player straight at S3.
• S3 raw uploads: raw-uploads-bucket is a separate, private bucket for pre-transcode uploads — deliberately kept apart from encoded-bucket so raw source video is never reachable through the public CDN. CORS is scoped to PUT only, from the cms ALB's own origin — correct back when cms served the upload page itself, but stale now that it serves no UI (see Known gaps).
• IAM roles — five distinct roles/users, each scoped narrowly, don't conflate them:
  • ecs-task-execution-role — shared by both services' ECS agent (image pull, log write, Secrets Manager read for DB password). Not usable by application code inside the container.
  • cms-task-role — the cms container's own AWS identity: S3 ListBucket/PutObject/GetObject on raw-uploads-bucket only, mediaconvert:CreateJob, iam:PassRole scoped to the MediaConvert service role (iam:PassedToService condition), receive/delete on cms-mediaconvert-events, and sns:Publish on the catalogue-events topic — the only thing it writes to. It has no access to discovery-catalogue-events, the queue subscribed to that topic: cms publishes, it does not reach into a subscriber.
  • discovery-task-role — the discovery container's own AWS identity, and its only one: receive/delete/get-attributes on discovery-catalogue-events. No S3, no MediaConvert, and no sns:Publish — it consumes the catalogue, it does not add to it. The search cache is absent from it on purpose: ElastiCache is reached over the Redis protocol inside the VPC, so search-cache-sg is the grant, not IAM.
  • mediaconvert-service-role — trusted by mediaconvert.amazonaws.com, not by ECS; the role MediaConvert itself assumes (passed as CreateJobInput.Role) to read raw-uploads-bucket and write encoded-bucket. Distinct from cms-task-role by design: one is "cms calling AWS", the other is "AWS calling AWS on cms's behalf".
  • gitea-ci-user — an IAM user (static access keys, not OIDC — the Gitea Actions runner doesn't support instance-profile auth) scoped to just ECR push (cms/discovery repos) and ecs:UpdateService/ecs:DescribeServices on the two ECS services. Never broaden this to ecr:*/ecs:*.

CI/CD (.gitea/workflows/)

• cms-deploy.yml: push to main touching cms/**. Builds/pushes the Docker image to ECR using gitea-ci-user, installs the AWS CLI (not preinstalled on the runner image — via AWS's official install script, not a third-party action), then aws ecs update-service --force-new-deployment. Cluster/service ARNs come from pulumi stack output and are set as repo variables (not secrets — ARNs aren't sensitive).
• discovery-deploy.yml: the same pipeline for discovery, on pushes touching discovery/**. One difference: the ECS service ARN is read from the DISCOVERY_ECS_SERVICE_ARN repo variable rather than pinned in the workflow, so it has to be set (from pulumi stack output discoveryServiceArn) before the first deploy can succeed.
• infrastructure-deploy.yml: push to main touching infrastructure/**. Runs pulumi up using a separate, broader AWS credential (PULUMI_AWS_ACCESS_KEY_ID/SECRET) than gitea-ci-user, since provisioning IAM/RDS/ECS/CloudFront needs wider permissions than pushing images and forcing deployments. Guarded with a concurrency group since Pulumi state isn't safe to update concurrently.
• The runner's ubuntu-latest label maps to a docker image configured on the runner host (outside this repo) — currently minimal, lacking the AWS CLI, hence the manual install step in cms-deploy.yml.