FEAT: Video Search In Discovery

CLAUDE.md

@@ -18,11 +18,12 @@ draining `discovery-catalogue-events`, the queue subscribed to the catalogue
 topic `cms` announces ready videos on, and keeps its own copy in its own
 database. Neither service reads the other's tables and nothing polls `cms`.
 
-It serves `GET /health`, `GET /api/videos/{id}` and the Swagger UI. Migration
+It serves `GET /health`, `GET /api/videos/{id}`, `QUERY /api/videos` (the
-`0001_init` still creates nothing (it predates the domain and exists only
+catalogue search — see below) and the Swagger UI. Migration `0001_init` still
-because `internal/db/migrate.go` embeds `migrations/*.sql`, which will not
+creates nothing (it predates the domain and exists only because
-compile against an empty directory); `0002_create_videos_table` is where the
+`internal/db/migrate.go` embeds `migrations/*.sql`, which will not compile
-schema actually starts. It now has an `internal/services` (SQS only) and an
+against an empty directory); `0002_create_videos_table` is where the schema
+actually starts, and `0003`–`0005` add what the search needs. It now has an `internal/services` (SQS only) and an
 `internal/consumers`, and a task role scoped to that one queue — receive,
 delete, get-attributes, and nothing else. It cannot publish back onto the
 topic: it is a subscriber, not a participant.
@@ -72,6 +73,57 @@ same video the same way). `categories` is a denormalized `TEXT[]` of names:
 so the same announcement can arrive twice, and a plain `INSERT` would fail on
 the second, wedging every message queued behind it. A later announcement wins.
 
+Three more columns/indexes exist purely for the search (`0003`–`0005`):
+`search_vector` is a **generated** `tsvector` column (`to_tsvector('arabic',
+title)`, `STORED`) with a GIN index; `categories` has a GIN index for the `&&`
+overlap; and `videos_recent_idx` is a btree on `(created_at DESC, id DESC)`.
+The text search configuration is `'arabic'`, not `'english'` or `'simple'`: it
+stems Arabic while leaving Latin-script tokens as written, which suits a mixed
+catalogue. The cost is that English is **not** stemmed, so "documentaries" does
+not find "documentary". It is named explicitly in the column *and* in every
+query — a query built under a different configuration stems its terms
+differently and silently matches nothing.
+
+### Catalogue search (`QUERY /api/videos`)
+
+Served on **QUERY**, not GET or POST: a search is safe and idempotent (which
+POST is not), but its parameters are a structured document that does not belong
+in a query string. Go's `ServeMux` routes the method fine. Two consequences to
+know before touching it:
+
+- It **cannot appear in the OpenAPI spec**. Swagger 2.0 and OpenAPI 3.x define
+  `PathItem` with a fixed field per method and there is no slot for QUERY;
+  `swag` rejects the annotation outright with `invalid method: QUERY`. So the
+  handler carries **no `@Router` annotation** and the endpoint is described in
+  prose in `main.go`'s `@description` instead. Adding a `@Router … [query]`
+  line breaks `swag init` for the whole service.
+- QUERY is still an IETF draft (`draft-ietf-httpbis-safe-method-w-body`), so
+  expect intermediaries that have never heard of it, and no HTTP caching of the
+  kind a `GET` with a query string would get.
+
+Request `{title, categories, limit, cursor}`, response `{videos, nextCursor}`.
+`title` is matched lexically, ANDing the words with the **last one as a prefix**
+(`desert & fal:*`) so a part-typed word still matches; `categories` narrows to
+videos filed under **any** of the names (`&&`, not `@>`); `limit` defaults to 20
+and is **capped at 100** rather than refused. Paging is **keyset**, not offset:
+the opaque cursor carries the last row's `(rank, created_at, id)`, so pages stay
+stable while the catalogue is being written to. The query asks for `limit+1`
+rows and hands back a cursor only when the extra row appears, which is how the
+last page reports itself as last.
+
+`tsQueryFor` builds the tsquery by splitting input on everything that is not a
+letter or digit. That drops every character tsquery gives a meaning to, which is
+what makes it safe to interpolate — `websearch_to_tsquery` parses user syntax
+but cannot express a prefix match.
+
+A measured caveat: for a **ranked** search, deep paging is not a seek. `ts_rank`
+is computed per row, so the keyset comparison lands as a filter rather than an
+index condition and every matching row is still scanned. Keyset paging still
+buys stability and avoids OFFSET's growing discard cost, but only the unranked
+browse path (no title term) is a true index seek — that is what
+`videos_recent_idx` is for, and it takes a 200k-row browse from ~15 ms to
+~0.03 ms.
+
 `cms`, `discovery` and `infrastructure` hold no test files of their own. The
 only tests in the repo are the black-box BDD scenarios in `tests/` — see below.
 

discovery/docs/docs.go

@@ -145,7 +145,7 @@ var SwaggerInfo = &swag.Spec{
 	BasePath:         "/",
 	Schemes:          []string{},
 	Title:            "Thamanyah Discovery API",
-	Description:      "JSON API for browsing the Thamanyah catalogue. Read-side counterpart to the CMS, which is what ingests videos.",
+	Description:      "JSON API for browsing the Thamanyah catalogue. Read-side counterpart to the CMS, which is what ingests videos.\n\nNot listed below: `QUERY /api/videos`, the catalogue search. It takes a JSON body `{title, categories, limit, cursor}` and answers `{videos, nextCursor}` — title is matched lexically with the last word as a prefix, categories narrow to videos filed under any one of them, limit defaults to 20 and is capped at 100, and cursor is the opaque `nextCursor` of a previous search. It is absent here because OpenAPI has no slot for the QUERY method, not because it is unsupported; see handlers.SearchVideos.",
 	InfoInstanceName: "swagger",
 	SwaggerTemplate:  docTemplate,
 	LeftDelim:        "{{",

discovery/docs/swagger.json

@@ -1,7 +1,7 @@
 {
     "swagger": "2.0",
     "info": {
-        "description": "JSON API for browsing the Thamanyah catalogue. Read-side counterpart to the CMS, which is what ingests videos.",
+        "description": "JSON API for browsing the Thamanyah catalogue. Read-side counterpart to the CMS, which is what ingests videos.\n\nNot listed below: `QUERY /api/videos`, the catalogue search. It takes a JSON body `{title, categories, limit, cursor}` and answers `{videos, nextCursor}` — title is matched lexically with the last word as a prefix, categories narrow to videos filed under any one of them, limit defaults to 20 and is capped at 100, and cursor is the opaque `nextCursor` of a previous search. It is absent here because OpenAPI has no slot for the QUERY method, not because it is unsupported; see handlers.SearchVideos.",
         "title": "Thamanyah Discovery API",
         "contact": {},
         "version": "1.0"

discovery/docs/swagger.yaml

@@ -43,8 +43,10 @@ definitions:
     type: object
 info:
   contact: {}
-  description: JSON API for browsing the Thamanyah catalogue. Read-side counterpart
+  description: |-
-    to the CMS, which is what ingests videos.
+    JSON API for browsing the Thamanyah catalogue. Read-side counterpart to the CMS, which is what ingests videos.
+
+    Not listed below: `QUERY /api/videos`, the catalogue search. It takes a JSON body `{title, categories, limit, cursor}` and answers `{videos, nextCursor}` — title is matched lexically with the last word as a prefix, categories narrow to videos filed under any one of them, limit defaults to 20 and is capped at 100, and cursor is the opaque `nextCursor` of a previous search. It is absent here because OpenAPI has no slot for the QUERY method, not because it is unsupported; see handlers.SearchVideos.
   title: Thamanyah Discovery API
   version: "1.0"
 paths:

discovery/internal/api/videos.go

@@ -11,3 +11,34 @@ type Video struct {
 	PlaybackURL string   `json:"playbackUrl" example:"https://d111111abcdef8.cloudfront.net/videos/abc123/index.m3u8"`
 	Categories  []string `json:"categories" example:"documentary,news"`
 }
+
+// SearchRequest is the body of QUERY /api/videos: what a reader is looking
+// for. Every member is optional — an empty body browses the whole catalogue.
+type SearchRequest struct {
+	// Title is matched lexically against video titles: the words are all
+	// required, and the last is treated as a prefix so a part-typed word still
+	// finds something.
+	Title string `json:"title" example:"desert fal"`
+	// Categories narrows the search to videos filed under any one of these
+	// names — not all of them, so it widens the choice of category while
+	// narrowing the results. Empty means every category.
+	Categories []string `json:"categories" example:"documentary,news"`
+	// Limit is how many videos to return. Absent or zero takes the default of
+	// 20; anything above the maximum of 100 is capped to it rather than
+	// refused, and the cursor still reaches the rest.
+	Limit int `json:"limit" example:"20"`
+	// Cursor asks for the page after the one a previous search ended at. Send
+	// back the nextCursor from that search, unchanged; it is opaque, and the
+	// only thing to do with it is return it.
+	Cursor string `json:"cursor"`
+}
+
+// SearchResults is one page of search results.
+type SearchResults struct {
+	// Videos are the matches, most relevant first. Never null: a search that
+	// matches nothing is an empty list.
+	Videos []Video `json:"videos"`
+	// NextCursor reaches the page after this one. Empty on the last page,
+	// which is how a reader knows there is no more to ask for.
+	NextCursor string `json:"nextCursor"`
+}

discovery/internal/db/migrations/0003_add_video_search.down.sql

@@ -0,0 +1,2 @@
+DROP INDEX IF EXISTS videos_search_vector_idx;
+ALTER TABLE videos DROP COLUMN IF EXISTS search_vector;

discovery/internal/db/migrations/0003_add_video_search.up.sql

@@ -0,0 +1,25 @@
+-- Lexical search over video titles.
+--
+-- The vector is a generated column rather than something the consumer writes:
+-- the catalogue's only writer is SaveVideo, and a title that arrives in an
+-- announcement should be searchable because it is stored, not because a second
+-- statement remembered to index it. STORED because a GIN index needs the value
+-- on disk to index it at all.
+--
+-- 'arabic' is the text search configuration, not 'english' or 'simple': it
+-- stems Arabic (الوثائقي -> وثايق) while leaving Latin-script tokens as written,
+-- which is the right trade for a mixed catalogue. The cost is that English
+-- words are not stemmed, so "documentaries" does not find "documentary".
+--
+-- The configuration is named explicitly rather than left to default_text_search_config,
+-- which is a per-session GUC — an expression that reads it is only STABLE, and
+-- a generated column requires IMMUTABLE. Naming it is also what keeps the
+-- column and every query in step: a query built with a different
+-- configuration would stem its terms differently and silently match nothing.
+ALTER TABLE videos
+    ADD COLUMN search_vector tsvector
+    GENERATED ALWAYS AS (to_tsvector('arabic', title)) STORED;
+
+-- GIN, not GiST: the catalogue is read far more than it is written, and GIN
+-- answers @@ faster at the cost of a slower update — the right way round here.
+CREATE INDEX videos_search_vector_idx ON videos USING GIN (search_vector);

discovery/internal/db/migrations/0004_add_video_categories_index.down.sql

@@ -0,0 +1 @@
+DROP INDEX IF EXISTS videos_categories_idx;

discovery/internal/db/migrations/0004_add_video_categories_index.up.sql

@@ -0,0 +1,7 @@
+-- Narrowing a search by category.
+--
+-- categories is a TEXT[] of names, and the search overlaps it against the
+-- names a reader asked for (&&). GIN is the index type that operator can use;
+-- without it the overlap is a filter applied after the rows are read, which is
+-- the whole table once the title term is broad.
+CREATE INDEX videos_categories_idx ON videos USING GIN (categories);

discovery/internal/db/migrations/0005_add_video_recency_index.down.sql

@@ -0,0 +1 @@
+DROP INDEX IF EXISTS videos_recent_idx;

discovery/internal/db/migrations/0005_add_video_recency_index.up.sql

@@ -0,0 +1,17 @@
+-- Browsing the catalogue, and paging through it.
+--
+-- Every search orders by (rank, created_at, id). When there is no title term
+-- the rank is a constant, and Postgres folds a constant sort key away — so the
+-- order that remains is exactly this index, and a browse becomes an index-only
+-- scan that stops as soon as the page is full instead of sorting the table.
+--
+-- Measured at 200k rows: a first page of a browse goes from a 15 ms parallel
+-- sequential scan with a top-N sort to 0.03 ms. It matters more for the cursor
+-- than for the first page — the keyset comparison (created_at, id) < (…)
+-- becomes an index condition rather than a filter, so a deep page is a seek
+-- to the right place in the index rather than a walk through everything
+-- before it.
+--
+-- DESC on both columns to match the ORDER BY: a btree can be read backwards,
+-- but only as a whole, so a mixed-direction ordering could not use it.
+CREATE INDEX videos_recent_idx ON videos (created_at DESC, id DESC);

discovery/internal/db/repositories/errors.go

@@ -7,3 +7,8 @@ import "errors"
 
 // ErrVideoNotFound reports that the catalogue holds no video with that id.
 var ErrVideoNotFound = errors.New("video not found")
+
+// ErrInvalidCursor reports that a paging cursor could not be read. It is the
+// caller's mistake, not a fault in the catalogue, so it must not reach the
+// handler as an indistinguishable query failure.
+var ErrInvalidCursor = errors.New("invalid cursor")

discovery/internal/db/repositories/videos.go

@@ -3,8 +3,13 @@ package repositories
 import (
 	"context"
 	"database/sql"
+	"encoding/base64"
+	"encoding/json"
 	"errors"
+	"strings"
 	"thamanyah/discovery/internal/models"
+	"time"
+	"unicode"
 
 	"github.com/lib/pq"
 )
@@ -57,3 +62,191 @@ func (svc VideoRepository) GetVideoByID(ctx context.Context, id string) (models.
 
 	return v, nil
 }
+
+// searchConfig is the text search configuration the search_vector column is
+// generated with. Every query has to name the same one: a tsquery built under
+// a different configuration stems its terms differently and matches nothing.
+const searchConfig = "arabic"
+
+// tsQueryFor turns what a reader typed into a tsquery.
+//
+// The words are ANDed, and the last one is given a :* prefix match — someone
+// typing into a search box is usually part-way through their last word, so
+// "desert fal" should still find "Desert Falcons". Earlier words are matched
+// whole, because they have been finished.
+//
+// The input is split on everything that is not a letter or a digit, which
+// drops every character tsquery gives a meaning to (&, |, !, :, *, parens) and
+// leaves nothing that could change the shape of the query. That is what makes
+// it safe to interpolate the result into to_tsquery: the alternative,
+// websearch_to_tsquery, parses user syntax but cannot express a prefix match.
+//
+// It returns "" when there is nothing to search for, which callers read as
+// "no title filter" rather than "match nothing".
+func tsQueryFor(term string) string {
+	words := strings.FieldsFunc(term, func(r rune) bool {
+		return !unicode.IsLetter(r) && !unicode.IsDigit(r)
+	})
+	if len(words) == 0 {
+		return ""
+	}
+
+	for i, word := range words {
+		words[i] = strings.ToLower(word)
+	}
+	return strings.Join(words, " & ") + ":*"
+}
+
+// searchCursor is where a page of results stopped: the sort key of its last
+// row. Paging resumes by asking for the rows that sort after it.
+//
+// It carries the rank as well as the row's identity because the results are
+// ordered by relevance first, and "the row after this one" is only a
+// well-defined place if every term of the ordering is pinned.
+type searchCursor struct {
+	Rank      float64   `json:"r"`
+	CreatedAt time.Time `json:"t"`
+	ID        string    `json:"i"`
+}
+
+// encode renders a cursor as an opaque string. Opaque on purpose: it is a
+// place in a result set, not a number a client should do arithmetic on, and
+// making it look like one invites callers to guess a page.
+func (c searchCursor) encode() (string, error) {
+	raw, err := json.Marshal(c)
+	if err != nil {
+		return "", err
+	}
+	return base64.RawURLEncoding.EncodeToString(raw), nil
+}
+
+func decodeCursor(encoded string) (searchCursor, error) {
+	raw, err := base64.RawURLEncoding.DecodeString(encoded)
+	if err != nil {
+		return searchCursor{}, ErrInvalidCursor
+	}
+
+	var c searchCursor
+	if err := json.Unmarshal(raw, &c); err != nil {
+		return searchCursor{}, ErrInvalidCursor
+	}
+	if strings.TrimSpace(c.ID) == "" {
+		return searchCursor{}, ErrInvalidCursor
+	}
+
+	return c, nil
+}
+
+// SearchVideos finds catalogued videos whose title matches the given term and
+// which are filed under any of the given categories, most relevant first.
+//
+// It returns the page and the cursor that reaches the page after it; that
+// cursor is empty on the last page, which is how a caller knows to stop.
+//
+// The title predicate is skipped entirely when the term yields no words, and
+// the category predicate when no categories are named, so a search with
+// neither browses the whole catalogue newest-first.
+//
+// Paging is by key rather than by offset. OFFSET makes the database walk and
+// discard every row it skips, so deep pages get steadily more expensive, and
+// it addresses a page by its position — which moves whenever a video is
+// announced, repeating or skipping rows for a reader mid-browse. Comparing
+// against the last row's sort key has neither problem: the work is the same at
+// any depth, and the page after a given row is that same page however much has
+// been added since.
+func (svc VideoRepository) SearchVideos(ctx context.Context, term string, categories []string, limit int, cursor string) ([]models.Video, string, error) {
+	query := tsQueryFor(term)
+
+	// Never nil: a nil slice reaches Postgres as NULL, and cardinality(NULL)
+	// is NULL rather than 0, so the "no category filter" branch would not fire
+	// and every search would come back empty.
+	if categories == nil {
+		categories = []string{}
+	}
+
+	// A zero cursor is passed as NULL, which the query reads as "start at the
+	// beginning" rather than as a row to seek past.
+	var after *searchCursor
+	if strings.TrimSpace(cursor) != "" {
+		decoded, err := decodeCursor(cursor)
+		if err != nil {
+			return nil, "", err
+		}
+		after = &decoded
+	}
+
+	var (
+		afterRank *float64
+		afterTime *time.Time
+		afterID   *string
+	)
+	if after != nil {
+		afterRank, afterTime, afterID = &after.Rank, &after.CreatedAt, &after.ID
+	}
+
+	// One row more than asked for: if it comes back, there is another page and
+	// the extra row is discarded. That is what lets the last page say so,
+	// rather than handing out a cursor to an empty page.
+	//
+	// The rank is computed in the subquery and referenced by name in both the
+	// cursor comparison and the ORDER BY, so the two cannot drift apart — if
+	// they did, "the row after this one" would mean something different from
+	// the order the rows are actually in, and pages would overlap.
+	rows, err := svc.SQLDB.QueryContext(ctx, `
+		SELECT id, title, playback_url, categories, created_at, rank
+		FROM (
+			SELECT id, title, playback_url, categories, created_at,
+			       CASE WHEN $1 = '' THEN 0::float8
+			            ELSE ts_rank(search_vector, to_tsquery('`+searchConfig+`', $1))::float8
+			       END AS rank
+			FROM videos
+			WHERE ($1 = '' OR search_vector @@ to_tsquery('`+searchConfig+`', $1))
+			  AND (cardinality($2::text[]) = 0 OR categories && $2)
+		) ranked
+		WHERE $4::float8 IS NULL
+		   OR (rank, created_at, id) < ($4::float8, $5::timestamptz, $6::uuid)
+		ORDER BY rank DESC, created_at DESC, id DESC
+		LIMIT $3
+	`, query, pq.Array(categories), limit+1, afterRank, afterTime, afterID)
+	if err != nil {
+		return nil, "", err
+	}
+	defer rows.Close()
+
+	// Never nil: a search that matches nothing is an empty list on the wire,
+	// not null.
+	videos := []models.Video{}
+	ranks := []float64{}
+	for rows.Next() {
+		var (
+			v    models.Video
+			rank float64
+		)
+		if err := rows.Scan(&v.ID, &v.Title, &v.PlaybackURL, pq.Array(&v.Categories), &v.CreatedAt, &rank); err != nil {
+			return nil, "", err
+		}
+		videos = append(videos, v)
+		ranks = append(ranks, rank)
+	}
+	if err := rows.Err(); err != nil {
+		return nil, "", err
+	}
+
+	if len(videos) <= limit {
+		return videos, "", nil
+	}
+
+	videos, ranks = videos[:limit], ranks[:limit]
+	last := videos[len(videos)-1]
+
+	next, err := searchCursor{
+		Rank:      ranks[len(ranks)-1],
+		CreatedAt: last.CreatedAt,
+		ID:        last.ID,
+	}.encode()
+	if err != nil {
+		return nil, "", err
+	}
+
+	return videos, next, nil
+}

discovery/internal/handlers/videos.go

@@ -1,6 +1,7 @@
 package handlers
 
 import (
+	"encoding/json"
 	"errors"
 	"log"
 	"net/http"
@@ -47,3 +48,76 @@ func GetVideo(w http.ResponseWriter, r *http.Request) {
 		Categories: append([]string{}, video.Categories...),
 	})
 }
+
+const (
+	// searchPageSize is how many videos a search returns when the caller does
+	// not say. Small enough to keep the common request cheap.
+	searchPageSize = 20
+
+	// maxSearchPageSize is the largest page the catalogue will build, however
+	// big a page the caller asks for. The endpoint is read constantly, and a
+	// page size is a request for work: without a ceiling, one client asking
+	// for the whole catalogue in a single response makes every other reader
+	// wait behind it.
+	//
+	// Asking for more is not an error — the page is simply capped, and the
+	// cursor still reaches the rest — because a client that wants everything
+	// is doing something legitimate, just not in one request.
+	maxSearchPageSize = 100
+)
+
+// SearchVideos serves the catalogue's lexical search.
+//
+// It is served on QUERY rather than GET or POST: a search is safe and
+// idempotent, which POST is not, but its parameters are a structured document
+// that does not belong in a query string. QUERY says both. It is specified in
+// draft-ietf-httpbis-safe-method-w-body, not in RFC 9110 — still a draft, so
+// expect intermediaries that have never heard of it. Note it also cannot be
+// expressed in the generated OpenAPI spec — Swagger 2.0 and OpenAPI 3.x define a fixed set of
+// per-method fields with no slot for QUERY, and swag rejects the annotation
+// outright — so this endpoint is deliberately absent from /swagger and
+// documented here instead.
+func SearchVideos(w http.ResponseWriter, r *http.Request) {
+	var request api.SearchRequest
+
+	if err := json.NewDecoder(r.Body).Decode(&request); err != nil {
+		writeProblem(w, http.StatusBadRequest, "Malformed Search",
+			"The request body could not be read as JSON. Send a JSON object describing what to search for; an empty object browses the whole catalogue.")
+		return
+	}
+
+	limit := request.Limit
+	if limit <= 0 {
+		limit = searchPageSize
+	}
+	if limit > maxSearchPageSize {
+		limit = maxSearchPageSize
+	}
+
+	found, next, err := repositories.VideoRepo.SearchVideos(r.Context(), request.Title, request.Categories, limit, request.Cursor)
+	if errors.Is(err, repositories.ErrInvalidCursor) {
+		writeProblem(w, http.StatusBadRequest, "Malformed Search Cursor",
+			"The cursor could not be read. Send back the nextCursor from a previous search unchanged, or omit it to start from the first page.")
+		return
+	}
+	if err != nil {
+		log.Printf("Something Went Wrong Searching The Catalogue: title=%q: %s", request.Title, err)
+		writeProblem(w, http.StatusInternalServerError, "Something Went Wrong Searching The Catalogue",
+			"The catalogue could not be searched. This is a server-side fault and the request was not processed; retrying in a few moments may succeed.")
+		return
+	}
+
+	// Translated field by field, for the same reason GetVideo does it: the row
+	// and the published schema are different things.
+	videos := make([]api.Video, 0, len(found))
+	for _, v := range found {
+		videos = append(videos, api.Video{
+			ID:          v.ID,
+			Title:       v.Title,
+			PlaybackURL: v.PlaybackURL,
+			Categories:  append([]string{}, v.Categories...),
+		})
+	}
+
+	writeJSON(w, http.StatusOK, api.SearchResults{Videos: videos, NextCursor: next})
+}

discovery/main.go

@@ -23,6 +23,8 @@ import (
 //	@title			Thamanyah Discovery API
 //	@version		1.0
 //	@description	JSON API for browsing the Thamanyah catalogue. Read-side counterpart to the CMS, which is what ingests videos.
+//	@description
+//	@description	Not listed below: `QUERY /api/videos`, the catalogue search. It takes a JSON body `{title, categories, limit, cursor}` and answers `{videos, nextCursor}` — title is matched lexically with the last word as a prefix, categories narrow to videos filed under any one of them, limit defaults to 20 and is capped at 100, and cursor is the opaque `nextCursor` of a previous search. It is absent here because OpenAPI has no slot for the QUERY method, not because it is unsupported; see handlers.SearchVideos.
 //	@BasePath		/
 
 func main() {
@@ -81,6 +83,9 @@ func runServer() {
 
 	mux.HandleFunc("GET /health", handlers.Health)
 	mux.HandleFunc("GET /api/videos/{id}", handlers.GetVideo)
+	// QUERY, not GET: a search is safe and idempotent but carries a structured
+	// body. See handlers.SearchVideos for why it is absent from the spec.
+	mux.HandleFunc("QUERY /api/videos", handlers.SearchVideos)
 
 	// Swagger UI and the generated spec. The UI assets are embedded in the
 	// binary by swaggo/files, so this needs no static directory on disk.

tests/README.md

@@ -77,6 +77,19 @@ Both steps of the upload flow, end to end:
   not there at all, and a video announced a second time is updated rather than
   colliding with its own row
 
+- searching that catalogue over `QUERY /api/videos`: finding a video by a word
+  in its title and by a part-typed one, narrowing to a category and being left
+  out of another, several categories meaning "any of", paging by cursor so the
+  pages tile the results exactly once, refusing a cursor no search issued, and
+  capping a page size that asks for the whole catalogue
+
+Search scenarios give every video they catalogue a nonce in its title, and
+assert relative to the video they created ("returns that video") rather than on
+absolute counts. Nothing cleans up between runs, so a fixed title accumulates a
+copy per run and a count would stop meaning anything. The paging scenario shares
+one nonce across its videos so that searching for it matches that run's cohort
+and nothing else.
+
 The publication scenarios read `tests-catalogue-events`, the suite's **own**
 queue on the catalogue topic — not discovery's. A consumer destroys what it
 reads, so sharing discovery's queue would have the suite and the running

tests/client_test.go

@@ -139,6 +139,55 @@ type catalogueVideoBody struct {
 	Categories  []string `json:"categories"`
 }
 
+// searchRequest is the body of QUERY /api/videos on discovery: what a reader
+// is looking for, plus where in the results to carry on from.
+//
+// Every field is optional. An empty body is the whole catalogue, newest first.
+type searchRequest struct {
+	Title      string   `json:"title,omitempty"`
+	Categories []string `json:"categories,omitempty"`
+	Limit      int      `json:"limit,omitempty"`
+	Cursor     string   `json:"cursor,omitempty"`
+}
+
+// searchResultsBody is what that query answers with: the page of videos, and
+// the cursor that reaches the page after it. NextCursor is empty on the last
+// page, which is how a client knows to stop.
+type searchResultsBody struct {
+	Videos     []catalogueVideoBody `json:"videos"`
+	NextCursor string               `json:"nextCursor"`
+}
+
+// ids lists the video ids in the page, for asserting on which videos came back
+// without caring about the rest of their fields.
+func (b searchResultsBody) ids() []string {
+	found := make([]string, 0, len(b.Videos))
+	for _, v := range b.Videos {
+		found = append(found, v.ID)
+	}
+	return found
+}
+
+func (b searchResultsBody) holds(id string) bool {
+	for _, v := range b.Videos {
+		if v.ID == id {
+			return true
+		}
+	}
+	return false
+}
+
+// queryJSON sends a QUERY request — the method the catalogue search is served
+// on, chosen because a search is safe and idempotent like a GET but carries a
+// structured body no query string would hold comfortably.
+func (c *client) queryJSON(path string, payload any) (response, error) {
+	encoded, err := json.Marshal(payload)
+	if err != nil {
+		return response{}, err
+	}
+	return c.send("QUERY", c.base+path, "application/json", encoded)
+}
+
 // postJSON marshals payload and posts it to a path on the CMS.
 func (c *client) postJSON(path string, payload any) (response, error) {
 	encoded, err := json.Marshal(payload)

tests/features/catalogue_search.feature

@@ -0,0 +1,78 @@
+Feature: Searching the catalogue
+  Discovery holds a copy of every video cms has announced as ready. Reaching
+  one by id is no use to a reader who does not have an id — finding something
+  to watch means searching for it by what it is called and what it is filed
+  under.
+
+  The search is lexical: it matches the words of a title, not its meaning. The
+  last word a reader types is treated as a prefix, because someone typing into
+  a search box is usually part-way through a word.
+
+  Background:
+    Given the CMS API is available
+    And the Discovery API is available
+
+  Scenario: Finding a catalogued video by a word in its title
+    Given the catalogue holds a video titled "Desert Falcons"
+    When I search the catalogue for that video's title
+    Then the search returns that video
+
+  # The last word is matched as a prefix, so the results keep up with someone
+  # still typing. Earlier words are matched whole — they have been finished.
+  Scenario: Finding a video from a part-typed word
+    Given the catalogue holds a video titled "Desert Falcons"
+    When I search the catalogue for "Desert Fal"
+    Then the search returns that video
+
+  # Categories narrow a search rather than widening it: a video filed elsewhere
+  # is not an answer to a reader who asked for one category in particular.
+  Scenario: A search narrowed to another category leaves the video out
+    Given the catalogue holds a video titled "Desert Falcons" filed under "documentary"
+    When I search the catalogue for that video's title in category "news"
+    Then the search does not return that video
+
+  Scenario: A search narrowed to the video's own category still finds it
+    Given the catalogue holds a video titled "Desert Falcons" filed under "documentary"
+    When I search the catalogue for that video's title in category "documentary"
+    Then the search returns that video
+
+  # Several categories mean "any of these", not "all of these": naming more of
+  # them offers the reader more, rather than demanding the video be filed under
+  # every one at once.
+  Scenario: Naming several categories matches a video filed under any of them
+    Given the catalogue holds a video titled "Desert Falcons" filed under "documentary"
+    When I search the catalogue for that video's title in categories "news, documentary"
+    Then the search returns that video
+
+  # Paging is by cursor rather than by offset: the catalogue is added to while
+  # people are reading it, and an offset silently repeats or skips a video when
+  # something is announced between one page and the next. A cursor names where
+  # the last page stopped, so the page after it is the same page whenever it is
+  # asked for.
+  Scenario: Paging through the results a page at a time
+    Given the catalogue holds 3 videos titled "Mountain Wolves"
+    When I search the catalogue for that title 2 at a time
+    Then the search returns 2 videos
+    And the search offers a cursor to the next page
+    When I follow the cursor
+    Then the search returns 1 video
+    And the search offers no further cursor
+    And the pages together hold every one of those videos exactly once
+
+  # A cursor is opaque, so a client that makes one up has made a mistake rather
+  # than found a fault. Saying 400 is what tells the two apart.
+  Scenario: Refusing a cursor no search issued
+    Given the catalogue holds a video titled "Desert Falcons"
+    When I search the catalogue for that video's title from the cursor "not-a-cursor"
+    Then the request is rejected with status 400
+    And the problem title is "Malformed Search Cursor"
+
+  # A page size is a request for work, and an uncapped one is a request for all
+  # of it — the catalogue is read constantly, and one client asking for every
+  # row at once should not be able to make everyone else wait. The cap applies
+  # silently rather than as a rejection: the reader still gets a page, and the
+  # cursor still reaches the rest.
+  Scenario: Capping a page size that asks for the whole catalogue
+    When I search the whole catalogue 5000 at a time
+    Then the search returns at most 100 videos
+    And the search offers a cursor to the next page

tests/steps_test.go

@@ -3,8 +3,10 @@ package tests
 import (
 	"bytes"
 	"context"
+	"fmt"
 	"net/url"
 	"slices"
+	"strconv"
 	"strings"
 	"time"
 
@@ -902,3 +904,316 @@ func theCatalogueEventuallyHoldsThatVideoWithNoPlaybackURL(t gobdd.StepTest, ctx
 	t.Errorf("the catalogue still points a player at %q for video %q after %s; the later "+
 		"announcement carried no playlist and should have replaced it", last, w.video.ID, catalogueSettleTimeout)
 }
+
+// --- Catalogue search ------------------------------------------------------
+
+// searchLimit is the page size the search scenarios ask for unless they are
+// about paging itself.
+const searchLimit = 20
+
+// uniqueTitle appends a nonce to the title a scenario names.
+//
+// Scenarios write real rows and nothing cleans up after them, so a fixed title
+// accumulates a copy per run and "the search returns exactly this video" stops
+// meaning anything. The nonce keeps every run's video findable on its own
+// while the feature file still reads in plain words.
+func uniqueTitle(title string) string {
+	return fmt.Sprintf("%s %s", title, strings.ToUpper(strconv.FormatInt(time.Now().UnixNano(), 36)))
+}
+
+// theCatalogueHoldsAVideoTitled puts one video into the catalogue under a
+// title the scenario chose: register it, report its transcode finished, and
+// wait for the announcement to land on the read side.
+func theCatalogueHoldsAVideoTitled(t gobdd.StepTest, ctx gobdd.Context, title string) {
+	catalogueVideoTitledUnder(t, ctx, title, "other")
+}
+
+// catalogueVideoTitledUnder is the whole write-to-read round trip for one
+// video, which is what "the catalogue holds …" costs: cms issues the id, the
+// job completes, cms announces it, discovery ingests it.
+func catalogueVideoTitledUnder(t gobdd.StepTest, ctx gobdd.Context, title, categories string) {
+	w := worldOf(t, ctx)
+
+	full := uniqueTitle(title)
+
+	requestUploadSlot(t, ctx, "searchable.mp4", "video/mp4")
+	uploadTheFile(t, ctx)
+	registerUploadedVideo(t, ctx, full, categories)
+
+	if w.last.status != 201 {
+		t.Fatalf("could not register a video to search for: %s", w.last.summary())
+		return
+	}
+
+	w.searchTitle = full
+
+	mediaConvertReportsState(t, ctx, "COMPLETE")
+	theCatalogueEventuallyHoldsThatVideo(t, ctx)
+}
+
+// searchTheCatalogueForThatTitle searches for the exact title the scenario's
+// video was catalogued under, nonce included — the scenario says "that video's
+// title" precisely so it does not have to know about the nonce.
+func searchTheCatalogueForThatTitle(t gobdd.StepTest, ctx gobdd.Context) {
+	w := worldOf(t, ctx)
+
+	if strings.TrimSpace(w.searchTitle) == "" {
+		t.Fatalf("no video has been catalogued, so there is no title to search for")
+		return
+	}
+
+	searchTheCatalogue(t, w, searchRequest{Title: w.searchTitle, Limit: searchLimit})
+}
+
+// searchTheCatalogue performs the query and decodes the page, leaving both the
+// raw exchange and the decoded results on the world so a "Then" step can
+// assert on either.
+func searchTheCatalogue(t gobdd.StepTest, w *world, request searchRequest) {
+	if w.discovery == nil {
+		w.discovery = newDiscoveryClient()
+	}
+
+	result, err := w.discovery.queryJSON("/api/videos", request)
+	if err != nil {
+		t.Fatalf("could not search the catalogue: %s", err)
+		return
+	}
+
+	w.last = result
+	w.results = searchResultsBody{}
+
+	// A rejected search has no page to decode; the scenario asserting the
+	// rejection reads w.last instead.
+	if result.status == 200 {
+		if err := result.json(&w.results); err != nil {
+			t.Fatalf("could not decode the search results: %s", err)
+		}
+	}
+}
+
+func theSearchReturnsThatVideo(t gobdd.StepTest, ctx gobdd.Context) {
+	w := worldOf(t, ctx)
+
+	if w.last.status != 200 {
+		t.Fatalf("the search did not succeed: %s", w.last.summary())
+		return
+	}
+	if !w.results.holds(w.video.ID) {
+		t.Errorf("the search for %q did not return video %q; it returned %v",
+			w.searchTitle, w.video.ID, w.results.ids())
+	}
+}
+
+// searchTheCatalogueFor searches for a term the scenario spells out, rather
+// than for the title of the video it catalogued.
+func searchTheCatalogueFor(t gobdd.StepTest, ctx gobdd.Context, term string) {
+	searchTheCatalogue(t, worldOf(t, ctx), searchRequest{Title: term, Limit: searchLimit})
+}
+
+// theCatalogueHoldsAVideoTitledUnder is theCatalogueHoldsAVideoTitled for a
+// scenario that cares which categories the video is filed under.
+func theCatalogueHoldsAVideoTitledUnder(t gobdd.StepTest, ctx gobdd.Context, title, categories string) {
+	catalogueVideoTitledUnder(t, ctx, title, categories)
+}
+
+func searchTheCatalogueForThatTitleInCategory(t gobdd.StepTest, ctx gobdd.Context, category string) {
+	w := worldOf(t, ctx)
+
+	if strings.TrimSpace(w.searchTitle) == "" {
+		t.Fatalf("no video has been catalogued, so there is no title to search for")
+		return
+	}
+
+	searchTheCatalogue(t, w, searchRequest{
+		Title:      w.searchTitle,
+		Categories: []string{category},
+		Limit:      searchLimit,
+	})
+}
+
+func theSearchDoesNotReturnThatVideo(t gobdd.StepTest, ctx gobdd.Context) {
+	w := worldOf(t, ctx)
+
+	if w.last.status != 200 {
+		t.Fatalf("the search did not succeed: %s", w.last.summary())
+		return
+	}
+	if w.results.holds(w.video.ID) {
+		t.Errorf("the search returned video %q, which it should have left out; it returned %v",
+			w.video.ID, w.results.ids())
+	}
+}
+
+// searchTheCatalogueForThatTitleInCategories is the several-categories form,
+// for pinning that they mean "any of" rather than "all of".
+func searchTheCatalogueForThatTitleInCategories(t gobdd.StepTest, ctx gobdd.Context, categories string) {
+	w := worldOf(t, ctx)
+
+	if strings.TrimSpace(w.searchTitle) == "" {
+		t.Fatalf("no video has been catalogued, so there is no title to search for")
+		return
+	}
+
+	names := []string{}
+	for _, name := range strings.Split(categories, ",") {
+		if name = strings.TrimSpace(name); name != "" {
+			names = append(names, name)
+		}
+	}
+
+	searchTheCatalogue(t, w, searchRequest{
+		Title:      w.searchTitle,
+		Categories: names,
+		Limit:      searchLimit,
+	})
+}
+
+// theCatalogueHoldsVideosTitled catalogues several videos under one shared
+// title, which is what a paging scenario needs: enough matches to fill more
+// than one page.
+//
+// The nonce is generated once and shared, rather than per video, so that
+// searching for it matches this scenario's videos and no others. Runs leave
+// their videos behind, so a term that also matched a previous run's would make
+// "the search returns 2 videos" mean nothing.
+func theCatalogueHoldsVideosTitled(t gobdd.StepTest, ctx gobdd.Context, count int, title string) {
+	w := worldOf(t, ctx)
+
+	w.searchTitle = uniqueTitle(title)
+	w.cohort = nil
+	w.pageSeen = nil
+
+	for i := 0; i < count; i++ {
+		requestUploadSlot(t, ctx, "searchable.mp4", "video/mp4")
+		uploadTheFile(t, ctx)
+		registerUploadedVideo(t, ctx, w.searchTitle, "other")
+
+		if w.last.status != 201 {
+			t.Fatalf("could not register video %d of %d to page through: %s", i+1, count, w.last.summary())
+			return
+		}
+
+		mediaConvertReportsState(t, ctx, "COMPLETE")
+		theCatalogueEventuallyHoldsThatVideo(t, ctx)
+
+		w.cohort = append(w.cohort, w.video.ID)
+	}
+}
+
+func searchTheCatalogueForThatTitleAPageAtATime(t gobdd.StepTest, ctx gobdd.Context, size int) {
+	w := worldOf(t, ctx)
+
+	w.pageSeen = nil
+	searchTheCatalogue(t, w, searchRequest{Title: w.searchTitle, Limit: size})
+	recordPage(w)
+}
+
+// followTheCursor asks for the page after the one just returned, with the same
+// term and page size — a cursor says where to carry on from, not what to look
+// for.
+func followTheCursor(t gobdd.StepTest, ctx gobdd.Context) {
+	w := worldOf(t, ctx)
+
+	if w.results.NextCursor == "" {
+		t.Fatalf("the last page offered no cursor, so there is no next page to follow")
+		return
+	}
+
+	searchTheCatalogue(t, w, searchRequest{
+		Title:  w.searchTitle,
+		Limit:  len(w.results.Videos),
+		Cursor: w.results.NextCursor,
+	})
+	recordPage(w)
+}
+
+// recordPage remembers what a page held, so a later step can check the pages
+// tile the results.
+func recordPage(w *world) {
+	w.pageSeen = append(w.pageSeen, w.results.ids()...)
+}
+
+func theSearchReturnsNVideos(t gobdd.StepTest, ctx gobdd.Context, count int) {
+	w := worldOf(t, ctx)
+
+	if w.last.status != 200 {
+		t.Fatalf("the search did not succeed: %s", w.last.summary())
+		return
+	}
+	if len(w.results.Videos) != count {
+		t.Errorf("expected %d videos in the page, got %d: %v", count, len(w.results.Videos), w.results.ids())
+	}
+}
+
+func theSearchOffersACursor(t gobdd.StepTest, ctx gobdd.Context) {
+	w := worldOf(t, ctx)
+
+	if strings.TrimSpace(w.results.NextCursor) == "" {
+		t.Errorf("the page offered no cursor, so there is no way to ask for the next one")
+	}
+}
+
+func theSearchOffersNoFurtherCursor(t gobdd.StepTest, ctx gobdd.Context) {
+	w := worldOf(t, ctx)
+
+	if w.results.NextCursor != "" {
+		t.Errorf("the last page still offered cursor %q, so a reader cannot tell they have reached the end",
+			w.results.NextCursor)
+	}
+}
+
+// thePagesTileTheCohort checks the pages between them held each of the
+// scenario's videos exactly once — the property a cursor exists to give, and
+// the one an offset loses as soon as the catalogue is written to.
+func thePagesTileTheCohort(t gobdd.StepTest, ctx gobdd.Context) {
+	w := worldOf(t, ctx)
+
+	seen := map[string]int{}
+	for _, id := range w.pageSeen {
+		seen[id]++
+	}
+
+	for _, id := range w.cohort {
+		switch seen[id] {
+		case 1:
+		case 0:
+			t.Errorf("video %q was catalogued but appeared in none of the pages; the pages held %v",
+				id, w.pageSeen)
+		default:
+			t.Errorf("video %q appeared in %d pages; a video should be on exactly one", id, seen[id])
+		}
+	}
+
+	if len(w.pageSeen) != len(w.cohort) {
+		t.Errorf("the pages held %d videos between them, but %d were catalogued: %v",
+			len(w.pageSeen), len(w.cohort), w.pageSeen)
+	}
+}
+
+func searchTheCatalogueFromCursor(t gobdd.StepTest, ctx gobdd.Context, cursor string) {
+	w := worldOf(t, ctx)
+	searchTheCatalogue(t, w, searchRequest{
+		Title:  w.searchTitle,
+		Limit:  searchLimit,
+		Cursor: cursor,
+	})
+}
+
+// searchTheWholeCatalogue searches with no term and no categories, which is
+// how a reader browses rather than searches.
+func searchTheWholeCatalogue(t gobdd.StepTest, ctx gobdd.Context, size int) {
+	searchTheCatalogue(t, worldOf(t, ctx), searchRequest{Limit: size})
+}
+
+func theSearchReturnsAtMostNVideos(t gobdd.StepTest, ctx gobdd.Context, most int) {
+	w := worldOf(t, ctx)
+
+	if w.last.status != 200 {
+		t.Fatalf("the search did not succeed: %s", w.last.summary())
+		return
+	}
+	if len(w.results.Videos) > most {
+		t.Errorf("the search returned %d videos, more than the %d it should cap at",
+			len(w.results.Videos), most)
+	}
+}

tests/suite_test.go

@@ -48,6 +48,24 @@ type world struct {
 	// catalogued is the read side's own copy of the video, once a step has
 	// waited for it to arrive.
 	catalogued *catalogueVideoBody
+
+	// searchTitle is the exact title the scenario's video was catalogued
+	// under. Scenarios name a plain title like "Desert Falcons"; the step
+	// registering it appends a nonce, because nothing cleans up between runs
+	// and a title reused across runs would make a result count meaningless.
+	searchTitle string
+
+	// results is the last page the catalogue search answered with.
+	results searchResultsBody
+
+	// pageSeen accumulates the ids from every page a paging scenario has
+	// walked, so it can assert the pages tile the results rather than
+	// repeating or dropping one.
+	pageSeen []string
+
+	// cohort is the ids of the videos a scenario catalogued together under one
+	// shared title, in the order they were created.
+	cohort []string
 }
 
 type worldKey struct{}
@@ -186,6 +204,24 @@ func TestVideoUpload(t *testing.T) {
 	suite.AddStep(`^the catalogue does not hold that video$`, theCatalogueDoesNotHoldThatVideo)
 	suite.AddStep(`^the catalogue eventually holds that video with no playback URL$`, theCatalogueEventuallyHoldsThatVideoWithNoPlaybackURL)
 	suite.AddStep(`^the catalogue's copy carries its title, playback URL and categories$`, theCataloguesCopyCarriesTheDetails)
+	suite.AddStep(`^the catalogue holds a video titled "([^"]*)"$`, theCatalogueHoldsAVideoTitled)
+	suite.AddStep(`^I search the catalogue for that video's title$`, searchTheCatalogueForThatTitle)
+	suite.AddStep(`^I search the catalogue for "([^"]*)"$`, searchTheCatalogueFor)
+	suite.AddStep(`^the catalogue holds a video titled "([^"]*)" filed under "([^"]*)"$`, theCatalogueHoldsAVideoTitledUnder)
+	suite.AddStep(`^I search the catalogue for that video's title in category "([^"]*)"$`, searchTheCatalogueForThatTitleInCategory)
+	suite.AddStep(`^I search the catalogue for that video's title in categories "([^"]*)"$`, searchTheCatalogueForThatTitleInCategories)
+	suite.AddStep(`^the catalogue holds (\d+) videos titled "([^"]*)"$`, theCatalogueHoldsVideosTitled)
+	suite.AddStep(`^I search the catalogue for that title (\d+) at a time$`, searchTheCatalogueForThatTitleAPageAtATime)
+	suite.AddStep(`^I search the whole catalogue (\d+) at a time$`, searchTheWholeCatalogue)
+	suite.AddStep(`^I follow the cursor$`, followTheCursor)
+	suite.AddStep(`^I search the catalogue for that video's title from the cursor "([^"]*)"$`, searchTheCatalogueFromCursor)
+	suite.AddStep(`^the search returns (\d+) videos?$`, theSearchReturnsNVideos)
+	suite.AddStep(`^the search returns at most (\d+) videos$`, theSearchReturnsAtMostNVideos)
+	suite.AddStep(`^the search offers a cursor to the next page$`, theSearchOffersACursor)
+	suite.AddStep(`^the search offers no further cursor$`, theSearchOffersNoFurtherCursor)
+	suite.AddStep(`^the pages together hold every one of those videos exactly once$`, thePagesTileTheCohort)
+	suite.AddStep(`^the search does not return that video$`, theSearchDoesNotReturnThatVideo)
+	suite.AddStep(`^the search returns that video$`, theSearchReturnsThatVideo)
 	suite.AddStep(`^I register the uploaded video titled "(.*)" under categories "(.*)"$`, registerUploadedVideo)
 	suite.AddStep(`^I register the uploaded video titled "(.*)" under category id (\d+)$`, registerUnderCategoryID)
 	suite.AddStep(`^I register a video titled "(.*)" with the key "(.*)" under categories "(.*)"$`, registerWithKey)