The technology behind Ninjatest

A Ninjatest match is nine questions — three VARC, three DILR, three Quant, or nine from a single section in a section-mode challenge. Each player traverses their own nine on their own clock. Every answer is scored on the server against a clock the server started, and the winner is decided only once both players have finished.

Architecturally the product is three layers, and the middle one is deliberately thin:

The unusual choice is layer two. There is no Node game server, no Edge Function tier, no queue worker. All game-critical logic lives in security definer Postgres functions — 95 of them across 104 migrations. The client can only call an RPC and render what comes back.

One rule governs the entire codebase: never trust the client for scoring, timing, ELO, or matchmaking. It sounds obvious and it is violated by almost every quiz product on the internet, usually in the same place — the timer.

A naive implementation sends timeTakenMsfrom the browser. That single field is the whole exploit: patch it to 400ms and every answer earns maximum speed bonus. In Ninjatest the client never sends a duration. The server wrote the question's start timestamp when it served the question, and computes elapsed time itself at submission:

taken = clamp(now() − q_started_<player>, 0, cap)

The same principle removes the answer key from the wire. The question-serving RPC strips correct_index and explanationbefore the row ever leaves Postgres, so "inspect the network tab" returns exactly what the UI shows. The explanation arrives only in the reveal call, after the answer is locked.

Forfeits work the same way. A client claiming "my opponent left" proves nothing, so forfeit_match requires a server-verifiable absence: the opponent must have missed a full question deadline plus a five-second grace with no submitted row, or already carry a cron-written timeout marker on the previous question. The client retries the call every ten seconds and the server keeps refusing until its own evidence is satisfied.

Pairing is the classic concurrency trap: two players poll at the same millisecond and both match with the same third player. The queue avoids it with a single atomic statement using FOR UPDATE SKIP LOCKED — the row a concurrent transaction is already claiming is skipped rather than waited on, so pairing never double-books and never deadlocks under load.

The rating band widens with wait time:

band = min(1000, 100 + wait_seconds × 20)   -- per player
match if |elo_a − elo_b| ≤ max(band_a, band_b)

A fresh entrant sees only ±100 ELO. After 45 seconds the window is ±1000 and effectively anyone. This trades match quality for queue latency on a curve rather than a threshold.

Three guards sit on top of the pairing:

• Rated-pair cap. The same two accounts can play at most three ratedmatches per day. Without it, two friends farm rating off each other indefinitely. Never-rated abandons don't count toward the cap.
• Live-match rejection. join_queue refuses a caller already inside an active match, which is also the definition the AI layer reuses.
• Exactly-nine guard. A match is refused unless the frozen question array holds exactly nine ids. If a section is too thin to fill its slots, the picker adaptively rolls those slots into another section rather than shipping a short array — because a short array lets the question index run past the end mid-play and corrupts the match. On a genuine content gap the caller simply stays queued rather than erroring.

Ninjatest originally ran matches in lockstep: one shared question index, both players advanced together, and the slower player dictated the pace. It was simpler and it was wrong. Reading speed is part of what a CAT aspirant is training, and lockstep either punishes the fast player with dead waiting time or rushes the careful one.

The current engine gives each player their own clock and their own position. The critical design decision was to make position derived, not stored:

position(user) = count(match_answers where match_id = m and user_id = u)
                 -- 0..9, and 9 means finished

There is no per-player index column, so there is nothing that can desynchronise from the rows that actually exist. A row written is a question advanced, by construction. Two columns hold the per-player question clocks; the serving and scoring RPCs read the caller's clock and advance only the caller. The match finalises when — and only when — both players hold nine answers.

Bot matches deliberately kept the old shared-index path. The bot has no independent clock to run and its difficulty is derived from the question's own rating, so lockstep is both correct and cheaper there. All three lifecycle RPCs branch on whether a bot is present.

Scoring is entirely server-side and every dial lives in a config table, never in application code. The per-question calculation:

base    = coalesce(question.duration_ms, section_cap_ms)
cap     = base + reading_ms       -- only for the first question of a passage group
taken   = clamp(now() − question_started_at, 0, cap)
bonus   = round(speed_mult × floor(least(cap − taken, base) / 5000))

correct → base_points + bonus
wrong   → −round((base_points + bonus) / (n_options − 1))
skipped → 0

Those multipliers aren't aesthetic. They were solved for section parity: with different time caps, a flat multiplier would make one section worth more per question than another, and rating would then measure which section you drew rather than how well you played. Tuned, every section maxes at 140 points per question.

The reading window is bonus-free by design. The first question of a passage group gets extra time to read the passage, but the bonus is computed against the base cap, so the extension is grace rather than free points.

The wrong-answer penalty is derived, not configured

This is the piece we're proudest of. The penalty started as a config value — a flat −30. On four-option questions that made instant blind guessing worth roughly +9 expected points per question. A snap-guess bot would have out-scored a careful human.

The penalty is now derived from the same speed curve as the reward, which makes a random guess exactly EV-neutral at every point in time and at any option count:

E[guess] = p_correct × (base + bonus)  −  (1 − p_correct) × (base + bonus)/(n − 1)
         = (1/n)(base + bonus)  −  ((n−1)/n)(base + bonus)/(n − 1)
         = 0

Guessing is worth exactly zero, whenever you do it, regardless of how many options the question has. As a bonus the ratio converges on CAT's own 1:3 marking scheme at the time cap. The retired config column is still in the table, unread — deleting it is a migration we haven't needed.

Ratings use a modified Elo with a margin-of-victory term and a favourite correction:

K        = 40 if games < 30 else 24 if elo < 2000 else 16
E_winner = 1 / (1 + 10^((R_loser − R_winner) / 400))
factor   = 0.3 + 0.7 × min(|score_margin| / FULL, 1)
factor  *= 2.2 / (0.001 × (R_winner − R_loser) + 2.2)   -- only if winner was favourite
Δ        = max(0, round(K × (1 − E_winner) × factor))
Δ_eff    = max(0, min(Δ, R_loser − 100))                -- rating floor
winner += Δ_eff ;  loser −= Δ_eff                       -- strictly zero-sum

• Margin matters, but not linearly. A 0.3 floor means a narrow win still moves rating; the remaining 0.7 scales with how decisive the win was, normalised against the match's own maximum possible margin so section-mode and mixed matches are comparable.
• Favourite-shrink. Margin-of-victory systems systematically overrate favourites, because a strong player's margins are autocorrelated with their strength. We apply the FiveThirtyEight-style correction to favourite wins only; underdog wins are untouched.
• Strictly zero-sum, with a floor. The transfer is one number applied in both directions, clamped so no one falls below 100. A consequence we accepted deliberately: beating a floored opponent gains nothing, which removes the incentive to hunt them.
• No forced +1. An earlier version guaranteed at least one point per win. That is a rating-printing press when you play many far-weaker opponents, so it was removed.
• Ratings are read at finalisation. Both players' current ratings are read under ordered row locks when the match ends, not snapshotted at match creation, so overlapping matches chain correctly instead of applying to a stale base. Ordered locking is what stops two simultaneous finalisations from deadlocking.
• Draws share one K. The lower of the two K-factors, applied once. Per-player Ks used to mint roughly +11 rating out of thin air on every mismatched draw.

Seasons apply a soft reset — elo = 1000 + (elo − 1000) / 2 — snapshotting the old standings to a results table and writing a reset marker into the rating history so the profile graph shows the discontinuity honestly rather than pretending it was a loss streak.

Every question carries its own rating, and it moves. When a player answers in a rated or bot match, one atomic UPDATE nudges the question's rating in the opposite direction of the outcome:

result = 1.0                      if wrong
       = 0.35 × (taken_ms / cap)   if correct   -- fast + correct ⇒ question is easy
K      = 32 while times_seen < 20 else 16
clamp  [400, 2800]

Over roughly twenty serves, a question converges on the rating at which players get it right half the time — an empirical difficulty measurement rather than an author's guess. Selection then biases toward the players' mean rating with random() × 300 jitter, so matches are challenging without being deterministic.

Three exclusions keep the signal clean:

• Answers faster than two seconds are discarded, correct or wrong. They're either a misclick or an automation attempt, and both would corrupt the difficulty curve.
• Unrated player-vs-player matches never nudge. An uncapped unrated challenge between two colluding accounts was a direct write channel into the question bank.
• In bot matches only the human's answer nudges. The bot's behaviour is derived from the question rating, so letting it write back would be a feedback loop measuring itself.

The starvation cliff: why selection can't be rating alone

CAT has two question types: multiple-choice, and TITA — Type In The Answer, with no options and therefore no guess floor. Identical content is genuinely harder as TITA, so TITA seeds sit 100 rating points above the MCQ scale. We sized that offset from the nudge's own steady state rather than guessing:

E[result] = 1 − 0.8p        (p = probability the player knows it)
gap(p)    = −400 × log10(0.8p / (1 − 0.8p))
          = +133 / +70 / +37   at p = 0.25 / 0.50 / 0.75

Then the offset created a worse problem. With selection driven by rating alone, on the live bank — 1,090 MCQ rated 1100–1500 against 52 TITA rated 1300–1600 — the measured TITA count per Quant block was:

A cliff, not a gradient. Below 1400 a player never saw a TITA at all — and the failure was self-sealing, because a question that is never served is never re-rated and can never migrate out of the dead zone. Rating is a one-dimensional signal and question type is a second dimension it cannot express.

The fix is a hard quota: one of every three Quant slots is reserved for TITA, with rating still choosing whichTITA. That holds ~33% at every rating, close to CAT's own ~36%. The quota lives in exactly one function so it can't drift between the match, challenge, and practice paths — a lesson learned when the practice drill silently served zero TITA across five sessions for precisely this reason.

Three realtime primitives are used, and the split between them is a trust boundary rather than a performance choice.

Broadcast never carries game-relevant data. It tells you your opponent has moved — not what they scored, not whether they were right. If the WebSocket layer were compromised tomorrow, the worst available outcome is a false liveness ping.

Spectators subscribe to a broadcast-only channel and are deliberately excluded from presence, so an audience arriving or leaving can never influence the forfeit logic of the players below.

And because realtime is a transport rather than a source of truth, every client rehydrates from the database on reconnect, on channel error, and on the browser's onlineevent. A dropped message costs a render, never a state divergence. A polling fallback backstops the queue for the case where the realtime publication doesn't cover a table at all — which is exactly how friend challenges work today.

Rating is only worth defending if it is hard to fake. The defences are layered, and each one closes a hole we actually found.

Per-player option shuffling

Two players in the same match see the same question with options in different orders, generated by a deterministic, IMMUTABLE permutation keyed on match, user, and question index. Nothing is stored; it is recomputed identically wherever it is needed. Screen sharing therefore doesn't transfer an answer — "it's C" is meaningless across two players.

Three functions must agree on that permutation: the one that serves shuffled options, the one that maps a submitted display index back to canonical, and the one that maps the canonical correct answer back to display for the reveal. A migration once recreated two of them from a pre-shuffle copy and silently desynced scoring — correct answers marked wrong, no error anywhere. There is now a dedicated invariant test that fails if the three ever disagree.

Row-level isolation of answers

Answer rows are readable by their owner only. The obvious policy — participants of the match can read the match's answers — leaked in a way that took a second look to see: with TITA, the typed answer is stored as plaintext, so whoever answered first handed the other player the answer while the question was still open. Every opponent-facing read (reveal, debrief, spectator view) goes through a definer function that applies its own timing rules, so tightening the policy cost nothing.

Clock synchronisation

The client renders a countdown, but it renders it against a measured offset from server time, sampled at the midpoint of the request so it corrects for absolute clock skew rather than just round-trip latency. At zero the client auto-submits a null. The display is a convenience; the server's own timestamp is what scores.

Behavioural telemetry

Clients may report exactly two events — tab_hidden and window_blur — from a server-side whitelist, rate-limited. The server adds its own signal the client can neither see nor suppress: fast_answer, recorded whenever a correct answer arrives in under two seconds. Alone each is noise; correlated across a match they are a legible pattern, surfaced to an internal review query.

Ninja AI is the coaching layer: a stat-grounded chat coach, a Socratic hint mode, per-match debriefs, a daily focus line, a seven-day study plan, and a PDF solver. Six user-facing routes, each doing its own authentication, its own per-user and per-IP rate limiting, and its own guard checks inline.

The rule that shapes everything: no LLM while you are playing

An AI that can solve CAT questions, inside a product where solving CAT questions faster than another human is the entire point, is a cheating engine unless it is gated. So every user-facing AI route is blocked while the caller is in a live match.

The important detail is what the gate keys on. It keys on the caller, not on the match the request names. Each per-match guard only inspects the match in its own arguments — so mid-match, a request pointed at an old, completed match passes those guards untouched. Second tab, live match running, LLM solving questions. The two mechanisms are not redundant copies of each other; they stop different attacks, and neither can be removed as duplicate.

The gate fails closed: if the check itself errors, the call is blocked. The widest holes were the routes taking arbitrary input — the chat coach (paste the live question) and the PDF solver (screenshot it into a document). The daily-focus route takes no input and isn't a cheat channel at all, but it rides the same rule so there is exactly one definition of "live" in the codebase — the same one matchmaking uses.

Cost engineering

LLM calls are the only unbounded-cost operation in the product, and they are treated as a resource to be budgeted rather than a feature to be sprinkled.

• Attempt ceilings are pre-spend. The three-explanations-per-question cap is checked before the model is called, not after. It is the only reason a fifteen-per-minute rate limit is financially safe.
• Caches are cost controls. Debriefs, daily focus, and study plans are stored on first write. Re-reading is free forever. They look like conveniences and are actually the billing model.
• The coach's data is bounded in app code. The agentic coach replays every prior tool result at each of up to six reasoning steps, so an unbounded rating history costs tokens quadratically in conversation length and linearly in career length. The rating curve handed to the model is capped at 30 points — measured, that is 16,590 → 2,658 tokens per turn for a 200-match player, and 82,590 → 2,658 at 1,000 matches. Deliberately not fixed in SQL: the profile graph legitimately wants the whole curve. The bug was only ever in handing it to a language model. And the trim takes the tail, because taking the head would feed the model a stale trend and silently invert the advice — a rising player read as sliding.
• Learning without per-match inference. The coach's knowledge of a player comes from a bounded-by-construction aggregate — three sections × two question types × three rating bands, plus one trend object — computed in SQL. Every match teaches the coach something and no match costs an LLM call. Timeouts are excluded from accuracy rates rather than counted as skips, and season-reset markers are excluded from the trend.

Reasoning tokens are a budgeting hazard

The production model emits reasoning tokens. They bill as completion and consume the output budget before any answer text exists. Measured: a trivial arithmetic prompt spent 83 of 108 completion tokens on reasoning; a real Quant solve spent 224 of 417. At a 300-token cap the response content came back null — not an error, not a truncation warning, just empty.

So output caps are never sized to the visible answer. Routes that need one short line still request a large budget and truncate afterward. The model fallback exists specifically for this failure mode — the upstream gateway already load-balances across 28 providers, so availability is not what a fallback buys. It only helps if the retry loop treats empty text as a failure rather than a success, which one route did not, leaving its fallback as dead code until we measured it.

Every question in the bank carries a 1536-dimensional embedding in a pgvector column, backfilled by a single idempotent writer that selects only rows with a null embedding — which makes re-running it the repair path rather than a risk.

Staleness is handled by a database trigger, not by caller discipline: any real change to a question body nulls its embedding, from any writer, including ones that don't exist yet. The trigger is scoped so it never fires on the rating and times-seen updates that the match hot path performs.

A 140× storage lesson

pgvector defaults vector columns to EXTERNAL storage, which pushes every 6KB embedding into TOAST — so an exact scan pays a detoast per row. Measured at 1,247 rows: 677ms and 7,693 buffers TOASTed, versus 4.8ms and 1,372 buffers inline.

Two traps in the fix. First, altering the storage mode only governs rows written after the alter, so the existing bank stays TOASTed and the improvement never materialises. Second — and this one is genuinely subtle — update questions set embedding = embedding is a silent no-op: assigning a column to itself passes the same TOAST pointer through, Postgres sees an unchanged external datum, and never re-toasts it. Forcing a fresh datum requires a round trip through the type: embedding = (embedding::text)::vector(1536), verified lossless across all rows before it was run.

There is no HNSW index. At this bank size the exact scan is fast enough and has no recall loss — an approximate index would be complexity bought with accuracy, for a latency win nobody would feel.

RLS is enabled on every table. The interesting policies are the restrictive ones:

Every policy wraps the auth-uid call as a scalar subquery. That single syntactic detail moves it into the query's InitPlan so it evaluates once per query instead of once per row — a linear-to-constant improvement on any policy-covered scan, and one the Supabase linter flags if it drifts back.

Definer functions carry their own discipline: an explicit search path pinned inline on each function (a blanket pin is dropped the moment anything replaces the function), execute revoked from public and anon, then granted explicitly to the roles that need it. Read-only public functions — leaderboard, profiles — stay anon-callable on purpose so logged-out pages can be cached.

Two independent durable layers, both backed by Postgres tables rather than process memory — an in-memory limiter is per-instance, and serverless instances multiply.

• Per-user, inside the RPC. The check is the first statement of the function and raises on exceed, so the limit and the operation share one transaction and cannot be raced apart.
• Per-IP, at the route handler. Returns a retry-after rather than throwing, and fails open on infrastructure error — deliberately, for unauthenticated endpoints where a limiter outage must not become an outage.
• The metered AI routes fail closed. Opposite decision, opposite reason: fail-open on a route that costs real money per call is a billing incident.

The API routes sit outside the auth proxy's matcher entirely, which is a deliberate architectural fact rather than an oversight — and it means each handler is fully responsible for its own authentication, its own gating, and its own limits. That is the only thing standing between a public domain and the most expensive call in the product.

• Local JWT verification in the auth proxy. Claims are verified locally against asymmetric signing keys instead of making an auth-server round trip on every single request. This is the single largest per-request cost reduction in the app.
• The onboarding gate costs one lookup per session. A completed profile is cached in a cookie, so the database check happens at most once rather than on every authed navigation — which is what keeps the optimisation above from being immediately undone.
• Public pages are cookieless and cached. The leaderboard and profiles render through a session-less client with incremental static regeneration. "You"-highlighting is resolved client-side specifically so the page itself stays cacheable for everyone.
• Authed pages are never bfcached. A no-store header on authenticated responses, because the browser's Back button restores the last rendered page without re-running middleware — which after logout means an authed screen on screen.
• Foreign keys are indexed. Nine were added in one hardening pass after the linter flagged them; unindexed FKs turn every cascading delete into a sequential scan.

Three cron jobs run inside Postgres. No external scheduler, no worker fleet.

The timeout drain writes its skip rows with a null elapsed time. That null is not missing data — it is the marker distinguishing a cron-written timeout from a genuine client submission, which always records a duration. Several downstream systems key on it: forfeit evidence, debrief accuracy, and the coach's learner profile, which excludes timeouts from accuracy rates rather than counting them as deliberate skips.

Game-critical logic is verified by SQL harnesses that run inside a transaction and roll back — real rows, real functions, real constraints, no residue. They assert the properties, not the implementations:

• Rating transfers are strictly zero-sum, and never breach the floor.
• Guessing is EV-neutral, at every point on the time curve and at every option count.
• The three option-shuffle functions agree — the specific regression that once shipped silently.
• Exactly one time-cap source is consulted by every function in a match context, so timers cannot desync.
• Self-paced progression: each player advances independently, and the match finalises only when both hold nine answers.
• Every user-facing AI route is blocked mid-match, including when it points at an unrelated completed match.

Alongside them: a deterministic Monte-Carlo simulation backing the question-rating constants, a zero-network self-test for the prompt-construction branches, a self-test for the coach's token cap that asserts both the cap and that the trim takes the tail, and a live end-to-end probe that spends about $0.004 to grade a real model answer against a real key.

One repository builds two production Vercel projects from the same branch, differing only by environment variable. The public waitlist site and the full battle app are, by construction, the identical commit — the front door is chosen by one variable at runtime, not by a code branch or a long-lived release branch.

That means the launch is the removal of one environment variable: no deploy, no merge, no migration. It also means an accidental deletion of that variable launches the product, which is precisely the kind of fact that belongs written down rather than discovered.

An earlier setup used a build-ignore guard to freeze the public site while the trunk moved. It was deleted once both domains were meant to track the trunk — a guard permanently defeatable by an environment variable, and invisible in the repository, is worse than no guard, because it is exactly the failure it was written to prevent.