Skip to content

ADR-028: Server-owned fields on whole-document aggregates

Status

In-Review, 2026-11-05. Delivered by the four stacked PRs of #4771's stored PDF classification, starting with #4838. The owner chose this design on 5 November 2026 ("Exact, split into stacked PRs").

Context

The Studies workspace's PDF filter and per-search counts classify every study in C# (#4719). At 215k studies that takes 2.7–4.8 times as long as the unfiltered list (#4831). Storing the classification's inputs on pmStudy makes each option an indexed predicate. But pmStudy has three properties that a naive denormalisation would break on:

  1. Studies are saved by whole-document replace. Review saves, screening, reservations and the unit of work ReplaceOne or FindOneAndReplace the Study aggregate. A field written by another process (here, the PDF check, which finishes in the background just after a review retrieval) is erased or written back stale by the next save. Bumping Audit.Version with that write would make the saves conflict instead. Slice 8 kept the check result in its own collection to avoid exactly this.
  2. A build that doesn't map a field writes it back stale. Every entity captures unknown elements in ExtraElements ([BsonExtraElements] on Entity), and a whole-document save writes them back exactly as loaded. A build that predates a field therefore carries back whatever value it loaded, however stale, and a build that maps a field without serialising it drops it. ADR-023 keeps compatibility for the rollout window, when the API and PM can run different builds.
  3. ADR-020 hashes the whole stored study. Bulk Study Update compares StudyContentHash before and after its writes, and before a rollback. Any field that changes on a locked study fails the apply or quarantines the rollback.

Decision

  1. Server-owned fields. A field written by a process other than the aggregate's own save is server-owned. The aggregate maps it so it's read onto the aggregate (not captured in ExtraElements), and never serialises it. For #4771 these are PdfCheck (a mirror of the last recorded PDF check) and PdfCheckState (derived from it).
  2. Derived fields ride with their inputs. A field computed from the aggregate's own fields is a get-only computed property, explicitly mapped, as SessionTally.TotalAllocatedSessionCount already is. A whole-document save then always carries a value consistent with the fields it was computed from. For #4771 these are PdfLinkKind and PdfLinkKey.
  3. Preserve on replace. While server-owned fields exist (armed, decision 5), every whole-document write of the aggregate becomes an update pipeline:
  4. $replaceWith the literal document merged with the stored server-owned fields;
  5. then the stage that re-derives whatever depends on both sides.

Filters, upserts and results are unchanged. One seam registered per aggregate type, in the shape of AggregateWriteGuards, chooses the write model. Every direct replace call site goes through it, and a source-scan test fails on a new one. 4. Writers recompute in the same update. A targeted writer of a derived field's inputs, or of a server-owned field, sets the derived values and runs the derive stage in the same single-document update, with values as $literal. A source-scan test fails on a writer that isn't on its allow-list. 5. Arming by schema, not by flag. The new fields are written only once their index migration has run in that environment: the writers check for the index, as slice 8's admission checks for 0034. The migration is registered (0036 and 0037, #4873) only after builds that preserve and re-derive the fields are deployed. So no pod that would write them back stale, or drop them, serves while they're written, and production, which runs no ADR-023 migrations yet, is untouched. Arming reads no flag, and no migration reads a flag (ADR-023, flags amendment). The feature's runtime flag gates only the read. - A host never writes before it knows (#4845 review). A pod that starts after arming must not serve a single unarmed write: its plain replaces would drop the fields from an armed, backfilled database. So a host that writes the aggregate hosts the arming check as its first hosted service, and that service's start returns only with a definite answer, retrying a failed check with backoff. The host starts its later hosted services (consumers, schedulers) and its web server only after that start returns. A write that runs before the host starts (an init task such as project management's seeder) awaits the same first answer before it runs. - Fail closed. A write that asks while its host has no answer yet fails with a retryable exception and writes nothing. A database that no host in the process checks (a test, a tool) is unarmed, as before. 6. Hash exclusions. Server-owned and derived fields are excluded from StudyContentHash (ExcludedTopLevelFields): - server-owned fields are bookkeeping that rollback never restores; - derived fields are fully determined by hashed fields.

The bulk writer re-derives them after apply and after restore. 7. The change stream ignores them. Updates that touch only server-owned or derived fields are dropped in the study change stream's $match, so they cost no realtime traffic and no lookup read.

Consequences

  • Armed, every study save is a pipeline update, not a replace. Unarmed, it's today's ReplaceOne, byte for byte. That includes production, until the owner runs ADR-023 migrations there.
  • The next denormalised field on pmStudy, or on any other whole-document aggregate, follows this pattern: register it as server-owned or derived, extend the derive stage, the hash exclusions and the change-stream filter, and extend the writer allow-list.
  • A rollback, after arming, to a build that doesn't preserve the fields would let its saves write back stale values (a build older than #4838) or drop them (#4838 itself). ADR-023's forward-only deployments make that a planned compensation (re-running the backfill), not a supported state.

Alternatives considered

Alternative Why not
The check bumps Audit.Version Review and screening saves would conflict after every check, reversing slice 8's design
No mirror; store only the link kind Available, Without a confirmed PDF and the counts still join the availability collection with large id lists: about 1.3–2 s at 215k, above the unfiltered list
The mirror carried by the aggregate, with read-repair Not exact: a stale write-back persists until the next check, and an Available result is never rechecked
Leave the fields unmapped, carried by ExtraElements A save writes back whatever it loaded, so a stale mirror and stale link fields would survive every save; and ADR-023 makes unknown-field tolerance a backstop only
Write the fields everywhere at once It would break the rollout window for older pods, and change production data and its save path while nothing there reads them

References