Skip to content

Operate the checked PDF pipeline (API side)

This is the operator's guide to the API half of isolated PDF checking (slice P2 of the isolated PDF checking plan). The checker service itself (pdf-checker, slice P1) has its own chart and documentation. Nothing here is switched on anywhere: the flag is off in every environment, the private listener is off by default, and no environment sets a scanner yet.

What the API does

A user proposes a replacement PDF for a study. The API keeps the bytes in quarantine (MongoDB, at most 8 MiB), scans them itself with the environment's private clamd, and offers only clean-scanned bytes to the isolated checker through a private listener. The checker reports bounded facts (pages, rendering, encryption, active content). The API decides the verdict from its own scan plus those facts. Nothing is previewable or approvable until the API has decided "checked", and approval (P3) publishes exactly the checked bytes.

Piece Where
Feature flag checkedPdfProposals (Release, default off, deployment-managed, not production-eligible) src/charts/syrf-common/env-mapping.yaml, feature-definitions.yaml
Admission: the flag and migration 0039's lease index CheckedPdfAdmission (SyRF.ProjectManagement.Mongo.Data)
Proposals and quarantined bytes pmCheckedPdfProposal, pmCheckedPdfFile (CheckedPdfProposalRepository)
Checker sessions, the activation record, cap-refusal runs pmCheckedPdfCheckerSession, pmCheckedPdfCheckingState (CheckedPdfCheckingStore)
The API's clamd client src/services/api/SyRF.API.Endpoint/CheckedPdf/ClamdPdfVirusScanner.cs
The propose route (Studies origin; the screen is P3) POST /api/projects/{projectId}/studies/{studyId}/pdf-proposals/checked
The private checking listener CheckingListener and CheckingEndpoints, /checking/v1/* on the checking port only

Switches and gates

A new proposal is admitted only when every one of these holds. Each fails closed: anything missing means refused (503 pdf_checking_unavailable) before any claim.

  1. checkedPdfProposals decides on (otherwise the propose route is unroutable, 404, and every checking call answers 404).
  2. studiesWorkspace decides on (the Studies surface flag; otherwise the propose route answers 404).
  3. Migration 0039 (0039_pmCheckedPdfProposal_State_Stage_LeaseId_OfferedAtUtc) has run. Production runs no pre-deploy migrations yet, so even a mistaken flag-on there leaves admission closed (acceptance criterion 19). Once the index exists, admission opens on the next call without a restart; after a Down, restart the API to close it again.
  4. The scanner is ready: a fresh VERSION (signatures at most 72 hours old, at most 5 minutes in the future) and the EICAR canary returning FOUND, cached for up to 60 seconds. After any ERROR, timeout or truncated reply the API sends clamd SHUTDOWN and stays closed until a fresh check passes. This never feeds the API pod's Kubernetes probes.
  5. An inspect checker has polled (lease) in the last 2 minutes on a live session.
  6. The proposer has fewer than 3, and the environment fewer than 50, proposals checking (429 pdf-checking-busy).

Chart values (API chart)

Nothing renders unless an environment sets these, so the production API render is unchanged.

checkedPdf:
  clamdHost: pdf-checker-clamd        # the pdf-checker chart's clamd Service in the same namespace
  clamdPort: 3310
  checkingListener:
    enabled: true                     # opens the checking port and renders the syrf-api-checking Service
    port: 8090
    serviceName: syrf-api-checking
    inspectSessionCap: 2              # the inspect replica count plus one for the rolling-update surge
    allowedCheckerLabels:             # the pods the checking port's NetworkPolicy admits
      app.kubernetes.io/name: pdf-checker-inspect
    inspectCredentials:               # each accepted inspect bootstrap Secret, by name and key
      - secretName: pdf-checker-inspect
        key: token
  • clamdHost alone renders SYRF__CheckedPdf__ClamdHost and SYRF__CheckedPdf__ClamdPort.
  • checkingListener.enabled renders the named checking container port, the syrf-api-checking ClusterIP Service (in no ingress), a syrf-api-checking NetworkPolicy and the SYRF__CheckedPdf__Checking__* settings. The NetworkPolicy admits only pods labelled checkingListener.allowedCheckerLabels (default app.kubernetes.io/name: pdf-checker-inspect) to the checking port and keeps the application and metrics ports open to everyone, as today. It is declared intent until the cluster enforces NetworkPolicy (P8). Each credential arrives as a secretKeyRef; no value is ever written in values files. The chart refuses a checking port equal to the application or metrics port.
  • Helm tests: src/services/api/.chart/tests/checked_pdf_test.yaml.

The private listener

  • Requests that arrive on the checking port are answered in a branch that runs before request logging, routing, CORS, authentication and MVC; the port serves nothing else. The checking calls are not MVC actions, so the public port answers 404 for them whatever the Host header says (CheckingListenerTests).
  • The bootstrap credential opens a session; the role comes from which configured credential matched, never from the request. Every other call carries the session token. An inspect token may call session/renew, lease, bytes and result; a fetch token (P9, not configured in the MVP) only session/renew, lease and fetched. Anything else is 403.
  • Sessions are renewed every 60 seconds, are stale after 120 seconds without renewal and expire 15 minutes after the last renewal. Only live sessions count towards the role's cap. A new session for a checkerId that has any session, live or stale, replaces it in place with no cap check (a compare-and-set on the old token, so two API replicas cannot both issue one; the loser answers 429), and the old token answers 401 from then on.
  • A replaced session's process is gone (a container restart after a crash), so any lease it held ends at once and is handled as its expiry would be: an attempt while the flag stayed on (offered again once, then failed "checks did not complete"), not counted when it was issued before the latest switch-on. The restarted container gets the job again straight away, with no second-lease alert.
  • One lease at a time per session: each lease call takes the session's lease slot (a conditional update on the session record) while it checks for a held lease and takes a job, so concurrent calls on one session never get two jobs; a call that finds the slot taken, or a lease already held, answers 204 and counts a second-lease attempt.
  • While the flag is off every call answers 404. A checker's session, session/renew and lease calls (a configured bootstrap credential, or a token this API issued, stale or not) also mark the activation record off, at most one write per off period; an unauthenticated call writes nothing. An API pod with the listener enabled marks it at start-up as well. Expiry and attempt counting run only while the flag is on, so time spent off never blocks a job or counts as an attempt.

Contract version 1 (what P1 implements)

The plan's section 3 is the contract; these are the wire details the API fixes. The shared worked examples are in src/testing/pdf-checking-contract/v1/ (the same files as P1's PR); CheckingEndpointsTests reads them.

Item Value
Bearer Authorization: Bearer <bootstrap secret> for session; Bearer <sessionToken> for every other call
session {"checkerId"} → 200 {"sessionToken","expiresAtUtc"}, 429 at the cap, 401 for an unknown credential
session/renew no body → 200 {"expiresAtUtc"}, 401 when stale, expired or replaced
lease {"checkerId","contractVersions":[1],"toolVersions":{"qpdf":"11.9.0",...}} → 204 (no work, rate-limited, index missing, a lease already held, or no common version) or 200 {"contractVersion","jobId","attempt","leaseId","leaseExpiresAtUtc","limits":{"maxBytes","maxPages"},"sha256","size","bytesPath"}
Ids jobId and leaseId are GUIDs written as 32 hex digits; any GUID form is read back
bytes GET {bytesPath}?leaseId=<leaseId> → 200 octet-stream with exact Content-Length and X-Content-SHA256; 409 without a matching current lease
result POST /checking/v1/jobs/{jobId}/result with {"leaseId","attempt","sha256","inspection":{"outcome","pages","renderedPages","encrypted","activeContent","pdfHeaderVersion"},"failureCode","durations"} → 204 (also for an identical replay), 409 for a stale lease, wrong attempt, job not in the inspect stage or already terminal
fetched PUT /checking/v1/jobs/{jobId}/fetched: 403 for an inspect token, 409 for a fetch token (no fetch stage in this build)
Bodies at most 16 KiB (413 above), strictly typed (400), unknown fields ignored (a forged scan field has no effect)
Unknown values an unknown outcome, failureCode or active-content name reads as incomplete: blocked
Attempts incomplete with timeout, tool-crashed, output-limit or leftover-process counts an attempt like an expired lease: offered again once, then failed; tools-unavailable and scratch-unavailable block it as "checking unavailable"

Production prerequisites (written down, not executed)

None of these is done by this PR, and none may be done without Chris's approval:

  1. Enforced NetworkPolicy in the cluster (P8) and an operator security review (acceptance criterion 16).
  2. The checker and clamd Applications for production, with generated per-role secrets (External Secrets, the Valkey pattern; rotation by renamed Secret as in the plan's section 8).
  3. The API checking listener values above in production, the migration 0039 run by the pre-deploy Job, and checkedPdfProposals switched on, each a reviewed cluster-gitops change.
  4. Alert rules on the SyRF.API.CheckedPdf meter (syrf_checked_pdf_session_cap_alert, syrf_checked_pdf_live_session_replaced, syrf_checked_pdf_second_lease_attempt, syrf_checked_pdf_bytes_without_lease, syrf_checked_pdf_lease_rate_limited, syrf_checked_pdf_session_refused_at_cap) with an owner. They are not in this PR.
  5. A retention rule for quarantined, failed and blocked bytes (an open follow-up, plan section 8).

Troubleshooting

Symptom Look at
Every proposal answers 503 pdf_checking_unavailable The API log names the gate: "the virus scanner is unavailable", "no inspect checker has polled", or the index warning from CheckedPdfAdmission
The checker gets 429 from session Live sessions fill the role's cap. A lost pod's session stops counting 120 seconds after its last renewal; refusals lasting 4 minutes raise the ALERT log line and the cap-alert counter. Rotate the bootstrap secret if the cap is being filled by someone else
The checker gets 404 from every call The flag is off in this environment
A proposal is blocked with checking-unavailable No checker leased it within 10 minutes of its offer (measured from the latest switch-on), or the checker reported its tools or scratch space unavailable

Audit lines start with Checked PDF checking audit: and carry the call, status, session, checkerId, job id and SHA-256, never bytes or a credential.