Motivation
Until v8.36, AskMyDocs read what a file said it contained.PdfConverter
extracted the PDF text layer; SourceType knew Markdown, text, PDF and DOCX
and nothing else. That is the right posture for a knowledge base fed by wikis
and repositories — and the wrong one the first time a scanned contract arrives
as an IMAP attachment. The text layer is empty, the document ingests as
nothing, and an image is a 422 Unsupported file type.
A competitor audit (Annota AI)
named this the one seam where a data-preparation product was ahead of us:
file → reviewed Markdown. The v8.36 cycle closes it in two steps. This page
is the first: OCR as a converter, with the same idempotency, provenance,
PII and cost discipline every other entry point already has.
Everything on this page is off by default (
KB_OCR_ENABLED=false, R43).
With the flag off the system behaves exactly as v8.35: images are refused at
every entry point and a scanned PDF ingests as an empty document. The design
rationale is ADR 0029.Theory & background
OCR is not one problem. A text-less PDF, a phone photo of a whiteboard and a 300-dpi scan of a table each want a different engine, and the quality gap between engines is wider than the gap between text-layer parsers:
Two facts follow from that table and shape the whole design:
- The engine is a deployment decision, not a code decision — so it is a driver behind a registry, selected by configuration, validated at boot.
- Two of the four engines send the document out of the tenant before any text exists — so the PII seam that protects embeddings cannot protect the scan itself. That is a final-egress decision and needs its own switch.
Design
One converter, one fallback, no overlap
PipelineRegistry resolves converters first-match by MIME and refuses
overlapping supports() predicates at boot (R23). A converter cannot look at
the bytes to decide, so “OCR when the PDF is scanned” cannot be a second PDF
converter. The split is therefore:
- Every entry point dispatches an image with its exact raster MIME
(
image/jpeg,image/tiff,image/webp,image/png) read from its bytes, never from the filename — the folder walker and the upload staging sniff the magic bytes (FileTypeSniffer::imageMimeOf(); a JPEG namedscan.pngis staged as.jpgand dispatched asimage/jpeg), connectors with the MIME they carry — so the converter registry andknowledge_documents.mime_typename what the bytes are; the OCR drivers still read the magic bytes themselves before they build a data URL or pick an input format, so a declared label is never trusted for that. OcrConverterclaims only the four image MIMEs (image/png,image/jpeg,image/tiff,image/webp) and only while the flag is on.PdfConverterstays the soleapplication/pdfmatch. When OCR is on it runsPdfTextLayerProbe, which decides per page over every page up toKB_OCR_MAX_PAGES(KB_OCR_PROBE_PAGES=0, the default; a positive value bounds the window and is a documented trade-off — a scanned page beyond it is not seen): a page with fewer thanKB_OCR_PROBE_MIN_CHARSextractable characters that carries an image is a scanned page; one that only paints (drawn content with no text object behind it — text outlined into paths by a design export, a decorative rule) counts as scanned only when no page has a text layer, and beside typed pages is a divider; one that paints nothing is blank, a separator, never a reason to OCR by itself. No text page but at least one scanned or painted page isempty; text pages and image pages ismixed— a typed cover over scanned body pages — and the whole document is routed to OCR so no page is silently lost (reasonmixed_pdf, the scanned page numbers recorded; a painted divider never promotes a text PDF tomixed, which would bill the whole document); otherwisepresent— including a window of blank pages alone, which is nothing to OCR and never a billed run over empty pages.empty,mixedand an ingest carryingmetadata.ocr.force = trueroute the document to the sameOcrServicethe image path uses. A file the parser cannot read at all (unreadable) is not treated as a scan: the converter first runs thepdftotextfallback it always had (KB_PDFTOTEXT_BIN, bounded byKB_PDFTOTEXT_TIMEOUT— a run past it is the deterministicrun_too_longrefusal of the document, never a silent hand-off to a billed OCR run, and the upload estimate says so before commit), keeps the text path when that yields text, and goes to OCR only when neither parser can read text from the file. “Yields text” is the probe’s own per-page rule (PdfTextFallback::hasText()): at least one page withKB_OCR_PROBE_MIN_CHARSnon-whitespace characters — several short pages that merely add up to the threshold are scanned pages and still go to OCR. The probe verdict —present,empty,unreadable:pdftotext,unreadable:pdftotext_emptyorunreadable:pdftotext_failed— is recorded inextractionMeta.text_layer_probe.
PdfPageChunker, which now claims the image source type
too: one chunk per ## Page N, so a citation like “page 3 of contract.pdf”
maps to exactly one row.
Drivers and the egress switch
OcrDriverRegistry is built from config('kb.ocr.drivers') and validates at
boot that every FQCN implements OcrDriver. Two rules are enforced in
resolve(), not in prose:
fake— the deterministic driver the test suite and the E2E harness use — is refused in production.- A driver whose
isRemote()is true (mistral-ocr,vision-llm) is refused unlessconfig('kb.ocr.allow_remote') === true. The env value is cast withFILTER_VALIDATE_BOOLEANand then compared strictly: the boolean-true spellings that filter accepts —true,1,yes,on(case-insensitive) — open the gate; everything else (false,0,no,off, an empty value, a typo) keeps the door closed.
remote: true|false on every OCR’d document, so an
auditor can answer “did this scan leave our infrastructure?” per document.
The switch is necessary, not sufficient. Where the bytes go and how much of
them is bounded in code before any egress (SEC-LLM-001 gates 2 and 7):
vision-llmresolves provider and model throughAiManager— the same choke point every chat call passes — so the platform’s provider policy applies unchanged and an unknown provider is a refusal, not a fallback.tesseractandvision-llmrasterise a PDF themselves with Poppler (pdftoppm+pdfinfo): the preflight (OcrDriver::unavailableReason(forPdf)) names a missing binary for a PDF — the estimate per staged PDF item, the status and re-run for a PDF document — while an image is never refused for a dependency it does not use. The page geometrypdfinforeports is the only bound applied before the render, so it must be complete: a failed report, or fewer page sizes than the pages to render, is a refusal (rendered_page_too_large), never a render checked only afterwards.mistral-ocrposts a source image as is anddoclingdecodes it as is, so both measure it against the same bounds the rasterising drivers apply to a rendered page — the pixel boxKB_OCR_RASTER_MAX_PAGE_PXand the page byte capKB_OCR_RASTER_MAX_PAGE_BYTES— before the request is built or the engine starts: a small file declaring bomb-sized dimensions, or one heavier than a page may weigh, never leaves and is never decoded (rendered_page_too_large). The upload estimate takes the same decision on the staged bytes, after the page-count and frame gates, so the modal states the refusal before commit instead of pricing a run that cannot start.mistral-ocrposts only to anhttpsURL whose host is in the exact allow-listKB_OCR_MISTRAL_ALLOWED_HOSTS(defaultapi.mistral.eu, api.mistral.ai), validates the response’s content type and size, caps each returned figure atKB_OCR_MAX_FIGURE_BYTES, stores a figure under the format its bytes are (a blob that is no raster is dropped), validates every page entry (each one an object; everyindexa whole number inside the returned list, no number claimed twice — a duplicate would collide two figures on one path, an out-of-range one would persist a bogus page, a malformed entry is an invalid response, never one silently dropped from a result recorded as complete) and treats an empty page list as an invalid answer, never a recorded run of zero pages.- A remote driver’s response is validated against the same page number the input cap admitted: more pages than the document has is an invalid answer — discarded before anything is stored, recorded or metered — so the cap bounds response-side storage and spend, not only egress.
- The persisted Markdown may cite only the figures the driver extracted and
the store wrote (
images/fig-{page}-{n}.{ext}). Every other image link in an engine’s, a provider’s or a model’s output — an unmatched placeholder, an external URL, a path that is not there — becomes an italic text description (OcrMarkdown::stripForeignImageLinks(), one rule for every driver), so a document never loads an arbitrary URL through the Markdown renderer and never cites a missing file. - Every run — local or remote — is refused before the driver starts when
the document exceeds
KB_OCR_MAX_PAGES(counted by the probe’s parser) orKB_OCR_MAX_BYTES; the refusal carries a machine-readable reason (too_many_pages/too_many_bytes) that the estimate shows in advance. A multi-page TIFF counts every frame as a page, and a driver that hands an image to its engine as one picture (tesseract,vision-llm,mistral-ocr) would be billed for all of them and transcribe the first: such a driver refuses a multi-frame TIFF outright (multi_frame_image, shown by the estimate too — split it into one image per page);doclingdecodes every frame itself and accepts it. - A PDF the parser cannot read has no verified page count — the probe reports
a
/Type /Pageobject count as a floor (pages_exact: false), and a lower bound cannot enforce a maximum. Such a document runs only where the work is bounded by construction: never on a remote driver (refused before any byte leaves, reasonpages_uncountable, shown by the estimate too), nor on a local driver that hands the whole file to its engine (docling— same reason), and among local drivers only on those that render page by page —tesseractrasterises withpdftoppm -l KB_OCR_MAX_PAGES, so the file renders at most the cap whatever its object table claims, each page under the driver’s timeout;doclinghands the whole file to its engine and is refused the same way, andvision-llm, which also rasterises page by page, is a remote driver and is refused before egress likemistral-ocr. Where it runs, the estimate marks the price as inexact.
Same bytes, same driver, no second bill
A run directory is immutable, so its first write is reserved atomically: a cache lock on the directory is held from the recorded-run check throughresult.json, with a lease sized per run from the driver’s declared worst
case for the verified page count (OcrDriver::maxDurationSeconds() — per-page
timeouts multiply, per-document timeouts count once) plus a write margin,
never below 1 200 s — so the lease is provably longer than the work it
protects, and a worker that dies mid-run blocks the directory for at most the
time its run could legitimately have taken. A second worker ingesting the same bytes at the same time waits
for it, looks again, and reuses the run the first worker recorded — one bill,
one directory, never two nondeterministic remote results interleaved in it. A
forced re-run has its own attempt identity and never touches a recorded run.
A recorded run is looked up by the driver’s identity (name, fingerprint,
capabilities), not by its availability: identical bytes re-ingested after a
deployment that switched remote egress off, or lost a binary, still reuse the
run that driver recorded — a reuse is a read, no egress, no bill — and the
egress gate / availability check applies only when a driver call is actually
needed. The run key also carries KB_OCR_MAX_PAGES (a page-by-page driver on
an unverified PDF records at most the cap), so a changed cap is a new run.
The write phase of a run (figures + result.json, never the driver call)
and a purge of the {source}.ocr/ directory exclude each other on a second,
per-directory lock: a purge holds it for the whole removal and re-checks the
directory before deleting it, so a run that starts writing after the purge
enumerated an empty directory is never deleted with its parent; a purge that
finds the lock held defers to the next sweep. The queue timeout of the
ingest job follows the same worst case: an image or a PDF while OCR is on is
dispatched with IngestDocumentJob::$timeout sized from the configured
driver’s maxDurationSeconds(KB_OCR_MAX_PAGES) capped by the run budget
KB_OCR_JOB_TIMEOUT (the driver enforces it), plus the lease margin
(OcrService::jobTimeoutFor()), never the 300 s default that would kill a
Docling call or a page-by-page run its lease still reserves; text, Markdown
and every ingest with OCR off keep 300 s. The run key also carries the
figure caps (KB_OCR_MAX_FIGURE_BYTES, KB_OCR_MAX_FIGURES,
KB_OCR_MAX_FIGURES_TOTAL_BYTES): a changed cap is a new run, never a
reused result that exceeds today’s limit or lacks the figures it would admit.
A recorded run lives at {source}.ocr/{run}/result.json next to its figures.
When the same bytes arrive again through the same driver — an identical
re-ingest, an IMAP backfill, a GitHub-Action full sync — OcrService reuses
the recorded pages instead of calling the driver: no spend, no FinOps row, the
document metadata says reused: true. The run key embeds the engine — driver
name, its fingerprint (model, language, DPI, the executables a local driver
runs — tesseract, pdftoppm, pdfinfo, the full docling path) and the
caps that shape the output (the page cap, the raster bounds, the figure
switch and budget) — so the same bytes through another engine are a
different, immutable run, switching the model never serves stale text, and a
lowered cap never reuses a run recorded under a wider one. kb:ocr (metadata.ocr.force)
bypasses the reuse on purpose; a run whose figures went missing is re-done.
Two consequences are stated, not hidden. First, result.json is the raw
OCR text on the KB disk: it follows the source file’s posture (ADR 0020 keeps
the vector store, not the disk, as the protected surface), sits under the
same ACL, and is purged with the source — the redacted text lives only in
the chunks. A deployment that must not hold raw OCR text beside its scans
sets KB_OCR_REUSE_ENABLED=false: every ingest then runs the driver, records
no result.json and is a new run with its own attempt identity (its own
{run} directory, like a forced re-run), so a re-ingest never rewrites the
figures a previous document version still references; and, exactly like a
forced re-run, that fresh run replaces in place the byte-identical
version it re-produced (OcrService::isFreshOcrRun() drives
replaceExisting on both ingestion paths), so the row always names the run
that was billed and the superseded run is left for the sweep — that knob governs the
recorded run (the raw OCR text and its reuse) only; figures follow
KB_OCR_FIGURES_ENABLED and the retention mode, so a deployment that must
hold no OCR asset at all beside its scans turns figures off too — and the
same happens by policy when the effective KB_SOURCE_RETENTION is
reference_only — once KB_CONVERSION_ARTIFACTS_ENABLED wires the
retention mode; with that flag off the knob stays the inert foundation it
was, and figures and reuse are unchanged (R43) — the mode that promises no
local copy at all: no run is recorded, no figure is stored, the Markdown
carries no images/ reference (ADR 0029 §5, ADR 0030 §3). Second, on a shared disk two tenants with the same source
key and bytes share the run: the second tenant’s ingest is reused: true
and carries no FinOps row — no data crosses (both already hold the bytes),
but the first tenant carries the cost; a deployment that isolates tenants by
disk or prefix isolates the runs with them.
Provenance — two orthogonal facts
The tool firewall keeps filtering on the first. The second is what a review
UI (W3) and the wiki export (W4, frontmatter key
extraction) read. Keeping
them apart is the difference between “this text was typed by an outsider” and
“this text was read by a machine from an insider’s scan”.
Figures
Figures the engine extracts are written at conversion time throughOcrFigureStore — the Flow persists step outputs to the database, so binary
blobs cannot ride in ConvertedDocument::mediaItems; the store writes them
and mediaItems lists the paths. The layout is:
{run} is the full 64-hex sha256(bytes · driver name · variant), where the
variant is the driver’s fingerprint() plus the page cap (;pages=N), the
raster bounds (;raster=<px>:<bytes>), the figure switch and budget
(;figures=0 or ;figures=1:<bytes>:<count>:<total>) and, for a forced
re-run or a run with reuse off, a fresh per-attempt salt (;attempt=…) — so
such a run is a new immutable directory, never the recorded one rewritten. A
truncated digest would be an identifier two different inputs could share, and
one run directory would then serve the wrong text.
Two versions of the same source path therefore never overwrite each
other’s pixels, a re-run on identical bytes lands on the same directory (the
same idempotency the ingest itself has), and a stored artifact (W2) can point
at the run that produced it. The Markdown references images/fig-3-1.png
relative to the run directory — the shape the wiki export copies verbatim.
The assets share the source file’s lifecycle: a soft delete keeps them, the
hard delete of the last row referencing the source removes the whole .ocr/
directory through the same reference gate DocumentDeleter already applies to
the file.
PII
ChunkRedactor (ADR 0020)
runs unchanged on the OCR output before embedding — scans are where the
codici fiscali live. Note what it does not cover: the pixels of a figure,
and the bytes a remote driver received. The first is why figures stay on the
KB disk behind the same ACL as the source; the second is why the egress switch
exists.
Cost — metered, and estimated before commit
Every OCR run is a FinOps line item withpurpose_tag = ocr:
- locally-priced drivers (
tesseract,docling,mistral-ocr) are metered byOcrCallMeteraspages × KB_OCR_RATE_PER_PAGEin the FinOps base currency; vision-llmis metered per token by thelaravel/ailifecycle hook, like any chat call, and is deliberately not double-counted by the page meter.
GET /api/admin/kb/uploads/{batch}/estimate in the
review step — before commit — and shows how many files and pages would be
OCR’d, with which driver, at what estimated cost — for a per_page driver;
for an Sdk-metered driver (vision-llm) the response says metering: sdk,
carries no page price (rate_per_page and every cost are 0) and the modal
says the provider meters tokens, so no page price is ever invented. The estimate reads the
staged bytes and runs the probe; it never runs a driver, so it is free. It
With the flag off the estimate still probes each staged PDF: one with a text
layer answers text_layer_present (it is ingested as text either way), only
an image or a scan is ocr_disabled — the OFF answer is honest, never
“every PDF would need OCR” (R43). The flag is read again at commit, not
only at staging: an image staged while OCR was on and committed after a
deployment turned it off is refused before the move (item failed with
ocr_disabled), never moved and dispatched only for the converter to refuse
it. It
also says when the run would be refused (driver_available: false — a remote
driver with the knob off, a missing binary — or a file over the page/byte
caps, or a staged object whose bytes are no raster — or no PDF — any more,
unrecognised_bytes), so the modal never promises a run the server will not
perform (R14). The driver verdict is per kind of input: every item carries
its own driver_available (a PDF needs the rasteriser’s Poppler binaries, an
image does not), the batch flag is false as soon as one staged item that
needs the driver (it would OCR, or OCR itself refused it) is blocked —
a text PDF or a text file never blocks a batch — and the modal names only
the files that would fail while still quoting the ones the driver can take. The modal’s file picker filters on the extensions
GET /api/auth/me delivers under kb_upload.accepted_extensions — the
backend’s SourceType::knownExtensions(), images only while OCR is on —
so the SPA keeps no second list of formats (R18).
Conversion artifacts on the Time Machine (W2, ADR 0030)
The v8.7 Time Machine browsed, diffed and restored versions — but a version was aknowledge_documents row plus its chunks, so “diff” meant a diff of two
chunk reconstructions (the indexed body, redacted, re-joined with blank
lines), never of the document itself. W2 stores the exact Markdown the chunker
received, per version, and makes every read of the Time Machine prefer it.
- One core, both paths. The write lives in
DocumentIngestor’s shared persistence (persistDrafts()for the Flow saga,persistFromDrafts()for the direct path): the temp file is written before the transaction, the row commits withmarkdown_pathpointing at the final path, and the move happens after commit. A rollback discards this attempt’s temp and nothing else; a concurrent identical ingest that already published the same bytes wins — the final is re-hashed against the temp (the path is the content hash; a truncated or replaced final is overwritten by the verified temp, never kept) and the loser drops its own temp. A publish that fails after commit keeps the pointer (statemissing, never rolled back) and throwsArtifactPublishFailedException: the ingest job retries, and the retry — an identical re-ingest — repairs the pointer through the same-hash path; a connector sync reports a failed document, never a silently degraded one. The retry is a new flow run (the job salts its idempotency key per attempt, so a persisted failed run is never handed back to the attempt meant to repair). On the direct path the canonical indexer is dispatched before the publish; on the Flow saga the failedpersist-chunksstep skips the indexer step of that attempt and the repairing retry runs it. A writer holds a cache lease on its temp (KB_CONVERSION_ARTIFACTS_TMP_LEASE, the primary guard — at least the age threshold) from the write to the publish or discard, so the temp sweep never removes a file a live writer is about to move into place. The publish itself runs under the artifact path’s lock after re-checking that the row still points there, and every removal of an artifact (hard delete, prune, orphan sweep) re-checks the references under the same lock before deleting — an identical ingest that recreated the same content-addressed path meanwhile keeps its artifact (artifacts_keptfor a pruned or deleted row,artifact_orphans_keptfor an orphan candidate). The path lock needs a lock-capable cache store (CACHE_STORE=redisin production): on a store that cannot lock, a publish is refused (thrown and its temp discarded) and a removal is reportedfailed— never run unguarded — while only the temp lease degrades to the age threshold. Thenullstore, which implements the lock contract and grants every lock without excluding anyone, is detected and refused exactly like a store that cannot lock; what stays undetectable is a provider whose locks do not exclude ACROSS processes (the array store, process-local by design). Neither lock is renewed: every critical section asserts, right before its irreversible step, that it still owns its lock (HeldLock) — a lapsed TTL is a refusal (the commit rolls back, the original is kept, the publish discards its temp, the removal isfailed) — a check right before the step, not a renewal, so the window shrinks to the step itself. A lock that cannot prove ownership — ownership is read by capability, from any lock exposing a callableisOwnedByCurrentProcess()that answers a boolean, so a probe that is missing, not public, or answers anything else (a third-party store registered withCache::extend()) does not qualify — has its guarded step refused as well — a stop rather than a degradation: every artifact-enabled ingest rolls back and retries and every prune reportsfailed, untilCACHE_STOREnames a lock-capable store. Reported once per class. The source file of a hard delete takes the storage key’s lock the same way: the reference scan, the.ocr/purge and the delete all run under it, each irreversible step asserting the TTL has not lapsed. Every refusal — a key another writer holds right now, a store that cannot lock, a lapse mid-section — keeps the file for the orphan sweep and reportsfile_deleted: false; the row is already committed, and a stale file never fails a deletion. With artifacts off nothing else takes that lock, so the hard delete takes none either. - The key is tenant- and project-namespaced, as safe segments. A segment
is used verbatim only when it matches
^[A-Za-z0-9][A-Za-z0-9._-]{0,119}$, and does not start with the reservedh-prefix; otherwise it becomesh-+ the full 64-hex SHA-256 of the value (injective: a verbatim segment can never spell an encoded one); the composed path is normalised and must stay inside.artifacts/, which the folder walker and the orphan sweeps never read back as a source. - Three nullable columns record the version’s provenance:
version_actor(user:{id},system:ingest,system:ocr, …),version_reason(free text, 1024 chars) andcontent_hash(SHA-256 of the stored artifact, recorded when it is written: equal todocument_hashby construction on every ingested version — a correction is an ordinary new version with its own hashes — and null where no artifact is stored; it is the integrity check on the stored bytes, not a second identity). The actor is derived server-side: the HTTP ingest and connector boundaries strip any client-suppliedversion_actor, and the trusted caller sets it (the authenticated principal forPOST /api/kb/ingestandrestore, the CLI / connector defaultsystem:ingest,system:ocrfor akb:ocrre-run). - Reads prefer the artifact and say so.
contentFor()returns the stored Markdown when the row has one and the file is there, otherwise the chunk reconstruction;diffreportsfrom_source/to_source(artifact|reconstruction, additive keys) andfrom_integrity/to_integrity(verified·mismatch· null) so the UI labels a diff faithful only when both sides are stored documents verified against their recorded hashes, stored but not verified when both are stored documents but a side has nocontent_hashto check against (a legacy pointer), and an index diff whenever a side is reconstructed. A missing file behind a non-nullmarkdown_pathis logged and degrades, never a 500. - Restore keeps the creation provenance and records itself apart.
version_actor/version_reasonsay who created the version and never change; a restore appends{actor, at, previous_live_id}tometadata.restoresand the versions surfaces expose the last one asrestored_by/restored_at(additive). The artifact is left untouched. The restored version’s canonical identity is its own, reconstructed from the frontmatter the archive retained and re-validated by the same parser the ingest path runs: a restore is the re-ingest of older bytes, so a canonical version restored over a non-canonical live one reclaims its slug, and a version that never declared one does not inherit the live row’s. Only a legacy row archived before the frontmatter was persisted still carries the outgoing version’s identity — there it is the only copy left. If the slug or doc_id is already held by another document (an archived sibling, or another source path in the same project), the content is restored without the identity and a warning names the holder — never a failed restore, and never a slug taken from its current owner. “Held” is judged the way the unique index judges it, not the way this reader may read: a soft-deleted or ACL-hidden holder still occupies the slot, so the probe lifts both global scopes. It keeps the tenant filter, because the uniques themselves are(tenant_id, project_key, slug)/(tenant_id, project_key, doc_id)— a holder in another tenant is not a conflict and does not cost this restore its identity. - A tampered artifact is never served as faithful.
content_hashis the integrity check of the stored bytes:contentFor()re-hashes what it reads and, on a mismatch, logs, falls back to the reconstruction and says so (integrity: mismatchon the content endpoint,from_integrity/to_integrityon the diff;verifiedwhen the hash matched,nullwhen there was nothing to check against). A stored pointer must already be canonical (no./.., no//, no\\) or it is refused on every disk — on an object store that lexical check is the whole check; on a local disk every read, delete and publish also resolves the real path and refuses one that a symlink under.artifacts/makes resolve outside the root, and containment is asserted before any existence probe. markdown_onlyhonours every row’s contract. Each row records the mode it was ingested under (metadata.source_retention); a shared original is dropped only when every row referencing that storage key — any tenant, trashed included — was ingested under a mode that does not require it and has its artifact present on disk. Afull_copyrow (or a pre-v8.36 row without the stamp, which counts asfull_copy) blocks the drop; areference_onlyrow never needed the local source. The stamp is host-owned —source_retention/source_droppedsent by a client or a connector are stripped likeversion_actor. An unknown value resolves to the configured mode only when a NEW version is being stamped; a row ALREADY persisted with an invalid or absent stamp reads asfull_copy, the conservative mode, and keeps its original whateverKB_SOURCE_RETENTIONsays today. Do not expect a historical row with a malformed stamp to follow a later change of the setting. The drop is gated on the row’s own stamp, never on the configured mode of the day: afull_copyversion re-embedded afterKB_SOURCE_RETENTIONmoved tomarkdown_onlykeeps its original, and a version being replaced (a forced re-embed, a fresh OCR run) keeps the contract it was born under —full_copyfor a pre-v8.36 row without a stamp; only a new version gets the configured mode, and with the artifacts flag off it is stampedfull_copy(nothing was stored, dropped or withheld).- Retention follows
KB_SOURCE_RETENTION.full_copy(default) stores the artifact next to the original;markdown_onlystores it and drops the original binary after the artifact commit (a Markdown source is its own artifact and is never dropped);reference_onlystores nothing. - Erasure covers it. A hard delete removes the row’s artifact; the prune
removes it with each pruned version, purges the OCR run directories no
remaining version references, and sweeps temp leftovers older than
KB_CONVERSION_ARTIFACTS_TMP_MAX_AGEthat no live writer leases (a leased temp is reportedartifact_temps_in_flight, whatever its age) plus artifacts no row (trashed rows included) references any more. Both sweeps are deliberately cross-tenant (the root is one shared tree; a file is deleted only at zero references across every tenant) and cover the configuredKB_FILESYSTEM_DISK/KB_PATH_PREFIXnamespace plus every(metadata.disk, metadata.prefix)a row with an artifact pointer recorded (a disk this deployment cannot resolve is counted asartifact_namespaces_skipped, see below). A soft delete leaves the artifact in place. markdown_onlykeeps the sweeps honest. The original is dropped only when every row referencing that storage key — any tenant, trashed rows included — already has an artifact (a shared disk can hold the same key for two tenants); the rows are then stampedmetadata.source_dropped, sokb:ingest-folder --prune-orphansnever reads the missing file as an orphan, and a PII-policy re-embed (kb:reembed-project) reads the artifact instead. The backfill applies the same contract: once a row’s artifact is verified (written by this run, or already there) amarkdown_onlyrow’s original goes through the same gate and the rows are stamped —full_copyrows keep theirs,--dry-runnever drops, and the summary line counts what went (originals_dropped).- Backfill is operator-only and judges each row on its own contract.
kb:artifacts-backfillwalks every live row: a row stampedreference_onlyisintentionally_missingwhateverKB_SOURCE_RETENTIONsays today (a row without the stamp predates v8.36 and counts asfull_copy, so afull_copyhistory is still backfilled after the knob moved); a pointer whose file is readable and hashes todocument_hashisalready_stored; any other row — no pointer, or a pointer whose file is missing or corrupt — is re-converted and gets its artifact only when the reconversion hashes to the row’sdocument_hash(hash_mismatchis reported and nothing is written — an artifact must agree with the version’s chunks); a missing source is reported, not invented. The command exits non-zero when any row endedsource_missing,conversion_failedordisk_unavailable— a row whose artifact it could not produce —hash_mismatchbeing informational, and--dry-runpredicts the same exit code. An identical re-ingest with the flag on repairs or publishes a version’s artifact the same way.
KbDocumentVersionsTool returns metadata only: an MCP read of the content
(or a diff of two) would hand the model un-redacted text. The agent-facing,
redacted rendering of artifacts is the W4 export.
Data model / contract
Three nullable columns onknowledge_documents (version_actor,
version_reason, content_hash, mirrored in the SQLite test migrations) plus
the markdown_path column ADR 0014 already declared; everything else lands in
JSON columns that already exist:
knowledge_documents.metadata.converter (excerpt)
knowledge_chunks.metadata gains provenance: "ocr" and
ocr_confidence (the page’s confidence) next to the existing page.
Configuration (config/kb.php → ocr, env in .env.example)
Tri-surface (R44)
All three sit on
OcrService (status(), rerun()); none re-implements the
lookup or the gating.
The Time Machine (W2) is tri-surface over DocumentVersionService:
Decision rationale (ADR-style)
- Why a fallback inside
PdfConverterand not a content-aware resolver? The registry’s contract is MIME-only and mutex-checked; teaching it to open files would move a per-document decision into boot-time infrastructure and break the overlap guard that has caught real re-routing bugs (R23). The probe lives where the bytes already are. - Why is remote OCR a second switch and not part of
KB_OCR_DRIVER? Because selecting a driver is an operator convenience and sending documents to a third party is a data-subprocessor decision (SEC-LLM-001 gate 3). A setting is not a security boundary; the registry enforces the policy in code and fails closed on anything buttrue. - Why is the extraction origin not a
provenance_tiervalue? ADR 0028’s tier answers “who wrote this” and drives the tool firewall; addingocrthere would either bypass the firewall for external scans or firewall internal ones. Two questions, two fields. - Why content-addressed figure runs? Two tenants or two versions can share
a source path; a fixed
images/directory would let one conversion overwrite another’s pixels and a purge remove the wrong asset. Hashing the input bytes makes the directory a function of the version, with re-run idempotency for free. - Why no MCP write? “The agent proposes, a person confirms.” OCR re-runs cost money and change what the model is grounded on; an agent may see the status and ask, not spend.
- Why gate the connector bridge separately?
HostIngestionBridgehands connector files straight to the job, bypassing controller and staging validation. Without its own gate an image would be queued and die in converter resolution; with it the refusal is a recordedconnector_ingest_refusedaudit event withreason: ocr_disabled.
Worked example
A three-page scanned lease arrives twice — once dropped on the admin upload modal, once as an IMAP attachment — on a deployment withKB_OCR_ENABLED=true,
KB_OCR_DRIVER=docling, KB_OCR_ALLOW_REMOTE=false.
- Upload modal. The staged batch’s review step calls the estimate:
“OCR will run on 1 file (3 pages) with the
doclingdriver — estimated 0.004/page.” The operator commits. - Job.
PdfConverterprobes 3 pages, finds 0 characters, recordstext_layer_probe = "empty"and hands the bytes toOcrService. Docling returns three pages and one figure (a signature block). - Disk.
contracts/lease.pdf.ocr/1c9e…/images/fig-3-1.pngis written; page 3’s Markdown referencesimages/fig-3-1.png. - Chunks. Three chunks,
heading_path = "Page N", each withprovenance: ocrand its page confidence. The tenant’s PII policy replaces the tenant’s codice fiscale with a surrogate before embedding. - FinOps. One ledger row,
purpose_tag = ocr,cost = 0.012. - IMAP. The same attachment arrives through the connector. Same bytes →
same
version_hash→ the ingest is the usual no-op. Had OCR been off, the bridge would have writtenconnector_ingest_refusedand confirmed the UID so the mailbox is not re-presented every sync. - Audit.
kb:ocr 4711 --status --tenant=acmeprints the driver, the reasonscanned_pdf, three pages with their confidence andremote: no.
Gotchas & operations
-
Turning artifacts on populates nothing by itself. Rows ingested before
KB_CONVERSION_ARTIFACTS_ENABLED=truekeep diffing by reconstruction until their next identical re-ingest (which publishes the artifact for a pointerless row under its own retention contract) or akb:artifacts-backfill --tenant=…run. -
A “stored” badge is a verified claim, memoized.
has_artifactis derived from the same read + hash check the content endpoint uses and is true only forverified— memoized forKB_VERSIONS_ARTIFACT_STATE_CACHEseconds so a timeline page does not re-read one object per row on every listing (the key carries the bytes’ identity; the content and diff endpoints never read the memo);artifact_statesaysmissingormismatchwhen the pointer no longer resolves to the recorded bytes (the UI shows a warning badge), andunverifiedfor a legacy pointer without acontent_hash— the next identical re-ingest orkb:artifacts-backfillrecords the hash and makes itverified; the backfill repairs the missing and mismatching rows. -
kb:artifacts-backfill --dry-runnever spends. The dry-run marker rides the conversion, so a row that would need a fresh OCR run to verify is reported asocr_unverifiedinstead of run (a recorded run is still read back). -
The
markdown_onlydrop is serialized with writers. The reference scan and the delete run under the storage key’s lock the persist paths hold around their row commit (every non-Markdown commit while the flag is on, areference_onlyversion included since it still requires the shared original — nothing waits on it with the flag off, in a dry run or for a Markdown source), and a sibling artifact counts only when its bytes verify. Like the OCR run lock it needs an atomic lock store (Redis) in production;KB_CONVERSION_ARTIFACTS_SOURCE_LOCK_WAIT/KB_CONVERSION_ARTIFACTS_SOURCE_LOCK_TTLtune it (10 s / 60 s). The gate runs on every path that leaves a verified artifact behind — the fresh ingest, an identical re-ingest whose artifact was just verified, repaired or published for the first time, akb:artifacts-backfillwrite — so an original re-uploaded after a drop, or kept because the key was locked, is dropped the next time round. -
The sweeps cover every artifact namespace the corpus records.
kb:prune-archived-versionssweeps temps and orphans on the configured(disk, prefix)and on every(metadata.disk, metadata.prefix)a row with an artifact pointer recorded (one line per namespace when there is more than one; a disk this deployment cannot resolve is reported per namespace and counted asartifact_namespaces_skippedin the summary line, not swept). The stale-OCR-run gate ofkb:prune-orphan-filesjudges all candidates of a namespace with one bounded query per 500 candidate(source, run)pairs, never one query per run. -
A refused delete is a failed prune.
kb:prune-archived-versionsreportsartifacts_failed/artifact_temps_failed/artifact_orphans_failed/ocr_failedand exits non-zero when the disk refused a removal (or a planted symlink under.artifacts/stopped a sweep): the rows are gone, the bytes are not. -
A version restored during the prune is never pruned. Every candidate is
re-read and locked in the deleting transaction and pruned only if it is
still archived; a version a Time Machine restore activated between the
selection and the delete is skipped and counted (
restored_meanwhile, additive, printed when non-zero). -
Turning it off is a stop, not a rollback. Nothing already stored is
discarded or hidden;
contentFor()keeps reading it and the UI keeps saying faithful for those pairs. -
markdown_onlyis a one-way choice per ingest. The original binary is deleted once the artifact is committed; a laterkb:ocrre-run of that document reportssource_missing-class refusals because the scan is gone. -
Enabling the flag does not re-process history. Documents that ingested
empty before v8.36 keep their empty text until re-ingested;
kb:ocr {id}queues that per document. -
Only a boolean-true spelling opens the egress gate.
KB_OCR_ALLOW_REMOTEacceptstrue,1,yesoron(case-insensitive,FILTER_VALIDATE_BOOLEAN); a typo or any other value keeps remote OCR closed, and the registry error says how to open it. - The estimate is a probe, not a promise. It counts pages the driver will see; a driver may still skip an empty page, which is why the ledger row is written from the actual result.
-
A dry run costs nothing and writes nothing.
Flow::dryRun()of the ingest flow reaches the converter, but the OCR core sees the dry-run mark and returns a page-shaped preview instead of calling the driver: no remote egress, no.ocr/write, no ledger row. A run already recorded for those bytes is shown read-only. The preview shows what the run would do: for a PDF the parser could not count, a bounded local driver renders up toKB_OCR_MAX_PAGES, so the preview carries the cap’s worth of## Pagesections and thatpage_count, never the parser’s floor. - Figures are not redacted. They inherit the source’s ACL and stay on the KB disk; the wiki export (W4) omits them under the tenant PII policy by default.
-
Docling and Tesseract are host binaries.
KB_OCR_DOCLING_BIN,KB_OCR_TESSERACT_BINandKB_OCR_PDFTOPPM_BINmust resolve on the queue worker, not only on the web pod;OcrDriver::isAvailable()is reported by every status surface asdriver_available, andOcrDriver::unavailableReason()names the cause (binary off PATH, missing API key, an endpoint outsideKB_OCR_MISTRAL_ALLOWED_HOSTS) in the estimate error and the re-run 422 — a misconfiguration is refused at preflight, never queued to fail in the worker. -
fakeis refused outsidelocal/testing/development. The gate is an allow-list, soAPP_ENV=prodor a misspelt name counts as production. -
Tests and the E2E harness use
fake, which is refused in production — aKB_OCR_DRIVER=fakeleft in a production.envfails loudly at the first OCR, not silently with synthetic text.