Retrieval pipeline

​

Motivation

​

Theory & background

1. Over-retrieval + rerank. Embedding similarity alone is a coarse filter. Retrieving a wider candidate set (candidate_multiplier × limit) and then reranking lets weaker-vector-but-strong-keyword matches survive.
2. Weighted-sum fusion. Multiple normalised signals (vector, keyword, heading, tag, recency, …) combine linearly into one comparable score.
3. Trust gradient. The canonical layer adds a priority boost and a status penalty, so a ratified accepted decision outranks a superseded one and an auto-compiled page sits below human-curated content.
4. Graph + counter-evidence. Beyond the top-k, a 1-hop graph walk pulls in structurally related docs, and a separate channel injects rejected approaches so the model stops re-proposing dismissed options.

​

Design

​

The reranker fusion

base      = vector_weight·vec + keyword_weight·kw + heading_weight·head
additive  = tag_overlap_weight·tags + preamble_match_weight·preamble
          + recency_weight·recency + status_active_weight·active
          + mention_boost (if the doc is @mentioned)
canonical = priority_weight·retrieval_priority − status_penalty − auto_tier_penalty
score     = base + additive + canonical

​

Hybrid fusion mode

​

Graph expansion & rejected injection

​

Data model / contract

final class SearchResult {
    public readonly Collection $primary;   // reranked top-k
    public readonly Collection $expanded;  // 1-hop graph neighbours
    public readonly Collection $rejected;  // rejected-approach docs (⚠ block)
    public readonly ?Collection $runnerUp; // candidates below top-k
    public readonly array $meta;           // counts, timing, strategy, stats
}

​

Decision rationale (ADR-style)

• Why over-retrieve then rerank, not just top-k by cosine? Pure cosine drops lexical-exact matches whose embeddings sit slightly further out. A 3× candidate window costs one wider SQL read and lets keyword and heading signals rescue them. Rejected as alternative: raising default_min_similarity (loses recall uniformly).
• Why a linear weighted sum, not a learned reranker? Transparency and tunability. Every weight is a config knob an operator can reason about and the kb:benchmark harness validates; a learned cross-encoder is a future option but would forfeit the auditable trust gradient.
• Why bake canonical status into the score, not filter on it? Filtering would hide superseded decisions entirely; penalising keeps them retrievable (for “why did we move off X?”) while ensuring the current decision ranks first. See ADR 0001 (canonical layer) and grounding & evidence tiers.

​

Worked example

query: "what cache layer did we standardise on?"

candidates (multiplier 3 → 24 for limit 8):
  dec-cache-v2      vec 0.81  kw 0.40  status accepted     → score 0.71
  dec-cache-v1      vec 0.78  kw 0.38  status superseded   → 0.71 − 0.40 = 0.31
  blog-redis-notes  vec 0.74  kw 0.20  (non-canonical)     → 0.49

primary top-k:        dec-cache-v2, blog-redis-notes, …
graph expansion:      dec-cache-v2 --depends_on--> mod-session-store
rejected injection:   rej-memcached-2023 (cosine 0.52 ≥ 0.40)  → ⚠ block

​

Gotchas & operations

• Weights are tuned together. Raising vector_weight without lowering the rest skews the whole gradient — re-run kb:benchmark after changes.
• Penalties dominate boosts by design. A 0.40 status penalty outweighs the ≤0.10 canonical priority boost, so a superseded doc never outranks an active peer on priority alone.
• No canonical docs → identical to plain hybrid RAG. The boost, graph expansion, and rejected injection all no-op.