Program View — Defense-News Intelligence

Status: Living Date: 2026-06-28 Author: San Lee

The program-management companion to the product one-pager: the workstreams, how they depend on each other, what's planned, and what could go wrong. Consolidated here for now; split into roadmap.md / risks.md once it outgrows one page.

Workstreams

Workstream	What it is	Repo
Knowledge base	Domain service (REST + async enrichment) that stores and serves notes	notes-api
Classification	LLM classifier with an eval harness	defense-news-classifier
Agent	RAG + tool-use agent over the system (the hub)	kb-agent
Concepts	Plain-language notes on the AI techniques behind the system, with an interactive concept map	learning-notes
Cross-cutting	ADRs, this program view, evals-as-CI, OTel observability	architecture (+ each repo)

Dependency map

graph TD
  kb["kb-agent<br/>RAG + tool-use agent"]
  notes["notes-api<br/>knowledge base"]
  clf["defense-news-classifier<br/>LLM classifier"]
  infra["K8s"]
  evals["evals-as-CI"]
  otel["OTel observability"]

  kb -->|tool call| notes
  kb -->|tool call| clf
  notes -->|BackgroundTask → /classify| clf
  notes -->|writes labels back as tags| notes
  notes --> infra
  clf -.->|eval harness| evals
  kb -.->|RAG eval| evals
  notes -.->|traces| otel
  kb -.->|traces| otel

The two load-bearing dependencies: kb-agent can't be "one system" until notes-api and the classifier are callable as tools — the contract for this is set (system/SYS-003, accepted) and both tool seams now work (classify_snippet → classifier over HTTP, frozen by system/SYS-004; and search_notes → notes-api over HTTP, frozen by system/SYS-006), each enforced by contract tests on both sides. And the classify-and-writeback loop is now closed — after POST /notes, notes-api fires a BackgroundTask that calls {CLASSIFIER_URL}/classify, reads the two labels, and writes them back as namespaced tags via PUT /notes/{id}/tags (system/SYS-005). Scaling that loop to a durable task queue is the remaining reliability step. Everything else is cross-cutting.

Roadmap — Now / Next / Later

Shipped (the foundation under everything below): SYS-001–SYS-010 recorded; the three code repos wired into one system (kb-agent ↔ notes-api ↔ defense-news-classifier), with the tool-layer and wire contracts frozen (SYS-003/SYS-004/SYS-006) and contract-tested on both sides; the classify-and-writeback loop closed (SYS-005, idempotent namespaced writeback, R1 mitigated); CI green across all three repos; the classifier at v3.2.1 (three-axis output, human-labeled gold eval plus a validated Opus judge, the autonomy ladder built and measured end to end at L1–L4, every axis measured at n=300 as well as on the n=54 human gold set, and the global-boundary prompt clause adopted after a pre-registered re-run cleared all four of its rules at n=595 — classifier/ADR-024); the documentation portal live (architecture/ADR-001, then SYS-008); SYS-009 setting how work cascades across surfaces, and SYS-010 recording the security posture; evals-as-CI, piloted in the classifier — its v2 capability evals now gate every PR (free offline scoring-regression gate) plus a paid weekly live-capability gate (classifier/ADR-007, R6 first pilot closed); and the prompt-optimization loop (rung 1) built — Level 3 of the autonomy ladder now shipped, not just spec'd (classifier/ADR-005, classifier/ADR-006); and the v2 eval modules' orchestration tests backfilled — the run-loop and main() coverage the pure-function tests deliberately skipped, lifting those four modules from 58–86% to 99% and overall src/ from 90% to 97% (the v2.0.2 hardening, riding the next tag rather than a standalone release).

Now (in flight)

• [product] Capstone narrative stub: the last artifact of the gap-closing pass.

Next

• ~~[classifier] v2.1.0 scale the gold eval with the validated judge (shrinks the n≈54 noise floor).~~ Shipped 2026-07-17 — 300 judge-graded DVIDS snippets, category 93.3% [89.9, 95.6] and domain 90.3% [86.5, 93.2], roughly halving the n=54 CI width. This sat under "Next" until 2026-07-19: the release carried no Downstream surfaces section naming this file, so nothing swept it. ~~The successor is scale the region eval — unblocked (judge-vs-human region agreement 96.3%) but unscheduled.~~ Shipped 2026-08-02 as v3.2.0 — the judge cleared classifier/ADR-014's gate on that agreement figure (a perfect score at the time; the marker above tracks the live artifact, which moved when v3.2.1 re-ran the gold set), and the n=300 run then narrowed the region interval from 18 points to 7 and sized the one named error cluster the region axis has. That cluster was then closed as far as a prompt can close it: the fix was measured, reverted as marginal, re-run at double the power, and adopted as v3.2.1 (classifier/ADR-023 → classifier/ADR-024). The figures are deliberately not restated here: they are a frozen dated measurement living in the classifier's own artifacts (evals/scale_eval_v3.txt, evals/region_clause_rerun.txt), and this repo points at them rather than quoting numbers it cannot assert against — evals/metrics.json publishes the n=54 gold block only.
• [cross-cutting] Evals-as-CI for kb-agent: extend the pattern piloted in the classifier (classifier/ADR-007) to kb-agent's own RAG — capability/regression evals beyond the existing deterministic shape-grader (closes the rest of R6). Half shipped 2026-08-02 — kb-agent reached SYS-017 tier 1 (kb-agent/ADR-012): CI reconstructs the notes corpus and runs both retrieval-eval arms on every push and PR, reporting. It stays under Next rather than moving to Shipped because tier 1 blocks nothing; the remaining move is tier 2 — floors measured from several CI runs, a gate script, and the required-status-check setting — and that is a separate job, not a leftover of this one.
• [program] Start the weekly status cadence, harvested from real progress.

Later

• ~~[classifier] v2.2.0 tiered model routing, then v3.0.0 add a region field.~~ Both shipped 2026-07-18 and should move out of Later: v2.2.0 shipped as a measured negative result — routing moved +0 rows at ~1.97× cost, so the shipped classifier stays single-model (classifier/ADR-013) — and v3.0.0 shipped the region field (classifier/ADR-014). The v3.0.0 schema change breached system/SYS-004; that breach was closed 2026-07-19 by the provider-published schema and the coordinated kb-agent update. See R8 below.
• [classifier] Loop demo rung 2: an agent-driven ML loop (a tiny AutoML) where an outer agentic loop wraps a classical TF-IDF + logreg baseline and does error-driven feature engineering against the LLM. Consumes the parked classical-ML bake-off.
• [notes-api] Phase 1 containerize + local K8s; Phase 2 a durable task queue (Celery + Redis or outbox) if the reliability SLA tightens beyond best-effort BackgroundTasks.
• [cross-cutting] ✅ OTel tracing shipped across all three services — the kb-agent tool-use loop, the classifier /classify LLM call, and the notes-api enrichment seam (opt-in per service, GenAI/HTTP semconv attributes). Drift detection over the emitted traces is the remaining observability frontier.
• [ops] Operational-maturity track: Linux, ssh, health checks ("can I operate what I built?").
• [non-goal] Other verticals (banking, etc.) — written down as a direction, not shipped.

Risk register

#	Risk	Severity	Mitigation / next action	Tracked in
R1	Duplicate enrichment / lost writeback — BackgroundTasks is best-effort; a worker crash loses the enrichment, and a requeued task re-runs the writeback	Low	✅ Mitigated: the writeback uses namespaced replace, so re-running converges to the same tags (idempotent). Lost enrichment is acceptable at this scale; upgrade path is a durable task queue (Celery + Redis). Frozen in system/SYS-005. (Mechanism corrected 2026-07-18: this row previously named PUT /notes/{id}/tags. The background task does not make that HTTP call — it opens a fresh session and writes through the ORM directly, notes-api/src/notes_api/tasks.py:206-214. The idempotency claim is unaffected; merge_tags does the namespaced replace. SYS-005 described the same HTTP mechanism and has since been corrected — its Writeback section now records the ORM-direct path and notes that the PUT endpoint exists but is not the enrichment path.)	system/SYS-005, notes-api/ADR-001
R2	Classifier accuracy ceiling — capped by label ambiguity, not model horsepower. At v3.2.1 (n=54 gold): category 94.4% / macro-F1 0.930, operational-domain 98.1% / macro-F1 0.982, region 94.4% / macro-F1 0.975. The ceiling claim held across three versions, and two measured escalations were declined on the strength of it — BM25 grounding (classifier/ADR-012) and tiered routing (classifier/ADR-013), the latter at ~1.97x cost for +0 rows. v3.2.1 is the first release to move it, and it moved it with a prompt clause rather than a model: the region axis's second ceiling mechanism was one named cluster (gold global pulled to a specific region — the no-guessing rule, not label overlap), and a one-bullet clause closing it was pre-registered, measured, reverted as marginal, re-run at double the power and adopted (classifier/ADR-024). That sharpens the risk rather than retiring it: what remains is label ambiguity proper, and the residual region errors have inverted into over-calls of global. (Numbers restated 2026-08-03 for v3.2.1. They are asserted against the classifier's generated evals/metrics.json by scripts/check_program_metrics.py — do not retype them by hand.)	Medium	Don't escalate the model (per system/SYS-002); refine taxonomy or use an LLM judge on boundary cases; set the expectation in product metrics	classifier/ADR-001, system/SYS-002
R3	Breadth creep — adding verticals/techniques without depth, eroding the through-line	Medium	"Deep on one vehicle, articulate transfer"; other verticals are an explicit non-goal; this doc + the one-pager are the guardrail	product/one-pager.md (Non-goals)
R4	Planning theater — gap artifacts drift from delivery and become hollow docs	Medium	Keep artifacts thin and living; attach each to Phase 0; feed the capstone from real decisions only	this roadmap (Now/Next)
R5	Simulated program — a solo project has no real cross-team coordination, so program evidence is simulated	Low (honesty)	Treat repos as workstreams with tracked deps; be explicit in the capstone that it's simulated, but the reasoning and artifacts are real	capstone (pending)
R6	RAG ships unmeasured — kb-agent integration could go out with no quality eval	Medium	🔄 In progress, first pilot closed: SYS-003 sets an eval acceptance gate, the deterministic shape-grader is in kb-agent/tests, and the classifier's capability evals are wired into CI as an enforced gate (free offline scoring-regression gate on every PR, paid weekly live-capability gate). A gate still runs in one repo only — defense-news-classifier/.github/workflows/evals.yml has no counterpart that blocks a merge. Corpus provenance — the precondition SYS-017 named, and the piece this row called the real work — closed 2026-08-02 (kb-agent/ADR-012): KB_AGENT_NOTES_DIRS overrides the absolute workstation path, an absent notes dir now errors instead of skipping, and CI clones learning-notes, builds the index against it, and runs both retrieval-eval arms on every push and PR. That puts kb-agent at SYS-017 tier 1 — measured in CI, not gated by it. What is left is the back half of the same order (reproduce the corpus → baseline → floors → gate): the first CI run exists but one run cannot supply the run-to-run noise a floor sits above (classifier/ADR-014), so floors, a gate script, and the required-status-check setting are all still outstanding. The risk is genuinely reduced — a retrieval regression is now visible on every PR rather than never — and not closed, because nothing stops one merging. (Scope corrected 2026-07-19: this row previously read "CI runs across all three code repos", the same overclaim SYS-017 caught on the portfolio.)	classifier/ADR-007, system/SYS-017, kb-agent/ADR-012, Next → evals-as-CI for kb-agent
R7	CLASSIFIER_URL unset silently skips enrichment — tag writeback is a no-op when the env var is absent, which is easy to miss in a deployed environment	Low	Document the env var prominently in notes-api README; the no-op is a deliberate safe default for dev/tests, but must be set explicitly in any environment where enrichment is expected	notes-api, system/SYS-005
R8	Silent contract drift on the /classify seam — classifier (provider) and kb-agent (consumer) are separate repos, so a renamed response field or changed enum could mis-read at runtime with nothing failing	High — materialized 2026-07-18, CLOSED 2026-07-19	✅ Mitigated, the hard way — it occurred first. defense-news-classifier shipped v3.0.0 adding region to the /classify response with no coordinated consumer update, and no build went red. The root cause was that the "contract tests on both sides" claim was wrong in kind: each repo asserted against its own private copy of the shape, so neither could observe the other. Closed 2026-07-19 by the shared-artifact fix this row called for: the provider publishes contracts/classify-response.schema.json and consumers fetch and assert against it (SYS-018); kb-agent/agent/tools.py now carries all three fields. Residual, deliberately accepted: the consumer check fails open — an unreachable or unpublished schema warns and passes rather than reddening an unrelated build. Drift is loud when the artifact is reachable, silent otherwise. See the SYS-004 closure note.	system/SYS-004 (amended)
R9	Loop optimizes against the eval (Goodhart) — the prompt-optimization loop tunes the prompt to the very metric it is scored on, so it can game the eval instead of genuinely generalizing	Medium	Mitigated by design: a 3-way split (optimize / validation / held-out real gold), with the done-signal riding the validation set and the untouched held-out number reported honestly whichever way it moves. The overfitting gap is the artifact's centerpiece, not a hidden failure	classifier/ADR-005, classifier/docs/specs/prompt-optimization-loop.md

On the "simulated program"

This is a solo build, so there's no real cross-team coordination to manage — the program layer is simulated. That's stated plainly on purpose: the workstreams, dependencies, sequencing, and risk reasoning are real and transferable, even though the org around them isn't. Naming the limitation is more credible than pretending it away.