Engineering View — Defense-News Intelligence

Status: Living Date: 2026-06-22 Author: San Lee

The technical-craft companion to the product one-pager and the program view. Per SYS-007, engineering is the substrate both P-tracks stand on — the credibility floor under "technical" PM/TPM — not a third parallel lane. So this view captures the AI-era craft the system exercises, the standards that make the repos cohere, and where the depth actually lives.

The skill substrate — six clusters

The competency map from SYS-007. The meta-skill beneath the first five: every classical engineering skill has an AI-era mutation, because outputs are probabilistic — testing → evals, monitoring → drift detection, integration test → eval gate. The sixth runs that idea in reverse: keep the classical skill intact on purpose, and use it as the ruler.

Cluster	What it means here	Where it lives	Maturity
Evals & quality bars (keystone)	golden sets, LLM-as-judge, regression gate in CI	classifier eval harness + gold set + validated judge, gating PRs (classifier/ADR-007); system-wide pattern adopted in SYS-017, which places every repo on a four-rung ladder	✅ shipped in the classifier (tier 3), 🔄 kb-agent at tier 0, blocked on corpus provenance
Context engineering	retrieval, chunk/result caps, grounding	kb-agent RAG; SYS-003 rule 4	✅ in use (unnamed)
Agents & orchestration	tool design, the tool-use loop, error recovery, handoff between agents	kb-agent loop; SYS-003 tool-layer contract; the classifier's autonomy ladder L1–L4, including a critic that hands a label backward for reclassification (classifier/ADR-018, ADR-020)	✅ shipped; both top rungs measured negative and declined
Baselines & build-vs-buy	fit the classical model you are not going to ship, to price the one you are; paired significance tests instead of eyeballed deltas	classifier bake-off — TF-IDF + logistic regression trained and scored against the same human gold set as the LLM (classifier/ADR-017)	✅ measured once, not yet a practice
Observability, cost & reliability	tracing, token/latency/drift, model-tier	OTel tracing across kb-agent, classifier /classify, notes-api (opt-in); SYS-002	✅ tracing shipped
Security, safety & governance	prompt injection, tool-exfil surface, output hardening	threat model of the kb-agent tool seam as a regulated deploy (SYS-016); SYS-010 posture	📝 threat model documented

What's next — the learning sequence

Priority order. It rhymes with the program roadmap and adds the two skills the system uses or needs but never named:

1. Evals (keystone, partly shipped) — the classifier has this: a 54-snippet hand-labelled gold set, a judge validated against it, and threshold gates that fail a PR (classifier/ADR-007). SYS-017 is settled — adopted 2026-08-02 with a tier ladder that places each repo honestly. What remains is the work it makes chippable: extending the pattern to kb-agent's retrieval, which is blocked on making its eval corpus reconstructible in CI before any floor can be set.
2. Observability / OTel (shipped) — OTel tracing across all three services (kb-agent loop, classifier /classify, notes-api enrichment seam), opt-in per service with GenAI/HTTP semconv attributes. Drift detection over the traces is the remaining refinement. You can't improve what you can't see.
3. Context-engineering depth — past naive RAG: retrieval quality, reranking, memory.
4. AI security (threat model documented) — the agent tool seam is modeled as a regulated deployment (SYS-016); building its tenancy + audit controls is the next step a real deployment would take.
5. MCP / interop (stretch) — candidate future ADR: MCP-ify the SYS-003 HTTP seam. A learning target, not current practice.

Cross-repo engineering standards

The house style every repo inherits — the conventions that make separate repos read as one system:

• Two-tier ADRs — decisions recorded, not lost (SYS-001). The habit is the craft.
• Model-tier discipline — Sonnet by default, Opus only where an eval shows it pays (SYS-002).
• Tool-layer contract — one observation shape + error-recovery + an eval gate for agent-callable tools (SYS-003).
• Eval-as-acceptance — a change ships when a gate says so, not when it "looks right."
• Context-budget discipline — cap and scope results, cite the source, never dump a document when a chunk answers.
• Google-style docstrings in all first-party Python — Args: / Returns: / Raises: / Yields:, one-line imperative summary, types in annotations and never repeated in the docstring. Module docstrings required; the house pattern (what the module is, why it is shaped that way, a Run locally: command where useful) is preserved, not flattened. One-liners are fine where the name and type hints already say it. No NumPy style, no reST/Sphinx field lists. Enforced by ruff's D rules with convention = "google" in defense-news-classifier, notes-api and kb-agent; tests are exempt (D1 per-file ignore) and D105/D107 are ignored globally.

Re-tiered here from SYS-014 on 2026-07-18. It is a convention, not a decision, by SYS-001's own definition — its text says it was "ratifying a de-facto standard, not imposing a new one", zero conflicting docstrings existed to convert, and nothing real was foreclosed. It fails prong 2 of the promotion bar. Its CI enforcement is not a counterargument: SYS-001's own correction, written about this exact document, concedes that enforcement is the wrong axis to sort a decision log by — a lint rule that fails builds in three repos is still a lint rule.

• Agent-facing CLIs are designed against a token budget — first-party CLIs an agent is expected to drive (the classifier's eval harness, netops-lab provisioning, desk's dashboard commands, anything new) treat context cost as a design constraint, not an afterthought. Adopted from AXI (MIT), 2026-07-26, as design guidance — not a dependency. Ten rules in three groups:

Efficiency	Token-efficient output · minimal default schemas (3–4 fields per list item, more on request) · truncate long fields with a size hint and a --full escape hatch
Robustness	Pre-computed aggregates so the agent needs no follow-up call · definitive empty states (say "0 results", never print nothing) · structured errors and clean exit codes, idempotent mutations, never prompt interactively
Discoverability	Ambient context before the agent acts · content-first (bare invocation prints data, not help) · contextual disclosure (suggest the next command after output) · concise per-subcommand --help as the fallback

"Never prompt interactively" is the one with existing teeth here — it is the same property the house command shapes already require, and a CLI that blocks on a prompt is indistinguishable from a hung agent.

Recorded as a convention, not a SYS number. It crosses repos, so it clears prong 1 of SYS-001's promotion bar, but the alternative to "design against a token budget" is "don't" — the null option, not a costly road not chosen. It fails prong 2, exactly as SYS-014 did. The genuinely foreclosing question next door — whether agent-facing tooling should be a CLI at all rather than an MCP server — is a real decision with a costed alternative and is not settled here.

AXI also ships tools, and taking one is a separate decision from taking the principles. Above is design guidance for CLIs we write. The project additionally publishes reference implementations for CLIs we use — gh-axi (wraps the official gh), quota-axi, chrome-devtools-axi, lavish-axi — plus an installable Agent Skill for scaffolding new ones, and 48+ community implementations. The principles are guidance and cost nothing; a tool is a dependency and gets audited before it lands. The community list is a directory, not an endorsement, and anything that touches credentials gets the full treatment or it does not get installed.

Measured, so the win is not misread as smaller output. On a two-row pr list, gh-axi emits 376 characters against plain gh's 250 — bigger, because the schema header and next-step hints are fixed overhead. The benchmark's win is turns, not bytes (3 versus 6–8): gh-axi pr view returns checks: "2 passed, 2 failed, 4 total" inline, where plain gh pr view carries no check status and needs a second gh pr checks call. Those hint lines cost tokens here to save a round-trip there. On short one-shot commands that trade loses; across a multi-step agent task it wins.

The evidence, stated honestly. AXI's author published a 425-run benchmark (17 tasks × 5 reps × 5 conditions) in which an AXI-shaped gh beat raw gh and three GitHub-MCP variants on success, cost, latency and turns. Reconstructing Wilson intervals from the published rates, the AXI-versus-everything gap survives (Newcombe intervals all exclude zero, +6.8pp to +27.1pp). Two caveats worth carrying: the study's agent and its judge are the same model with no reported agreement against human labels, and every non-AXI comparison in it overlaps zero — including CLI versus MCP on reliability, where only the ~3× cost difference is real.

• Published figures are asserted, never retyped (the convention half of SYS-019, which is the rule) — any eval number or version claim quoted outside the repo that measured it must be checked against that repo's published artifact. The classifier publishes evals/metrics.json; a figure opts in with a marker, and CI fails if it drifts.

Marker (HTML)	<span data-metric="KEY">92.6%</span>
Marker (Markdown)	category <!-- metric:KEY -->92.6% — renders invisible
Version claims	<!-- version:classifier -->**v3.0.0** — never in backticks: code spans are stripped before scanning, so a marker inside them matches nothing
Keys	the artifact's gold object (category_accuracy, domain_macro_f1, …)
Fails on	value mismatch · unknown key · zero markers found · zero of a marker type · a marker that starts a line
Warns and passes on	artifact fetch failure — an outage must not redden an unrelated build. Placement is still checked: it reads no artifact, and a rule that stops running during a blip is a gate that did not run

Historical figures are deliberately unmarked. A v1 baseline or a superseded column is a frozen record of a past run; marking it would make the guard rewrite history on every release. That is SYS-009's guarantee-vs-dated-observation rule applied to numbers, and it is why decisions/ and adr/ are outside every check that asks "is this number still current?": an ADR records what was true when it was written, and re-syncing it to today's artifact would be rewriting the record.

The one rule that does reach them is placement, and the difference is why it is split out. A marker at the start of a line opens a CommonMark HTML block, which closes the paragraph above it and suspends inline formatting until the next blank line — so the page renders with literal ** and backticks while the source looks fine and the checker passes clean. That is true whether the figure beside it is live, historical or frozen, so placement asks "does this file render?" and is swept over every .md in the repo. It can never ask anyone to restate a number: the fix is always to move the marker onto the end of the previous line. Markers inside code spans and fences are documentation of the convention, not uses of it, and are skipped.

Where the repo owns the artifact, generate instead of check. The classifier's own README table is produced by scripts/gen_readme_metrics.py, not verified by a checker — a generated number cannot be wrong, where a checked one is still wrong between a careless edit and the next CI run. Same reasoning as ADR-002.

Written down 2026-07-19, after the same stale number was found on five surfaces across three repos in one day. This section is the shared artifact; the implementations are deliberately not shared. Four small checkers (portfolio/scripts/check-published-metrics.cjs, architecture/scripts/check_program_metrics.py, learning-notes/scripts/check_published_metrics.py, and the classifier's generator) each stay self-contained, because a vendored guard that falls out of sync is worse than a duplicated one — it reports green from stale logic. Duplicated logic is cheap at this size; a duplicated convention is what actually hurt, so that is the part centralized here.

• Repo layout the portal depends on — every app repo keeps README.md, docs/ and decisions/ at its root, under exactly those names. scripts/build_portal.py copies those three and nothing else (APP_COPIED, and the for sub in ("docs", "decisions") loop that skips silently via if s.exists()). Rename decisions/ to adrs/ in any repo and the portal drops that repo's entire decision log with no error and no warning — the build goes green and the content is simply gone.

Written down 2026-07-18, having previously existed only inside the build script. This is a genuine cross-repo binding: it constrains six repos' directory layout, and it is the reason it did not ride down with the portal decision when that was re-tiered to ADR-001. A constraint recorded only in the code that enforces it is one nobody outside this repo can see — the same class of failure as a contract whose guard was never wired.

Where the depth lives — a reading map

For going deeper (and for downtime reading) — the real artifacts, by repo:

Repo	Read it for
defense-news-classifier	the eval harness, the synthetic-data / circular-eval trap, the /classify service, the classical baseline the LLM was priced against, and the autonomy ladder L1–L4 with the measured declines at its top two rungs; ADR-001 onward
kb-agent	the manual tool-use loop, RAG retrieval, the observation-shape refactor
notes-api	Python/FastAPI REST API; SQLAlchemy 2.0 ORM; async tag enrichment via BackgroundTasks (CLASSIFIER_URL seam); ADR-001–002
architecture	the system decisions (SYS-001 onward), the program view, the product one-pager, the case study
learning-notes	the plain-language concept behind each technique, and the D3 concept-map graph

Note: this table is the closest thing to a single reading surface today, and it's hand-kept. A browsable surface that stitches concepts + ADRs + program/product into one place — the concept graph has no node type for ADRs yet — is a tracked idea in the program view's Later and the case study's "portfolio site" line.