How AUA-Veritas works

System overview

AUA-Veritas is a macOS desktop application that wraps the Adaptive Utility Agent (AUA) framework in a consumer-friendly chat interface. It runs multiple large language models simultaneously, applies mechanism design to select the best answer, and maintains a persistent correction memory that improves routing over time.

The system has four distinct layers, each adding correctness on top of the previous:

Technology stack

The backend binary is built with PyInstaller in --onedir mode (not --onefile). This eliminates the 8–15 second cold-start extraction penalty of single-file builds — files are read directly from disk at the binary's path.

Data model

All state is stored in a single SQLite file at ~/Library/Application Support/AUA-Veritas/veritas.db. Key tables:

VCG mechanism design

AUA-Veritas implements the Vickrey–Clarke–Groves (VCG) mechanism from the AUA framework. In a standard VCG auction, a social planner selects the allocation that maximises total welfare. Here, "welfare" is the expected correctness of the answer a model will produce for this query in this domain.

The welfare formula

The domain probabilities p(j|q) are derived from model self-reports — each model appends a DOMAINS: tag to its response listing the 1–3 domains it considers most relevant. The router strips the tag, aggregates across all models (equal weight split per model's list), and normalises to produce a probability distribution.

Why VCG is the right mechanism

Utility function — effective_u

The domain-specific effective utility u_i(j) is a volume-normalized win rate. Raw win rates are unreliable with few observations — a model with 1 win from 1 query has the same win rate as one with 20 wins from 20 queries, but very different reliability.

At n=0, the utility is 0.5 — neutral, with no advantage to established models. At n=20, the observed win rate fully dominates. Between these values, observations count less per-observation than later ones — a form of Bayesian shrinkage toward the prior.

Hierarchical utility (Phase 12)

For queries classified to deep domain nodes (e.g. molecular_biology → biology → science), the effective utility walks up the tree to find the deepest level with sufficient data:

Deeper nodes need fewer observations because they represent rarer, more specific queries. A model that has answered 8 protein-folding questions has demonstrated enough signal to use that node's effective_u rather than falling back to the biology or science level.

Dynamic domain ontology

The domain taxonomy is a rooted tree that grows naturally from model self-reports. Ten L0 root domains are fixed anchors. Everything below is dynamic.

Three-tier normalization

When a model reports a domain string (e.g. "molecular biology"), the system resolves it through two stages:

Candidate promotion

Strings in the candidate queue are evaluated by a background job every 5 minutes. A candidate is promoted to a full node when ALL of:

The divergence threshold is depth-relative because a 15% performance gap at the root level indicates a meaningfully distinct domain, while the same gap at depth 6 is likely sampling noise in a very specific sub-domain.

Compliance monitor

Models periodically stop following system prompt instructions — most commonly the DOMAINS: tag under context window pressure. The compliance monitor detects this and escalates automatically.

Rule categories

Streak-based escalation

Each rule maintains a consecutive-failure counter per model. When a counter reaches the threshold (default 2), the next system prompt highlights the violated instruction in bold:

The streak resets to 0 immediately when the rule passes. The system self-heals without user intervention.

Keyword indexing and real-time search

Every message is keyword-extracted and stored in a local SQLite table. Searches across months of conversation history complete in microseconds — because search never touches the DB at query time. Everything runs from three co-maintained in-memory structures.

Three in-memory structures

Both structures live in Python memory. Keypress search is pure Python set operations — no I/O, no SQLite, no disk. Memory footprint at scale:

Message-level scroll-to (clicking a result) uses a targeted DB query on (conversation_id, keyword) — both indexed, called once on click not on keypress. The memory vs single-DB-lookup tradeoff is deliberate: keeping message-level sets in memory would add 3–4× overhead (measured: 364MB at 20k messages) for a feature used far less frequently than search-as-you-type.

Inverted index — how it works

When the keyword "postgres" is extracted from a message, it is added to _KW_INDEX["postgres"] as a set containing that conversation's ID. If "postgres" appears in 47 conversations, _KW_INDEX["postgres"] is a set of 47 conv_ids.

A search for "postgres index" becomes:

Prefix search via bisect

_KW_SORTED is a sorted list of all unique keywords maintained in parallel with the inverted index. Prefix search (typing "post" matching "postgres", "posting", "post-hoc") uses Python's bisect module:

This makes real-time sidebar search work correctly — each keystroke triggers a prefix search that returns results before the user finishes typing, with no perceptible latency.

Message-level index for scroll-to

_KW_MSG_INDEX maps each keyword to a nested structure: keyword → conv_id → {msg_ids}. When you click a search result, the frontend scrolls to the exact message that matched — not just the conversation. The best match is the earliest message containing all query words:

Extraction pipeline (async, zero response latency)

Extraction runs in a background asyncio worker — never on the response path:

spaCy POS-tags each message (capped at 2000 chars) and keeps nouns, proper nouns, and non-stopword tokens. Code tokens (camelCase, snake_case, dotted paths) are identified by pattern and given a 2× weight boost so architectural terms surface reliably in long prose responses.

Schema

The SQLite table is the persistent store — the in-memory index is rebuilt from it at startup. The DB index on keyword makes the startup load fast even with millions of keyword rows. After startup, the DB is only written to (never read) during normal search operations.

Context recovery

AUA-Veritas is designed so that resuming a conversation after months requires no user action. Three independent mechanisms layer together to restore full context automatically.

Layer 1 — Correction injection (always active)

This is the primary mechanism. Before every query — including the first message in a resumed conversation — all stored corrections are scored and the relevant ones injected into the system prompt. Because corrections are global and persistent, a preference you expressed six months ago in a different conversation applies to today's query if it's semantically relevant.

Layer 2 — Model recovery prompts

Each model generates and stores its own recovery prompt — a self-written system message that summarises the active corrections, preferred domains, and its own reliability trajectory. The model writes this in its own style, incorporating the full set of active corrections it would need to behave correctly.

Staleness detection:

Layer 3 — Context window backups (structured handoff notes)

For long-running conversations, each model periodically writes a structured handoff note — not a raw summary, but a purpose-built document for resuming in a new window. When context pressure is detected, this note is injected before the system context so the model picks up immediately.

The backup prompt forces the model to capture five things explicitly:

The backup is injected with a framing header that tells the model it is resuming a conversation and should use the note silently without mentioning it to the user.

Memory architecture

The memory system stores corrections in the corrections table with 12 fields including type, scope, domain, confidence, decay class, and corrective instruction. Corrections are never deleted — they may be superseded (scope set to "superseded") but the audit trail is preserved.

Correction types

Decay classes

Corrections have a decay class (A, B, or C) that determines how quickly they lose relevance in the scoring function. Class A corrections (explicit user instructions) never decay. Class C corrections (inferred from implicit signals) decay over 30 days without reinforcement.

Correction detection pipeline

Implicit corrections (natural language replies like "No, that's wrong — it's 53") go through a three-stage pipeline before storage:

Context injection scoring

Before every query, IncludeUtilityScorer scores all stored corrections and selects which to inject. The scoring function is a weighted combination of 8 factors:

Corrections below the threshold are not injected — this prevents irrelevant memories from consuming context window. Pinned corrections always pass regardless of score.

The relevance score uses the same semantic similarity model as the correction detector, comparing the stored correction's canonical query against the current question.

Full query flow

A single query through AUA-Veritas traverses this path:

Peer review

In High and Max accuracy modes, a designated reviewer model independently evaluates the winner's answer. The reviewer receives the original query and the winner's response — without knowing which model produced it or what the VCG scores were.

Reviewer isolation

The reviewer is deliberately isolated from the scoring system (reviewer_no_leak compliance rule). This prevents the reviewer from being influenced by knowledge of the competition outcome.

Review verdicts

Context prompts

Each model generates its own recovery prompt — a system message incorporating all active corrections, the reliability trajectory, and top domains. The prompt is written by the model itself (it knows its own style best) and saved to model_context_prompts.

Staleness detection

The corrections hash is a fast-path check — comparing a 16-character SHA256 hex digest against a stored value avoids a full timestamp scan of the corrections table on every staleness check. Only stale models are called for regeneration — fresh models return their saved text immediately.

A background asyncio job runs every 15 minutes to regenerate stale prompts silently. The manual Generate / View prompt button is a fallback for the window between staleness and the next job run.

Empirical validation

The AUA framework underlying Veritas has been validated across three dimensions:

The utility–correctness correlation of r=0.461 (p<10⁻⁴⁰) confirms that the VCG welfare score is a meaningful predictor of answer correctness — not just a post-hoc rationalization. Models with higher effective_u in a domain genuinely produce more correct answers in that domain.