Building Local Agents with Local Models

Many products are called local agents because the desktop application, tools, or state run on the user's PC, while the model responsible for reasoning still lives behind a cloud API. Those systems do have a local execution layer, but they face a different set of memory, latency, privacy, and availability constraints from an agent driven by a local model.

This article is about the latter. The model weights and inference runtime live on a consumer PC or workstation, and the core reasoning happens on the device. Putting a language model on a PC is only the starting point. The hard problems appear when it begins to read local information, call tools, and change external state, while remaining understandable after failure and staying out of the way of a game, IDE, or creative application.

Correct generation is no longer enough. A local agent has to manage at least three groups of constraints:

There is no fixed ordering among them. Read-only question answering often depends most on context quality and latency. File edits, device control, and account operations are different: a short prompt can trigger a high-impact action, so authority, state correctness, and recovery may come first. When a foreground game occupies the GPU, resource contention moves to the front.

I use harness to mean the control system around model execution. It manages routing, context, tools, permissions, state, validation, recovery, and observability. This is broader than a prompt with a few tools and more concrete than any particular agent framework. When I evaluate a local-model agent, I care about whether the model and harness can maintain these constraints together. Model benchmarks alone cannot answer that question.

1. What does "local" mean for an agent?

Several kinds of locality are often collapsed into one term:

A desktop agent that reads and writes local files but uses a remote reasoning API is not a local-model agent in the sense used here. The reverse is also true: a local-model agent need not be completely offline. With explicit user authorization, it may call search, calendar, music, or other remote services. Fully offline should describe a specific task path that needs no network connection, not a vague property of an entire product.

A hybrid agent chooses among such paths. It might use a local model for simple commands and sensitive work, then request permission before escalating a task that exceeds its capability or resource budget to a remote model. Fallback matters, but the subject here remains an agent whose core reasoning can run locally. Only under that condition do KV cache, prefill, and competition with foreground workloads for RAM and VRAM become direct system constraints.

The model maps input tokens to output tokens. The inference runtime loads weights, allocates memory, executes prefill and decode, and exposes controls for sampling, batching, and context. Tools connect model output to file systems, applications, browsers, devices, or remote services. The environment holds the real state outside the model.

An agent starts with model calls but continues through environmental feedback until a stopping condition is met. Anthropic's classification of agentic systems separates workflows, where code follows predefined paths, from agents, where the model dynamically directs its process and tool use. The distinction identifies whether control flow is fixed in code or selected by the model at runtime.

Loop, graph, harness, and multi-agent describe different layers: how execution repeats, how state and control flow are represented, what system surrounds the model, and whether work is distributed across execution units.

LangGraph's documentation describes a graph in terms of state, nodes, and edges. A node may call a model or run ordinary code. A graph is therefore a representation of control flow and state, not an agent by itself.

A harness answers a broader question. Anthropic's work on long-running agents includes tools, compaction, environment initialization, progress files, version history, tests, and recovery across sessions within the harness. That makes it a runtime control plane rather than a thin wrapper around a model.

A minimal tool-using agent may need only one loop in its harness. Stable branches, checkpoints, or human intervention can justify a graph that makes control flow and state explicit. Multi-agent structure is useful when decomposition, parallel exploration, or context isolation provides a measurable benefit. They can be combined: a harness may execute a graph whose loop delegates selected nodes to subagents.

There is no universal upgrade path. A graph improves visibility and control but raises the cost of state design. A subagent adds handoff, synchronization, prefill, and evaluation work. The structure should follow the application rather than a fixed threshold for task length.

2. A context window is not a context budget

From this point, the equations assume that model inference runs locally. With a remote model, the agent process no longer owns local VRAM and KV cache, but provider limits, network latency, data egress, service availability, and API policy take their place. Authority, state correctness, and recovery remain.

The context length on a model card is an upper bound supported by the model, not a deployment promise. A runtime may need a shorter window because of GPU memory, system memory, KV cache, scratch space, or concurrent sessions. Games, rendering, and video processing can occupy the same GPU, so the available budget changes while the machine is running.

A local agent therefore has two context numbers: the maximum supported by the model and the amount usable on the current hardware under the current workload. The second number matters more. The prompt must fit, but prefill, decode, and foreground performance must also remain acceptable.

The agent's input working set can be written as:

System instructions, tool schemas, history, retrieval results, screenshots, and tool output all compete with the user's request. Generation also needs reserved output space:

A prompt that fills the window may leave too little room to finish a plan, explain a tool error, or produce the final answer. Output reserve belongs inside the budget rather than being whatever happens to remain.

2.1 The KV cache memory term

For a conventional decoder-only Transformer that caches a key and value at every layer, uncompressed KV cache can be estimated as:

• B is the number of resident sequences or the batch size.
• S is the number of cached tokens.
• L is the number of layers.
• H_KV is the number of KV heads.
• D_H is the dimension of each head.
• P is bytes per element.
• The factor of two accounts for keys and values.

This is not an exact formula for every architecture. GQA and MQA reduce H_KV; KV quantization changes P; sliding-window attention does not retain the full history; latent or compressed attention uses another representation; prefix sharing and offload change residency. KIVI shows that KV quantization can reduce memory substantially, but its quality and throughput results apply to the models and workloads evaluated in that paper. They cannot be converted directly into a general gain for every local agent.

The memory condition also includes weights, runtime overhead, scratch space, tool processes, and the foreground application:

The pool index p matters because weights, KV cache, tools, and the foreground application may occupy VRAM, system RAM, or a unified memory pool. The time variable matters too. An idle desktop, a large compilation, and a running game leave different resources available on the same PC. A fixed percentage of physical VRAM has little explanatory value without the workload.

2.2 Prefill, decode, and tool latency

Long context costs more than KV cache. Prefill processes the input, while decode generates one token at a time. Their compute patterns differ, so a single tokens-per-second number cannot summarize both. FlashAttention shows how attention's IO path affects practical performance. PagedAttention shows that KV cache fragmentation, allocation, and sharing affect serving capacity.

For an agent, end-to-end time also includes routing, retrieval, tool execution, and verification:

Fast local inference does not guarantee a fast local agent. In a task with several model turns, browser actions, and external state checks, any term may become the bottleneck.

2.3 Fitting is not the same as using well

Lost in the Middle found that performance on multi-document question answering and key-value retrieval changed with the position of relevant information in long contexts, and often degraded when that information appeared in the middle. Models can therefore struggle to use context well before reaching a hard limit. The paper does not establish a universal threshold such as 60, 70, or 80 percent. Model, task, position, and data type all affect the result.

Context quality includes relevance, freshness, provenance, position sensitivity, contradictions, and compression loss. A deployable context is better described as a feasible set:

This definition deliberately avoids a table that maps a graphics card to a token count. Change the architecture, quantization, runtime, batch size, tool payload, output reserve, or machine state, and such a table becomes obsolete. The equation can instead be recomputed for the deployment at hand.

3. Treat context as a working set

An append-only transcript is the simplest context policy and one of the easiest to lose control of. Appending every message, tool, and observation can work for short tasks. Over longer runs it creates simultaneous problems in tokens, latency, and information quality.

A better approach treats context as an actively managed working set. At each turn, the harness decides what remains in the prompt, what becomes a reference, what moves into durable state, and what can be discarded.

3.1 Routing depends on the application and task

Requests should not all enter the same agent loop, but openness is not the only routing criterion. Complexity, predictability, risk, available resources, and product goals all matter.

The same request may take a different path in another product. A game assistant may prioritize response time and frame time. A coding agent cares more about tests and recoverable state. A knowledge assistant may put retrieval quality first. Routing should select a path that is reliable enough at an appropriate cost for the application, not default to the most elaborate pipeline.

The same reasoning applies to model selection. Intent classification, parameter extraction, and fixed commands that a small model handles reliably do not need a large model with the full tool catalog. An uncertain router needs an escalation path so that latency optimization does not send complex work into an underpowered branch.

3.2 Progressive disclosure, skills, and just-in-time retrieval

Tool schemas consume context and create selection ambiguity. Instead of exposing every capability at once, a harness can provide a capability index and load a full schema when it becomes relevant. The model first sees what exists, then reads detailed descriptions, parameters, and examples on demand. This is progressive disclosure.

Agent Skills are one implementation of the idea. In Claude Agent Skills, the runtime initially loads a skill's name and description. It reads the complete SKILL.md only after those metadata match the task, and larger references or scripts can wait until they are needed. Domain knowledge does not have to occupy the initial context.

Progressive disclosure is broader than Skills. Tool catalogs, API documentation, datasets, MCP servers, and application capability schemas can all use layered loading. Skills are a file-based package, not the only expression of the design principle.

Data can also arrive just in time. Anthropic's context engineering guidance discusses retrieval, compaction, structured external notes, and subagents as different ways to manage long work. A path, query, object ID, or version can remain as a light reference until the raw content is needed.

This saves context but adds exploration latency and can hide information that the model does not yet know to request. A practical design needs to test capability discovery recall and provide escalation when discovery fails.

3.3 Durable state should not hide in a summary

A conversation summary can record what the agent is doing. It should not be the sole source of truth for the external world. Task status, user approvals, resource versions, unfinished actions, and tool postconditions belong in structured storage. The prompt should carry a current view and versioned references.

Large observations can move to files or a database, but a path alone is insufficient. Store a content hash, source or version, and a way to reread the material. Otherwise the agent may later resolve the same path to different content.

Compaction is lossy. Fluency is not the main concern; losing unfinished constraints, failure reasons, or provenance is. Suppose a tool reports that eight operations succeeded and two failed for lack of permission. A summary that says only "the batch operation ran" can cause the next turn to repeat the first eight or mark the task complete. Compaction evaluation should favor recall before compression ratio.

3.4 Prefix caches and subagents carry state costs

A prefix or KV cache can avoid repeated prefill, but it creates an invalidation problem. If prompt rendering, tool schemas, model versions, or history state change, the runtime must decide whether the old cache is compatible. A cache hit is both a performance event and a state compatibility decision.

A subagent isolates exploration in a separate context. This works well for parallel research and clearly decomposable tasks, but its condensed handoff risks losing task-relevant detail. The handoff should state evidence, conclusions, unresolved questions, and scope. The parent should independently verify high-impact claims. Compressing one hundred thousand tokens into one thousand cannot preserve every detail.

4. The harness is the control plane

If the loop decides what to do next, the harness decides under which rules that step runs. Its responsibilities include:

• Selecting a direct, retrieval, deterministic, or agent path.
• Assembling the working context and maintaining durable state.
• Exposing narrow, typed, versioned tools to the model.
• Enforcing policy, validation, deadlines, and cancellation outside the model.
• Recording replayable model, tool, state, and resource events.
• Recovering, compensating, or terminating explicitly after failure.

One framework need not provide all of this, and the components need not share a process. Each responsibility does need a clear owner.

4.1 Narrow tools are more reliable than long prompts

With a tool such as manage_files(path, operation, options), the model must infer the operation, valid parameter combinations, and side effects together. Splitting it into read_file, write_versioned_file, and delete_file_with_backup narrows the interface and gives policy a clearer binding point.

A tool schema should define inputs, return states, error types, and a verifiable postcondition.

4.2 success is not a postcondition

A model receiving success: true does not prove that the external action finished. A tool may have sent a request successfully or returned success after a partial write. The harness should verify the postcondition through the owning system, whether that means rereading a file hash, querying a setting, or checking a resource by its external ID.

Some guarantees should not be learned by the model:

• Parameter type and range checks.
• Capability and permission checks.
• Confirmation for destructive actions.
• Deadlines, rate limits, and resource quotas.
• Idempotency keys and retry policy.
• Artifact signatures, versions, and compatibility checks.

Models are useful for interpreting ambiguous intent and proposing plans. Code or a policy engine should enforce deterministic constraints.

4.3 Hooks are only as useful as their signals

Hooks often run checks before and after tool calls. A useful hook stays quiet on success. On failure, it identifies the violated invariant, the observed value, and a repair path. Returning boilerplate after every successful call consumes context and makes real faults harder to see.

Hooks need versions, tests, and observability too. A faulty pre-tool hook can block all work. An incomplete post-tool hook can record partial failure as success. Moving logic from a prompt into code makes it testable, not automatically correct.

4.4 Observability should identify the owner

A final transcript rarely explains why an agent failed. A trace should correlate:

• The agent, harness, and model versions that received the request.
• The context, retrieval results, and tool schemas used.
• The proposed tool call and the policy decision.
• Tool execution, external state verification, and state transitions.
• Latency, queueing, RAM or VRAM pressure, and cancellation state.

OpenTelemetry's GenAI semantic conventions define developing fields for agent invocations, tool execution, conversations, errors, duration, and call counts. These fields remain marked Development, so I treat them as shared vocabulary rather than a stable schema.

Observability has a cost. Prompts, tool output, and trajectories may contain private data or secrets. Logging everything by default can turn the privacy benefit of a local agent into a high-value local archive. Structured correlation and minimal content capture need to be designed together.

5. Permissions, state, and reliability

Context management controls what the model sees. Permissions and state management control what the system allows and whether an action actually happened. A question-answering agent may need little machinery here. An agent that edits files, controls devices, or calls external services does not.

The model should propose an action; the harness or application should decide whether to execute it. A tool may need an API credential, but the credential need not appear in the prompt. The model generally needs the tool name, arguments, and purpose. The application can inject credentials during execution. MCP security guidance similarly warns against token passthrough and preserves user consent.

A timeout creates an instructive state problem. If an agent asks to change a game setting and the call times out before returning, repeating the call may duplicate the action, while declaring failure may be false. The harness should reread the setting. If the target value is present, it can finish. If it is absent, it can decide whether the tool supports a safe retry. File writes, messages, and paid API calls have the same shape, though their recoverability differs. Idempotency tokens and bounded retries help only when the tool or service explicitly supports them, as described in AWS reliability guidance.

A sandbox can limit the files, network, and system resources available to a plugin or tool process. The right mechanism depends heavily on platform and product integration. A desktop hardware assistant and a coding agent that executes unknown code need different isolation. Local execution grants neither least privilege nor trustworthy tool output.

6. Resource contention and graceful degradation

A local model shares CPU, GPU, NPU, RAM, VRAM, and power with the target application. How the agent yields resources depends on its role.

Allocation should follow the primary workload and its service objective. A game assistant serves the game. An in-application assistant should not noticeably damage the application's main function. A standalone agent may use more resources when the user permits it.

Graceful degradation preserves the most important capability under pressure. The harness might select a smaller model, reduce concurrency, trim nonessential context, narrow retrieval, defer background work, or request permission for a cloud fallback. The right choice depends on the application. Safety checks and user confirmation should not disappear silently to save resources.

Local hardware spans a wide range. antirez/ds4 primarily targets Macs with at least 96 GB of memory and also supports CUDA and ROCm. It shows that large unified-memory or multi-device systems can run model and context configurations far beyond a typical 8 GB, 12 GB, or 24 GB GPU. It also reinforces why context budgets must be tied to a hardware envelope.

That resource model raises a practical question: can retrieval and external capability definitions let a small local model do useful work without carrying the full knowledge base and tool catalog in every prompt? G-Assist provides a concrete case.

7. Project G-Assist: connecting a small model to a real system

Project G-Assist is a useful example of a local-model agent. In the installation I examined, the high-level model roles were a Qwen 3 4B local generation model and a Qwen embedding model. This observation identifies the model families and their broad roles, not the exact call order for each request or a guarantee about every release.

7.1 Narrow the problem before generation

A 4B model can handle many narrow instructions, but it need not read the full product knowledge base and every tool definition on each turn. One useful architecture retrieves a small set of relevant knowledge or functions, then gives those candidates to the generation model. This reduces prefill and limits interference from unrelated tool descriptions. I use that pattern to interpret the model roles here, rather than claiming G-Assist follows one fixed internal pipeline for every request.

It is a natural application of progressive disclosure. Model weights provide general language and reasoning ability. External knowledge can update separately, and the tool catalog changes as plugins are added or removed. They do not need to share one release cycle.

7.2 Manifests keep capabilities outside model weights

G-Assist's public plugin documentation describes discovery through a plugin manifest containing a name, description, and functions. Functions declare tags, properties, and required fields. The manifest is a call interface for the runtime and a capability description for routing.

Public documentation also describes natural-language routing and direct plugin invocation. Direct invocation narrows the namespace first, leaving the model to select the function and arguments. Natural-language routing is useful when the user knows the goal but not the tool name.

A new plugin can therefore add capabilities through its public interface without retraining the generation model. The harness discovers and calls the plugin, while the model continues to use the function-calling contract. External capability definitions matter especially for small models because a tool catalog can grow faster than the prompt budget.

7.3 A local model does not make every path offline

The public Protocol V2 documentation specifies engine-driven health monitoring and execution deadlines. If a plugin stops responding, the engine can terminate its process. These deterministic controls contain plugin failure without asking the model to notice it.

Plugins may operate locally or connect to remote services. Public examples include local peripheral control as well as online tools. In G-Assist, local describes core model inference rather than every data path.

My conclusion from the example is modest. A small generation model can sit inside a much larger product system built from retrieval, explicit capability descriptions, and harness-managed plugins. The harness also produces the execution evidence needed for the next problem: deciding whether a failure belongs to the model at all.

8. A trajectory is not training data

An agent loop naturally produces trajectories. A record becomes a candidate for training only after authorization, validation, and attribution.

Suppose an agent selects the right plugin but changes the wrong setting. To locate the failure, replay needs the model and harness versions, retrieved functions, proposed arguments, policy decision, tool result, external state after execution, and termination reason. Those fields let us distinguish a bad model choice from stale retrieval, an ambiguous tool schema, or a tool that reported success too early.

The repair follows the owner. A stale schema is fixed in the tool contract; a missing postcondition belongs in the harness; a repeated wrong function choice under correct context may justify new model data. The repaired system is evaluated inside the same harness, rolled out gradually, and traced again. Over time, model improvements should allow obsolete prompt scaffolding to be removed rather than making the harness grow forever.

8.1 Decide whether to capture before capture begins

"Anonymize it later" is not enough. Consent, retention, permitted uses including training, provider terms, and provenance should be decided before capture. Private paths, tool arguments, screenshots, and error logs can reveal identity, source code, or credentials.

Data minimization affects observability, so a trajectory schema should separate correlation metadata from payloads that may contain sensitive content. Secrets should be excluded at the source rather than left for a regular expression before training.

8.2 Replay should identify the owner

Replay should begin with deterministic validators for schema validity, tool execution, external postconditions, state transitions, and deadlines. Model judges or human preferences are more appropriate for style, helpfulness, and choices among several valid answers.

If a tool schema is stale, adding ten thousand training examples teaches the model to accommodate an interface that should be fixed. If the model repeatedly selects the wrong function under a strict schema and correct context, post-training becomes a reasonable option.

8.3 Keep more than successful paths

Clarification, refusal, no-tool decisions, cancellation, partial failure, recovery, and correct termination are useful behavior. Training only on successful trajectories can teach a model to act at any cost because the data never shows a safe stop.

Training and hold-out sets should not be split only by random rows. Rollouts of the same user task, minor paraphrases, and shared tool state can leak across the boundary. Split by task lineage, environment, and time, and maintain a separate safety regression set.

9. Match post-training to the error class

Training enters only after attribution leaves a model-owned error. SFT, preference optimization, RL, and distillation require different evidence; they are choices for different failure classes, not a fixed maturity sequence.

9.1 SFT: the target behavior can be demonstrated

Targeted SFT fits cases where the model has not seen a required format or has not formed stable domain behavior. Examples include domain terminology, tool-call formats, multi-turn clarification, and recovery patterns. LoRA and QLoRA reduce the cost of parameter-efficient fine-tuning. They change how training is performed, not whether training is the right repair.

SFT data needs boundary cases. Showing only correct tool calls, without no-tool decisions or requests for confirmation, teaches an incomplete policy.

9.2 Preference optimization: hard constraints already hold

When multiple outputs satisfy hard constraints but differ in concision, clarification quality, refusal, or recovery explanation, chosen and rejected pairs can express that preference. DPO optimizes such preferences with a classification objective, avoiding the explicit reward model and online policy sampling used in the paper's comparison setup.

The original DPO experiments do not show that it automatically improves agent trajectories. Pair quality, context, tool state, and preference consistency still govern the result. Treating a failed tool execution as a rejected response can teach the model to compensate for a harness defect.

9.3 Verifiable-reward RL: outcomes can be checked reliably

The evidence for GRPO in DeepSeekMath comes from mathematical reasoning. DeepSeek-R1 further demonstrates how RL can encourage reasoning on mathematics, coding competitions, and other verifiable STEM tasks.

Agent tool use resembles that setting only when outcomes are verifiable. Code can run tests. A structured query can be checked against a result set. A game can expose state. Whether a sent message was appropriate, a deletion matched the user's intent, or a user was satisfied is much harder to reduce to a reliable reward.

RL cannot repair a faulty simulator. If the reward omits a hidden side effect, the policy may exploit the omission. A reward is verifiable only when it covers the task invariants that matter, not merely because code returns a number.

9.4 Distillation: the teacher works but is too heavy to deploy

If a teacher can complete the task but is unsuitable for the target PC, selected behavior can be distilled into a smaller student. Knowledge distillation gives the basic method. Minitron shows one route that combines pruning and distillation to produce compact language models.

The student must be evaluated inside the target harness. Generic benchmarks do not reveal regressions in tool schemas, context policy, cancellation, resource contention, or postconditions. TinyAgent is a focused example: it curates data for edge function calling, retrieves tools, fine-tunes small models, and uses quantization. It shows that task-specific co-design can work, not that a small model replaces larger models in every agent setting.

Quantization is also a deployment intervention. It changes memory and latency and may alter tool selection, long-context behavior, or numerical stability. The final gate should evaluate task quality, latency, memory, authority, and recovery on the intended hardware.

10. My take

I would start with the deployable working set, not the advertised context length. The budget has to survive the actual foreground workload. Then I would spend engineering effort on a testable harness before adding another agent layer: explicit routes, narrow tools, durable state, and postcondition checks make failures easier to assign and repair.

Resource policy should follow the product. A game assistant protects gameplay; a dedicated workstation may treat the agent as its primary workload. Multi-agent structure has a similar burden of proof. Context isolation and parallel exploration are useful only when they repay the costs of handoff, coordination, authority propagation, and evaluation.

Finally, I would judge the learning loop by whether the system becomes simpler. Clarification, refusal, cancellation, and recovery traces often teach more than another successful run. When the model improves, obsolete prompt scaffolding should disappear, while permissions, idempotency, artifact verification, and postconditions remain deterministic. The goal is a smaller system whose failures are easier to understand.