How to Cut Claude Usage by Pairing It With a Local LLM

Most of what a coding agent does to your context window is read files. Not reason about them, not write code, not make decisions. Read them. At roughly 4-8 tokens per line of source, a single 600-line file costs somewhere around 2,400-4,800 tokens to pull in. It then stays in the conversation for the rest of the session, whether or not you needed more than one line of it.

The economics have moved sharply in the wrong direction. Across roughly 100 trillion tokens of routed traffic, OpenRouter found programming grew from about 11% of total token volume in early 2025 to over half by late 2025. Over the same period the average prompt roughly quadrupled, from ~1,500 tokens to over 6,000. Programming requests frequently exceed 20,000 input tokens (OpenRouter, The 2025 State of AI Report). Agentic sessions are overwhelmingly input-heavy. Vantage's cost modeling of an illustrative 50-turn coding session puts it near 1M input tokens against 40K output, a 25:1 ratio (Vantage, 2026). The firm labels those figures illustrative rather than measured, and models per-turn input as growing stepwise across phases. The compounding is in the cumulative total: because each turn re-sends the context the previous turns built, total input across a session grows with roughly the square of the turn count.

At the Sonnet-tier API rate of $3 per million input tokens (Anthropic pricing, retrieved 2026-08-04), that session costs about $3 in input alone, nearly all of it re-reading things the model already saw. The absolute figure differs by tier and changes over time; the input-heavy ratio is what holds.

A local model can do that reading for free. It runs on hardware you already own and has no usage limit. And for the narrow question you usually want answered, which file calls this, what line is it on, does this import that, a mid-size local model is entirely adequate.

The pairing is obvious once stated. What is less obvious is that setting it up is the easy part, and almost nobody gets the second part right: making the agent actually use it, and then proving the local side is fast enough to be worth using. This guide covers all three. By the end you will have a working split between agent and local model, a way to tune the local side with evidence instead of guesswork, and a method for checking whether you actually saved anything.

Key Takeaways

• Route bulk, low-judgment work to the local model and keep judgment work in the agent. The split has to be a closed list, not a vague guideline.
• Setup is easy; adoption is the hard part. A permissive rule produces about one delegation per session and then the agent reverts.
• Point the agent at the local model as a measurement subject: it can mine your server logs, build a benchmark harness, and run controlled experiments.
• Shaping the input beats every speed knob. Sending excerpts instead of whole files cut attached tokens by more than an order of magnitude in testing.
• Verify savings from your own logs. Without a baseline you are guessing.

On this page

1. Before You Begin
2. Step 1 - Decide What to Offload
3. Step 2 - Give the Agent One Command to Call
4. Step 3 - Make the Agent Actually Use It
5. Step 4 - Have Claude Find Your Ceiling First
6. Step 5 - Have Claude Mine Your Server Logs
7. Step 6 - Have Claude Build a Benchmark Harness
8. Step 7 - Test the Levers That Actually Move
9. Step 8 - Verify You Actually Saved Something
10. Common Mistakes
11. What Success Looks Like
12. Reader Questions
13. Where to Start

Before You Begin

You need three things, and none of them are specific products:

1. A local inference server exposing an OpenAI-compatible chat endpoint. Most local runtimes do this now. If yours can accept model and messages and return a completion, it works. You may already have one: in Stack Overflow's 2025 survey of more than 49,000 developers, local-inference tooling was the most-used agent orchestration option among developers building AI agents, at 51.1% adoption (Stack Overflow 2025 Developer Survey).
2. A local model you have already chosen. Pick it once, before you start tuning, and stop revisiting it. Model shopping mid-task produces a setup nobody can reason about later, and a smaller model's failures tend to be plausible-looking wrong answers, which cost more to unwind than a slow correct one.
3. An agent that can run shell commands and load persistent rules: a skill file, a project instruction file, or equivalent.

This guide names no hardware and no model deliberately. The numbers quoted from my own testing are labeled as such and are illustrative; every step includes the method for measuring your own, which is the part that actually transfers.

Step 1 - Decide What to Offload

Write the split down before you write any code, because the decision rule is what makes the system usable.

Send to the local model: reading files to answer a narrow question, sweeping the same question across many files, summarizing logs and diffs, drafting boilerplate and test scaffolds, mechanical transforms, extracting structured data from unstructured text, first-pass triage.

Keep in the agent: security, authentication and permission logic; anything touching money or personal data; architecture and design decisions; work that depends on the conversation's history, which the local model does not have; and any file the agent is about to edit.

That last exclusion is a hard technical constraint, not a preference. Most agent harnesses require a real in-conversation read of a file before they will let you edit it. A local-model read does not satisfy that requirement. The correct shape is: local model narrows down which file and which lines, agent does a tightly ranged read of just that span, then edits.

The critical design choice is that this must be a closed list. Write it as: work not named in the exclusions goes to the local model, and unsure means delegate. If you instead write "delegate when it makes sense," you have built something that will be used approximately once. Step 3 explains why.

The split is defensible on capability as well as cost. On structured relation extraction across nine benchmarks, the best sub-billion model, a fine-tuned Qwen2.5-0.5B, reached a general-domain positive-class micro-F1 of 0.83, against 0.69 and 0.66 for two frontier models prompted zero-shot (arXiv 2606.22606, June 2026). Read the caveat carefully, and note the authors state it themselves: this "does not imply that SLMs are intrinsically stronger; rather, targeted task adaptation" is doing the work. Those numbers come from fine-tuned models, and an off-the-shelf local model will not match them. What the result establishes is the shape of the tradeoff: extraction and locate-and-cite work does not need a frontier model, which is exactly the work Step 1 sends locally.

(Related deep-dive on writing prompts that small models follow reliably, forthcoming.)

Step 2 - Give the Agent One Command to Call

The whole integration is a thin command-line wrapper around your local endpoint. The agent calls it like any other shell command. Keep it to one script, and make it do these things:

Attach files by path, never by pasting contents into the prompt. This is the entire point. If the agent reads a file and pastes it into the wrapper's prompt argument, those bytes have already entered the agent's context and you have saved nothing. The wrapper reads the file itself.

Stamp real line numbers onto attached content. Ask a model to cite a line from a bare code block and it will count the fence and the preamble and land a line or two off. Every citation then needs manual re-checking, which is exactly the verification cost delegation was supposed to remove. Number the lines going in, and citations come back greppable.

Derive the input budget from the context the model actually loaded, not the context length it advertises. Those differ a lot, and the loaded value changes whenever the model is reloaded. Reserve roughly a third of the window for the reply.

Refuse oversized input instead of letting the server truncate it.

The truncation trap: a silently truncated prompt does not produce an error. It produces a confident, well-formatted answer about the half of the file the model actually saw. That is far more expensive than a refusal, because nothing about the output looks wrong.

Print compact output. One line per file when sweeping, with full answers written to disk for anything that needs a closer look. A sweep that prints every full response hands the agent all the transcripts at once and spends the context the sweep existed to save.

Add a sweep mode that takes a list of paths (from a find, a grep -l, or a glob) and asks the same question of each. This is the highest-value operation in the whole setup, because the answer is short and the input is enormous.

Step 3 - Make the Agent Actually Use It

This is the step that decides whether any of the rest matters, and it is the one most setups skip.

A skill written in balanced, reasonable language does not get used. Consider guidance like "delegate when the work is bulky and low-judgment," "don't delegate if verifying costs as much as doing it," "don't delegate if the user is waiting." It reads as sensible. In my own sessions it also produced roughly one delegation per session, after which the agent quietly reverted to reading files itself. Each caveat is individually defensible. Collectively they authorize skipping every single time, because the user is always waiting.

Four things change that:

Invert the default. State that delegation is what happens unless an exclusion applies, and that unsure means delegate. The reasoning is asymmetric: a wrong local answer costs one grep to catch, while a declined delegation costs context that never comes back.

Use an objective threshold, not a judgment call. "Files over roughly 150 lines" is checkable. "Large files" is not, and gets re-litigated every time.

Add a rationalization table. List the thoughts that precede skipping, and answer each one. This device works better than any amount of explanation:

• "Faster to just read it myself." Faster this turn. Then the context is gone for the next twenty.
• "It's only one file." One file is 300 lines. Check the threshold; don't estimate it.
• "I already used it this session." There is no quota.
• "I'm just reading one more range." The second and third range of one file is the hunt. That was one question.

Then add enforcement that survives a long session. A rule read once at session start is buried by turn ten. If your harness supports hooks, a per-turn line carrying the server status and a visible count of local-model calls this session is remarkably effective: a counter stuck at 1 after fifty tool calls is a signal you cannot miss. Pair it with a non-blocking advisory when a large file is about to be read whole.

One trap worth knowing before you hit it: subagents do not inherit skills. If your agent spawns exploration subagents, each one starts with a fresh context that never saw your delegation rule. It then reads files natively, no matter what the main thread knows. In my own transcripts, sessions that leaned on exploration subagents made zero local-model calls despite the rule being loaded. The fix is to put the rule in the subagent's own system prompt: either by defining a custom exploration agent that carries it, or by pasting the rule into every spawn.

Step 4 - Have Claude Find Your Ceiling First

Now switch roles. Instead of using the agent as a consumer of the local model, use it as an instrument for measuring one.

Before tuning anything, have the agent compute the theoretical ceiling:

max_decode_tokens_per_sec  ≈  memory_bandwidth / model_size_in_bytes

Generating each token requires reading the model's weights once, so token generation is memory-bandwidth-bound on essentially every local setup. Divide your hardware's memory bandwidth by the on-disk size of the quantized model and you have an upper bound that no configuration change can beat.

This is a structural property of transformer inference, not a quirk of any one runtime. Decode must stream the weights from memory for every single token. Prefill processes all input tokens in parallel and is compute-bound (DigitalOcean, The LLM Inference Trilemma; the mechanism is independently described in SARATHI, arXiv 2308.16369, a 2023 paper, but the asymmetry it documents is unchanged).

That asymmetry is why a local model that looks slow on paper is still fine for bulk file reading. Reading is mostly prefill, which parallelizes. Only the short answer is decode.

Then measure the real number, ask for a fixed-length output, time it, divide, and compare. The bands below are my working heuristic, not a published benchmark: efficiency varies by runtime, quantization format, and hardware class, so calibrate them against your own setup.

• Roughly 60-80% of theory: normal. There is likely no large win hiding in your backend configuration. Stop looking for one, and go reduce token counts instead. This single comparison is the highest-value five minutes in the whole process, because it rules out an entire category of work.
• Far below that: worth investigating. Check that the model is fully offloaded to the GPU, that the machine is not in a low-power mode, that memory pressure is not forcing swap, and that nothing else is competing for the accelerator.

The practical consequence: if you are already near the ceiling, the only remaining lever is generating and processing fewer tokens. Steps 5 through 7 are all variations on that.

Step 5 - Have Claude Mine Your Server Logs

Most local inference servers write per-request timing to a log: tokens processed, tokens generated, and milliseconds for each. If yours does, you are sitting on a large benchmark of your actual workload that costs nothing to collect and is not biased by whatever hypothesis you are currently entertaining.

Point the agent at the log directory and ask it to parse out, per request: input tokens and the time to process them, output tokens and the time to generate them, and a timestamp. From that, three things worth knowing:

Median input and output rates. Your real baseline, across real requests, rather than a synthetic benchmark.

The time split between reading input and generating output. This tells you which half to attack, and it is the number people most often get wrong. Do not infer it from a single request: in my logs one pathological request with a very long reasoning trace made the workload look almost entirely generation-bound, while across all requests the split was much closer to even. Both halves were worth optimizing.

Your request-size distribution. The median tells you what to optimize for; the maximum tells you what your setup must survive. That maximum matters later. It is the number that makes proposals to shrink your context window dangerous, because the largest requests in your history would start failing outright.

(Related deep-dive on parsing inference server logs, forthcoming.)

Step 6 - Have Claude Build a Benchmark Harness

Ad-hoc timing produces confident wrong answers. Ask the agent to build a small harness instead, and be specific about what it must include.

A corpus of real files, stratified by size, drawn from a project you actually work on. Synthetic inputs will not match your token distribution.

Ground truth derived at runtime by a deterministic tool: have the harness run ripgrep (or equivalent) at benchmark time to compute the correct answer, rather than hard-coding answers that silently drift as the files change.

Negative cases. Include files where the correct answer is "no". Without them, a model that always answers yes scores well.

An accuracy score on every single run. This is the part people leave out, and it is the part that makes the whole exercise meaningful.

Then the five controls that separate a real result from a story:

1. Repeats. Two minimum, three preferred, and report the median. One run is not a measurement.
2. Cold and warm cache, separated. Local servers reuse the key-value cache across requests with a shared prefix. Send the same prompt twice and the second one skips most of the input processing entirely. Bust the cache with a unique nonce for cold runs, and never compare a warm run against a cold one.
3. Shuffled condition order, so warmup and thermal drift do not line up with one particular setting.
4. A cooldown between conditions, and run your baseline both first and last. If those two disagree by more than about 15%, the machine was throttling and the whole session is suspect.
5. An accuracy gate. A configuration that is faster and less accurate is a regression, not a win. Decide that before you see the numbers.

One more instruction worth giving explicitly: have the harness record every wrong answer verbatim. An accuracy delta with no examples is a number you cannot diagnose. With the misses captured, you can tell the difference between a model that is genuinely wrong and a benchmark whose ground truth was too narrow, which does happen.

Step 7 - Test the Levers That Actually Move

Test one lever at a time against a fixed baseline and let winners compose forward. A full cross-product is hundreds of conditions and days of compute.

Shape the input first

This is almost always the largest win, and it is the one most people skip because it sounds too simple: send only the parts of the file that could possibly contain the answer. Match a pattern, keep a few lines of context around each hit, mark the elided regions, and preserve real line numbers.

In my testing this cut attached tokens by more than an order of magnitude across a corpus, and end-to-end time by several-fold, larger than every generation-side optimization combined. You can estimate the win before spending any compute at all, since input processing time is just tokens divided by your measured input rate.

It also, unexpectedly, improved accuracy. Asked which line called a particular function, the whole-file run answered with the line of the import statement; the excerpt run, which had the import and the actual call in one tight window, got it right. A smaller window is an easier question.

That is not an isolated result. Accuracy on multi-document QA and key-value retrieval follows a U-shaped curve against where the relevant content sits: performance is highest when the answer is at the beginning or end of the input and "significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models" (Liu et al., "Lost in the Middle," arXiv 2307.03172). The paper reports the effect qualitatively rather than as a single headline percentage, so treat it as a reliable direction, not a number to quote.

Whether that effect can be engineered away at the attention level is less settled than it might appear. A 2026 study across six small models found that an intervention which verifiably increased attention mass toward function tokens produced null effects on two models, harm on a third, and a mixed result netting to "approximately zero" on a fourth and, more tellingly, that degradation rate did not predict retrieval accuracy at all. Its conclusion is that mean attention degradation is "largely descriptive rather than prescriptive" (arXiv 2607.20524, July 2026). That cuts both ways: it is evidence against attention-score tuning as a fix, not evidence that the positional effect is immovable. The practical reading is unchanged either way. A bigger context window is not a substitute for sending less.

Two limits are worth stating. A slice cannot answer anything the pattern did not match, so this only applies to pattern-anchored questions. And if grep alone fully answers the question, skip the model entirely. It earns its keep when the question is anchored but semantic: "is this called at module scope or inside a function?"

Then concurrency

Sweep worker counts (1, 2, 4, 8, 16) and watch aggregate throughput. Where it saturates is a property of your hardware and it varies enormously between setups, so there is no number to copy from anyone else. Watch per-request latency as you climb: past the saturation point, every individual answer gets slower while the total finishes no sooner.

A metric note that will save you confusion: if your model emits variable-length reasoning, raw wall-clock is too noisy to rank conditions. Normalize by dividing a fixed token count by measured aggregate throughput.

Then output length

If your model produces a reasoning trace before answering, that trace is often the overwhelming majority of generated tokens, even for trivial questions. Capping it is tempting, and the results are less straightforward than they look.

In testing, capping the trace barely moved the median but nearly halved the 95th percentile. For a sweep you are waiting on, the slowest request is the latency you actually feel, so that is a real win, but you would miss it entirely if you only recorded medians.

Two cautions, both measured the hard way. Capping too aggressively costs accuracy unless you also constrain the output format. And in my testing a cap of zero was slower than no cap at all: correct answers, far fewer tokens, and worse wall clock, because the model burned time without emitting. Do not assume the curve is monotonic; test the extreme so you can rule it out.

Step 8 - Verify You Actually Saved Something

Close the loop, and be strict with yourself here, because this is easy to fake.

Count input tokens processed locally. Your server logs have this. That total is, roughly, reading that never entered the agent's context.

Subtract the agent's own overhead. Each delegation costs the agent something to issue the command and read the compact answer back, on the order of a hundred-odd tokens. With a good ratio this is noise, but count it.

Price it against cached input, not list price. If your agent already uses prompt caching, repeated context bills at roughly a tenth of the base input rate, so the honest comparison for displaced reading is against the cached rate. That narrows the gap, but only for content that actually stays cached across turns, which growing file reads generally do not.

Watch the call counter. If your per-turn reminder shows the session's delegation count, that number climbing into the tens on ordinary working days is the strongest signal the setup is alive.

Two self-deceptions to avoid:

Do not count your benchmarking traffic as savings. When I first totalled my local server's throughput, the largest single day was almost entirely benchmark runs: the same nine files read hundreds of times while tuning. That is test traffic, not displaced work, and including it would have inflated the number several-fold.

Do not claim a percentage without a baseline. If you did not measure your agent's consumption before the change, you cannot compute a reduction after it. You can honestly report tokens displaced; you cannot honestly report "40% less usage" from memory. If you want the real number, run a week of normal work with no benchmarking and compare against a matched week before.

Common Mistakes

Setting a default from one benchmark run. Two back-to-back runs with identical prompts are not an A/B test. The second one is reading the first one's cache. I shipped a wrong concurrency default from exactly this, twice, in one day.

Trusting a resource estimator without checking it responds. One tool's memory estimate returned an identical figure for every context length and every concurrency setting, meaning it modeled the weights only and could not answer the question it appeared to answer. Vary the input; confirm the output moves.

Carrying tuning numbers across machines or backends. Covered in Step 7: same weights and same quantization can produce opposite results on different runtimes. Re-measure on each.

A health check that passes while real calls fail. If your wrapper's status check exercises a different code path than actual requests, it will eventually report green on a completely broken setup. Make the check use the same path.

Telling the model "do not think." Asking politely in the system prompt is worse than useless. In a single observation it produced upward of ten times more reasoning tokens, because the model reasoned at length about whether to comply. That is one data point, and the multiplier varied across prompts. Use a mechanical control instead.

Treating research estimates as results. Published or generated guidance routinely overstates by an order of magnitude, usually by extrapolating from a worst-case sample. Log every claim as a hypothesis to test.

What Success Looks Like

You will know the setup is working when the session's delegation counter reaches double digits on an ordinary working day rather than sitting at one. Whole-file reads of large files become rare and deliberate, usually just before an edit. Questions that used to mean opening six files become a single sweep command returning six short lines.

And you will have a short performance document with your numbers in it: your ceiling, your input/output split, your saturation point, rather than numbers copied from someone else's hardware. That document is what makes the next tuning decision cheap, and what stops you re-litigating the same question in six months.

Reader Questions

Does this work with any local inference server? Any server exposing an OpenAI-compatible chat endpoint will work for the wrapper in Step 2. Steps 5 through 7 assume your server writes per-request timing logs; if it does not, you can still run the harness from Step 6, you just lose the free historical data.

What if my hardware is slow? Then the ceiling calculation in Step 4 matters more, not less. A slow setup with sliced inputs can be perfectly usable, because you have removed most of the work rather than made the machine faster. Do Step 7's input shaping first and judge afterwards.

Won't a smaller model give me wrong answers? Sometimes, which is why every step here insists on an accuracy gate and on verification. The tiered rule that works: navigational answers need no separate check, because opening the cited line is the check; anything you repeat as fact gets a grep first; anything that ships gets read fully and tested. On the narrow extraction work this guide sends locally, the capability gap is smaller than intuition suggests. See the benchmark in Step 1.

Will 4-bit quantization wreck accuracy? Not for the small-to-mid models used this way, though the honest answer is that it depends on the model and the benchmark. The peer-reviewed result is GPTQ, which reports reducing weights to 3-4 bits "with negligible accuracy degradation relative to the uncompressed baseline" (Frantar et al., arXiv 2210.17323, ICLR 2023). Published vendor comparisons since then have been mixed: some benchmarks show 4-bit landing within about a point of full precision, others show several points lost, and the spread is wide enough that any single figure is worth distrusting until you have reproduced it. The relation-extraction result in Step 1 was itself measured on 4-bit models, which is the more useful signal: on the narrow extraction work this guide sends locally, 4-bit was sufficient. Measure it on your own corpus with the accuracy gate from Step 6 rather than taking a number from anyone's blog, including this one.

How do I stop it being used for things it shouldn't? The closed exclusion list from Step 1, stated in the rule the agent reads, and kept short enough to remember. Long lists of qualified guidance are what produced the one-delegation-per-session failure in the first place.

Where to Start

If you do only one thing from this guide, do Step 4. Compute your ceiling and compare it to your measured speed. It takes five minutes and it tells you whether to spend your effort on configuration or on sending less text. For most setups the answer is the second one, and Step 7's input shaping is where the real win is hiding.

Then come back and do Step 3 properly. A local model your agent ignores is worth exactly nothing, and adoption fails quietly: the rule is loaded, the setup is correct, and the work still happens in the wrong place.

(Related deep-dive on the measurement harness and setup checklist, forthcoming.)