Deploy LLM evaluation locally, keep the code on your machine

Deploying LLM evaluation locally usually means one of two things: running a model-scoring eval suite on your own hardware, or evaluating the code an LLM writes without sending it anywhere. This post is about the second. RedMirror Reflection runs as an MCP server on your machine, drives your coding agent on a local model, and a compiled kernel proves whether the generated code can actually reach a bad state. Your code never leaves your hardware, the verification uses no tokens, and what comes back is a reachable, replayable path, not a score you have to trust.

People searching for a way to run LLM evaluation locally almost always have the same reason underneath: the code cannot go to a third party. It might be regulated, it might be a client's, it might just be yours and you would rather it stayed that way. The cloud tools that review or grade code want you to upload it, and that is the exact step you are trying to avoid. So the useful question is not "which SaaS scanner is best," it is "what can I run entirely on my own machine, with my own model, and still trust the result."

First, two things "LLM evaluation" can mean

The phrase is overloaded, and the two meanings need different tools, so it is worth separating them before you pick anything.

Evaluating a model. You have prompts and expected behaviours, and you want to score how a model answers them: accuracy, refusals, regressions between versions. That is the world of eval harnesses like promptfoo, deepeval, or lm-evaluation-harness, and yes, you can run them locally against a self-hosted model. If that is your goal, those are the right tools and RedMirror is not one of them. We are happy to say so plainly.

Evaluating the code a model writes. Your agent just generated a handler, a query, an auth check, and the question is whether that code is correct and safe, not whether the model sounds smart. This is the harder, more consequential job, and it is the one RedMirror is built for. The rest of this post is about doing it locally.

Evaluating AI-written code, entirely on your hardware

The setup is deliberately boring. Serve a model locally with Ollama, vLLM, or LM Studio, point your coding agent at it, and expose RedMirror Reflection to the agent as an MCP server. Now the loop closes on your machine: the agent, prompted by you or by RedMirror's own guidance, proposes a place a bug might live; the kernel takes that proposal and decides it. The proposal step uses your local model. The deciding step uses no model at all, because it is a compiled verifier, not a prompt. That is why a modest local model is enough to run the whole thing: it only has to be a good proposer.

Nothing about that loop reaches for the network. The code stays where it is, the model stays where you put it, and the licence check is a local signature verification rather than a call home. It runs the same on a laptop with no connection as it does on a build box behind a firewall.

Why "locally" is the whole point here

• The code never leaves. No upload step means no copy of your source sitting in someone else's logs, and nothing to explain to a security team or a client.
• It works air-gapped. Offline binary, self-hosted model, local licence check. The environments that most need a review are usually the ones that forbid sending code out, and this fits them by construction.
• Verification is free of tokens. The expensive part of most agent workflows is model calls. Here the proving step is a native kernel, so you can point it at a large codebase without watching a meter.
• You own the model choice. Swap in whatever you can run, from a small quantized model on a laptop to something larger on a GPU box, without changing the verifier.

What you get back: a proof, not a score

A cloud reviewer, or a bare LLM, hands you prose: a paragraph, a severity, sometimes a reproduction that never ran. Run the evaluation locally through RedMirror and the output is different in kind. The kernel lifts the code to a state machine and searches the reachable states for a way to violate the property. It either returns a concrete path into the bad state or it returns nothing.

invariant: only_the_owner_reads_a_record  (viewer_id == record.owner_id)
REFUTED
  1. <init>                     session.role = "user", viewer_id = 8137
  2. GET /api/orders/1002        record.owner_id = 4471   // not the viewer
  3. handler returns 200         viewer_id != owner_id   // invariant broken, path returned

You can replay those steps. That is the difference between an evaluation you have to believe and one you can check. There is no confidence score to inflate, because there is no opinion in the loop; either a path drives the code into the bad state or it does not. Findings are proofs, not opinions.

Which local model should you run?

Because the model is only proposing, you do not need a frontier system to get real results. We tested this directly: several LLMs, one harness, a compiler grading every answer, and a small locally-runnable model held its own against much larger ones. The kernel does the deciding, so the model's job is to be interesting, not to be right. If you want the numbers and the head-to-head, the benchmark is written up here, and the practical shortlist is in which local model to run and can a small model find security bugs.

The honest scope

This proves reachable logic, authorization, and state bugs, the class that hides behind a clean-looking diff, and it reproduces taint it can drive to a sink. It is not a secrets scanner or a dependency checker, and it is not an eval harness for grading model quality. Where the local model is only proposing, it can still propose nonsense; the difference is that nonsense never reaches you wearing the word "finding," because anything the kernel could not prove is filed as a candidate with the reason it was ruled out. Run locally, the trade is simple: you keep your code, you keep control of the model, and you only accept a finding you can replay yourself. For the full setup and where it fits an air-gapped workflow, see air-gapped review with a self-hosted model.

Frequently asked questions