The Judgment Layer: Why Prompts and RAG Aren't Enough

Sources checked November 3, 2026: Polanyi, The Tacit Dimension, Feigenbaum, "Knowledge Engineering", Lewis et al. on RAG, Barnett et al. on RAG failure points, Liu et al., "Lost in the Middle", OpenAI's accuracy guide and Zheng et al. on LLM-as-a-judge.

What is the Judgment Layer?

Think of an AI answer as three stacked parts. At the bottom is knowledge: what the model learned in training, plus anything you hand it through a prompt, a file or a retrieval system. At the top is output: the email, the plan, the review. In between sits a set of decisions most setups leave to chance. Which point goes first? What gets cut? Is this claim strong enough to make? Is the draft finished?

A senior person makes those calls without thinking about them. A model makes them too, but from generic defaults, because nobody gave it anything better. The Judgment Layer is the name for that middle part when you fill it on purpose with a specific expert's criteria.

The distinction matters because the two layers fail differently. Missing knowledge produces wrong answers, which you can usually spot. Missing judgment produces answers that are correct and mediocre: the right facts in the wrong order, a safe recommendation where a sharp one was needed, a review that flags ten small things and misses the one that matters. Those are harder to catch, and they're where most of the editing time goes.

Knowledge vs judgment: the tacit knowledge gap

Judgment is hard to hand to a model for the same reason it's hard to hand to a new hire: much of it was never written down. The philosopher Michael Polanyi built The Tacit Dimension on the observation that "we can know more than we can tell." His publisher summarizes tacit knowledge as "tradition, inherited practices, implied values, and prejudgments" (University of Chicago Press).

In practical terms, tacit knowledge is the part of expertise that shows up in decisions, not in documents:

• An editor knows a headline is weak before she can say why.
• A sales lead knows which objection is the real one.
• A compliance reviewer knows which phrasing will draw a regulator's attention, even if it's technically accurate.

Your SOPs, wikis and past work hold the explicit part. The tacit part comes out only when the expert reacts to a specific piece of work. That's the key to capturing it, and it's the reason a pile of documents, however well indexed, doesn't add up to judgment.

Expert systems tried this before

The idea that expertise is mostly judgment isn't new. It's the founding claim of expert systems, the rule-based AI programs built at Stanford and elsewhere from the 1960s on, such as DENDRAL for chemistry and MYCIN for infections.

Edward Feigenbaum, of Stanford's Heuristic Programming Project, split an expert's knowledge into two types in a 1980 Stanford report. The first is the facts of the domain, "the widely shared knowledge that is written in textbooks." The second is "heuristic knowledge," which he described as "the rules of good practice, the judgmental rules of the field," and added that these rules "are rarely written down." He went further: this knowledge is "private" to an expert, "not because the expert is unwilling to share publicly what he knows, but because he is often unable to" (Feigenbaum, 1980).

His conclusion still holds: "The power resides in the knowledge," meaning the judgmental kind, not the inference method. What didn't hold up was the capture process. Knowledge engineers had to interview experts and hand-code rules, and Feigenbaum himself called knowledge acquisition "a critical problem" and a bottleneck. Rule bases were brittle and expensive to keep current.

Two things are different now. A language model can apply criteria written in plain language, so nobody has to translate an expert's judgment into IF-THEN rules. And Feigenbaum's own method for surfacing heuristic knowledge, working "in the context of a large number of highly specific performance problems," is exactly what a calibration loop does: show the expert real requests and real answers, and record what they'd change.

What prompts and RAG add, and where they stop

Prompts and retrieval are both ways of getting information into the model's context window. That's useful, and it's also the limit.

Prompts

A prompt states the task and whatever context you remember to include. A good one can carry some criteria, but it carries them one chat at a time. Every tool and every teammate gets a different version, and the criteria quietly fall out on busy days. That ongoing cost is what we call Prompt Debt: the time and quality lost from re-explaining the same context to AI in every chat.

RAG

Retrieval-augmented generation, introduced in a 2020 paper by Lewis and colleagues, pairs a model with a searchable index so it can pull in passages before it answers. The authors noted that "providing provenance for their decisions and updating their world knowledge remain open research problems" for models on their own, and reported that RAG produced "more specific, diverse and factual language" than a model without retrieval (Lewis et al.). For facts that change or are private, that's the right tool, and it's how most AI knowledge base products work.

But RAG answers the question "what's relevant?" It doesn't answer "what's good?" The research on where it struggles points the same way:

• Relevant isn't the same as right. A study of three RAG systems, built for research, education and biomedical work, catalogued seven failure points. Two of them aren't about missing facts at all: "Wrong Format," where the model ignores the requested format, and "Incorrect Specificity," where the answer is "not specific enough or is too specific to address the user's need" (Barnett et al.).
• More context isn't more attention. Researchers testing long contexts found models perform best when relevant information sits at the start or end of the context and "significantly degrades" when it's in the middle. In their open-domain test, going from 20 to 50 retrieved documents improved GPT-3.5-Turbo by only about 1.5% (Liu et al.). Anthropic makes a similar point: context is "a finite resource with diminishing marginal returns" (Anthropic).
• Quality only shows up in use. The same RAG study concluded that "validation of a RAG system is only feasible during operation" (Barnett et al.). Someone still has to judge the answers.

Retrieval can surface a style guide if it happens to match the query. Criteria you need on every task shouldn't depend on ranking well in a similarity search.

Where fine-tuning fits

OpenAI's accuracy guide splits optimization into two axes. Context optimization is for when the model lacks knowledge, has outdated knowledge or needs proprietary information. LLM optimization is for when the model gives inconsistent formatting, the wrong tone or style, or doesn't follow reasoning consistently. The guide says the methods are "additive, not exclusive" (OpenAI).

Fine-tuning sits on the behavior axis. It works by showing the model "many examples of that task being performed correctly," per the same guide. That can bake in a style, but it has three gaps for judgment work:

• Examples don't explain themselves. The model learns the pattern of your good work, not the reasons it's good, so it can't tell you why a draft fails.
• Changes are slow. A new standard means a new dataset and a new training run.
• It's tied to one model. Switch tools or models and the training doesn't come with you.

Prompts vs RAG vs fine-tuning vs the Judgment Layer

These aren't rivals. A solid setup often uses a prompt for the request, retrieval for the facts and a Judgment Layer for the standard the result is held to.

What goes into a Judgment Layer

Criteria are more specific than "be clear" or "sound professional." Useful ones tend to fall into five groups:

1. Priorities. What matters most, in order. "For a pricing page, clarity on what's included beats persuasion."
2. Rejections. What a good expert never does. "Don't open a cold email with a compliment about the prospect's company."
3. Thresholds. When something is good enough. "A claim needs a named source or it comes out."
4. Trade-offs. How to choose when two goals conflict. "If brevity and completeness clash in an exec summary, cut detail and link to it."
5. Examples with reasons. A good and a bad version, and what makes the difference.

The examples above are illustrations, not rules to copy. The point is that each one would change an output, and none of them is a fact a retrieval system would find.

How to build one

Asking an expert to list their criteria rarely works, for the reason Polanyi and Feigenbaum gave: they know more than they can say. Build it from cases instead.

1. Pick one area of work. Choose a task where you keep correcting AI output for the same reasons.
2. Collect the explicit layer. Write down the rules you already know you follow. This is the easy part.
3. React to realistic requests. Have an AI answer requests that look like your real work, then note what's off and what you'd do instead. This is where tacit criteria surface.
4. Compare side by side. Put a generic answer next to one shaped by your criteria. If you can't see the difference, the criteria aren't specific enough yet.
5. Store it once, outside any single chat. This is the idea behind the Brief-Once Workflow: encode your standards, examples and preferences once so every AI tool reuses them instead of being re-prompted. That can be a file you load, an agent skill or an MCP connection your tools call.
6. Fix the layer, not the prompt. When an answer misses, add the correction to the criteria. A fix typed into one chat disappears; a fix to the layer carries into every future answer.

That's close to how Callstand is set up. You describe up to five areas you know well, then calibrate: Callstand writes realistic requests, an AI answers them, and you say what's off and what you'd do instead. Ten calibrations per area get it ready, and a side-by-side test shows a generic AI's answer next to your Callstand's before you publish.

The Judgment Layer and LLM-as-a-judge

Using one model to grade another's output is now common. In one widely cited study, strong judges like GPT-4 matched human preferences "over 80%" of the time, about the same rate humans agree with each other. The same researchers documented position bias, verbosity bias and possible self-enhancement bias. On a set of math questions, giving the judge a reference answer cut its failure rate from 70% to 15% (Zheng et al.).

That last result is the Judgment Layer argument in miniature: a judge does better with a standard to judge against. Anthropic says the same about its evaluator-optimizer pattern, where one model drafts and another critiques. It's "particularly effective when we have clear evaluation criteria" (Anthropic).

So the judge is the mechanism, and the Judgment Layer is the standard it grades against. Without an expert's criteria, a model judge grades against generic taste, which is how you end up with output that scores well and still reads like everyone else's.

In Callstand, that shows up as three calls your AI can make: Plan it returns a prioritized plan, Make it returns the finished piece, and Check it tells you what to fix, what's fine and what to stop worrying about. All three draw on the same criteria. Answers come back without revealing the underlying notes, and when a request falls outside what it was taught, it says so instead of guessing.

When you don't need one

A Judgment Layer is overhead. Skip it when:

• The task is factual lookup. If the answer is right or wrong, retrieval and a good prompt cover it.
• Any reasonable answer will do. Brainstorms and rough first passes don't need an expert's standard.
• You haven't settled what good looks like. Iterate in chat until your corrections start repeating, then capture them.
• It's a one-off. If you won't do the task again, the setup won't pay back.

It earns its place when the same kind of work comes back often, when quality depends on experience, and when more than one person or tool needs to hold the same standard.

Frequently asked questions