What is Harness?

Think of an AI agent flow like bowling: the task is executed as a ball on the surface – which is the LLM, and the harness like the bumpers on the sides of the lane. Without them, one bad roll and you’re in the gutter: a hallucinated citation, a malformed output, a runaway loop that burns your budget. The harness is what keeps the ball in the lane.

When you play, you can play wild – without guardrails, lanes, borders – and still hit the target. But it is not guaranteed. For agentic flows in production, you want this to be a guarantee and maximize the chances of the task executing successfully.

At monday.com, we learned this firsthand while building the feedAgent – a deep agent that curates a personalized activity feed for every workspace member, surfacing what matters and needs attention, while framing the next step, to enable closing the loop fast. This article walks through the harness we built, phase by phase, and what we learned about making AI agents work in the real world.

We ended up realizing there were three main places where things could go wrong:

(1) Data & Context: What goes as input to the agent, and what it knows about the world it is working in.

(2) Agent Flow: The flow itself and its boundaries

(3) Feedback Loop: How it improves.

There are some very strong harnesses out there, but we believe when you adopt one, you need to (1) know it very well, (2) understand your business domain, and enrich it with your specific needs.

We used LangChain’s agent architecture with the built-in harness. Then we expanded and built our own harness around it, which is business-specific.

The Case Study: FeedAgent

Let’s drill down to a brief product background: a workspace is a shared project space where people and agents collaborate. The feedAgent curates a personalized activity feed for each workspace member – deciding what to surface from everything happening around them, making sure the feed is accurate, actionable, fresh, and reliable.

What makes it a useful case study is the combination of constraints it operates under. It reasons in a loop, calls tools, and decides when it’s done. It fetches real data from multiple sources. Its output must be structured and grounded – every feed item cites the specific activities behind it, and every feed item is actionable. And its output is what users actually see and act upon.

One of the biggest challenges here is to dig up only what’s most important and relevant out of all workspace data – activity on boards, docs, agent runs and so on, which in numbers is a huge amount of data.

The Method: Walk the Flow

We decided to build the harness by ‘walking the flow’ – going over each phase of the agent’s execution flow, understanding the angles and turns it can take, and at each step asking, “What can go wrong here, and what does the harness do about it?”

This method respects that an agent isn’t a pipeline; it’s a complex loop, different for each specific feed we are building.

Data retrieval – Before the agent is invoked

The first phase is the data retrieval and the prompt building. We start by building the system prompt with some background on the task, the flow, what is considered success, and so on. After that comes the specific feed’s data- a big, cross-asset amount of data: user activity across different assets, agent activity, existing feed items, and more.

But the model will only get what the harness chooses to expose.

The harness question: Make sure the input is accurate, with limited size and without PII:

1. Limited size and PII-Safe: Activity preprocessor – The goal here is to reduce input tokens while maintaining all important data, for both context management and costs. The functionality we created collapses user and agent event bursts into single events, keeps only important aggregated changes, and drops setup noise. This phase is deterministic; the model never sees the raw flood. This step also removes all PII (Personally Identifiable Information) data before the LLM ever sees it.
2. Fighting hallucination: Alias system – We replaced raw IDs in the prompt with short aliases and created a hallucination detection surface, so any reference the model invents rather than genuinely retrieves gets caught and dropped. When it produces output, it cites those aliases as evidence for each feed item, then filters out every hallucinated activity it invented on the way.
3. Prompt caching – The system prompt is loaded before every run, on every feed generation. Without caching, every single invocation pays the full input token cost for the same static text. With prompt caching at the AI gateway level,  the system prompt is transmitted once, and subsequent calls within the cache window pay a fraction of the cost.

Agent flow – Running the agent with boundaries 

1) Initialization: The moment the agent is created, just before it runs

With the prompt assembled, the agent is created: a model is instantiated with runtime config (model name, temperature, token limits), tools are wired up, the system prompt is attached, and the middleware stack is applied. This is the moment the agent’s capabilities and constraints are defined – what it can do, how far it can go, and what shape its output must take.

The harness question: does the agent start with the right tools, the right limits, and a clear contract for what it must produce?

1. Output schema – Without a schema, the model returns free-form text. Since the agent’s output needs to be consumed by downstream code, it must be well-defined – the output schema forces the model to emit its response via a schema-validated tool call, rather than trusting the model to produce valid JSON. It’s the difference between a contract and a hope.
2. Retry on schema failure – If the model misses the schema, retry with the validation error fed back as context. With the validation error appended, the model has the context to correct its output.
3. Model call limit – We used LangChain’s modelCallLimitMiddleware with exitBehavior: ‘end’. The agent can’t run indefinitely, and hitting the limit triggers a safe stop rather than an unhandled failure.

2) Inside the agent run

This part will be split into two: The run and the tools.

• 2.1 – The run: The agent is invoked. It will reason, call tools, reason again, and eventually emit its structured output – or it won’t. The harness question: what bounds the run, and what happens when it ends without producing output?
  • Recursion limit – separate from the model call limit set at initialization. The model call limit counts how many times the model is invoked. The recursion limit caps how many loop iterations the graph can execute – it’s the outer ceiling on the whole run. Together, they create two independent bounds: the model can’t be called more than N times, and the loop can’t cycle more than M times.
  • No structured response/exception handling – every run resolves cleanly: either it produces validated output, or the failure is caught and logged rather than surfacing to the user.

• 2.2 – The tools: The agent has a few tools available. Each one is a potential failure point – a network call that can fail, a DB query that can return nothing, and so on. The harness question is different for each. Here are a few tool-harness patterns we applied:
  • Freedom in choosing tools – only when needed. 
    • Don’t give the option to choose a must-have tool: If there is a must-have tool down the line, we separate it from the agent run and add it later in the flow to make sure it happens.
    • Clean context, minimal danger: Narrow down the tools and permissions of the available tools only to the required actions.
  • Tell the model what to do next – There’s a difference between { error: “failed” } and { available: false, message: “proceed with available data” }. The first is a signal the model must interpret. The second is an instruction. Good tool error messages are actionable – they tell the model how to behave in the failure case.
  • Tool lifecycle management – Tools that hold connections need a defined close path, and that close must be guaranteed regardless of how the run ends. 

3) Output: Before it becomes real data

The agent has finished. It returned a structured response – a list of new feed items and updates to existing ones, each citing the aliases it used as evidence.

The harness question: can we trust what the model returned? How does the harness verify it?

1. Hallucination filtering – Anything that doesn’t resolve to a valid activity ID is dropped and logged. The model may have cited an alias that was never in the original data – those citations are hallucinations.
2. Deduplication – The same activity cannot appear in two feed items. If the model grouped the same event twice, we have a priority system to keep it in the more relevant one. This is enforced through a set that tracks every resolved ID across the full output pass.

The Feedback Loop

1) Observability: Can you see what happened?

The run is over. The harness question shifts from correctness to visibility: after the run, what signal do you have, and where does it live?

The harness question: after the run, can you see what actually happened inside it?

1. LangSmith tracing – The entire agent invocation is wrapped in traceable(). LangSmith captures every intermediate step: tool calls, model responses, token counts, reasoning chains. This is where tool-level visibility lives: which tools were called, how many times, and what they returned.
2. Events – A summary event fires on every run, success or failure. It carries run-level metrics: duration, activity counts, feed item count, model name, tool-call trajectory, a full prompt-length section breakdown, and other relevant data.

2) Learning loop

The harness, as we’ve described it so far, governs a single run – what goes in, what happens during, what comes out, and what you can see. But a harness that resets every run has no memory of what worked. Feed Memory closes that loop: behavioral signals from how users interact with the feed flow back into the agent’s input on the next run.

The harness question: does the agent learn from past runs, or does it start from scratch every time?

1. Explicit rules – The user tells the feed-agent something directly. The chat extractor converts it into a structured rule, high authority, applied deterministically before curation begins.
2. Inferred observations – The user never said anything, but the behavior aggregator reads patterns across dismissals, CTA clicks, and dwell times. Lower authority, given to the agent as soft context, informs borderline decisions; it never overrides explicit rules.

And the read path closes the loop: back at Phase 1, the agent reads accumulated rules and observations before curating.

3) Evals

That’s what keeps us confident as we make changes. Since the agent is not deterministic, it will not be enough to test the deterministic parts of it, and we need to have confidence that the full output quality is maintained. 

We try to look at the evaluation on 2 levels:

1. Offline evals – We created a strong dataset that we continuously enrich with use cases we encounter in production. We use an LLM as a judge with specific metrics for each part, and run it on the CI to make sure we have no regressions before deploying to production.
2. Online evals –  Ongoing analysis of production data detects potential issues and suggests fixes. The metrics are similar, but the data is real-time data.

For both, we used monday.com’s internal skills and infrastructure that enables robust evals.

To Sum-Up

The bumpers don’t guarantee a strike. They guarantee the ball stays in the lane — every single time, across every run, for every user. Walking the feedAgent flow phase by phase, the pattern that kept appearing was the same: the failure modes weren’t in the model; they were in the assumptions we made around it. What goes into the prompt, what bounds the run, what gets verified before output ships, what gets remembered for next time — none of that is the model’s job. It’s the harness. Agentic flows in production are all about building a strong harness around them.