Don't approve the model, approve the app

Doneyli De Jesus leads AI engineering solutions at Langfuse, the open-source agent observability and evals platform that is now part of ClickHouse. In this 19-minute expo session from AI Engineer New York (14 October 2026), he explains how his team helped a top-five bank decide whether an AI agent is ready for production. The method is general, but it was shaped by the constraints of a regulated industry.

The core idea fits in one line from the bank: “we don’t certify models here, we certify applications.”

Why public benchmarks don’t settle it

Every new model launch comes with a leaderboard. Those numbers tell you what a model can do on open datasets. They don’t tell you whether it will work on your internal data and your use cases.

Banks care about this gap more than most. Their data and their way of working are their “secret sauce”, and much of it never leaves the building. The example use case was a 10-K analyst for investment bankers. As soon as an earnings call ends, the analyst should produce an analysis of the transcripts and filings. Summarising a transcript is easy. The hard part is bringing in what a human analyst knows about how the firm operates, and then proving the output is right often enough to clear a quality bar.

So the question changes from “how good is the model?” to “can this application ship?”.

The constraints: on-prem, gated, air-gapped

Every solution had to meet three requirements before evaluation even started.

• On-premises. After years of moving to the cloud, banks are re-commissioning their own data centres and standing up GPU infrastructure, because of how sensitive the data fed to these models is. Some have research teams with PhDs who train and fine-tune their own models.
• Gated. Nothing goes out to the internet. Zero-data-retention promises aren’t enough; the bank wants to avoid even the appearance that a provider could train on its data.
• Air-gapped. Only the people who need the data, inside the bank’s own infrastructure, can see it.

This is why the tooling matters as much as the method. Langfuse is open source and can be self-hosted inside that boundary, so the traces and scores never leave it either.

What “ready” has to prove

Once the platform and models are in place, the bank needs a pass/fail stamp of approval for an application. It should answer three questions:

1. Does the application work the way we intended?
2. Does it meet our compliance requirements as a regulated business?
3. Will users find it usable? If they don’t, adoption stalls and sentiment about AI tools turns sour.

The stamp has to be re-issued whenever the agent’s configuration changes: a new system prompt, a different model or a new tool.

Certify the application, not the model

The application is the unit that gets approved. It includes the prompt, the model, the tools and the retrieval sources. Because the controls sit around the model, you can swap the model tomorrow and the application should hold its quality, as long as it clears the same gate again.

If you come from software engineering, this is ordinary QA. The golden dataset is your test suite, and nothing goes to production without passing it.

How the pipeline works

Mechanically there are four parts.

1. A golden dataset. This is the anchor. Many banks already have one; what they lack is a way to run test cases against it routinely.
2. Runs that produce traces. Running the agent over the dataset produces a trace for every item. The traces are the evidence of what the agent was evaluated on and what it did.
3. Scores. Each trace is scored on several dimensions, such as accuracy, groundedness and regulatory compliance.
4. A gate. Each dimension has its own threshold. Anything that doesn’t clear the gate isn’t promoted.

Around this sits a feedback loop with domain experts, covered further down.

The golden dataset

Each item is a key–value record with inputs, expected outputs and metadata. The metadata lets you judge behaviour as well as the answer, for example which tools the agent should have called. The bank ran one dataset of about 150 questions and another of about 500. In the public reproduction, the 10-K analyst uses FinanceBench (150 Q&A items drawn from SEC filings), with a 10-item sample for quick iteration. Only full runs count as review evidence.

Scoring: three kinds of evaluator

As agents get more complex, a single LLM judge isn’t enough. You need a mix of deterministic and non-deterministic evaluators.

The model that was grounded but bad at maths

The team started with a range of small and large models. One small model stood out with two numbers:

• 97% groundedness. It was very good at pulling information from internal sources and basing its answers on the bank’s real data.
• 60% numerical accuracy. Whenever it had to quote a financial figure or do simple arithmetic, it slipped. LLMs predict the next token; they don’t do maths.

High groundedness with low numerical accuracy points to a reasoning problem, not a retrieval one. So the fix wasn’t a bigger model. Arithmetic is deterministic, so they gave the agent a calculator tool: whenever a number has to be computed, it runs a small piece of Python instead of predicting digits. Numerical accuracy rose to around 95%. That comfortably cleared the bank’s 85% gate for this use case.

The evaluation framework matters as much as the tool. Every score is tied to a specific configuration, so you can show a reviewer exactly which change (this tool, that model) moved accuracy from 60% to over 90%.

Averages hide a veto

You don’t pass an application because its scores average above 90%. Each dimension gets its own threshold, and some are binary. In one full run, the agent passed groundedness and completeness comfortably but scored zero on regulatory compliance. Averaged, it might look fine. For a bank, a single compliance breach means penalties and reputational damage, so that one dimension shuts the gate on its own.

The public write-up’s gates for its three example agents show how each dimension, and each use case, gets its own bar. Every dimension must pass:

The gate check itself is simple: average each score across the run and compare it with that dimension’s threshold. Shortened from the Langfuse post:

values = [ev.value for result in item_results for ev in result.evaluations
          if ev.name == score_name and ev.value is not None]
avg = sum(values) / len(values) if values else 0.0
passed = avg >= threshold

The run passes only if every dimension’s passed is true.

Domain experts set the bar

The thresholds aren’t an engineering decision. For a risky AI use case, the risk and compliance people decide what bar the application must meet before they’re comfortable putting it in front of staff or customers. The bar varies by use case. You first agree the dimensions, then set a threshold per dimension for each application.

Experts also close the loop on failures, using annotation queues:

1. Collect the failure modes, for example every trace that failed regulatory compliance, into a queue.
2. A domain expert opens each trace, sees what the agent did, and leaves feedback.
3. Engineers use that feedback to change the agent (prompt, tools, model), which triggers a fresh gate run.
4. Reviewed traces become candidates for the golden dataset, because they’re real agent runs with expert judgement attached. Over time the benchmark gets sharper on exactly the dimensions that matter.

From weeks to minutes

Risk and compliance reviews of an application used to take weeks, sometimes months. With the dataset, scorers and gates automated, a re-certification after a change takes minutes, and the review can focus on the dimensions that actually failed.

There’s no template. Each organisation has to decide what it evaluates and which metrics matter. Doneyli’s takeaways:

• Write your own benchmark. Public benchmarks weren’t built on your data.
• Gate the application, not the model. With the controls in the application’s configuration, swapping a model is just another gate run.
• Let domain experts set the thresholds that decide promotion to production.
• Your data is your bar. This matters most in regulated industries, where data stays on your premises and public information isn’t what you’re judged on.

Try it yourself

The talk links to a full reproduction: doneyli/langfuse-llm-certification-finance, described in the Langfuse post Building Deployment Gates for LLMs and AI Agents in Financial Services. The steps below follow its README.

1. Clone and configure

git clone https://github.com/doneyli/langfuse-llm-certification-finance.git
cd langfuse-llm-certification-finance
cp .env.example .env
uv sync    # or: pip install -r requirements.txt

Fill in .env:

• LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY: from your Langfuse project settings.
• LANGFUSE_BASE_URL: defaults to https://cloud.langfuse.com. Use http://localhost:3000 for a self-hosted instance.
• ANTHROPIC_API_KEY: for the Claude agents and the LLM judge. Use LLM_API_KEY for OpenAI-compatible models.

To mirror the bank’s on-prem setup, run Langfuse yourself:

docker compose -f selfhost/docker-compose.yml up -d

How to check it worked: docker compose -f selfhost/docker-compose.yml ps lists every service as running, and http://localhost:3000 shows the Langfuse sign-in page. Create a project there and copy its keys into .env.

2. Load the golden dataset and register scores, queues and prompts

uv run python setup_datasets.py --dataset financebench --sample
uv run python setup_score_configs.py
uv run python setup_annotation_queues.py
uv run python setup_prompts.py

How to check it worked: in the Langfuse UI, Datasets lists certification/financebench-sample with 10 items. Open one and you’ll see its input, expected output and metadata. The score configs appear in the project’s score settings, Annotation Queues shows the review queue, and Prompts lists the templates.

3. Run the gates

Model gate (one LLM call per item):

uv run python run_certification.py --dataset certification/financebench-sample \
  --model claude-sonnet-4-6 --queue-failures

Application gate (the multi-step 10-K analyst agent):

uv run python run_usecase_certification.py --list
uv run python run_usecase_certification.py --use-case 10k-analyst \
  --dataset certification/financebench-sample --queue-failures

How to check it worked: open the dataset in Langfuse and switch to its runs. The new run should list each item with its scores, and each item links to its trace. Open a trace from the agent run and you should see the steps, including any calculator tool calls. Because of --queue-failures, any items that missed a threshold appear in Annotation Queues, ready for review.

4. Export the evidence pack

uv run python export_results.py --dataset certification/financebench-sample
uv run python export_results.py --dataset certification/financebench-sample --format json --output report.json

How to check it worked: report.json exists and python3 -m json.tool report.json | head -40 prints the run’s dimensions, scores and gate result. This export is the artefact a reviewer signs off against.

The repo also includes a review portal (uv run python -m portal.app, at http://localhost:8050, after building portal/frontend with npm install && npm run build). It has cicd/thresholds.json and GitHub workflows for running the gate in CI, and promote_trace_to_dataset.py for turning reviewed traces into new golden items.

How it ties together

• The golden dataset defines “correct” for your data.
• Traces record what the agent actually did on each item.
• Scores of three kinds (code, path and LLM judge) turn traces into numbers per dimension.
• The gate applies a separate threshold to each dimension, so a single veto (like compliance) can’t be averaged away.
• Domain experts set those thresholds, review failures in annotation queues, and feed reviewed traces back into the dataset.
• Any configuration change re-runs the gate, which is what lets you swap models without re-arguing the whole approval.